Scheduling Data Pipelines at Criteo — Part 3

The Proven Model in Production

Building a successful Platform is a quest of the good abstraction level.

If you’ve missed it, check out the previous articles in this series:

Scheduling Data Pipelines at Criteo — Part 2

This week we deep dive into the key ideas leveraged by BigDataFlow

medium.com

Scheduling Data Pipelines at Criteo — Part 1

Introducing Criteo’s BigDataFlow project

medium.com

After 2+ years of general availability we can say that this model is a great success at Criteo for both the users and the team handling the platform.

It fastened time to market

This achievement is in part due to the platform approach which was new for data pipeline scheduling at Criteo: prior to this each team had to create their own scheduler instance which meant provision a Database, create a git project, deployment and monitoring configuration.

It is also mainly due to how we leverage that model to give data-pipeline developers ways to experiment and build confidence in their changes.
When creating or editing a dataset SQL query, a set of functionalities is provided:

format to format the code
show <dataset> [partition] that we saw in the previous chapter
check which validates the whole data flow
dry-run <dataset> <partition> [upstream_datasets_to_include] executes one partition of a dataset on behalf of the human user launching it and writes the results into a temporary table.
If given a list of upstream datasets it also executes the partitions of those datasets that are depended on (directly as well as transitively) by that dry-runned partition.
test that runs unit tests: test definitions in json format can be defined per dataset to test the data transformation logic as well as the scheduling.
diff <left dataset+partition> <right datase+partition> <configuration file> that compares two partitions data, schema, size in a report. It is known as the deep-diff because one of its most valuable strength is in how it reports differences in nested data.
non-reg for non-regression tests

It made efficient collaborations

With our users

The declarative nature of SQL allowing to statically detect issues is leveraged to give highly contextual error messages. To give an example:

Let’s say I’m creating a new change to add an hourly dataset with 3 years of retention.
Automated validation will detect that once merged BigDataFlow will start computing a significant number (= 26280) of new tasks, and block my code review.
From there I can either:
• change my dataset to add a “start date” (an optional argument to the TIMESERIE function so it doesn’t output timestamps prior to that value) to today because i don’t need the data to be retro-computed over the 3 last years (and just forgot to do so in my initial commit)
• or confirm it’s my intent by adding a flag in my commit message to let the validation pipeline know.

Taken individually those might not seem essential things but the sum of those makes the platform self explanatory to some degree (following tutorials and reading the documentation is still highly encouraged). This was a great deal for the team owning the service. It gave us confidence in our ability to support a vast number of users building and maintaining a large number of datasets with minimal recurrent user requests, incidents. Something worth pointing out is we have no toil operations on our hands. Our only service support tasks are reviewing the occasional addition of a new User Defined Function signatures in our SQL dialect, answering our users questions and improving documentation.
The limited expressiveness of our SQL dialect compared to a general purpose programming language bounds the complexity of the pipelines. It leads to the reuse of a limited number of pipelines design patterns which helps maintainability. A drawback is we sometimes identify scheduling needs that are not properly addressed natively in the language thus require development in BigDataFlow. This is rare enough to not be a concern, especially since our amazingly smart users found ways to implement their scheduling patterns anyway using side effects in FTIs until we have proper support for those.

With our fellow SREs

Being a platform as well as the declarative nature of SQL give us the ability to change implementations of those integrations under the hood without impacting users.
Criteo is running on bare metal, so we have SRE teams maintaining many services that our platform interacts with: Hadoop, Hive, databases, security, build, deployment, metrology, etc ..
Bumping versions and changing technologies to use better ones is part of the maintenance every living software must undergo. Being both owners of the BigDataFlow platform and support team for the Cuttle/Hippo libraries we went through several of those. We saw a clear gain in efficiency with the BigDataFlow platform as it reduces the number of teams involved in rolling out a change.

Adoption grew over time

2.5 years after opening BigDataFlow to all Criteos, the platform now has:

210 distinct users with 60 monthly active users
2250 datasets in BigDataFlow’s DAG
• 640 external datasets made available by other systems
• 1100 internal datasets that do data transformations
• 510 exports of data outside of Hadoop
On average 50k tasks executions every day.
Datasets produced by BigDataFlow weight 6.6 PB of raw data (= without accounting for replication) which amount to 20% of Criteo’s offline data-pipelines outputs.

More to say and more to do !

This 3 parts blog-post introduced the BigDataflow project, the design principle that makes it different from pre-existing workflow schedulers as well as an overview of the user experiences. We did not get into all topics that make the model work in practice, such as:

Schema evolution.
Usage of git in our APIs supporting the development workflow.
Depending on the latest available partition with LAST and scheduling based on data values.
Code generated datasets used to compute AB tests metrics.

Today the platform is still under active development, with the goal to have most of Criteo datapipelines migrated to it. The code base with its workflow management and all the integrations is significant. Despite the project being 3.5 years old we still do massive refactoring to adapt to growth, simplify it every chance we get and add/improve features. We do so with ease and take this opportunity to thanks the scala ecosystem at large and the authors of the libraries that we use: cats (data types and type classes for functional programming), cats-effect (effect system library), fs2 (functional streaming library), fastparse (to build our parser), paiges (for formatting our SQL), doobie(for database queries), circe (for json APIs and configuration).

We still have lots of work ahead of us:
One field of work which is well under way is the switch from Hive to SparkSQL thanks to Spark 3 robustness. Spark offers a modern dialect with lambda functions and state of the art features enabling faster executions and more intelligent resource usage. We expect that the Adaptive Query Execution feature will reduce the manual tuning effort needed in prior versions of Spark and allow users to have very similar experience as when running their jobs using Hive.
An interesting applied research this project allows is optimization of the DAG. Because the SQL DAG is a data structure, there are many analysis that can be done on it, which can lead to optimization like selecting subexpressions to materialize. The research team at Microsoft which published that paper produced several other interesting ones that we want to take inspiration from to reduce our costs, especially when we will run on Spark. The team has an open position for an intern to work on this topic.

Many people have been involved in designing, implementing and on-boarding data-pipelines onto this platform. Our fellow SRE and Platform teams as well as our users help us shape the future of BigDataFlow and by the way several teams are looking for new Criteos to join us!

Careers at Criteo | Criteo jobs

Find opportunities everywhere. Choose your next challenge.

careers.criteo.com

Raph (@heapoverflow ) on behalf of the Data Processing team

References:

Airflow / Luigi / Azkaban / Oozie / Cuttle / https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 / https://flyte.org/ / Directed Acyclic Graph / F1 query / Trino / Presto / SREcon19 — SDKs Are Not Services and What This Means for SREs / NABD Conf’19 — Fixing the Big Data Development Cycle with SQL / https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-comparing-flyte-and-airflow-72c40d143aad / https://engineering.atspotify.com/2022/03/why-we-switched-our-data-orchestration-service/ https://dl.acm.org/doi/10.14778/3192965.3192971 / https://www.gerritcodereview.com/ / https://typelevel.org/cats/ https://typelevel.org/cats-effect/ / https://com-lihaoyi.github.io/fastparse/ / https://github.com/typelevel/paiges / https://tpolecat.github.io/doobie/ / https://circe.github.io/circe/ / https://fs2.io/ / https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf / https://careers.criteo.com/