Scheduling Data Pipelines at Criteo — Part 1

Introducing Criteo’s BigDataFlow project

Data @ Criteo

Data is a core asset for Criteo as it feeds our ML engine, it is the source of the reports that we provide to our clients and it is explored internally to gain insights.
Every interaction of our clients with the Criteo platform is logged and all this data is transferred to our Hadoop clusters, where it is transformed, joined and learned on by many different teams across the organization.

From each raw byte that is logged we generate another 3 bytes of transformed data. We produce almost 1PB of new data per day for a total of 85 PB of storage (ignoring the replication needed to be resilient to hardware failure).
This data pipeline represents thousands of automated processes (let’s call them tasks) per day, that need to be orchestrated. Those tasks are also often named “jobs”, and even though it feels weird to talk about “Spark task” I’ll stick with the word task in this post for consistency.

Workflow management systems propose a solution to implement that orchestration.

Widely used workflow management systems

There are multiple existing open-source projects in the industry like Airflow, Luigi, Azkaban, Oozie. At Criteo we use Cuttle which has similar capabilities.

To our knowledge these systems are based on a DAG (Directed Acyclic Graph) of tasks. For each task, users of such systems are asked to provide, among other things:

the dependencies of the task
some executable to concretely run the task

With this information a workflow management system launches a given task as soon as all of its dependencies are available. In naive workflow management systems a task can check if its dependencies are ready and fail if they are not, to be re-scheduled later. By being aware of dependencies, a workflow management system can mutualize this dependency state checks work and avoid retrying a task because its dependencies are not ready. Once the listed dependencies are met, the task execution is launched. It is assumed that task inputs are available but also correct and in the proper format.

Workflow management systems provide great value thanks to:

their built-in integrations with various ecosystems like Hadoop
their UI allowing to visualize the state of the pipelines
their support to operate the pipelines of tasks by pausing and backfilling tasks.

Backfill definition = It happens that one wants to re-execute a task, discarding the initial output and replacing it by the new execution output. Backfills are often due to changes in the task inputs (new data arrived, invalid data got fixed/removed), or to bug-fixes / changes in the task code.

Searching for a better model

Our experience with Cuttle is that some issues remained:

Consistency: Having to maintain both the task and the DAG is a source of errors. It’s easy to forget to declare a dependency or get a time-shifting computation wrong leading to mismatches between dependencies declared in the DAG and the ones actually used by the task.
Lineage: There is no support for introducing breaking changes in tasks like removing a column from an output. Data producers have to detect that a change in their task will break the task of one of its clients. Doing so on their own is challenging for a data producer. Good lineage tooling and structured data ownership can help them reach out to their clients and involve them in the identification of impacts. Data producers and data consumers teams owning clients that need some adaptation need to negotiate both the update of the clients task and orchestration of the update release. From our experience, this is by far the most time consuming task required to rollout a breaking change.
Testing: Testing tasks independently from the scheduling is relatively easy: the developer has to build a set of parameters passed to the task, using an output destination he/she is allowed to write to and launch it with his/her credentials. However integration tests of a task (or even a chains of dependent tasks) and scheduling need some dedicated code in the projects and toil operations.

Because mistakes happen, those usability issues can lead to detection of problems once changes are merged. That’s why we have preprod environments, right ? Well yes, but detecting issue at runtime when they could have been detected at build time slow down the development cycle. Furthermore preprod isn’t an exact replica of production, especially given Criteo scale. Testing data-pipelines in conditions very close to production requires some investment from the teams owning the Cuttles: most of the subtleties presented in this blog post series on web-services tests apply to data-pipelines.

Other companies faced similar issues. Lyft and Spotify recently shared some very interesting blog posts on data pipelines scheduling. They detail some of those issues as well as others and explain the improvements they got using Flyte. Among other innovations, Flyte leverages function calls to define workflows (aka DAG) and python typing support to identify schemas of input/output. Within Criteo we use Hippo, a scala library, which follows a similar approach to generate Cuttle workflows. This approach relies on existing programming language compilers to detect some of those issues at compile time. Our Hippo library is used by numerous projects though it demands some expertise in Scala and software packaging (data dependencies become compile dependencies thus add maintenance complexity) which makes it unfit for general usage by all Criteos data producers.

Ease of use is a challenge for data workflow schedulers. Ease of use with minimal training even more so. Alongside data engineer roles who work on reliably transforming data day in and day out, there are a lot of roles that have punctual need to automate data transformations. For example, data analysts work is focused on building insights and knowledge from data. A data analyst will identify some need for data transformation automation useful to his task, essentially a mean to an end like receiving daily data extract or sending them to a client. A self-service web application called Dataflow was built at Criteo to address a set of common use cases/scenarios. Through wizards the user can create and edit SQL query templates and configuration files. This web application has a review system build-in so users can remain in the same application and avoid using Gerrit (the code review software used at Criteo). Dataflow, like Hippo, generates Cuttle workflows. It has proven to be very useful even though it did not solve any of the consistency, lineage and testing issues. At peak, around 2 thousand tasks were running on Dataflow instances: mostly tasks with few dependencies at the end of Criteo data graph. Lack of support for maintenance and deprecation of those tasks is not as critical as it is to ones at the root of the data graph.
DBT is a popular open-source project that started with a similar approach as Dataflow. Its roadmap is a great perspective in the ecosystem and resonate on several topics with the story we share here.

Hippo and Dataflow were both useful to niche use cases.

Aiming to increase the velocity of the development of data pipelines for all Criteos, we decided to go further and create a workflow management system language with the following goal in mind:

users should only write the task

From the task, the project infers statically:

the type of inputs and outputs
the DAG of tasks

This enables detecting impacts of breaking schema changes instantly. With this information it’s possible without any extra work from the user than providing a period of time to test: to launching the task, or chains of dependent tasks, on actual prod data to validate runtime behavior as well as output data values. Both of those fastening the development feedback loop and thereby productivity.
Orchestration of the release of different versions of tasks DAGs can then be resolved with schema evolution integrations.

This idea arose from numerous internal and external discussions and was presented in conferences:

Remainder of this 3 parts blog post series will detail the project and share our experience with it now that it has passed the test of time.

Introducing: the BigDataflow project

Let’s start with a concise description of what the project is:

A workflow management system

Scheduler for timeseries and cron tasks
Process tasks and their dependencies, as a graph (DAG)
Operation tooling: user can pause and backfill datasets
Developer tooling: dry-runs, unit tests, non-regressions, etc…

A SQL Language

HiveQL with a superset for scheduling
Can invoke code from SQL such as Spark jars, Python PEX through Foreign Transformation Interface as well as integrations to read data from variety of sources.

COPY functions to export data to various systems: databases, file-servers, metrics, email, etc ..

TAGS to define latency SLO
No code required, only SQL

A Platform

Centralized
• One application instance
• One repository for all
Self Service: Multi-tenancy is not achieved at the cost of ownership
•Impersonation: Tasks run using teams user accounts using predefined Hadoop resources.
• Reviews: Teams autonomously review changes to their datasets.
• Continuous Deployment: BigDataFlow handles releases for user, once a commit is merged, it will land in production at the next paced deployment.
• No need to bootstrap / deploy / maintain per team
• Is operated by one single team with on calls
With interfaces to build and operate pipelines
• Web Interface to visualize and operate pipelines
• Web IDE to edit pipelines
• Command Line Interface to edit and operate pipelines

In the following posts we will first explain the design choices by diving a bit into the implementation of the BigDataFlow project and then share our experience with the project.

Raph (@heapoverflow ) on behalf of the Data Processing team

Scheduling Data Pipelines at Criteo — Part3

The Proven Model in Production

medium.com

Scheduling Data Pipelines at Criteo — Part 2

This week we deep dive into the key ideas leveraged by BigDataFlow

medium.com

Careers at Criteo | Criteo jobs

Find opportunities everywhere. Choose your next challenge. Find the job opportunities at Criteo in Product, research &…

careers.criteo.com