A new article in the Criteo DevXDays series. You can find all the articles about Developer Experience at Criteo here.

Data at Criteo is a core asset and the source of reports we provide to both audiences, external and internal. We are talking about a massive amount of data daily and need a proper workflow management system to orchestrate every piece involved.

After trying and experimenting with different solutions, an internal project started to create a better system that would fit our needs. The BigDataFlow project was born.

The BigDataFlow project

To better understand why we decided to create something and what we managed to achieve, check the three articles we published about it last year 👇

Scheduling Data Pipelines at Criteo — Part 1

Introducing Criteo’s BigDataFlow project

medium.com

Scheduling Data Pipelines at Criteo — Part 2

This week we deep dive into the key ideas leveraged by BigDataFlow

medium.com

Scheduling Data Pipelines at Criteo — Part 3

The Proven Model in Production

medium.com

Continuous Delivery of data pipelines

As a platform, BigDataFlow handles releases for users, once a commit is merged, it will land in production at the next paced deployment. To make continuous deployment possible without causing incidents, BigDataFlow has a couple of useful tools:

• Static Analysis, which helps to identify errors introduced by a new release.
• Command Line Interface, which allows editing, testing, operating and showing info about the pipelines.

The following talk covers those two previous points, showcasing the safe experience users have when editing data pipelines. This talk was presented at Criteo DevXDays’ latest edition past year. 📹