Data
BigDataFlow: Continuous Delivery of data pipelines
A new article in the Criteo DevXDays series. You can find all the articles about Developer Experience at Criteo here. Data at Criteo is a core asset and the source of reports we provide to both audiences, external and internal. We are talking about a massive amount of data daily and need a proper workflow…
Scheduling Data Pipelines at Criteo — Part 3
The Proven Model in Production Building a successful Platform is a quest of the good abstraction level. If you’ve missed it, check out the previous articles in this series: Scheduling Data Pipelines at Criteo — Part 2 This week we deep dive into the key ideas leveraged by BigDataFlow medium.com Scheduling Data Pipelines at Criteo — Part 1 Introducing…
Scheduling Data Pipelines at Criteo — Part 2
The key ideas leveraged by BigDataFlow Bringing database query planner inside Workflow Management Systems Going back to the goal we introduced in Part 1 for our data pipeline platform: Users should only write the task How does the project infer statically ? the type of inputs and outputs the DAG of tasks The first idea leveraged…
Scheduling Data Pipelines at Criteo — Part 1
Introducing Criteo’s BigDataFlow project Data @ Criteo Data is a core asset for Criteo as it feeds our ML engine, it is the source of the reports that we provide to our clients and it is explored internally to gain insights.Every interaction of our clients with the Criteo platform is logged and all this data is…
