Big Data
-
BigDataFlow: Continuous Delivery of data pipelines
A new article in the Criteo DevXDays series. You can find all the articles about Developer Experience at Criteo here. Data at Criteo is a core asset and the source of reports we provide to both audiences, external and internal. We are talking about a massive amount of data daily and need a proper workflow…
-
Scheduling Data Pipelines at Criteo — Part 2
The key ideas leveraged by BigDataFlow Bringing database query planner inside Workflow Management Systems Going back to the goal we introduced in Part 1 for our data pipeline platform: Users should only write the task How does the project infer statically ? the type of inputs and outputs the DAG of tasks The first idea leveraged…
-
Scheduling Data Pipelines at Criteo — Part 1
Introducing Criteo’s BigDataFlow project Data @ Criteo Data is a core asset for Criteo as it feeds our ML engine, it is the source of the reports that we provide to our clients and it is explored internally to gain insights.Every interaction of our clients with the Criteo platform is logged and all this data is…
-
Serving growing user needs with automated tooling
Running a Big Data platform can offer a great deal of value to users, but only if that value is simple and easy to access. At Criteo, we’ve found that making our massive data sets accessible to our growing and geographically-distributed sales user base has come with certain challenges; not least the ability to scale…