Engineering
-
Monitoring microservices — Central Monitoring: A tool for a global view of things
A bit of history Some years ago, Criteo switched from monolithic applications to microservices. With this new architecture comes challenges like monitoring hundreds of applications, all interacting with each other. At Criteo, there are several ways to introduce innovation. One of them is the yearly Hackathon! At the 2020 event, the fantastic Firewatch team aimed to…
-
Scheduling Data Pipelines at Criteo — Part 3
The Proven Model in Production Building a successful Platform is a quest of the good abstraction level. If you’ve missed it, check out the previous articles in this series: Scheduling Data Pipelines at Criteo — Part 2 This week we deep dive into the key ideas leveraged by BigDataFlow medium.com Scheduling Data Pipelines at Criteo — Part 1 Introducing…
-
Scheduling Data Pipelines at Criteo — Part 2
The key ideas leveraged by BigDataFlow Bringing database query planner inside Workflow Management Systems Going back to the goal we introduced in Part 1 for our data pipeline platform: Users should only write the task How does the project infer statically ? the type of inputs and outputs the DAG of tasks The first idea leveraged…
-
Scheduling Data Pipelines at Criteo — Part 1
Introducing Criteo’s BigDataFlow project Data @ Criteo Data is a core asset for Criteo as it feeds our ML engine, it is the source of the reports that we provide to our clients and it is explored internally to gain insights.Every interaction of our clients with the Criteo platform is logged and all this data is…
-
5 Do’s and Don’ts to restart a Hadoop cluster with no downtime
Our Hadoop cluster plays a pivotal role in our business operations. Operating a network of 3,000 advertisers, 5,000 publishers and 1.5 billion active shoppers simply wouldn’t be possible without a storage and compute environment that scales. Since beginning our Hadoop journey in 2012 we’ve continued to build on our big data operations, with a strong…
-
Serving growing user needs with automated tooling
Running a Big Data platform can offer a great deal of value to users, but only if that value is simple and easy to access. At Criteo, we’ve found that making our massive data sets accessible to our growing and geographically-distributed sales user base has come with certain challenges; not least the ability to scale…
-
Monitor Finalizers, contention and threads in your application
This post of the series details more complicated CLR events related to finalizers and threading. Part 1: Replace .NET performance counters by CLR event tracing. Part 2: Grab ETW Session, Providers and Events. Introduction In the previous post, you saw how the TraceEvent nuget helps you deciphering simple ETW events such as the one emitted when…
-
The need to scale bigger
At Criteo, hypergrowth is something we have gotten used to. Our business has grown from a single data center in Paris back in 2005, to a globally distributed, multi data center infrastructure, with over 35 000 servers and what is one of the largest Hadoop clusters in Europe. This blogpost is the story of how…
-
How to beat !dumpheap -stat?… with ClrMD
When you are dealing with large memory dumps, figuring out what instances of which types (i.e. the list of types sorted by size of their instances with their count) are stored in memory takes time. Sos in WinDBG provides the dumpheap -stat command that can take minutes. For example, on a 16.7 GB production dump,…
-
Grab ETW Session, Providers and Events
This post of the series shows how to easily listen to CLR events with the TraceEvent package. Part 1: Replace .NET performance counters by CLR event tracing. In the previous post, you saw that the CLR is emitting traces that could (should?) replace the performance counters you are using to monitor your application and investigate when…