Based on a talk given at Devoxx France 2025 by Emmanuel Guérin

At Criteo, we serve more than 22,000 customers and deliver billions of ads every day. Behind that business is a large engineering organization and, over time, a codebase that grew from a few projects in one repository to more than 1,000 repositories and 6,500 modules.

At that scale, CI is no longer only a question of compiling code and running tests. The real challenge is dependency management: keeping a large multi-repository system consistent while code changes continuously in many places at once.

To explain the solution we ended up with, we need to go back to the way our build evolved. This article is a brief history of that evolution, and of the system we built around it: MOAB, our “build from main” approach.

MOAB is a play on words between MOAB (“Mother of All Bombs”) and “Mother of All Builds”.

Our dependency pipeline

Humble beginnings

In the early days, things were simple enough. A few modules lived in a single repository, everything was built from source, and what was tested was also what was assembled. As long as the number of projects stays limited, this model works very well.

The monorepo model has a useful property: at build time, there is only one version of each module in the workspace. The build system resolves everything from source and the final result is consistent by construction.

But like any model, it starts to show limits as the organization grows. Build times increase, cloning becomes heavier, and development workflows become harder to scale. Around 2011, with about 150 developers, we tried to absorb that growth by creating more branches in the same repository. It solved very little and created a lot of merge pain.

In 2012, we moved to a multi-repository model. Repositories could now evolve more independently, have their own pipelines, and publish their own artifacts. It looked like the right answer at the time.

It was not long before it created a different kind of problem.

Versions in source code

Once repositories are built independently, internal dependencies usually become explicit versioned artifacts. A project no longer depends on “module A”, but on “module A version 1.2.3”.

That works well enough when the graph is small, but it quickly becomes difficult to manage when many teams publish and consume libraries at the same time.

The first issue is that upgrading a library becomes a coordination problem. A library team can publish a new version, but it has no practical way to validate all downstream users before they decide to upgrade. Some clients upgrade quickly, some much later, and some not at all. The larger the graph becomes, the less clear it is what is really compatible with what.

The second issue is transitive dependencies. Two modules may be individually correct, but when they are assembled in the same application, they may expect different versions of the same library.

What B was built and tested with:

  Module B
     |
     v
  Module A v1.x


What C assembles after upgrading A:

        Module C
       /        \
      v          v
Module A v2.0  Module B
                 |
                 v
           expects A v1.x

  direct dependency in C: A v2.0
  transitive expectation from B: A v1.x
  => mismatch hidden inside CCode language: JavaScript (javascript)

In this example, module B was validated against A v1.x. Later, module C upgrades its direct dependency to A v2.0 while still depending on B. The resulting application contains a combination that was never really validated together.

This kind of mismatch is easy to miss because it is introduced transitively. The build can still look correct from the point of view of C, while the inconsistency only appears later, during integration or at runtime.

By the end of 2012, the situation had become serious enough that propagating one library update through the system could take weeks. That was the point where we had to rethink the model.

The red pill moment

In December 2012, after several months during which our old model had effectively blocked releases, we launched what became known internally as Red Pill, a reference to The Matrix: the moment we chose to confront the real limits of that model instead of continuing to work around them.

We moved to a single main branch for development and release. We pushed harder on engineering practices such as code review, testing, and feature flags. We created a dedicated team to work on build and automation. And, most importantly for this article, we decided to remove internal dependency versions from source code entirely.

That decision became the basis of MOAB: “build from main”.

It also came with a more standardized workflow:

• propose a change
• validate it in CI before merge
• review it
• merge it to the main branch
• build and publish it through the usual pipeline

None of this was especially exotic, but getting every team to follow the same path was an important part of making the model work.

This point is worth mentioning because the solution was not only technical. It required dedicated people, a common workflow, and enough management support to apply the same rules across the organization.

Introducing the MOAB: Build from main

The core principle is simple: when building a repository, use the current state of the main branch for all internal dependencies.

In other words, developers declare what they depend on, but not which version of that dependency they want. The build system determines the versions consistently for the whole codebase.

This changes the ownership model quite a bit. Application teams no longer control the version of internal libraries they consume. Library teams, on the other hand, must be much more careful about compatibility because their changes are quickly validated against the current state of the rest of the system.

In practice, the difference is visible directly in build files.

Traditional internal dependency declaration in Gradle looks like this:

plugins {
  id("java-library")
}
dependencies {
  implementation "my.other:project:3.2.5"
  testImplementation "junit:junit:4.12"
}Code language: JavaScript (javascript)

With MOAB, internal dependencies no longer carry a version:

plugins {
  id("com.criteo.moab-module")
  id("java-library")
}
moabDependencies {
  implementation ":my:repository:my:project"
}
dependencies {
  testImplementation "junit:junit:4.12"
}Code language: JavaScript (javascript)

The same idea applies in .NET. A traditional declaration might look like this:

<PackageReference Include="MyDependency" Version="1.0.2" />Code language: HTML, XML (xml)

With MOAB, the internal dependency becomes:

<PackageReference Include="MyDependency" />Code language: HTML, XML (xml)

External dependencies still need explicit version management. We will come back to that later.

How does it work?

A virtual trunk

Even though the code lives in separate repositories, the build system needs a global view of what exists and what depends on what.

For that, MOAB maintains what we call a virtual trunk. In practice, it is a registry of the repositories that are part of the system, together with the commit SHA selected for each of them. Each change updates that registry and creates a new global snapshot of the codebase.

Conceptually, this gives us something very close to a monorepo commit, but built on top of many repositories.

Source-based versioning

Once we have a global view of the sources, we still need to assign versions to modules. Those versions are not chosen manually. They are derived from the source code itself.

The process is roughly the following:

1. Assign a build number to the global build.
2. Extract the list of modules from the repositories involved.
3. Compute a source fingerprint for each module from its own sources.
4. Derive a version fingerprint by combining that source fingerprint with the resolved versions of the module’s dependencies.
5. Choose a published version from that version fingerprint.
6. Record the mapping between the version fingerprint and the published version so the same fingerprint can reuse the same version later.

The result is a module registry for that build: a complete list of modules and their corresponding versions.

The following diagram illustrates that process:

One useful consequence is that unchanged modules can reuse existing versions. Once combined with caching, this makes a large part of the system effectively free to rebuild.

Developer workspaces

The build model also has to remain usable for developers locally.

Here we use two modes. The first one is full build from source: all repositories involved in the change are checked out and built together. The second one is partial checkout: only a subset of repositories is present locally, while the missing dependencies are resolved from artifacts produced by CI.

This is important for day-to-day work. If a developer needs to change two repositories at once, the workspace can treat them as source dependencies. If only one repository is checked out, the build falls back to the last known compatible artifacts for the rest of the graph.

Distributed CI and caching

As the system grew, our CI architecture also had to evolve.

The first version was close to a giant monorepo build: one large workspace trying to build everything. It quickly inherited the same drawbacks. A failure in one area could impact many others, and scaling the whole thing on a single machine was not realistic.

The second version introduced distributed CI operations, one per repository, with task-level caching.

The third version was made possible by source-based versioning. Because each CI operation can be fingerprinted from the module versions it contains, we can determine whether it has already been built successfully before. If it has, the whole operation can be skipped.

This is also true for version assignment itself. If a module fingerprint has already been seen, the corresponding version can be reused. In practice, this means that a significant share of CI executions never need to happen at all.

Today, between 50% and 75% of CI executions are skipped thanks to caching.

Quality and validation

A green repository build is not enough to tell whether the system is healthy.

In a setup of this size, repository status is too coarse. What matters is the effect of a change on the individual components that will later be assembled into products.

For that reason, MOAB produces validation events at the component level. A CI operation does not only say “this repository passed” or “this repository failed”. It emits validation steps for the components it contains, and those components can themselves depend on validation results from other components.

This gives us a quality graph that follows the dependency graph much more closely.

It solves an important operational issue: a repository may still look green in isolation while one of the components it depends on is already broken elsewhere. Without component-level propagation, the system can appear healthier than it really is.

This information is exposed through Evergreen, an internal dashboard that shows which components are broken and which ones are merely victims of an upstream failure. That distinction matters when deciding what is safe to deploy and where the actual root cause sits.

Some numbers

The system today supports:

These numbers help explain why simple per-repository version management eventually stopped being practical for us.

What we learned

A different contract between teams

One of the interesting outcomes of this model is the way it changes responsibilities.

Application teams give up direct control over internal dependency versions. In return, they no longer need to spend time manually upgrading internal libraries or wondering whether they are behind.

Library teams take on more responsibility. Their changes are propagated much more directly through the system, so backward compatibility and validation become even more important than before.

This only works if developers trust the CI. If the system is unstable or flaky, people very quickly stop believing the result. In a model where so much responsibility is delegated to automation, that trust is essential.

Easy dependencies are not always good

Another lesson is that making dependencies easy to add also makes them easy to abuse.

A common failure mode is the small helper hidden inside a much larger module. A developer notices a convenient utility in module A, adds a dependency on A from module B, and unintentionally pulls a large chunk of logic and transitive dependencies into B.

The direct effect is larger dependency graphs, longer feedback loops, and more rebuilds. The longer-term effect is architectural erosion. Over time, applications become more coupled than they need to be.

That is one of the trade-offs of the model: dependency management becomes much smoother, but dependency discipline becomes even more important.

How about external dependencies?

Internal dependencies fit the “build from main” model very well. External dependencies do not.

For open-source libraries or third-party SDKs, there is no realistic way to pretend that one version of everything will always work for everyone. Those dependencies still need explicit lifecycle management.

Our approach there is much more traditional: propose upgrades automatically, follow their effect through the dependency graph, and make ownership clear when an upgrade breaks something.

MOAB removes a large class of coordination problems, but not all of them.

This article was a brief overview of how our multi-repository CI evolved at Criteo and why we ended up removing internal dependency versions from source code.

The main lesson is that, at large scale, dependency management becomes a central part of the build problem. As long as versions live in source code, the distance grows between what teams validate locally and what the final system actually assembles.

Build from main is our answer to that problem. It required a significant tooling investment, a dedicated team, and a fairly strict engineering model. It is not a universal solution, and many organizations do not need this level of machinery.

But once the codebase reaches hundreds or thousands of repositories, the coordination cost of versioned internal dependencies grows surprisingly fast. For us, removing those versions from source code was one of the most impactful decisions we made.

Emmanuel’s talk in French 👇

Further reading

Earlier Criteo Engineering posts covered other steps of the same journey:

• Why you don’t need a mono-repo but could just build from source
• Implementing build from source with a mono-repo vs a poly-repo