In a world where distributed systems have become the norm and development cycles are accelerating, software quality can no longer be left to chance: it must be measured, verified, and integrated into the daily work of teams. At Criteo, this requirement is even more critical because services are numerous, interdependent, and essential to the proper functioning of the advertising platform.
This is the context in which AppChecker was developed — an internal tool designed to automate application compliance audits. Its goal is to provide an objective, standardized, and scalable evaluation framework to measure service maturity while supporting development, operations, and reliability teams. AppChecker is both a technical and strategic tool, helping strengthen the robustness of the company’s software ecosystem.
Why AppChecker?
Before AppChecker, compliance checks relied on manual processes — often lengthy, inconsistent, and open to interpretation. Each team applied standards in its own way, leading to quality gaps, frequent oversights, and even regressions in application monitoring. These checks were seen as cumbersome, sometimes arbitrary, and rarely documented centrally.
As reliability requirements grew, it became necessary to have a tool capable of providing automated, traceable, and shared evaluations. AppChecker meets this need by establishing a common language between teams, making checks more transparent, and helping prioritize corrective actions. It shifts the approach from ad-hoc control to continuous improvement.
AppChecker is also a driver of DevOps culture. By making standards visible and actionable, it encourages teams to adopt best practices while reducing friction between development and operations. It becomes a tool for dialogue, education, and accountability.
It also addresses internal compliance requirements, ensuring that services meet defined quality levels. In an environment where compliance is a strategic issue, AppChecker acts as a guarantor.
How did we build it?
Key Features
AppChecker stands out for its features, designed to integrate naturally into teams’ daily workflows. It runs automated checks on services, covering a wide range of criteria: adherence to development standards, deployment configuration, monitoring integration, documentation, alerting, permissions, and more.
Checks are grouped into thematic “groups” (e.g., observability, security, deployment), which are organized into “profiles” tailored to different contexts (such as reactivating CD or assessing maturity) or teams. This structure allows teams to run targeted or global checks as needed, customizing parameters at the profile level.
The web interface offers a smooth, intuitive experience. Results are clearly presented with explicit statuses (success, failure, ignored), links to error sources, and sometimes even suggested fixes. AppChecker doesn’t just flag issues — it guides teams toward resolution with actionable instructions.
Slack integration makes tracking corrective actions easier, by providing a report summary and a detailed list of failures. The tool becomes a true quality assistant, embedded in existing workflows.
Technical Architecture
Behind its simplicity lies a robust, modular architecture. It integrates with many internal services — Consul for service discovery, Git for source code analysis, Sentry for error monitoring, Marathon for container configuration, Slack for notifications — to collect the data needed for audits.
The architecture is designed for scalability: each check is an independent, easily testable, and extensible module. Profiles and groups are defined in configuration, allowing teams to customize audits without changing code.
AppChecker also includes an automatic scheduling mechanism for audits. Checks can run at regular intervals without manual intervention, based on rules defined by teams (by service, profile, or domain). Results are compiled into detailed reports sent directly to teams via Slack or email. This feature helps maintain consistent compliance over time and enables proactive tracking of gaps and corrective actions.
Types of Checks
AppChecker offers a rich, evolving library of checks. Some are technical, such as verifying the use of internal SDKs, updating JavaScript dependencies, or ensuring configuration files are present. Others are production-oriented, like validating Prometheus alerts, checking service name consistency in Consul, or verifying Grafana dashboards.
There are also governance-related checks: documentation presence, validation of internal repository links, access permission checks, and compliance with deployment best practices.
Experimental checks are available too, to test new standards or evaluate emerging practices, keeping AppChecker at the forefront of internal innovation.
Challenges
Developing AppChecker involved major challenges, starting with aligning standards across teams. Each team had its own practices, often tailored to specific contexts but sometimes conflicting with global standards. Common checks had to be developed for all teams. This required significant integration work with internal services like Consul, Git, Sentry, and Marathon, each with its own technical and security constraints.
Constantly evolving internal standards demanded a modular, extensible architecture to absorb changes. Managing errors and false positives required fine-tuning, improving audit relevance, trust in the tool, and corrective action efficiency — cementing AppChecker as a cornerstone of software quality at Criteo.
Another major challenge was adoption. To avoid being seen as a control tool, the project team focused on an intuitive interface, transparent communication, and strong operational value.
AppChecker in Action
Example: A team wants to reactivate its Continuous Delivery pipeline, suspended due to stability issues. It must prove its application meets maturity requirements: sufficient test coverage, change management processes, observability in place.
With AppChecker, the team selects the relevant profile, runs the audit, and gets a detailed report in minutes. The report lists executed checks, their status, detected errors, and recommended actions. The team can fix issues, rerun the audit, and validate compliance.
Another example: An SRE team wants to audit all services in a domain to identify weaknesses. It uses AppChecker to run bulk audits, aggregate results, and produce a maturity map. This approach helps prioritize efforts, plan improvements, and track impact over time.
Results and Benefits
Since its deployment, AppChecker has transformed how Criteo teams approach software quality. What used to be an occasional, urgent audit is now seamlessly integrated into the development cycle. The tool has shifted the mindset from reactive to proactive compliance.
Benefits are tangible: audits are faster, more reliable, and better understood. Checks that once took hours are now automated, freeing time for design and service improvements. AppChecker also detects gaps early, tracks maturity progress, and helps prioritize corrective actions systematically.
Finally, the tool has strengthened collaboration between developers and SREs. By establishing quality standards, it facilitates communication, aligns expectations, and turns audits into drivers of continuous improvement.
What’s next?
This flexibility paves the way for intelligent standardization, where each team retains some autonomy while adhering to a common framework. AppChecker is becoming a true software quality platform for evaluating, improving, and managing service reliability.
Eventually, it could integrate more deeply into CI/CD pipelines, trigger audits automatically on each merge, or feed organization-wide maturity dashboards.
The goal is to make AppChecker a technical governance foundation for managing quality at scale — a central reference, reporting tool, and driver of organizational transformation.
AppChecker is not just an automated audit tool. It’s an essential component of Criteo’s development ecosystem. By making quality standards measurable, easing their adoption, and supporting teams in implementation, it helps make reliability a shared, concrete, and achievable goal.
Its ambition is to provide a clear, adaptable, and intelligent framework for evaluating and improving service quality. In a world of growing software complexity, AppChecker offers a pragmatic, elegant, and effective solution. It doesn’t replace human expertise — it amplifies it. It doesn’t dictate best practices — it makes them visible, actionable, and shared.
And perhaps its greatest strength: making quality a collective effort, supported by technology, in the service of excellence.
