In recent years, our focus has been on defining Service Level Objectives (SLOs) for HTTP API endpoints. While this approach has proven beneficial for development teams, it has also had less impact on external customers and product teams. Users typically expect SLOs to be associated with features rather than individual endpoints or services. Communicating the reliability of a feature is more straightforward and meaningful than discussing the reliability of an endpoint or service.
The Shift to Feature-Based SLOs
The primary goal of our project was to establish SLOs for individual features, which are significantly more meaningful to both internal and external users compared to SLOs focused solely on service endpoints. Our internal and external customers have clearly expressed a desire for more regular visibility and accountability regarding the availability of our applications. They want consistent reports that detail the quality of service we provide, ensuring they are always informed about the performance and reliability of our features.
Given that Criteo places an extremely high emphasis on technical performance, it is crucial for us to have robust applications in place, as a consequence our monitoring and alerting systems must accurately measure and respond to any under-performing applications. This is essential not only for maintaining our high standards but also for assuring our end clients of the stability and reliability of the features they depend on.
Introducing Service Level Objectives is a pivotal step in enhancing the technical performance of our Commerce Growth features. SLOs are critical for maintaining a healthy and complex system of applications. They provide clear targets and benchmarks that help us ensure our applications are performing optimally.
Internal Objectives and Response Processes
Internally, we aim to define specific objectives for latency and error rates. We already established a response process for when these internal objectives are not met. This process will translate both technically, in terms of endpoints, and functionally, in terms of features. By doing so, we can ensure that any issues are addressed promptly and effectively.
We also want to ensure that our reporting is transparent and accessible at the client level. This means providing regular updates so that everyone is aware of what is happening and can feel confident when discussing these issues. With this solution, it will be straightforward to determine the SLO of a feature because we will have a clear understanding of all the endpoints involved in that feature. Furthermore, when an alert is triggered for a specific endpoint, it will be easy to identify which feature is affected. This clarity will help us respond more quickly and maintain the high level of service our clients expect.
Feature Monitoring: A New Approach
Feature Monitoring is a web application, made of an Angular UI and a C# API. The main part of the UI is a dashboard displaying a tree view of:
- The features that need to be monitored, with their status, either “success” or “failure.”
- The SLOs of the features, which are based on available SLIs.
- The components involved in the features, with the current value of their SLI.
Features can be added as top-level features or sub-features. As the dashboard is a tree view, the status of the components is used to compute the status of the feature and is propagated up to the top-level feature, giving a status to all features: if the status of a leaf of the tree is “failure,” all the nodes above are also “failure.”
Components are the technical parts of the features. For now, they are data centers or endpoints of services.
SLOs are based on available SLIs: the pass rate of an endpoint or the latency of an endpoint. They are reviewed and approved/rejected by the team owning the service, to make sure they are aware of any changes in their scope.
Some features may share the same component. In this case, a unique SLO is computed for this component with the most strict value. For example:
- With an SLO on the latency, the lowest SLO is used.
- With an SLO on the pass rate, the highest SLO is used.
Prometheus Integration
Once the configuration of services, features, SLOs, and components is complete, Prometheus takes over to automatically create the necessary alerts. This process involves Prometheus loading data directly from the Feature Monitoring system. Here’s how it works:
- Threshold Calculation: The threshold for each alert is the most strict SLO among the SLOs of the components. This ensures that the alerting system is finely tuned to the performance expectations and requirements of each feature.
- Receiver Configuration: The receiver, which is responsible for handling the alerts, is set up according to the service configuration. This means that alerts are directed to the appropriate teams or individuals who can take immediate action to address any issues.
Challenges in Adoption
The adoption of this new tool has presented several challenges that we needed to address. One of the primary difficulties was that the system heavily relied on a deep understanding of the relationships between various features and their corresponding endpoints. This dependency meant that we first had to identify and enlist individuals who possessed the necessary knowledge to define the feature repository accurately.
Once the feature repository was created, the next step involved finding a significant number of people to carry out the detailed mapping process. This mapping was a bit tedious to establish because it required functional knowledge of every endpoints in a given service.
As a result, maintaining this mapping has become an ongoing responsibility for the teams involved. They must regularly update and verify the mappings to ensure they remain accurate and reflective of any changes in the system. This continuous maintenance is crucial for the tool’s effectiveness and reliability and is something we are looking to automate.
Future Improvements and Potential
In the future, this solution has the potential to be utilized for providing comprehensive data to report the status of individual features. This capability could be particularly beneficial for creating a status page that offers a high-level overview of key features, such as “Commerce Growth.” Such a status page would allow users to quickly and easily understand the current state and performance of these critical features.
Additionally, the user experience could be significantly enhanced by developing a more advanced mapping editor. This improved editor would facilitate the ongoing maintenance of the feature repository, making it easier for teams to update and manage the mappings between features and endpoints. A better mapping editor would streamline the process, reduce errors, and ensure that the repository remains accurate and up-to-date.
Furthermore, there is the possibility of integrating new Service Level Indicators (SLIs) into the application. By incorporating additional SLIs, we could provide a more comprehensive and nuanced view of feature performance. This would allow for more precise monitoring and reporting, ultimately leading to better decision-making and improved reliability of our services.
Conclusion
Thanks to this innovative tool, we now have the capability to monitor and report the performance of our services from a functional perspective rather than merely a technical one. This shift allows us to generate comprehensive reports that are more meaningful to our customers, providing them with clear insights into the quality and reliability of the features they use.
By basing our alerting system on this functional monitoring, we can ensure that any issues are promptly identified and addressed, maintaining high standards of service. This approach brings our technical teams closer to the needs of our customers, even within the complex environment of microservices. It enables them to focus on what truly matters to the end-users, ensuring that the services we provide are both reliable and efficient.
Product managers also benefit significantly from this tool. They can gain a clear view of feature performance without needing to delve into the technical details of each implementation. This allows them to make informed decisions that align with customer needs and expectations, ultimately enhancing the overall user experience.
For support teams, the tool provides a streamlined way to quickly identify which features are impacted during incidents. This rapid identification process enables faster resolution times and minimizes the impact on our customers. By having a clear understanding of the functional status of our features, support teams can provide more effective and timely assistance, ensuring that any disruptions are swiftly managed.
Overall, this tool represents a significant advancement in how we monitor, report, and respond to the performance of our services. It aligns our technical efforts with customer expectations, enhances decision-making for product managers, and improves the efficiency of our support teams. This holistic approach ensures that we continue to deliver high-quality, reliable services that meet the needs of our users.
The journey to implementing feature-based SLOs has been challenging but immensely rewarding. By shifting our focus from technical endpoints to functional features, we have created a more meaningful and user-centric approach to monitoring and reporting service performance. This tool not only enhances our ability to respond to issues but also aligns our technical efforts with the needs and expectations of our customers.
As we continue to develop and refine this tool, we remain committed to maintaining the highest standards of service reliability and performance. By leveraging the insights and capabilities provided by feature-based SLOs, we can ensure that our services not only meet but exceed the expectations of our customers. This approach allows us to be more proactive in identifying and addressing potential issues, ultimately leading to a more stable and reliable user experience. Our dedication to continuous improvement and customer satisfaction drives us to innovate and enhance our monitoring and reporting systems, ensuring that we deliver exceptional value and performance in every aspect of our services.
