Leveraging Commerce Data for Outcome-Based Relevancy in Agentic Recommendation Systems

TL;DR

Most agentic commerce recommendation services today rely on content embeddings or mainstream LLMs (e.g., Claude, Llama, GPT). While effective for semantic understanding, these models are not grounded in real shopping behavior and are not optimized for commercial objectives such as clicks or conversions.

In this article, we show that leveraging commerce data at both the retrieval and re-ranking stages leads to substantial outcome-based relevancy gains:

+60% uplift in outcome-based relevancy for SKU re-ranking when incorporating transaction-derived popularity signals.
+37% uplift in outcome-based relevancy for SKU retrieval using a content embedding fine-tuned on organic clicks, compared to state-of-the-art zero-shot text encoders (MiniLM-L12, Gemma, Qwen).

These results highlight the importance of commerce-trained models for building high-performing agentic recommendation systems.

Introduction

Agentic commerce recommendation services are tailored to recommend the best products fitting a user query.

Metrics of interest

Throughout this article, we use relevancy in a sense that goes beyond surface-level semantic matching. Specifically, we focus on outcome-based relevance, commonly referred to in the recommendation systems literature as performance-based recommendation. Rather than assessing relevance solely through textual or semantic similarity between a query and product content, performance-based recommendation measures relevance through observable user outcomes, such as clicks, purchases, or downstream engagement. This distinction is critical in commerce settings, as it directly shapes how recommendation systems are evaluated and optimized.

Accordingly, there are two main routes used to evaluate agentic commerce recommendation services:

Accuracy (semantic relevance) answers the question: “Does this product description semantically match the user query?” (e.g., price, color, material). This metric has been used, for instance, to evaluate OpenAI Shopping Research, where 64% accuracy was reported. High accuracy can be achieved using retailers’ catalog data alone.
Outcome-based relevance answers the question: “Does this recommendation satisfy the user’s underlying need, as evidenced by real user behavior?” While semantic similarity is a necessary prerequisite for a good user experience, it is not sufficient. Two products may be equally accurate from a content standpoint (brand, category, attributes) yet differ significantly in their ability to meet user expectations, drive engagement, or lead to a purchase. Outcome-based relevance therefore represents a stronger and more user-centric notion of relevance, as it captures user intent through real commerce outcomes rather than inferred intent from text alone. As a result, commerce data — clicks, sales, and other behavioral signals — are essential to accurately measure and optimize relevance in agentic recommendation systems. Without grounding models in these outcomes, recommendation quality remains limited to proxy signals that fail to reflect what truly satisfies users.

Architecture of agentic commerce recommendation services

The most common and efficient architecture is a two-step one, detailed below:

Product retrieval happens first and involves getting a list of candidate products to ensure accuracy and capture one part of accuracy. It’s done using content embeddings and KNN search, optimised on commerce data.
Product re-ranking then orders the candidates products for maximizing outcome-based relevancy. It’s done using both query recommendation accuracy and commerce-driven scores, such as how much sales a given product drove. It also comes with recommendation explanations, sometimes coined reasoning.

We will deep-dive into the benefits of leveraging recommendation systems, optimized for outcome-based relevancy, into an agentic commerce recommendation service, compared to only using the content information of the returned products (e.g., title, category, or description). More precisely, we will present two benchmarks focusing on the product retrieval and re-ranking stages to illustrate the benefits of leveraging commerce data on top of content to target outcome-based relevancy for the end user.

Product Retrieval

Experimental Design. We conducted our evaluation using a dataset of user search queries paired with the corresponding clicked products. To evaluate outcome-based relevancy for the end user, we measured the ability to identify the product actually clicked for a given query from alternative, negative products. We used the actual rank of the clicked product for each user query to assess outcome-based relevancy, from a set of K = 400 products. We then normalized this rank to end up with the so-called reciprocal rank metric. A higher value for this metric indicates that users are more likely to find what they’re looking for among the top results. Negative candidate products are defined as alternative clicked products (from other user search queries) sharing either the same brand or the same category as the target product. This forces the model to go beyond surface-level semantic similarity between the query and product text, and instead captures user intent and product relevance.

Results. We compared our CLEPR model against widely used open-source text encoders developed by major companies (Google, Microsoft, Meta, Alibaba). All of the embedding models are given the same data as input: the search query and the products, represented by (brand, category, title).

Relevancy here is a shortcut for outcome-based relevancy.

Our results show that CLEPR outperforms zero-shot text encoders by 37% in outcome-based relevancy on average, while being smaller and more efficient.

Product Re-Ranking

In this section, we aim to assess the uplift brought by commerce data for product recommendation outcome-based relevancy. We focused on the re-ranking stage and used a proprietary model CLEPR, leveraging commerce data, and compared the latter to baselines not trained on such data.

Experimental Design. As before, we assumed that a set of candidate products (here K = 400) has been selected after a SKU Retrieval step leveraging our embedding model CLEPR. These products, deemed relevant, have been selected by thresholding the normalized CLEPR score (defined as the inner product between the user search query and product embedding). We are now interested in comparing different product re-ranking strategies, including one leveraging commerce data, to maximise some outcome-based relevancy metric. More precisely, we used the product re-ranker as baseline taking the CLEPR similarity score to rank products. We compared this baseline strategy with the one leveraging a simple PSales model outputting the probability of a product being sold once displayed and clicked. This PSales model is simply the proportion of sales observed during the previous 7 days, which could be further enhanced using the output of a deep learning model. As an outcome-based relevancy metric, we considered the sales rank, which stands for the position of the purchased product in the ranked product recommendation list. As before, we normalized this ranking metric to end up with a user-friendly metric indicating the likelihood of finding the product users would buy among the top results.

Results. The following table shows the results of our benchmark regarding the product re-ranking stage. These results indeed outline that leveraging commerce data to maximise outcome-based relevancy for agentic commerce recommendation service constitutes an improved value for customers. More precisely, the relevancy, i.e., here the likelihood of finding the right product the user would buy in the recommended ones, increases by 60% using our commerce data.

The CLEPR solution is trained solely on organic clicks and does not include bias terms associated with sales events. Consequently, at this stage, we cannot confidently report a global uplift across the end-to-end pipeline based on sales conversion metrics commonly used by advertisers and retailers. Still, our benchmark allows us to provide sufficient independent uplifts for both the SKU retrieval and re-ranking steps, which provides confidence in at least a similar global uplift for an agentic commerce recommendation service.

What’s Next

Following our two recent articles on agentic commerce recommendation services, we will continue this blog series with deep dives into key enabling technologies, including multimodal embeddings and LLMs for efficient product search, LLMs and fine-tuning techniques for product recommendation explanations (reasoning), and deep learning–based product re-rankers.

How RecSys & LLMs Will Converge: Architecture of Hybrid RecoAgents

Authors: Flavian Vasile, Martin Bompaire

medium.com

Criteo Boosts Ad Performance with Latest AI Advancements

Author: Béranger Dumont

medium.com

Appendix

Accuracy Metric in Product Retrieval

In this section, we aim to assess the uplift brought by commerce data for product recommendation accuracy, defined previously. To this purpose, we focused on the product retrieval stage and compared one of our embedding model solutions, coined CLEPR (Contrastive Language Embedding for Product Retrieval), with state-of-the-art zero-shot embedding models from the MTEB leaderboard of Hugging Face. Our solution stands for a 120M parameters and two-tower bi-encoder model (we compute separate embeddings for the query and each of the products) using Multilingual-MiniLM-L12-H384 as a backbone. It has been fine-tuned on organic click data and currently powers our keyword-based product recommendations. Each product is characterised by its brand, category, and full textual product title.

Experimental Design. To assess product recommendation accuracy given a user query, we use a dataset of tuples (user search query, product, accuracy label), where labels are binary and assigned by human annotators. We report ROC-AUC (percentage) as an offline metric and perform a single train/test split across all models.

When fine-tuning or post-training models using human-annotated data, only the training split is used, and post-training is limited to 50% of the annotated set. While this dataset has known limitations (e.g., sample size), and ROC-AUC is not an ideal metric for recommendation quality, it currently provides a reasonable proxy. We expect to revisit both datasets and metrics as we continue to develop benchmarks for agentic AI.

Results. The table below depicts the results of our benchmark for the accuracy metric. Bold indicates the largest offline performance. Blue rows stand for our models, while purple rows refer to zero-shot embedding models from the Hugging Face leaderboard. Per this benchmark, we can take away one main learning: fine-tuning on commerce data helps in achieving better accuracy (as clicks are positively correlated to accuracy) compared to pre-trained zero-shot models, but the uplift is moderate (3 points with respect to better text encoders, 6 points with respect to the same backbone model). Still, we believe that such commerce data uplift could be improved by considering other commerce signals, such as co-events (e.g., co-views or co-sales), although it is not the main focus of this article.

Stay tuned for upcoming posts that share Criteo’s AI journey!