An Expert Community Where Every Voice Matters


Join Us
,

RecoAtlas: An Open Reinforcement Learning Environment for Training and Evaluating Shopping Agents on Behavioral Data

RecoATLAS, fine-tuning RL env for tool-using shopping assistants

AI assistants are becoming the new front door of commerce. But a shopping agent should be judged by more than how convincing its recommendations sound. At Criteo, we believe the real measure is whether it helps people find and buy what they actually need. Today, we’re open-sourcing RecoAtlas, a Reinforcement Learning (RL) environment for training and evaluating shopping agents using behavioral signals.

A ranked list of five guitars next to a set of guitar, tuner, picks, strap, case and strings.
Ask a traditional recommender to help you take up the guitar and it returns five candidates for the same role. A shopper needs a set, one role each.

From the vision of shopping agents to making them work

Every few years, commerce gets a new front door: first the search box (think Google or Amazon search), then the feed (Instagram, Pinterest, TikTok), and now the AI assistant. Instead of searching and scrolling through products, shoppers can describe what they want and let an agent find, compare, and assemble the right options. A growing share of shopping journeys will begin this way.

In our previous article, The Future of Commerce Recommendation: Why Hybrid Systems Will Win, we outlined our vision for bringing AI agents into real-world commerce: combining the language understanding and reasoning capabilities of LLMs with the behavioral intelligence of recommender systems.

Traditional recommenders already shape much of what we see when we shop. They learn from clicks, purchases, and returns, and have become remarkably good at predicting what someone might want. But they were not designed to handle requests like “Build me a complete trail-running setup under €500 for Iceland in October”. Such requests require more than ranking relevant products: they require understanding a goal and assembling products that work together and satisfy constraints.

LLMs bring capabilities that traditional recommenders lack. They can interpret complex requests, reason over constraints, call tools, and explain their choices. That makes LLM agents, language models that call tools such as product search, the natural first approach to conversational recommendation. But are they good recommenders? Semantically understanding what a shopper wants is not the same as knowing exactly what they would buy. The challenge is to combine these complementary strengths into agents that deliver useful recommendations at scale.

We believe this requires five design principles:

  1. Recommendation tools. A shopping agent needs more than general-purpose search, for two reasons. First, it must navigate catalogs of millions of products and find items the shopper is more likely to buy, not just semantically plausible ones. Second, it must understand which products work well together and avoid near-duplicates. Consider someone looking to take up the guitar. A traditional recommender might return five good guitars, but the shopper needs more than alternatives to choose from: a guitar, a tuner, picks, a strap, a case, and strings. This is bundle shopping, where the goal is to assemble a complementary set rather than rank similar products. RecoAtlas tackles exactly this challenge, equipping agents with specialized tools for product search, complementary-item recommendation, near-duplicate filtering, and result refinement.
  2. An objective learned from behavior. Two products can look nearly identical on paper yet lead to very different shopper outcomes: two phone cases with the same description, one of which is returned far more often because its cutouts don’t quite fit. Clicks, purchases, and returns reveal preferences that product descriptions alone cannot capture. The agent and its tools should learn from these signals, not just from how plausible the recommendations sound.
  3. An efficient, specialized brain. Serving millions of shopping sessions, each involving multiple model and tool calls, imposes strict cost and latency constraints. A small, specialized orchestrator (the model that plans the request, calls the tools, and chooses the final set) working with powerful recommendation tools offers a promising alternative to running a frontier model at every step.
  4. Personalization. Traditional recommenders learn representations of shoppers from their past behavior. Agents need to combine this long-term understanding with the goals and preferences expressed in the current conversation.
  5. Adaptation. Catalogs, prices, and tastes change constantly, and every retailer, brand, and product category brings different constraints: a new console launch reshuffles which accessories sell together within weeks, and a fashion catalog turns over every season. Recommendation agents must adapt both their recommendations and their tool-use strategies as these environments evolve.
The five design principles mapped onto a shopping agent
The five design principles mapped onto the agent. RecoAtlas covers the tools, the behavioral objective and the small orchestrator (1–3). Personalization and adaptation (4–5) come next, in an upcoming extension.

From plausible answers to useful recommendations

These principles describe the kind of shopping agents we want to build. But an architecture alone doesn’t tell us how to train an agent to use recommendation tools effectively, or how to measure whether its recommendations are any good.

Today’s AI assistants are largely trained and evaluated using human preference judgments or other language models acting as judges. These methods can assess whether an answer sounds helpful, but not necessarily whether the recommended products match what shoppers actually want and buy.

This is the gap we set out to address with RecoAtlas (Reco Agentic Tasks for Learning and Assessment in Shopping), an open reinforcement learning (RL) environment for training and evaluating shopping agents using behavioral data.

RecoAtlas takes the hybrid architecture we previously advocated and makes it trainable. Rather than simply prompting an LLM to recommend products, it provides an environment in which an agent can learn which recommendation tools to call to assemble product sets that satisfy a shopper’s goal. Crucially, its reward is grounded in behavioral data rather than solely in language-model judgments.

It focuses on the first three design principles: recommendation tools, behavioral learning objectives, and agent orchestration.

Our hypothesis at Criteo is not that small models inherently outperform large ones. It is that a small, specialized agent equipped with strong recommendation tools and trained on behavioral signals can deliver much of the value at a fraction of the cost. RecoAtlas provides an environment in which to investigate that hypothesis.

The code and paper are public. We also presented a tutorial at the CONSEQUENCES workshop at RecSys ’26, with accompanying slides and a full video recording.

Two kinds of difficulty

A shopping request can be hard in two independent ways. The need can be stated clearly but be hard to fulfill, or it can be easy to fulfill once the agent has understood it, which takes a conversation. Keeping them separate helps us understand where an agent fails and what it needs to learn.

The first is multi-call: delivering the result. Consider a shopper asking for a complete outfit, a set of compatible electronics, or a comparison across several alternatives. The agent must plan queries, call recommendation tools, inspect results, reformulate searches when necessary, and assemble the final set. These decisions depend on one another: what the agent searches for next depends on what it has already found. RecoAtlas focuses on this challenge, training the agent to coordinate retrieval and selection across multiple tool calls.

The second is multi-turn: understanding users as they reveal their preferences and intentions through conversations with the assistant. Shopping needs are rarely fully specified upfront. Users gradually express their preferences, refine their requirements, and correct earlier assumptions, often mixing long-term tastes with the immediate needs of a particular purchase. An effective agent must continuously build and update its understanding of the user as the conversation unfolds.

Two panels side by side. Multi-call: one shopper request for a trail-running setup under €500 leads the agent through four tool calls to a four-item set. Multi-turn: a three-turn chat narrows running shoes down to waterproof trail shoes under €150, found with a single search.
Multi-call and multi-turn side by side. On the left, the need is clear in one message and the work is a chain of tool calls. On the right, one search is enough, but the need only comes together over three turns.

RecoAtlas focuses on multi-call tool orchestration for a specific shopping task, with a planned extension to multi-turn intent discovery and personalization.

Two kinds of difficulty: tool calls per turn vs turns with the shopper
Two independent dimensions of shopping-agent difficulty: tool calls per turn and conversational turns with the shopper. RecoAtlas focuses on multi-call orchestration; an upcoming extension will focus on multi-turn interaction.

Why simulated environments

If we want an agent to learn a behavior, we need an environment that can measure it and provide feedback. Without a measurable objective, we cannot reliably tell whether the agent is improving. With one, we can use reinforcement learning to train the behavior directly.

In this sense, designing RL environments is becoming the new feature engineering. Just as feature engineering determined what signals a model could learn from, environment design determines what experiences an agent learns from and which outcomes it is rewarded for. Rather than handcrafting the behaviors we want, we design the tasks, tools, and feedback that allow those behaviors to emerge through learning.

Much of the recent progress in agents follows this approach. Models are post-trained through repeated interactions with simulated environments, where they take actions, receive responses from tools or simulated users, and are rewarded for successful outcomes.

The data is often semi-synthetic: real structure underneath, simulated interaction on top. The critical question is what the simulation is anchored to.

For commerce, we believe that anchor must be shopper behavior.

RecoAtlas keeps real what matters most: the product catalogs, recommendation tools trained on purchase data, and the sets of products customers actually bought. It simulates what historical logs cannot provide: the natural-language shopping requests that might have led to those purchases.

The reward comes from recovering historically purchased products, not from a language model deciding whether a recommendation sounds good. This gives us a way to apply modern agent-training methods to recommendation outcomes, including difficult tasks such as bundle shopping.

Building tasks from real purchases

Existing evaluation methods capture different aspects of recommendation quality, but none fully reflects the challenges of real-world shopping.

Candidate reranking benchmarks ask agents to select the best products from a predefined shortlist, without testing their ability to search large catalogs or assemble complementary and diverse sets. LLM-based judges enable evaluation at scale, but often assess whether recommendations sound appropriate rather than whether shoppers would actually choose them. Online A/B tests provide the strongest evidence of real-world impact, but require production deployment, substantial traffic, and time to collect reliable results and cannot serve as offline reproducible benchmarks.

RecoAtlas bridges these approaches by constructing shopping tasks from public commerce datasets.

The recipe assumes three ingredients: user interactions with timestamps, a catalog of items, and metadata describing each item. RecoAtlas was evaluated on the open H&M fashion catalog and Amazon reviews dataset from Musical Instruments, Electronics, and Video Games, but the same recipe applies to any catalog where people’s choices are logged, such as travel, jobs, or real estate.

Each task is built in four steps:

  1. Split the shoppers. We use 70% to train recommendation tools, 15% to develop and train agents, and 15% for held-out evaluation. No shopper appears in more than one group, so evaluation measures performance on previously unseen shoppers.
  2. Find purchased sets. We identify groups of three to seven items bought by the same shopper within a single week, then use an LLM to filter them for coherence. The LLM must accept or reject each set as a whole, judging whether it forms a coherent bundle; it cannot add or remove individual items.
  3. Generate the shopping request. An LLM writes a plausible request that might have preceded the purchase, such as “A setup for singing and playing at small coffee-shop gigs.” Automatic checks ensure that the request does not explicitly name the purchased products.
  4. Score the agent. The agent receives the request and the required number of items, K, but never sees the purchased set or a shortlist of candidates. It can make up to six rounds of tool calls before submitting its recommendations.

We measure performance using SetHit@K, the fraction of the hidden purchased set recovered by the agent’s K recommendations.

If an agent correctly identifies two of five products, its score is 0.4. Only exact product matches receive credit. A score of 100% would mean the agent recovered every item the shopper bought. Because a request can be met by many valid sets, no agent should be expected to come close.

This is deliberately demanding. The agent must search the catalog, decide which tools to use, and assemble the set without seeing the answer.

How a RecoAtlas task is built and scored
The RecoAtlas task-construction pipeline, from historical purchases to synthetic requests, agent tool calls and SetHit@K evaluation. The purchased set remains hidden from the agent.

Teaching an agent to shop from behavior

A RecoAtlas agent has two main components: recommendation tools that retrieve and organize products, and an orchestrator that decides how to use those tools and assemble the final recommendation.

Our approach puts behavioral intelligence into both.

Recommendation tools trained on purchases

The environment provides three types of tools: product search, complementary-product retrieval, and near-duplicate removal.

These tools share a catalog embedding space and begin with semantic relevance ranking. We then fine-tune them to match atomic search queries to purchased products.

The goal is not to abandon semantic relevance in favor of sales. It is to go beyond text similarity and capture what shoppers actually choose.

A product can match the words in a request perfectly and still be a poor recommendation. Behavioral training helps the tools distinguish between products that look similar in descriptions but differ in how well they meet real shopping needs.

A small orchestrator trained with reinforcement learning

The orchestrator interprets the request, plans searches, invokes tools, evaluates intermediate results, and selects the final set.

We train it using reinforcement learning, with reward derived from how much of the historically purchased set it recovers.

We deliberately investigate small models: commerce operates at a scale where millions of shopping sessions may each require several model and tool calls. Cost and latency are therefore not secondary considerations; they determine what can realistically be deployed.

Smaller models can also be less expensive to retrain as catalogs and tastes evolve, and to specialize for particular retailers or product categories.

The question is whether a small agent can learn to use powerful recommendation tools well enough to compete with much larger general-purpose models.

What the results show

Our initial results suggest that both the recommendation tools and the agent’s training objective matter substantially.

Without recommendation tools, even strong LLMs struggle

An LLM operating without recommendation tools recovers less than 1% of the products shoppers bought.

A fixed rule that combines the three recommendation tools, with no LLM at all, reaches 4.4% SetHit.

A 27B open model orchestrating the same tools reaches 9.2%, increasing to 11.6% with extended reasoning enabled.

This illustrates the value of combining language-model reasoning with purpose-built recommendation tools.

The LLM helps both to find and to choose

The agent and the fixed rule differ in two ways: how they retrieve candidate products, and how they pick the final set from those candidates. To separate the two, we handed the fixed rule every product the agent had come across during its tool calls (multiple search and complementarity calls).

With the agent’s candidates, the fixed rule improves from 4.4% to 6.4%, so the agent finds better candidates. On that same pool of candidates, the agent itself reaches 9.2%, so it also chooses better. Choosing accounts for 58% of the agent’s gain over the fixed rule.

Behavioral signals reveal what LLM judges miss

One of the clearest results comes from comparing an agent that uses semantic recommendation tools with another agent that uses the behavior-trained tools.

With semantic tools, SetHit reaches 8.1%. Replacing them with behavior-trained tools increases it to 11.6%. Yet an LLM judge gives both configurations essentially the same score, approximately 0.87. To the judge, both sets look equally plausible. But the purchase data reveals a meaningful difference.

This is why we argue that recommendation agents need behavioral evaluation, not just language-based judgments.

Can you trust the benchmark?

Because an LLM helps construct RecoAtlas, it is fair to ask whether the resulting tasks are sound. Its role, however, is narrow: it rejects real co-purchase sets that lack coherence, and it writes the shopping queries. It never adds or removes items, and SetHit, our primary metric, involves no LLM at all. We still checked the tasks by hand and results are shared below.

Seven people, three of them from outside the team with no AI background, reviewed 50 random fashion tasks. In 48 of them, the products fit the request; in 49, they formed one coherent bundle. Moreover, exact matching is strict: a perfectly good alternative gets no credit if the shopper didn’t buy it. So we also asked whether SetHit agrees with people. For 50 requests, the same annotators compared two bundles without seeing any scores, one that recovered some of the purchased products and one that recovered none. They preferred the former in 81% of their judgments.

Finally, to test how sensitive RecoAtlas is to the LLM used to build it, we regenerated it with a different model family, Gemini instead of GPT, at every LLM step. The ranking of the evaluated models did not change.

How small can the agent be?

Because our approach relies on small agents, we first established a performance floor: how well they do with no training at all. We ran the same agent with Qwen3.5 models ranging from 0.8B to 27B parameters on all 1,960 Musical Instruments tasks.

SetHit climbs steadily with size, from about 1.6% at 0.8B to 9.3% at 27B with thinking on, and thinking helps at every size. But below about 4B parameters, the agent does no better than the fixed rule running the same tools on its own (4.4%). From 4B up, the language model clearly adds something.

That is the size class we care about for deployment, and it’s why the reinforcement-learning results below start from a 4B model.

Reinforcement learning improves small agents

Training the orchestrator on behavioral rewards produces substantial gains. To showcase this, we trained two small models, Qwen3-4B and Gemma-4-E4B, three times each with different random seeds, on two catalogs.

In particular, Qwen3-4B’s agent improves from 6.2% SetHit before training to 10.8% after reinforcement learning, about 1.7 times its starting point. That beats the 27B model with reasoning off, at 9.2%.

Training also changes how the agents work. They retrieve more of the products shoppers bought during recommendation tool calls, and they keep more of what they retrieve in the final set. Gemma, which produced a valid report less than one time in five before training, now almost always does.

Training also changes how the agents behave. Their recommendation calls surface more of the products shoppers actually bought, and they keep more of those retrieved products in the final set. Gemma, which produced a valid report in fewer than one in five tasks before training, now almost always produces one.

The task remains far from solved. Even the strongest agent recovers less than a third of the products shoppers bought across every catalog.

The full experimental results and methodology are available in the paper.

What we are releasing

We’re releasing RecoAtlas as an open environment that researchers and practitioners can use to evaluate shopping agents, experiment with reinforcement learning, and build similar tasks using their own commerce data.

The release includes:

  1. The RecoAtlas Python package, under Apache 2.0. A single call, recoatlas.load("hm-clothing"), loads a catalog, its recommendation tools, and its tasks. The package ships with two catalogs: Amazon Musical Instruments and H&M Clothing.
  2. The benchmark, containing 5,030 held-out shopping tasks across four catalogs, each pairing a natural-language shopping request with a set of products historically purchased by one shopper.
  3. RL training adapters for OpenEnv, TRL, and Miles, allowing the environment to integrate with existing reinforcement-learning workflows.
  4. The build pipelines used to turn purchase logs into recommendation tools and shopping tasks. These can be adapted to a retailer’s own catalog and behavioral data.

What you can do with it

  • Choose a model for your shopping assistant. Compare candidate models and agent designs on what shoppers actually bought, before deploying them.
  • Improve your own model. Train a smaller agent with the RL adapters and measure what training buys.
  • Build the same test on your own data. Point the build pipelines at your catalog and purchase logs to get a purchase-grounded benchmark for your retailer or brand.
  • Research agents, tools, and rewards. Swap tools, rewards, or RL algorithms and compare them on the same held-out tasks.

Try it

Clone the repository and install the environment using:

uv sync --extra tools

A complete episode looks like this:

import recoatlas

env = recoatlas.load("hm-clothing", device="cuda")
# First call downloads approximately 2.8 GB

ep = env.episode(split="agent_hidden", index=0)

ep.search_products(
    "relaxed high-waisted shorts",
    top_k=10
)

ep.submit([
    {"product_id": "...", "reasoning": "..."},
    # Additional recommendations
])

ep.score()
# {"sethit": ..., "valid": ..., "reward_direct": ...}Code language: PHP (php)

The repository’s ELI5.md guide walks through running the paper’s agent and evaluating its reports. The examples/trl_grpo.py example demonstrates how to start a GRPO training run with TRL.

For a deeper walkthrough, see our tutorial, Training and Evaluating LLM Shopping Agents with RecoAtlas, presented by Flavian Vasile, Otmane Sakhi, and Imad Aouali at the CONSEQUENCES workshop at RecSys ’26.

The 35-minute tutorial covers the motivation for agentic recommendation, environment construction, benchmarking, and reinforcement-learning methods and results. You can watch the full recording below or read the slides (PDF).

What comes next

RecoAtlas is a step toward making shopping agents trainable and measurable using the same kinds of signals that have powered recommendation systems for decades.

Our results suggest three important shifts for commerce.

  • Evaluation moves from plausibility toward outcomes. Whether an agent recommends products people actually choose is a different question from whether its answer sounds helpful. Retailers will need behavioral evaluations to compare agents and optimize their recommendations before deploying them to shoppers.
  • The competitive advantage extends beyond model size. Strong recommendation tools, high-quality behavioral data, and effective orchestration can matter as much as the language model itself. Our experiments show that specialized training can make smaller agents substantially more competitive.
  • Shopping agents become more practical at commerce scale. If smaller models can deliver useful recommendations with lower latency and cost, agentic shopping can move beyond expensive, occasional interactions toward serving everyday commerce sessions.

These are early results, not the end of the research.

RecoAtlas addresses the multi-call problem: how an agent searches, selects, and assembles products to satisfy a shopping request. A planned extension will address the multi-turn problem: how agents discover shopper intent, maintain preferences, and adapt to different retailers, verticals, and use cases.

Our broader goal remains the same as in our original vision of hybrid recommendation agents: combine the reasoning capabilities of language models with the behavioral intelligence of recommendation systems, and train the resulting agents to deliver outcomes that matter to shoppers.

We hope the community will try RecoAtlas, build on it, and help us understand where it works and where it falls short.

Resources