This Research card explains how generative recommendation systems can use their own semantic IDs, short learned codes that group similar items, to make offline evaluation, testing with existing logs instead of live users, more reliable. By evaluating groups of items instead of individual items, teams can compare system or re-ranking changes using existing logs before an A/B test, even when the logs mostly contain popular items.

• Title: Off-Policy Evaluation for Semantic ID Recommenders: Does the Model’s Own Code Hierarchy Help?
• Author: Artem Betlei
• Team: CAIL – AI Foundations – Commerce Foundation Models
• Venue: Causality, Counterfactuals & Sequential Decision-Making Workshop at RecSys 2026
• Category: Semantic ID Recommenders

Why did we work on this topic?

Near-argmax recommender logging, which mostly follows the highest-scoring items, concentrates exposure on a small number of items. In the production diagnostic, item-level effective sample size, roughly the amount of independent evidence available, was only about 0.003 of the raw sample, making standard item-level offline evaluation unreliable.

The paper studies whether shared semantic-ID prefixes can provide more evidence. A semantic ID is one item’s code; the semantic-ID tree is the hierarchy formed by all codes and shared prefixes. This aligns evaluation with generative recommendation systems.

What did we achieve?

We showed that marginalizing evaluation, meaning evaluating groups instead of individual items, from items to semantic-ID prefix clusters restores support and improves accuracy. On the semi-synthetic KuaiRand testbed, combining real data with controlled simulation and using separate data splits to avoid information leakage, the true policy value is known. Item-level SNIPS, a method that corrects for unequal exposure, reached 0.155 RMSE, or average estimation error, compared with 0.087 for its grouped version at the best depth. On a production slice, going from the finest to the coarsest code level increased effective sample size for a greedy target by about 1.8 times.

The paper also shows that the best code depth, or prefix length, depends on available support: coarser levels help when data is scarce, while more data allows finer levels. Selecting the depth from logged data alone captured 67–81% of the improvement that oracle depth selection, choosing the best depth using held-out truth, achieves over item-level evaluation, and the support-dependent pattern was directionally reproduced on Amazon Reviews.

How did we proceed?

We treated prefixes of a semantic-ID tree as groups of items and reweighted logged rewards at the group level rather than at the individual-item level. The decoder, which builds an item code one part at a time, can return the probability of a prefix directly by combining the probabilities of its code parts. These group probabilities do not require enumerating the whole catalog, for either the logging or the candidate policy.

We compared several offline methods for estimating a new policy’s performance from existing logs, and code granularities, or group sizes, on a production log from a generative recommendation system, the leakage-controlled KuaiRand dataset, and an Amazon Reviews replication. Additionally, we also varied support size, embedding source, quantizer (the method that turns representations into discrete codes), and depth-selection strategy. We also compared the tree with flat clusters of the same size, meaning groups without multiple levels, while tracking estimation error and effective sample size.

What is the originality here?

Unlike prior cluster-based off-policy evaluation, meaning offline evaluation of a new selection rule using logs from another rule, where clusters are built separately, the paper uses the generative recommendation system’s own semantic-ID tree as its evaluation structure. The decoder returns each prefix probability by combining the probabilities of its code parts, so cluster probabilities are cheap, whereas an arbitrary clustering needs a sum over the catalog for every request.

The experiments separate two effects that are easy to conflate: the accuracy improvement comes from grouping items, not specifically from using a hierarchy, while the hierarchy makes that grouping natural and operational for systems that retrieve items by generating their codes. The work also connects the choice of code depth to the trade-off between statistical support and differences in reward between items inside the same group.