As our Northern summer wanes and autumn approaches, we take stock of what we've gathered and read the leaves from the last of our sun tea. In this issue, we check in on the coven conference circuit and look forward to the next round of offerings and rituals for 2027. Then, we consult other sages for wisdom on recommendation modeling and evaluation.
🧹 Broomstick Bulletins
A quick fly-by of happenings, events, and gatherings
With The Web Conf, SIGIR, and KDD behind us (and the proceedings now available), we're about half-way through the conference season. There's still some conferencing left to go:
Author acceptance notifications for RecSys have gone out already, so we're starting to see pre-prints and author versions of accepted papers pop up on arXiv and LinkedIn. Keep your eye out for those if you want to get an early peek at the conference ahead!
Submissions for 2027 conferences are also starting to open up:
- CHI 2027 (Pittsburgh)
- Paper submissions by Sept 10, 2026
- Poster and interactive demo submissions by Jan 28, 2027
- Web Conf 2027 (Dublin)
- Workshop and tutorial submissions by Sept 21, 2026
- Most submission deadlines in October and November
- UMAP 2027 (Chicago)
- Workshop/tutorial proposals by Dec 11, 2026
- Paper submission deadlines in January
- Industry track and demo submissions due in March
Submission schedules for SIGIR 2027, KDD 2027's Second Cycle, and RecSys 2027 are yet to be published, but our scrying suggests that the submission dates may fall somewhere in the first few months of 2027. Foretelling is a notoriously fickle branch of magic, though, so check back for updates if you're preparing to submit a manuscript!
The ACM Transactions on Recommender Systems journal has completed their annual rite of conference paper expansion for RecSys '24, and the resulting compendium is now available. We thought this analysis of sequential structures in popular public datasets and these proposed improvements to LightGCN's neighbor aggregation looked interesting.
🗃️ Curated Curiosities
Blogs, books, and broadcasts from nearby realms
[The] trade-off between task specificity and model generalizability is a fundamental issue in recommender system research. Highly specialized models, fine-tuned for a particular domain, can achieve state-of-the-art performance in their respective tasks but may struggle when applied to other recommendation settings. Conversely, more general models that are designed to work across various domains often sacrifice performance in any given task, as they cannot fully exploit the domain-specific characteristics that drive user preferences.
— Aixin Sun
One goal of our grimoire is to help address the gap between the research literature's narrow focus on scoring models on the one hand and the full breadth and complexity of real-world recommendation systems on the other, and we feel the core argument in this recent article in Communications of the ACM is closely aligned with that goal. Aixin Sun makes the case that the RecSys field has done an overly-narrow and often sloppy job of formalizing recommendation tasks for research, with negative consequences for how we understand and evaluate recommenders. We tend to agree: while recommendation contexts, goals, and constraints can vary widely, the standard generalized evaluation measures like predictive accuracy necessarily leave out the quirks and variations of specific domains. Sun illustratively explores some of that variation, advocating for greater task-specificity in research and broader inclusion of the full user interaction life-cycle in the recommendation task.
Sun acknowledges "post-interaction feedback" as part of the interaction life-cycle but doesn't give a thorough treatment, since many users do not provide explicit feedback. However, a variety of user controls have become semi-standard in industrial recommenders (e.g. thumbs up/down, category preferences), and making them function as users expect is an important aspect of building trust in the system. If I can't trust that you're listening to what I explicitly tell you I want and don't want, how can I trust you to pick up on more subtle signals?
These signals are largely absent from the research literature, both as algorithmic inputs and as part of evaluation methodologies. Typical evaluation protocols rely on exact-matching of held-out interactions using IR metrics like precision and NDCG, which don't readily account for alignment with users' explicit feedback and preferences. Better-formulated recommendation tasks—while helpful—may not be enough if we continue to evaluate them by such anemic measures of recommendation quality.
Note: we shared some techniques for building user controls in our last Handy Hexes post.
Storage, training, and serving costs all threaten to grow proportionally unless you’re deliberate about efficiency. The single highest-impact technique we’ve deployed to hold costs in check across all three dimensions is request-level deduplication: a family of techniques that ensures we process and store request-level data once, not once per item.
—Pinterest Engineering
The typical rituals and incantations for scoring operate over user-item pairs, providing features for each pair as model inputs and repeating the user features for each paired item. The naïve approach involves actually duplicating the user features each time, which works but can become quite costly over the training and serving lifecycle of a model. In Scaling Recommendation Systems with Request-Level Deduplication, Pinterest shares their approach to reducing the cost of duplicated user features and the accommodating downstream adjustments they made to their training process, loss functions, and model architectures.
One clever charm that caught our attention is sorting the training data by user, putting more of each user's positive interactions into the same batch. This makes it simple to mask out the user's other positive examples to prevent them from being sampled as in-batch negatives. The technique provides a nice alternative to either keeping a separate index to filter out false negatives (which typically only works at small scale) or ignoring false negatives altogether (which is scalable but degrades learned representations).
One of the key challenges in training two-tower models is choosing which negative posts to show the model. If all negatives are random or trivial, the model quickly learns to separate “obvious” non-relevant posts, but struggles with more subtle distinctions. If all negatives are other posts in the batch, then the model only learns embeddings for popular posts more likely to show up in batches of user-post interactions while ignoring the long tail of less popular posts entirely.
This spring, Ahmad Khan wrote about the two-tower model Glassdoor uses for the retrieval/candidate-generation stage of their recommender, highlighting some techniques they use to improve user and item embedding representations. The spellwork they shared includes mixed negative sampling, which combines in-batch negative samples with additional uniformly-sampled negatives, and a self-supervised loss that uses masking and drop-out to create alternate "views" of user and items. We were also pleased to see several concrete examples of model input features, which is one of the practical areas that is often glossed over in both industrial and academic RecSys writing.
Spotted: another use of the four-stage recommender concept in the wild!
[LLMs'] broad world knowledge and strong language understanding make it possible to represent user histories and item metadata directly as text, capture rich relationships in a shared semantic space, and steer recommendations via natural‑language prompts. However, off‑the‑shelf LLMs are still far from production‑ready recommenders: they often over‑recommend globally popular content, hallucinate out‑of‑catalog items, ignore business constraints, and provide only limited personalization.
We have sometimes encountered the expectation that an LLM can serve as a drop-in replacement for a full multi-stage recommender, which we have found a bit baffling and couldn't be further from the truth in our view. So, we were glad to see Netflix emphasize this point in their recent post outlining how they developed a recommendation scoring model backed by a purpose-built LLM. Rather than directly generating recommendation lists, Netflix trained their LLM for use as a user encoder in an (extremely fancy) implementation of the industry-standard two-tower model architecture. Their model encodes users and items one tower at a time and scores them with a plain old dot product, just like many "traditional" RecSys models (including matrix factorization).
This is really interesting work, and Netflix's approach certainly has its advantages (particularly around feature context engineering). But, if you were expecting LLMs to produce wildly different and wholly novel architectures that completely replace existing approaches, this post may serve as a bit of a wake-up call about how and where LLMs actually fit into an industrial recommender system.