The 41st edition of the International Conference on Machine Learning (ICML) took place from July 21 to July 27, 2024, at the Messe Wien Exhibition Congress Center in Vienna, Austria. This prestigious event, known for showcasing groundbreaking research in machine learning, artificial intelligence, statistics, and data science, adopted a hybrid format this year, allowing both in-person and virtual participation. Researchers, professionals, and students from around the globe gathered to share insights, network, and explore the latest advancements in the field. The conference featured an array of workshops, tutorials, keynote speeches, and poster presentations, making it a hub of innovation and collaboration.
As a silver exhibitor, we were proud to host a booth at the conference in the bustling exhibition hall of the Messe Wien Congress Center. Our booth became a focal point for engaging discussions, networking opportunities, and the exchange of ideas with fellow experts and enthusiasts in the machine learning community. We showcased our latest projects and innovations and were thrilled to connect with so many attendees. We extend our heartfelt thanks to everyone who visited our booth and interacted with our team.
In addition to our presence as an exhibitor, we were honored to present two papers that were accepted at the conference program. Our team also delivered a presentation during one of the workshops, contributing to the rich tapestry of knowledge and innovation that defines ICML. These presentations highlighted our latest research findings and sparked valuable discussions with peers and thought leaders in the field.
Our participation
We were very pleased that Criteo’s own Vianney Perchet was a co-author on two accepted papers at this year’s ICML:
Active Ranking and Matchmaking, with Perfect Matchings
The objective is to find, as quickly as possible, the optimal matching in a given population of items/players. Each individual has a “quality” and a pairing has no cost if the quality are close enough, otherwise it induces a fixed positive cost. Finding the optimal matching then somehow reduces to rank, actively, the different items using noisy feedback.
We, therefore, combined and developed techniques from bandits and sorting networks to achieve near-optimal results.
Non-clairvoyant Scheduling with Partial Predictions
The objective is to set up a machine that treats tasks in parallel by optimizing the ordering of the tasks (i.e., the scheduling). This is a well-known and studied problem, but we assume that the decision maker has some prior knowledge of some task durations (but not all of them). This is an “in-between” setting, different from the full prediction or the 0-prediction that are usually studied; in particular, it raises interesting new complexity questions that were absent in extreme cases.
We provide in this paper both lower and upper bounds on the problem complexity, and we show that they are almost equal.
Also, as part of the conference, Thibaut Issenhuth had the opportunity to present his work during a poster session at the Structured Probabilistic Inference and Generative Modelling Workshop:
Improving Consistency Models with Generator-Induced Coupling
The research focuses on consistency models, a subset of diffusion generative models. We propose a novel method that enables faster convergence during training and improved overall performance.
Thibaut highlights that presenting this work was a rewarding experience, especially engaging with a diverse audience that included experts in diffusion models and those familiar with only the basics. This diversity helped him refine her arguments and understand which aspects were most compelling to different people.
Highlights from our team
Andre Cunha
This is the first time I’ve attended ICML and I was impressed by its scale and by the quality of the papers and the talks. As expected, large language models (LLMs) were the most prominent topic, but other subjects were not neglected, including vision models, learning theory, optimisation, visualisation, reinforcement learning, causality, and many more.
The Criteo booth offered us the opportunity to have excellent interactions with other researchers and practitioners. During these conversations, it became apparent that the interest in federated learning and differential privacy has been increasing, despite — or maybe because of — the widespread difficulty in deploying these systems into production.
My only criticism is the fact that the scale of the conference: due to all the parallel sessions it is inevitable that everyone will miss some fascinating material.
Scientific Highlights
I came to the conference with a quite specific interest: tabular data. In that domain, a new architecture was proposed that outperforms all of its competitors on recommendation datasets, and a position paper defended the need for tabular foundation models. Since our problems are also class-imbalanced, we learned the best combination of downsampling the majority class and oversampling of the minority class.
The fascinating phenomenon of grokking, whereby a neural network generalises to unseen data well after achieving near-zero training loss, was shown to be more widespread than previously thought and techniques were proposed allowing us to anticipate when it occurs.
On the learning theory front, a proposal for a general framework for learning from weak supervision and an analysis of learning theory for distributional inputs caught my eye. Regarding optimisation, it was shown that spikes in the training loss can lead to better generalisation, and that cool-down can have the same performance as cosine scheduling, and how weight decay affects learning in neural networks (with a bonus explanation of the success of the duo AdamW + Weight Decay over Adam + L2).
We were also shown that feature visualisations might not be reliable, that untrained networks can favour some classes, and that it is mind-numbingly easy to fool a language model.
PS: readers looking for subjects to ponder over a glass of wine on a rainy evening might want to reflect on the possibility of creating silently suffering consciousness or of machines learning the true probability of events.
Artem Betlei
My first attendance at the main ICML conference was remarkable. I share the key highlights from this unique experience:
The scale of the conference is staggering — as Francis Bach mentioned, ICML is growing super-exponentially, with about 400 attendees and 120 accepted papers in 2004 (20x less than in 2024) and 1200 attendees and 310 accepted papers in 2014 (7x less than in 2024) 🚀
As for the trends, the focus on large language models and foundation models was obvious and therefore not surprising. I really appreciated though how the paradigm of foundation models is being spread across different research areas and data modalities.
Amidst a variety of concepts and problems that were new to me, a few particularly caught my attention. One of them is conformal prediction, a distribution-free method for quantifying uncertainty — there was a very informative tutorial on the topic. Another problem I found very promising was strategic ML. The idea here is to learn about strategic user behavior. Users, being the clever creatures they are, will naturally try to game the system to get what they want, so the challenge is to anticipate and adapt to that. The tutorial dove into how we can learn models that are robust to these kinds of strategic interactions. Lastly, I would highlight sharpness-aware minimization (SAM), a technique aimed to improve model generalization by minimizing both the loss value and loss sharpness at the same time. Researchers from Huawei effectively applied SAM to enhance the learning capabilities of transformers for multivariate forecasting.
Among the numerous impressive papers I have come across, there is a partial list of papers that in particular stood out to me:
The Platonic Representation Hypothesis (project page)
In this position paper, authors from MIT formulate the “Platonic Representation Hypothesis” that neural networks, trained with different objectives on different data and modalities, are converging in their representation spaces to a shared statistical model of reality, akin to Plato’s concept of an ideal reality. They demonstrate convergence across data modalities: as vision models and language models get larger, they measure the distance between data points in a more and more alike way.
Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experiments(poster)
Researchers from Netflix study inference on the long-term causal effect of a continual exposure to a long-term treatment, based on an experiment with short-term observations. They connect the problem with offline reinforcement learning, leveraging doubly-robust estimators to estimate long-term causal effects for long-term treatments and construct confidence intervals.
Discovering Environments with XRM (poster)
This paper from Meta highlights a need to develop algorithms for automatic environment discovery within datasets for Out-of-Distribution (OOD) generalization tasks. Authors propose Cross-Risk Minimization (XRM) approach by training twin networks, each learning from one random half of the training data, while imitating confident held-out mistakes made by its sibling. By learning algorithms on top of XRM environments, authors demonstrate oracle worst-group-accuracy, addressing a long-standing challenge in OOD generalization.
Imane Baha
At the outset of my machine learning career, attending ICML 2024 was profoundly impactful. It provided me with clarity and direction, reinforcing my commitment to further immerse myself in the ML research community. There’s no better way to be inspired to contribute meaningfully than by engaging directly with leading minds in the field — I even had the opportunity to discuss with Chris Bishop on the latest edition of his deep learning book.
The conference showcased an extraordinary array of innovations and state-of-the-art contributions. Here are some of my key highlights:
PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responses
This paper from Google Research introduces a novel algorithm for learning from aggregate responses, grounded in the concept of constructing aggregation sets (bags) for event-level loss functions. The authors demonstrate that the optimal bagging problem reduces to one-dimensional size-constrained k-means clustering for linear regression and GLMs. The PriorBoost algorithm adaptively forms bags of samples that are increasingly homogeneous with respect to unobserved individual responses to improve model quality.
Variational Learning is Effective for Large Deep Networks
This spotlight paper challenges the prevailing notion that variational learning is ineffective for large networks. The authors introduce the Improved Variational Online Newton (IVON) optimizer, which consistently matches or outperforms Adam in training large models like GPT-2 and ResNets from scratch, offering better predictive uncertainty with similar computational costs. Bonus point : A ready-to-use PyTorch library, ivon-opt, is also provided.
Neural Operators with Localized Integral and Differential Kernels
— Neural Operators ICML tutorial
This paper by Nvidia is excellent, but the ICML tutorial on Neural Operators by Kamyar Azizzadenesheli was even better! He introduced Neural Operators — an extension of neural networks designed to learn mappings between infinite-dimensional function spaces — and covered their theoretical foundations, including formulation and universal approximation capabilities. The tutorial highlighted numerous engineering applications, from CO2 storage mitigation and climate change mitigation to Seismic monitoring and Molecular Dynamics. Physics-Informed NO (PINO) seems to be a fast growing field with numerous open problems, and this tutorial surely reignited my interest in physics.
Karim Kassab
Attending ICML for the first time as a PhD student was a very interesting experience, particularly due to the conference’s vast scale and the high quality of both papers and presentations. It was also a delight to discuss with conference attendees at the Criteo booth, where I was happy to witness people’s interest in both the company and my work there.
While ICML is not a vision-oriented conference, I was personally pleased to see vision papers being presented, which I was able to discuss with the authors. I was particularly interested in papers targeted towards 3D modeling (e.g., Neural Radiance Fields, 3D asset generation). At Criteo, we have been exploring how to create a 3D scene given only its views (2D images — like our product catalog). Check our research card on that topic:
Research Card: Exploring 3D-aware Latent Spaces for Efficiently Learning Numerous Scenes
Paper: Exploring 3D-aware Latent Spaces for Efficiently Learning Numerous Scenes
Authors : Antoine Schnepf (Criteo AI…
techblog.criteo.com
Among the many impressive works I’ve encountered, here’s a non-exhaustive list of papers that I found particularly intriguing:
IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation
This paper targets text-to-3D generation, but utilizes video generative models to achieve this task, instead of the more usual image generative models. To do so, they fine-tune a text-to-video generative model to take as input an image of an object and generate a video of it where the camera is rotating around the object. Subsequently, they utilize the generated multi-view frames to train a Gaussian Splatting model. I was especially impressed with the speed at which they are able to do this generation, as it is much faster than the more typical Score Distillation Sampling (SDS) approach.
Make-A-Shape: a Ten-Million-scale 3D Shape Model
This paper targets 3D asset generation by utilizing a wavelet-tree representation to efficiently encode high-resolution SDF shapes. It also introduces a subband coefficient filtering and packing scheme to do diffusion-based generation. I was particularly impressed with the scale of their training, as they simultaneously train on many well-known 3D datasets, and support many input modalities.
LoRA+: Efficient Low Rank Adaptation of Large Models
This paper improves upon the performances of the original LoRA paper by implementing a simple idea: Using different learning rates for A and B matrices. The authors prove that this allows for more efficient feature learning, and improves fine-tuning speeds and performances.
Julien Zhou
As a Phd student, it was my first time attending a big international conference. I was impressed by the scale of it, it was intense and exhausting. An impressive diversity of works were present which made it difficult to see all the things I was interested in. It was an opportunity to discuss with more senior researchers, to network, and to meet the community which is crucial to build collaborations and find interesting problems to tackle.
Among all the things that I saw, I particularly appreciated the tutorial on Conformal Prediction. It is a framework enabling to get confidence intervals without assumptions on models. The subject is becoming popular lately and this talk was really enlightening.
ICML Tutorial Distribution-Free Predictive Uncertainty Quantification: Strengths and Limits of…
Edit description
icml.cc
I also really appreciated the two workshops concerning Reinforcement Learning. They delivered a lot of useful insights for my research: FoRLaC Workshop 2024 and Aligning Reinforcement Learning Experimentalists and Theorists.
Among all the papers presented, I also enjoyed two papers: Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise and What is the Long-Run Distribution of Stochastic Gradient Descent? A Large Deviations Analysis, for their endeavor to analyze empirically successfull algorithms from a theoretical viewpoint.
Thibaut Issenhuth
Attending my first ICML conference was an incredible experience. It was truly motivating to be immersed in this environment for a week, surrounded by the excellence and diversity of so many presented works.
Attending such a conference is a great way to observe the emerging trends. The most prominent trend is around Foundation Models, particularly their fine-tuning. There was even an oral session dedicated to Low-Rank Fine-Tuning of LLMs. The progress in this area is astonishing, as fine-tuning LLMs with limited resources is becoming increasingly feasible. Additionally, a significant number of papers focused on Diffusion Models, their extensions, and applications. A quick search revealed that 120 papers included the word “Diffusion” in their title! Lastly, there was a substantial body of work addressing AI for Science, with applications ranging from climate modeling to molecule generation. It was inspiring to see how researchers are developing new machine learning models tailored to the specific challenges of these domains.
Among all those works, I particularly appreciated the following papers:
Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations
This work, by researchers from Meta, addresses the task of recommendation systems. They propose a modern solution based on large transformers trained in an autoregressive manner and leveraging scaling laws. On the technical side, they first introduce a sequentialization of recommender systems’ data and then adapt transformers’ architecture to handle such data.
Switched Flow Matching
The authors identify a critical issue in flow matching and diffusion models: when the data distribution is a mixture of modes, “singularities” can occur in the transition areas between modes. To address this, the authors propose learning switching ODEs, where the model learns from a mixture of ODEs. They demonstrate that their model exhibits better properties than standard flow models, which could be helpful for learning complex distributions such as those in Text-to-Image models.
Lastly, as mentioned before, I could attend the Structured Probabilistic Inference and Generative Modelling Workshop. With the current surge of interest in generative models, the workshop drew a large crowd, so much so that the room was already packed for the first talk by Ben Poole on generative models for 3D, and I couldn’t even get in. Fortunately, I managed to get in for the second talk. The talks were truly inspiring, featuring high-profile scientists presenting their latest research. For instance, Molei Tao introduced an extension of Langevin Dynamics to manifolds, which enhances the performance of diffusion models on non-Euclidean data. Similarly, Ricky T.Q. Chen presented his recent work on Flow Matching, a new framework that generalizes score-based diffusion models to arbitrary source and target distributions. This approach was notably used in the latest Stable Diffusion 3 model.
Thibaud Rahier
As one of the top international machine learning conferences, ICML is a key moment in the year providing sort of a “snapshot” of the advancement of research in machine learning and artificial intelligence. The work presented at ICML’24 was heavily influenced by the lightning fast popularization of large language models (GPT, Gemini, Llama, Mistral, …) but was still very diverse and therefore exciting.
Most of the presented papers on LLM training were related to the fine-tuning and alignment stages of LLM training. These stages are both much more cost-efficient than the initial pre-training stage (which only a handful of companies and institutions can afford) and are also easier to grasp theoretically and, therefore, to principally improve. On that subject, I particularly liked WARM: On the Benefits of Weight Averaged Reward Models (A. Ramé et al.), which proposes an improved way to align models with Reinforcement Learning with Human Feedback (RLHF) using weight averaging to leverage different reward models efficiently. The results are very good, and I think that a more formal understanding of RLHF for alignment of large (language) models might be an interesting future research perspective for the online learning/bandit experts who we have among our ranks at Criteo.
On the subject of reinforcement learning (RL), which is central to both our applied and more fundamental research at Criteo, there were two interesting workshops (already mentioned by Julien) that provided a great opportunity to observe the current state of the field and to discuss with researchers studying similar topics as us. There also were insightful papers, such as Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (Farebrother et al.), defending the idea that tackling the value function training problem in RL through the classification lense rather than regression is both better performing and much more scalable, with diverse applications to multi-task RL (Atari, Chess, …). This is an interesting idea that could probably go beyond the RL scope!
Moreover, there was interesting work on privacy and safety, which is crucial in areas where research moves as fast as it does today in machine learning. Among others, I was very interested in Stealing Part of a Production Language Model (Carlini et al.), who study how to recover information from an LLM — namely its embedding projection layer — only using the model’s output, which is accessible from standard APIs. Such work gives important insights into how much can be revealed about a model by allowing its use by the public.
It’s a wrap!
And, with that, ICML 2024 is done 😃
Our participation as a silver exhibitor provided invaluable opportunities to showcase our work, engage with attendees, and foster new connections. The positive interactions at our booth and the enthusiastic reception of our presentations underscored the importance of collaboration and knowledge sharing in driving the field forward.
