Our experience at EACL’24

April 11, 2024

As we gathered at the 2024 edition of the European Chapter of the Association for Computational Linguistics (EACL), we were all eager to see where the community’s research was headed after a whirlwind year of Large Language Model (LLM) breakthroughs.

Although one could have thought that ChatGPT was the answer to all our questions, we now see in perspective that, on the contrary, it opened a multitude of new questions surrounding these models: biases, scalability, limitations… The conference promised to offer a good overview of what these questions are and how researchers are tackling them. Here are some trends that we observed:

Truly open-sourcing open-source models (like OLMo, with both accessible weights and dataset) as the only way to control bias, leakages, and other issues.
Synthetic data generation consolidates as the scalable way to instruction-tune and align models with no or less human feedback.

Also, it is not really a trend, but something that caught our attention is how the community still relies on templates in order to control the generation of the LLMs. See, for instance:

In this post, we will discuss a selection of papers and presentations that caught our eye.

Keynotes

Human vs. Generative AI in Content Creation Competition

Prof. Hongning Wang

Main takeaway
There might be an equilibrium where generative AI tools coexist with human creators. Human creators who create low-quality content will be displaced from the market since their place will be taken by generative AI. Human creators offering high-quality niche content are the ones who will be able to coexist with Generative AI.

Summary
EACL 2024 kicked off with an invited talk by Prof. Hongning Wang. The speaker analyzed a question that concerns many people: in the face of AI’s growing role in content creation: where do humans stand? The digital landscape is teeming with AI, from chatbots that mimic human conversation to algorithms that generate new music. Amid this digital renaissance, content creators are understandably nervous. They’re no longer just competing with each other, but now with AI trained on their own creations. The researchers modeled this problem as a Tullock contest — a model applied in econometrics to analyze competitive scenarios — , placing GenAI in direct rivalry with human creators to see how this affects the market dynamics between creators and content consumers, aiming for a Nash equilibrium. It is a nice modeling work, though it understandably ignores some factors, such as regulations or more complex consumer dynamics.

Quality data for LLMs

Prof. Hinrich Schütze

Main takeaway
Data is as important as architectures and algorithms. To create synthetic datasets for instruction tuning, creativity and understanding of the final task pays off.

Summary
In this keynote, professor Schütze showed us his work on how to synthetically create an instruction-tuning dataset from a basic dataset through the idea of reverse instruction (the related paper is called LongForm) For instance, given a document saying “A: I love pizza”, prompt a pre-trained LLM with “P: what kind of instruction could this be the answer to?” to obtain something like “Q: what kind of instruction could this be the answer to”, therefore completing the pair (instruction Q, answer A) necessary to instruction-fine-tune the model.

Prof. Schütze also mentioned some interesting follow-ups of the LongForm paper from other teams:

TeGit: Generating High-Quality Instruction-Tuning Data with Text-Grounded Task Design modifies the protocol and asks the model to generate not only the instruction but also the output, resulting in less noise in the final dataset.
Self-alignment with instruction back-translation fine-tunes a first model on a small amount of data to generate high-quality instructions, and then it generates the instructions for the full corpus.

Best Paper Awards

Best Paper Award

LOCOST: State-Space Models for Long Document Abstractive Summarization

Oour paper has been honored with the best paper award 🏆(1 award / 388 accepted papers with a total of 1271 submissions). It was a joint work with CNRS-ISIR, Sorbonne University, and A*STAR, Singapore. This was a truly memorable moment, as it is a rare occurrence that may happen only once in a lifetime. Giving the final presentation at the plenary session was also nerve-wracking. Kudos to our PhD students! 🎉

Our paper explores state-space models (SSMs) for long document abstractive summarization. While being significantly less demanding in terms of computing, our model achieves performance comparable to transformers. This work demonstrates that attention-free encoders like SSMs may still compete with the widely accepted transformers.

Best Non-Publicized Paper Award

Leak, Cheat, Repeat

We loved this work. The paper meticulously documents the pervasive issues in data contamination and evaluation practices in the use of Large Language Models (LLMs). By systematically analyzing the use of OpenAI’s GPT-3.5 and GPT-4 models across numerous studies, it exposes how approximately 4.7 million samples from diverse benchmarks have inadvertently seeped into these models, potentially skewing their performance metrics and the integrity of subsequent research. This makes a strong case in favor of really open-source models as a way to ensure that advancements in the field are built on solid, reproducible foundations.

You can check all the awards here 👉 https://2024.eacl.org/program/best-paper/

Other interesting papers

As for the papers, these are some that caught our eyes, many in the dialog and interactive system session:

Asking the right question at the right time: in this paper, researchers tackled the challenge of when and how a chatbot should ask clarification questions. Until now, chatbots were trained to ask for clarifications using human-annotated datasets. However, the authors found that humans and chatbots disagree on what needs clarification. They propose a methodology for triggering clarifying based on model (rather than human) uncertainty, showing (of course) positive results.
Leveraging Implicit Feedback from Deployment Data in Dialogue: another paper trying to get the most out of the available datasets. The paper proposes optimizing conversational agents by utilizing implicit feedback — such as the length of a user’s answer, the estimated sentiment, or whether the user replies at all, which might be indicative of whether they liked the chatbot’s output.
Commonsense-augmented Memory Construction and Management in Long-term Conversations via Context-aware Persona Refinement: the paper tackles the problem of user-inferred personas that might vary from session to session. For instance, one day, a user can be tagged as “lazy” because she is sick and says she stayed in bed until 10 a.m. Then, another day, she can be tagged as a “hard worker” when she talks about how often she goes to the gym or her professional achievements. The paper proposes exploiting the common sense from the LLMs and an iterative process to refine the personas with new conversations.

In the application session, we could see how people are already finding ways to exploit LLMs in different domains such as legal, pharmacy, interviewing, CV analysis, and presentations. Here are some pointers:

Attending conferences is consistently a treat as it provides opportunities to engage in research discussions with fellow scholars. This particular event holds special significance for us as we were honored with the best paper award.

We also appreciated having the opportunity to speak to Mirella Lapata, who delivered an impressive keynote on “Prompting is *not* all you need! Or why Structure and Representations still matter in NLP.

A very nice conference overall. We are looking forward to seeing what the community brings next year! 🚀

Generative Ai Tools, Large Language Models, LLM, NLP, Research