JEP-TALN RECITAL is the main gathering of French-speaking NLP researchers. It combines three historical French conferences.
- Journées d’Etudes sur la Parole: centered on speech.
- Traitement Automatique de la Langue Naturelle: centered on text.
- Rencontres des Etudiants Chercheurs en Informatique et Traitement Automatique des Langues: a space for students to showcase their work.
We attended to present our recent work, LOCOST: State-Space Models for Long Document Abstractive Summarization, to the French community. Our paper was awarded the best paper in EACL 2024.
Our experience at EACL’24
Our highlights from the European Chapter of the Association for Computational Linguistics latest edition where we won…
techblog.criteo.com
While the 2024 edition took place in Toulouse, no one had a poster comparing the embedding distance between “chocolatine” and “pain au chocolat”, “poche” and “sac”. Fast-research question: Does the following hold?
In any case, in this post, I will discuss a selection of talks, papers, and presentations that caught our eye.
Keynotes
OpenLLM-France: towards sovereign multimodal LLMs
Jean Pierre Lorre (LINAGORA)
Main takeaway
Open source is open weights, open training, and open data.
Summary
Jean Pierre Lorre, Research Director at LINAGORA, walked us through the challenges of open source in modern AI. Defining open source in LLMs is not trivial since models are not just pieces of software. The Open Source Initiative is proposing a three-pronged definition of open: open weights and architecture (the final software everyone can use), open training (the recipe to reproduce the same model), and open dataset (the source of information to train the model). The data part is often forgotten, and we don’t know the data used to train models such as Mistral or Llama. Yet, accessing these data can help us understand and prevent some biases. For instance, it’s estimated that the predominant language in most of these datasets is English, accounting for more than 75% in many cases. This impacts not only the culture-dependent outputs of the models but also their performance when tasked with non-English corpora. OpenLLM-France is a consortium of labs and companies that aims to create open, French-oriented LLMs.
Towards a Reappraisal of the Linguistic Perspective in the LLM Era
Veronique Hoste (Ghent University)
Main takeaway
Blindly trusting the power of LLMs will not solve some tasks. We need linguistics back.
Summary:
While the current zeitgeist is that LLMs will solve everything by means of more parameters and more data, Veronique Hoste defended the importance of computational linguistics in many tasks. She argued that many discourse-related tasks are still hard to solve with LLM-only systems: coreference resolution (determining whether “she” refers to “Maria” or “the teacher” in a given sentence), emotion detection, and irony detection. She called for an effort to unify expert linguistic knowledge with modern LLM architectures.
Foundation models for text and speech encoding in French and other languages
Laurent BESACIER (NAVER)
(slides: JEP-TALN-2024-Besacier.pdf )
Main takeaway
HyperCLOVA X, an LLM for Korean.
Summary
Laurent walked us through some of the “French-speaking” LLMs that NAVER has contributed to, such as HuBERT (speech) and FlauBERT (text, a French BERT). However, generalist LLMs are not as deficient in French as they are in other languages, such as Korean. HyperCLOVA X is an LLM trained by NAVER (a Korean company) on Korean user-generated text.
Having Korean-speaking LLMs is key for companies working with Korean clients (such as Criteo). When no Korean model is available, the poor man’s solution is to pivot into English and then translate the final result into Korean, but it’s very suboptimal, and many nuances are often lost.
Papers
Reducing repetitions in machine translation
This paper, authored by a team from Systran (a machine translation company), discusses the challenge of managing repetition in machine translation. Take the Spanish sentence: “Hemos luchado contra la infodemia combatiendo los mitos…”. Should we translate it into English as “We have combated against the infodemy by fighting the myths…” or rather as “We have fought against the infodemy by combating the myths…”? Variation in word choice makes the sentence more readable. However, LLMs are not explicitly trained to manage this, often resulting in excessive repetition (a problem particularly pronounced in encoder-decoder architectures). The authors propose a modification of the loss function to tackle this issue.
A personal thought: we are constantly tweaking the loss function to solve various isolated problems like repetition, hallucination, etc. While it might be tempting to stack these loss functions together, it’s important to note that the loss function designed to reduce hallucinations might, for instance, conflict with the one intended to reduce repetitions. How can we combine all these constraints? Perhaps through curriculum learning, mixtures of experts, or other strategies?
Detecting clinical entities in few-shot for three languages
This paper compares the performance of encoder-based (BERT) and decoder-based (GPT) LLMs in Named Entity Recognition (NER) tasks. Encoders generally underperform in general domains but excel in clinical domains, suggesting that LLMs have not seen enough data in certain specialized areas.
Why is NER in clinical data important? It is a crucial step in mapping unstructured clinical documents into more structured, usable data. Examples of entities that need to be detected include drugs and diseases.
Why do all LLMs underperform in this context? The primary issue is the lack of annotated clinical data. The dominant strategy has been to use encoder models (like BERT) pre-trained in general domains and then fine-tuned on limited clinical data. A second strategy involves pre-training a decoder (like GPT) and using prompts to adapt the model to NER tasks.
The problem with the encoder approach is that there isn’t enough high-quality data to fine-tune the models in the clinical domain. Prompting, on the other hand, is highly context-dependent, and its efficiency varies significantly depending on the scenario, making it slow and CO2-consuming.
Overall, fine-tuning BERT-like models on clinical data is the best and most ecological choice, as these models and datasets tend to be smaller.
On the difficulties of long-document machine translation
BLEU is the most widely used metric for translation evaluation. It is typically computed by comparing the output sentence or paragraph with reference texts. However, some variants of BLEU have been proposed to handle document-level evaluations since document-level BLEU doesn’t always align with the mean of sentence-level BLEUs. After analyzing BLEU in the context of long documents, the paper proposes ds-BLEU, a new variant designed specifically for long-document translation.
Small models are good: an empirical study of zero-shot classification
Imagine you have a classification task (e.g., spam/not-spam, category classification). Should you use the largest LLM available? Not necessarily. In this paper, researchers show that bigger is not always better. Fine-tuning a smaller model or changing the architecture (e.g., using an encoder-decoder or a decoder-only model) might yield better results.
Evaluation of current benchmarking for biomedical French language models
This paper emphasizes the importance of high-quality annotated medical data. It appears that medical datasets are often annotated by non-medical professionals, who may not fully understand the significance of a given medical condition or whether different terms refer to the same condition. As a result, medical data should ideally be annotated by health professionals.
A very nice conference with a very active community doing amazing research to make AI accessible to everyone. We look forward to seeing what our French-speaking family brings next year! 🚀
