I submitted my grasp’s thesis on Emotion Recognition in Dialog (ERC). The mannequin is emonetachieved weighted F1 39.18 with EmoryNLP — Competing with the general public PapersWithCode leaderboard on the time, sitting between TUCORE-GCN_RoBERTa (39.24) and S+PAGE (39.14), and an enchancment over CoMPM, my baseline of alternative. +1.81 F1.
Two years later, I checked the sector once more. Leaderboard will not be acknowledged. High entries are not encoder-only fashions with intelligent consideration heads. LLaMA-2–7B-based system with LoRA tweaks and search enhancement prompts: InstructERC, CKERC, BiosERC, LaERC-S. The strategies are totally different. Calculations are totally different. I’ve a unique mind-set.
Nonetheless, a cautious studying of those new papers reveals that The core concept I proposed in EmoNet seems inside EmoNet, simply carried out in one other layer of the stack. That is the story of what I constructed, the place I positioned it, and what I’d construct now if I had been to start out over.
What’s ERC and why text-only is tough?
Emotion recognition in dialog is the duty of assigning an emotion label to every utterance in a multi-turn interplay. This differs from sentiment evaluation for stand-alone texts in a single necessary method: The emotion of an utterance is formed by what occurred earlier than it and by who’s talking.
Think about this trade from the EmoryNLP dataset (sourced from a TV present). buddy):
Monica: Wendy, we’ve a deal! Sure, you promised! Wendy! Wendy! Wendy! [Mad]
Rachel: Who was that? [Neutral]
Monica: Wendy bailed. No waitress. [Mad]
Remoted and asking, “Who was that?” Emotionally impartial. label impartial it simply is sensible in context — That is situated between two indignant utterances by totally different audio system, and the ERC mannequin should seize the dynamics of this dialog.
There’s a second wrinkle: Multimodal data is lacking. In actual human conversations, tone of voice, facial expressions, and physique language are a giant a part of the emotional alerts. Textual content-only ERC removes all of that. The identical phrases: “Oh, that is superb.” — will be honest or sarcastic, and it is usually laborious to inform from the textual content alone.
This data loss is the largest problem. We have to extract feelings from alerts which might be noisier than human-level benchmarks.
Panorama of 2024
After I began writing my paper in late 2023, the EmoryNLP leaderboard was dominated by transformer-based architectures with numerous intelligent modifications. A fast tour:
– ket (Zhong et al., 2019) — A educated transformer centered on emotional graphs. The primary paper that introduced transformers to the ERC.
– Dialogue GCN (Ghosal et al., 2019) — a graph convolutional community that transforms interactions into node classification issues.
– RGAT (Ishiwari et al., 2020) — Relation-aware graph consideration utilizing relational positional encoding of speaker dependence.
– Dialogue XL (Shen et al., 2020) — tailored XLNet with utterance repetition and dialogue self-attention.
– excessive trance (Li et al., 2020) — a hierarchical transformer with pairwise utterance speaker verification as an auxiliary activity.
– Tucore-GCN (Lee & Choi, 2021) — Heterogeneous dialogue graph utilizing speaker recognition BERT.
– CoMPM (Lee & Lee, 2021) — combines dialogue context with pre-trained reminiscence monitoring of audio system.
I selected Based mostly at CoMPM For 2 causes. First, we explicitly modeled the speaker’s pre-trained reminiscence as a separate module. This matched my instinct as follows. who talking is simply as necessary what they’re saying. Second, its structure was sufficiently modular that it may very well be expanded with out having to rewrite it from scratch. The CoMPM paper confirmed that including pre-trained reminiscences to the context mannequin resulted in measurable enhancements, however speaker identification was nonetheless preserved. native to every dialog. The second a brand new dialog started, all the pieces the mannequin had realized concerning the speaker was discarded.
It appeared like an issue price fixing.
Three contributions of instinct
1. World speaker identification
drawback. In CoMPM and most earlier research, the scope of speaker ID is restricted to at least one dialogue. Speaker A Scene 1 has nothing to do with Speaker A In scene 14, even when it is the identical individual. Due to this fact, all interactions begin coldly.
Instinct. Folks have distinctive emotional patterns. Monica will get indignant about sure issues. Phoebe is admittedly hilarious. As anticipated, Ross is overcome with a way of hysteria. If the mannequin can maintain the next data this specific speaker All through the dialog, it’s best to be capable to make better-calibrated predictions when the speaker seems once more.
implementation. Every distinctive speaker inside the complete dataset will get a secure identification throughout the dataset. first time monica geller message, she is assigned an ID (for instance, ID 7) and retains it. After that, she stays ID 7 each time she seems throughout episodes, seasons, and scenes. The mannequin is now in a position to be taught speaker-specific patterns that persist.
That is clear on reflection. In 2024, the leaderboard mannequin did not work.
2. Speaker motion module
drawback. World Speaker Identification alone is only a label. For this to be helpful, your mannequin should: do one thing Together with the historical past that audio system have collected. How can we give the transformer entry to “all the pieces Monica has ever stated on this dataset” with out blowing out the context window or making coaching tough?
Instinct. recurrence. GRU is a pure match for sequentially compressing a speaker’s previous utterances right into a single, fixed-size illustration. Latest statements contribute additional. The previous ones step by step fade away. configurable double sliding window Restrict GRU’s enter (for instance, the final N utterances by this speaker) to maintain compute and reminiscence predictable.
implementation. Every utterance is individually encoded by a pre-trained RoBERTa spine. The ensuing embedding flows by way of the GRU in a single route. The ultimate hidden state of the GRU (which we name “kt”) represents the speaker’s conduct sample on the present second. That is projected to the identical dimension because the interplay context output and is appended to it. The mixed sign is fed to the ultimate classifier.
This structure is structurally much like CoMPM’s pretrained reminiscence module, however with two necessary variations. The speaker historical past pool is international (not native to the present dialog), GRU explicitly fashions temporal decay.
3. Weighted cross-entropy loss
drawback. EmoryNLP Imbalance — impartial giant quantity unhappy Roughly 4.5:1. Most papers handle this by knowledge augmentation or undersampling. Nevertheless, the dialog knowledge collection: Deleting or duplicating utterances distorts the pure stream of emotion. That is precisely the sign that the mannequin is attempting to be taught.
Instinct. In the event you can’t safely change the information, change it at a loss. A single misclassification happens as a result of uncommon lessons have larger weights unhappy The mannequin prices greater than a single misclassification impartial.
implementation. Cross-entropy derived from inverse class frequencies and containing normalized per-class weights. There’s nothing uncommon about it, however the dialogue of the dialogue sequence is clearly motivated, making this a principled alternative moderately than an arbitrary one.
Outcomes: What labored and what shocked me
The paper’s ablation desk is:
The end result that shocked me, and I feel probably the most trustworthy a part of this piece, is the second line. Simply including the World Speaker ID made the mannequin considerably worse (F1 dropped from 37.85 to 29.43). It appeared like a failure at first.
However that wasn’t the case. World speaker identification means — This permits the mannequin to be taught the patterns of long-distance audio system. That characteristic itself creates a representational burden that the remainder of the mannequin can’t soak up. Simply as soon as, speaker operation module has been added – offers a structured approach to mannequin use World identification — contributions surfaced. By the ultimate configuration, EmoNet had recovered and outperformed the CoMPM baseline by 1.81 F1.
That is the lesson realized from ablation. Options alone don’t have any worth. It has worth when mixed with the machines that eat it. Analysis papers that report “this addition resulted in a +X% impact” usually disguise ablation strains the place the addition alone made issues worse. I made a decision to go away that column alone.
Full mannequin dealing with impartial, pleasureand I used to be scared good. highly effective remained probably the most tough class — partly as a result of it was uncommon; highly effective and pleasure They’re nearly indistinguishable in textual content conversations with out audio cues. This can be a multimodal drawback disguised as a textual content drawback.
Reflection (2026): The sphere has moved, so we should transfer too
Two years later, EmoryNLP’s leaderboards look very totally different. The present main techniques are:
– Direct ERC (Lei et al., 2023) — reformulates ERC as a generative LLM activity. Higher mannequin interplay roles and emotional dynamics utilizing search-enhanced instruction templates and auxiliary duties akin to speaker identification and emotion prediction.
– CKERC (Fu, 2024) — Introducing ERC with enhanced frequent sense. For every utterance, LLM generates frequent sense annotations concerning the speaker’s intentions and presumably the listener’s reactions, offering implicit social and emotional inferences past the specific interactional context.
– BiosERC (Xue et al., 2024) — Inject speaker biographical data derived from LLM into the ERC course of, permitting the mannequin to make inferences not solely concerning the context of the utterance but additionally about speaker-specific traits.
– LaERC-S (Fu et al., 2025) — Two-step instruction tuning. Stage 1: Equip your LLM with: Speaker-specific traits. Stage 2: Use these traits throughout the ERC activity itself.
Look fastidiously on the final two.
BiosERC speaker bio data In spirit, it is a scaled-up model of World Speaker ID. As a substitute of an integer ID, it’s a textual content profile that LLM can accommodate. LaERC-S speaker traits In spirit, it’s a speaker conduct module (historic speaker patterns out there within the mannequin), however as an alternative of being carried out as a separate GRU, it’s constructed into the instruction tuning.
Architectural instinct got here in useful. The implementation layer has modified.
That is the half I discover actually attention-grabbing. After I was engaged on EmoNet in 2024, I used to be considering inside an encoder-only transformer paradigm. “How can we add one other module to the structure?” The 2024-2025 paper is considering contained in the LLM paradigm about “How can we encode this concept into an instruction tuning or search context?” The concept is analogous. Leverage factors are totally different.
If I had been to rebuild EmoNet nowI am not going to start out with RoBERTa-large. I’d begin with a small open supply LLM (LLaMA-3.2–3B, Qwen-2.5–3B, or Phi-3.5). LoRA Observe the InstructERC household of approaches and fine-tune them with EmoryNLP. World Speaker Identification would be the speaker’s biography in textual content format retrieved from the vector retailer. The speaker’s conduct module will probably be a few-shot immediate that reveals the speaker’s most up-to-date emotional historical past. The weighted loss stays largely unchanged. Class imbalance would not care concerning the mannequin you are utilizing.
The structure diagram will look utterly totally different. If you realize the place to look, the conceptual debt to the 2024 paper will turn out to be obvious.
From this research, I realized that the half-life of analysis debt is longer than I anticipated. Concepts can survive paradigm shifts, even when the implementation would not.
the place this leaves me
EmoNet is at present publicly archived at: DOI 10.5281/zenodo.20048006 Consists of full paper, protection slides, and PyTorch implementation. GitHub. I am at present engaged on a modernized port (a LoRA-tweaked LLM with search-based speaker context) as a follow-up venture. I’ll write about this quickly.
In the event you’re engaged on fine-tuning conversational AI, utilized NLP, or LLM, we might love to listen to what you are constructing.
