67.6%
of sentences got the same emotion whether we read them or listened to them.
Hover or tap an emotion or a flow.
Our 651 annotated test sentences, labelled twice: once from the transcript, once from the audio. Much of the emotion lives in the voice, which a text-only model never hears.
Problem
Emotion recognition is built mostly for English. We set out to do it in Bulgarian, our own language, on real conversation: reality-TV speech, labelled across seven emotions, with a model trained on 608,540 sentences drawn largely from books.
What we built
A pipeline that takes a YouTube link and returns a timestamped table: every Bulgarian sentence, its English translation, and its emotion, one of anger, disgust, fear, happiness, neutral, sadness or surprise. The emotions come from a fine-tuned XLM-RoBERTa transformer that reads the Bulgarian directly.
- Videoa YouTube link
- TranscribeBulgarian speech to text
- TranslateBulgarian to English
- ClassifyXLM-RoBERTa, seven emotions
My role
I annotated emotions for our test set and contributed to feature extraction. My main focus was modelling. I owned the RNN and LSTM track: eleven documented experiments, from a simple baseline to stacked BiLSTMs with attention, BPE tokenisation and focal loss. Together with a teammate, I then built our best model, the fine-tuned XLM-RoBERTa transformer. I also ran the prompt-engineering track with Llama models. I also presented the project.
Result
Our best model, and the one in the final pipeline, was the XLM-RoBERTa transformer I built with a teammate: 0.42 macro F1 on real speech, against 0.33 for the best LSTM. My LSTM track showed why it was needed: trained from scratch on translated written text, even the final tuned LSTM fell from 0.59 on validation data to 0.32 on real speech. A model pre-trained on far more varied text is better placed to handle that gap. The transformer still leans towards neutral and struggles with anger, fear, surprise and disgust. Prompting Llama 3.3 70B with self-consistency voting reached 0.59 on a separate 1,005-sentence set.
My experiments
Macro F1 on the same 1,005 labelled sentences. This set differs from the LSTM test set, so the two are compared by trend, not head to head.
What I'd do now
The experiments made one thing clear: better data would move the result more than a bigger network. Next, I would:
- Train on speech, not books. Even a few thousand labelled conversational sentences would likely close more of the gap between validation and real speech than any architecture change.
- Use the voice. Reading and listening disagreed on a third of sentences. Pitch and energy features, or a speech model, could recover the anger that text alone misses.
- Let the LLM label, then fine-tune. Use the 70B model to label much more real Bulgarian speech, and fine-tune our transformer on it, combining the reach of the large model with the speed of the small one.
- Evaluate on one shared test set. Score the LSTMs, the transformer and the prompted LLM on the same annotated sentences, so every approach can be compared directly.
Skills gained
- Emotion annotation: labelling a test set across seven emotions, by reading and by listening
- NLP features: POS tagging, TF-IDF, sentiment scores and FastText embeddings for Bulgarian
- Deep learning for text: RNNs, BiLSTMs, attention, BPE tokenisation and focal loss, and where they stop against a pre-trained transformer
- Experiment discipline: eleven logged iterations, each with a hypothesis, a result and a next step
- Prompt engineering: few-shot prompts, bias-correcting instructions, majority voting and self-consistency
- Presenting: turning a technical result into a clear story for a non-specialist audience
Contact
Looking for a data science or AI intern?
I'm open to internships in data science, machine learning and NLP for February 2027. Email is the quickest way to reach me.