Reads 22 letters of Dutch Sign Language fingerspelling from a webcam, in real time.
Learning mode and a game in the browser, with every frame streamed to the model over a WebSocket.
0.85 / 0.80
Accuracy and F1 a model must reach
before the pipeline will register it
3
Places it runs
local, on-premise and cloud, from one Docker image
21
Hand landmarks tracked
on every frame, with MediaPipe
2
Models working together
a landmark model steps in when the image model is unsure
15
Frames vote on each letter
10 to lock it, 7 to keep it, so letters don't flicker
6
Live health metrics
on the operations console, each with its own threshold
The system at a glance. Every number comes from the project's code and configuration.
Problem
Learning sign language on your own is hard: books and videos show you a handshape, but nobody tells you whether yours is right. We built an app that watches your hand through the webcam and gives feedback letter by letter, for NGT, the sign language of the Netherlands.
What we built
A full-stack recognition system for the 22 static letters of the NGT fingerspelling alphabet. MediaPipe tracks the hand, a fine-tuned EfficientNet-B0 classifies the letter, and a model trained on the 21 hand landmarks takes over when the image model is unsure. A voting window then steadies the result before the Vue frontend shows it, in a learning mode and a game.
- Camerawebcam frames over WebSocket
- Track21 hand landmarks
- ClassifyEfficientNet-B0, landmark fallback
- Steady15-frame vote
- Feedbacklearning and game views
Behind it sits a production setup. An Azure ML pipeline splits, augments and fine-tunes the model, and refuses to register any version below 0.85 accuracy or 0.80 F1. One Docker image, with three-tier authentication, runs in three places:
My role
All-round. I worked across the stack, from the backend's foundations and the cloud training setup to the frontend, and I built the operations dashboard that watches the live system: traffic, errors, latency and model confidence, each against an editable threshold, with an overall health status.
| Endpoint | Status | Count |
|---|
A working replica of the console I built, with example traffic. The thresholds, health logic and error breakdown match the real system: one breached metric is enough to mark it degraded.
Result
A working system end to end: real-time recognition of 22 NGT letters in the browser, a training pipeline that only promotes models clearing the accuracy and F1 gate, the same image running locally, on-premise and in the cloud, and monitoring that flags problems before users report them.
What I'd do now
The foundation is production-ready; the next gains are in what it recognises and how it learns:
- Add the letters that move. Letters signed with motion need a sequence model over landmark time series, not a single frame.
- Test with Deaf signers. Native signers, different hands, skin tones and lighting would show where the model is weakest, and whether it is fair.
- Monitor accuracy, not just health. Confidence shows how sure the model is, not whether it is right. A light feedback loop, where users confirm or correct a letter, would measure live accuracy and feed retraining.
- Move inference closer to the user. A quantised model in the browser would cut latency and server cost.
Skills gained
- Observability: structured logging, error handling and a monitoring dashboard with editable alert thresholds
- Operational dashboards: turning raw API and model metrics into charts and health signals a team can act on
- Cloud ML foundations: setting up Azure ML environments and data assets for a gated training pipeline
- Frontend development: laying the foundations of a Vue 3 application
- Production context: working inside a system with Docker, CI/CD and three deployment targets
- Teamwork at scale: sprint planning across a shared monorepo with code review and quality gates
Contact
Looking for a data science or AI intern?
I'm open to internships in machine learning, MLOps and computer vision for February 2027. Email is the quickest way to reach me.