Visitor Centre Robot Assistant

Visitors arriving at a university's Visitor Centre ask a small set of practical questions: where a room is, what lunch costs, who may park, how to register a guest. This project explores whether a social robot can usefully take some of those questions, with a staff member always nearby as the fallback.

Two constraints shaped the work. The space is public and noisy, and a robot that confidently gives a wrong direction does more harm than one that admits it does not know.

The starting point was a NAO with no knowledge base of its own, running on a BUas AI-assistants platform and listening through its own microphone. I built a local version around it, with a knowledge base the robot answers from, so I can tune and test it on site. A Python 3 loop on a laptop captures speech, transcribes it, calls the chatbot and sends the reply to a small Python 2 HTTP bridge on the robot, which exposes speak, acknowledge, thinking, listening and touch endpoints. Three interchangeable backends share one interface and one system prompt: a local Ollama model, an OpenAI-compatible model on the BUas LLM server, and the original platform. Only the model changes between them, which keeps every comparison fair.

The whole knowledge base is injected into the system prompt, which is simple and reliable at this size. Strict rules sit around it: the robot may reason over its facts, for example to total a bill or check opening hours, but it never invents a location, price or contact detail.

Most of the hard problems were about timing, not language. On the local model a model call took a median of 5 seconds, between 2 and 11 across the 67 calls in a full test run, so the rest of the loop had to hide the wait. A hostname lookup that resolved to an address the robot's server did not listen on was delaying every reaction, so the laptop now resolves it once to IPv4 at start-up. Sending an answer sentence by sentence left audible gaps in NAO's speech, so each reply goes out in one call. The acknowledgement is a clip synthesised to WAV at robot start-up, and transcription runs in a background thread while it plays, so the visitor hears a reaction at once. The robot's server handles one request at a time, so the laptop polls for the head touch instead of holding a connection open.

1. Listenlaptop microphone, speech-to-text
2. Correctcampus names fixed before the chatbot sees them
3. Answerchatbot reading from a knowledge base
4. Speak and pointNAO talks, gestures and shows cues

I am the sole developer. My employer sets the feature requests and reviews progress in regular meetings, my mentor, Margot Neggers, advised on model choices, and the implementation decisions are mine.

The pieces I own: the head-touch start, name capture, the listening, acknowledgement and thinking cues, pointing while speaking, the location-aware system prompt and knowledge base, speech-to-text tuning and mishearing correction, the local and server chatbot backends, configuration and logging, and both test suites.

The robot runs end to end at the Visitor Centre. I chose the model by benchmarking it against the safety suite, not by speed alone: a 4B model (gemma3:4b) was the fastest but leaked its instructions and invented directions, and qwen3:8b was safe but slower. The assistant first ran on a local Gemma 4 model and now runs on Qwen3.6-27B hosted on the BUas server, which passes both suites in full.

Two automated suites guard the behaviour: 67 chatbot cases across 13 categories, including 8 red-team and 5 refusal cases, and 24 goodbye-detection cases, 15 that should end a visit and 9 that should not. The chatbot suite is keyword-based, so passing means the expected-phrase and forbidden-phrase rules hold; I still read the answers myself rather than quote an accuracy figure.

In a full test run on the local model, the 67 model calls took about 5.6 minutes in total: a median of 5 seconds each, and between 2 and 11 seconds. With the model on the BUas server, the delay before NAO speaks is now barely noticeable.

The assistant is still being improved and tested with visitors at the Visitor Centre. Feedback so far is limited, so I make no claims about visitor outcomes yet.

Every behaviour above is guarded by automated test cases. Each tile is one case, and every one passes on the current model. The answer checks use keyword rules, so I read the answers myself as well; the goodbye checks are exact and involve no model.

• Enrich the knowledge base. Cover more of the campus and the visit, with every fact verified with the Visitor Centre and the test suite extended in step, so new facts and new refusals are checked together.
• Move from a prompt-stuffed knowledge base to RAG. Chunk and embed the source documents, retrieve only the few relevant passages for each question, and keep the rule that nothing retrieved means an honest "I don't have that information". The existing suite would show whether retrieval preserves the no-invention behaviour, and facts could be updated without restarting the assistant.
• Handle several people speaking. With two or more visitors talking at once, the assistant would need to separate speakers, for example with diarisation or a microphone array, and track whose question it is answering.
• Recognise Dutch names and speech. English speech recognition often mishears Dutch names, for example "Stijn" arrives as "stone". Running recognition in Dutch, or in both languages, is the real fix.
• Make pointing finer. Four preset directions from a fixed spot are coarse, and moving the robot invalidates them.
• Improve the audio input. The microphone sits on the laptop, so visitors have to speak towards it, and speech-to-text relies on an online service. A microphone near the visitor and on-device recognition would fix both.