Policy-Grounded Claim Decisions with RAG

The task was to optimise a return and warranty system. An agent uploads photos of an item, and the question is whether the claim should be approved, denied or sent to a person, according to the company's written policy.

Two things mattered. The decision had to follow the policy document, not a language model's memory of what such policies usually say, and uncertain cases had to go to a human instead of being guessed.

The system has three parts: a web app for agents, an API that orchestrates the checks, and models served from a separate GPU server. A vision model turns the photos into structured findings, and a second step decides the claim against the policy. That second step is the RAG service I built.

The policy document is split into overlapping chunks, embedded with BAAI/bge-m3 and stored in a persistent Chroma vector store, which fills itself on first start-up if it is empty. For each claim I build one search query from the product type, the customer's description and the vision model's findings, so retrieval sees both what was claimed and what was seen.

The three closest passages go to a Qwen2.5-7B model together with the claim and the findings. It answers in JSON with a decision, approved, denied or needs human inspection, plus its reasoning and a confidence score. The embedding and language models sit behind OpenAI-compatible endpoints, so either can be swapped without touching the pipeline, and changing a rule means editing the policy document rather than the code.

We were a team of two. I was responsible for the RAG part: ingesting and chunking the policy document, embedding and storing it, retrieving the relevant passages for each claim, and the policy-grounded decision step.

The RAG service is part of the claim flow end to end: a claim with photos goes in, and a decision with reasoning and a confidence score comes out, with unclear cases routed to human inspection.

• Show the evidence. The decision model already has a field for the policy excerpts it relied on, but the API returns only the decision, reasoning and confidence. Returning the cited passages would let an agent check any decision against the policy in seconds.
• Measure retrieval before tuning it. Build a labelled set of claims, note which policy passages should be found for each, and check whether they land in the top 3. Chunk size, overlap and the number of passages should be tuned against that, not by feel.
• Constrain the output. The JSON is currently cut out of the reply between its first and last brace. Schema-constrained generation with validation would remove that fragility, with an invalid reply falling back to human inspection.
• Improve the retriever. Combine keyword and embedding search, and rerank the candidates, so exact policy terms are not lost in a purely semantic match.
• Calibrate the confidence score. It is the model's own estimate. Comparing it with agents' final decisions would show whether it can be trusted for routing low-confidence claims to a person.