AI Engineering FoundryCompleted
SupportDesk AI
Customer support RAG chatbot in .NET 10 and React: cited answers, refusals when docs don't cover it, tenant-safe Qdrant search and a 59-question eval.
The problem
A support team's answers already exist, spread across a refund policy, a warranty PDF, a shipping FAQ and an internal escalation guide. Customers and new agents still ask the same questions, and a general chatbot will happily invent a return window that isn't in any of them. SupportDesk AI answers only from the documents a team uploads, cites the passages it used, and says so when the documents don't cover a question. It also has to respect who is asking: a customer must never see what is in the internal escalation guide, and one company's documents must never show up in another company's answers.
Engineering focus
- RAG in ASP.NET Core, step by stepTutorial
- Retrieval-augmented generationPrimer
- Local models with OllamaGuide
- Page- and heading-aware ingestion with OCR
- Access control inside the Qdrant query
- Cited answers streamed over SSE
- Follow-up question rewriting
- Deterministic evaluation, no LLM judge
In use
How it works
- Background ingestion
- Uploads return 202 and a background worker extracts, chunks, embeds and indexes them; anything still pending after a restart is re-queued, so SQL Server stays the source of truth. PDFs are read page by page with PdfPig, and a page with almost no text but with images is sent to Tesseract OCR.
- Chunks that keep their place
- Chunks are up to 1,200 characters with 150 of overlap and never cross a page or heading, so a citation can name the page and section. Each chunk is embedded together with its document title and section.
- One collection per embedding space
- Provider, model and dimension name the Qdrant collection, so vectors from two embedding models can never be mixed. Switching models means reprocessing into a new collection; a mismatch is rejected with a 409.
- Access filter inside the query
- The user's organization and allowed visibilities are must conditions of the same Qdrant search, on indexed payload fields. The search method has no overload without an access scope, and content a user may not see is never retrieved, ranked or sent to the model.
- Refuse before calling the model
- If no chunk passes the score threshold, the API returns a fixed not-found answer without a model call. Otherwise up to five numbered passages go into the prompt, which tells the model to answer only from them, cite them and treat them as untrusted data.
- Streaming with sources first
- The chat endpoint sends server-sent events in a fixed order: conversation, the rewritten question for follow-ups, sources, answer tokens, then done or error. The React client renders the citations before the first token arrives.
- Swappable providers
- Chat and embeddings switch independently by configuration between Ollama (qwen2.5:7b, nomic-embed-text) and OpenAI, through small typed HttpClients instead of AI SDKs.
Measured results
All numbers come from the repo's dated result files in eval/results/. Scope: a 59-question test set (44 single questions plus 15 follow-ups with a scripted history; 49 answerable, 7 unanswerable, 3 forbidden) over an 8-document test knowledge base for Northwind Outfitters, a fictional outdoor-gear shop: seven public documents in Markdown, text and a 2-page PDF, plus one internal escalation guide. Chat model qwen2.5:7b, embeddings nomic-embed-text, both on Ollama on an 8 GB GPU; top 5 chunks, MinScore 0.3. These are results on a small test set, not a benchmark.
Current run, 20261005-203035-dense-full.md (2026-10-05):
| Metric (59 questions, 49 answerable) | Result |
|---|---|
| Retrieval hit rate @5 | 100% |
| Answer accuracy | 95.9% (47 of 49: all 37 single questions and 10 of 12 answerable follow-ups) |
| Refusal accuracy (unanswerable + forbidden) | 100% |
| Citation accuracy | 97.9% |
| Hallucination rate | 0% |
| Forbidden chunks retrieved / access leaks | 0 / 0 |
| Median total time | 792 ms |
The two misses are follow-ups: one over-refusal (f02, the answer was in the retrieved passage) and one rewrite that dropped the word that linked the question to the right passage (f04). The refusal and leak figures rest on a handful of questions; the repo's own evaluation notes call a small number of passing refusal questions "limited evidence", which is why production telemetry also counts declined outcomes.
What the evaluation changed
The scoring is deterministic: each question lists the facts an answer must contain and the facts it must not, and citations are parsed from the [n] markers. The same answer gets the same score every run, and every failure can be read in the report. Each change below was kept or dropped on those numbers.
| Date (UTC) | Change | Result (result files) | Decision |
|---|---|---|---|
| 2026-10-04 | Baseline (33 questions, 27 answerable) | 92.6% answers (25/27), 6/6 refusals, 100% citations, 0% hallucination, 484 ms median. Both misses (q02, q12) were over-refusals: the right passage was retrieved but the model wouldn't apply a 30-day rule to "six weeks ago". (20261004-170538-baseline.md) | Fix the prompt, not retrieval |
| 2026-10-04 | Similarity threshold | Unanswerable questions scored 0.44–0.61, overlapping the weakest answerable question (0.54). No threshold separates them. (20261004-170409-baseline-retrieval.md, EVALUATION.md) | Keep MinScore at 0.3 as a floor; let the prompt refuse |
| 2026-10-04 | Prompt variant: "apply a covering passage's rules; refuse only when no passage covers the topic" (44 questions) | 94.6% → 97.3% answers, fixed q12, no change in refusals, hallucination or leaks. A different variant ("reason from what the passages state") broke another question and was rejected. (20261004-194224-dense-final.md, 20261004-200941-topic-prompt.md) | Adopted |
| 2026-10-04 | Follow-up rewriting (59 questions) | Follow-ups 13/15 with rewriting vs 10/15 without; questions that are only pronouns 3/3 vs 0/3. Answer accuracy 93.9% vs 87.8%; median over all questions 639 ms vs 548 ms. (20261004-202425-rewrite-on-2.md, 20261004-202518-rewrite-off-2.md) | On |
| 2026-10-04 | Hybrid search (BM25 sparse + dense, RRF k = 60; 44 questions) | Exact-code passage MRR 0.95 → 1.00, but answers 94.6% → 91.9% (95% on a repeat) and citations 100% → 97%. With top 5 the right chunk was already in the context. (20261004-194224-dense-final.md, 20261004-194259-hybrid-final.md) | Built, off by default |
| 2026-10-05 | Reranking (ms-marco-MiniLM-L6-v2 cross-encoder, ONNX on CPU; 59 questions) | Same answers (95.9%), citations 97.9% → 93.6%, median 0.8 s → 1.6 s. A score cutoff of 0.01 dropped answers to 88%. (20261005-203035-dense-full.md, 20261005-203138-rerank-full.md) | Built, off by default |
| 2026-10-06 | Chat model on the same 59 questions | See below | Keep qwen2.5:7b |
| Model (Ollama, 8 GB GPU, 2026-10-06) | Answers | Hallucination | Citations | Median / p90 latency |
|---|---|---|---|---|
| qwen2.5:7b (default) | 96% | 0% | 98% | 0.8 s / 1.2 s |
| llama3.1:8b | 90% | 0% | 98% | 0.8 s / 1.4 s |
| qwen3:8b, thinking off | 94% | 2% | 96% | 0.8 s / 1.7 s |
| qwen3:8b, thinking on | 94% | 2% | 88% | 5.3 s / 9.9 s |
All four models refused every refusal question and leaked nothing. Thinking mode cost about six times the latency and lowered citation accuracy on this set. Hybrid search and reranking are not wrong ideas: they are switched off because, on a small knowledge base where the right chunk is already in the top 5, they added cost without better answers. They are one setting away for a larger corpus with many near-duplicate chunks.
Key decisions
| Decision | Why |
|---|---|
Typed HttpClients instead of AI SDKs | One way to handle timeouts, cancellation, errors and streaming across Ollama and OpenAI |
| Access filter inside the Qdrant query | A filtered-out chunk can't leak through ranking, token counts or the prompt |
| Documents are Internal by default | Publishing to customers is a deliberate step |
| One Qdrant collection per embedding space | Mixing vectors from two models becomes impossible by construction |
| Chunks never cross a page or heading | Citations can name the page and section |
| Hybrid search off by default | Better ranking for exact codes, no better answers on this corpus |
| Reranking off by default | Same answers, lower citation accuracy, twice the latency |
| Follow-up rewriting on | 13/15 vs 10/15 follow-ups answered |
| Tesseract as a CLI, optional | Same behaviour on Windows and Linux; OCR is skipped if it isn't installed |
| API and web app in one container | Same origin: no CORS, and the SameSite=Strict refresh cookie keeps working |
Security
Every user, document and conversation belongs to exactly one organization. Roles decide visibility: customers see Public documents, agents and admins also see Internal ones. Sign-in returns a 15-minute JWT tied to a server-side session; refresh tokens are 256 random bits, stored only as SHA-256 hashes in an HttpOnly, Secure, SameSite=Strict cookie, and rotated on every use. Presenting a used refresh token again revokes the whole session. An integration test against a real Qdrant indexes documents for two organizations and both visibilities, then searches as each role; with the filter removed on purpose, the other organization's chunk appeared and the test failed.
For the wider picture of permissions in AI systems, see AI agent security for developers.
Architecture
React 19 (served by the API, same origin)
│ HTTPS, JWT + refresh cookie, SSE
ASP.NET Core 10 API ── minimal APIs, rate limiting, OpenTelemetry
├─ Application: RagService (rewrite → retrieve → refuse? → prompt → stream), ingestion, identity
├─ Infrastructure: Ollama/OpenAI clients, Qdrant search, EF Core, PdfPig, Open XML, Tesseract, ONNX reranker
└─ DocumentProcessingWorker (background): extract → chunk → embed → upsert
│ │ │
SQL Server Qdrant Ollama / OpenAI
(users, docs, (one collection per (chat + embeddings)
conversations, embedding space,
audit, sessions) org + visibility payload)Clean Architecture: Domain (entities and rules) → Application (use cases, RAG, ingestion) → Infrastructure (providers, storage) → Api (endpoints, SSE, security, telemetry). Dependencies point inward.
Limits
- A 7B model over-refuses some questions that need a small inference ("six weeks ago" against a 30-day window). The larger models tried did worse overall on this set.
- The test set is small: 59 questions over 8 documents. The refusal and leak results rest on few questions.
- OCR quality limits what scanned pages can answer.
- Changing the embedding model means re-processing every document into a new collection.
- Not built, on purpose: agents, MCP, GraphRAG, prompt management, multiple vector databases, Kubernetes, microservices.
Run it
Prerequisites: .NET 10 SDK, Node.js 20.19+, Docker, Ollama with qwen2.5:7b and nomic-embed-text pulled (or an OpenAI key), and optionally Tesseract.
git clone https://github.com/AfzaalLucky/SupportDesk-AI-Enterprise-RAG-Core-React.git
cd SupportDesk-AI-Enterprise-RAG-Core-React
cp .env.example .env # set Auth__SigningKey, the bootstrap admin, SQLSERVER_SA_PASSWORD
ollama pull qwen2.5:7b && ollama pull nomic-embed-text
docker compose --profile app up -d --build
# open http://localhost:5280The README covers running the API from source (dotnet run), the API endpoints, and running the evaluation yourself (dotnet run --project tools/SupportDesk.Eval -- --label my-run), which writes a dated report to eval/results/.
Related
- Build RAG in C# with ASP.NET Core: the step-by-step tutorial built from this code.
- What is RAG?: the concepts behind every stage above.
- AI Knowledge Base Chat: the same idea in Python with FastAPI. It is a single-user, local document Q&A app with LangChain loaders; SupportDesk AI is the .NET version with organizations, roles, follow-ups and a published evaluation.
- AI Database Agent, Part 4: hybrid retrieval over a SQL business glossary, where hybrid did help. Compare it with the result here.
- ASP.NET Core vs FastAPI for AI APIs: the trade-offs behind building this in .NET.
- Real Estate AI Chatbot with ASP.NET Core: a chatbot without retrieval, streaming Claude over SignalR. Use SupportDesk AI when answers must come from your own documents.
- More in RAG articles.
Operations
On a 132-question workload the model was 96% of the time. The first trace found a two-second IPv6 penalty on every new connection to Ollama and Qdrant; fixing it cut a cold retrieval from 4.14 s to 55 ms.
- Traces
- OpenTelemetry spans for each question, with rewrite, retrieve and generate stages, viewable in the Aspire dashboard.
- Metrics
- Retrieval and generation durations, time to first token, token usage and one outcome per request: answered, no_context, declined, failed or cancelled.
- Logs and audit
- One structured log line per question with IDs, counts, scores and timings, never the question or answer text, plus an audit trail of sign-ins and document changes.
- Container
- A Docker image and Compose file for the API and web app, SQL Server and Qdrant, with Ollama on the host for the GPU and Nginx in front on a server.