AI Engineering FoundryCompleted
AI Database Agent
Ask a SQL Server database a question in plain English and get back a validated, schema-aware SQL answer — built in five measured versions, from a naïve Text-to-SQL prototype to a secured, knowledge-augmented, tool-calling agent.
The problem
Natural-language database interfaces are easy to demonstrate but considerably harder to make reliable. The system needs to understand database structure, generate useful SQL, validate generated queries, control execution, and provide useful answers without blindly trusting model output.
In use
Versions and results
Each version is scored by execution accuracy — its result set must match hand-written gold SQL — on a 32-question development set and a 12-question held-out set written after the knowledge base was frozen. The held-out column is the generalisation claim; the development set was used to fix V4's failures. Only V1 ever executed unsafe SQL — an UPDATE and a DROP TABLE, both stopped by the database user's permissions.
| Version | What it adds | Dev (32) | Held-out (12) |
|---|---|---|---|
| V1 Text-to-SQL | Table and column names in, SQL out, run as a read-only database user. | 53% | 33% |
| V2 Schema-aware | Types, keys, join paths, descriptions, allowed values, business rules — and permission to decline. | 81% | 50% |
| V3 Validated | T-SQL parser allow-list, schema check, cost estimate, error-driven repair loop. | 84–88% | 50% |
| V4 Knowledge | Business glossary and verified examples, hybrid term + vector retrieval from Qdrant. | 100% | 75% |
| V5 Agent | Tool calling (describe, glossary, sample values, test queries) plus a harness self-check. | 97% | 75% |
Local qwen2.5-coder:7b (V1–V4) and qwen2.5:7b (V5, needs native tool calls). Each version name links to its write-up.
Operations
On a 132-question workload the model was 96% of the time. The first trace found a two-second IPv6 penalty on every new connection to Ollama and Qdrant; fixing it cut a cold retrieval from 4.14 s to 55 ms.
- Traces
- OpenTelemetry: one trace per question — retrieval, each model call with its tokens, validation, cost estimate, execution. Question text never goes on spans.
- Metrics
- Outcomes by version, per-stage latency, repairs, validator findings by kind, tokens.
- Audit log
- Every question in a separate AgentOps database, keyed by trace id. The agent never writes to the database it queries, and a failing audit log never costs an answer.
- Container
- Chiseled, non-root .NET 10 image; forwarded headers, per-client and concurrency limits for running behind Nginx; CI builds, tests and builds the image.
Architecture
Roadmap
Build roadmap