Add grounded Ollama rewrite flow for chatbot

This commit is contained in:
nmemmert
2026-04-14 12:46:56 -04:00
parent 50459b7234
commit 87967bf149
3 changed files with 333 additions and 9 deletions
+44
View File
@@ -57,6 +57,50 @@ Admin auth:
The app will be available at `http://localhost:4173`.
## Optional local LLM (Ollama) for grounded rewrites
You can keep deterministic retrieval as the source-of-truth and optionally rewrite responses with a local model.
1. Install and run Ollama on your host.
2. Pull a small model suited to older hardware, for example:
```bash
ollama pull qwen2.5:3b-instruct
```
3. Start the API with these environment variables:
```bash
CHATBOT_LLM_ENABLED=true
CHATBOT_LLM_BASE_URL=http://127.0.0.1:11434
CHATBOT_LLM_MODEL=qwen2.5:3b-instruct
CHATBOT_LLM_TIMEOUT_MS=25000
CHATBOT_LLM_NUM_CTX=2048
```
4. Call the rewrite endpoint from your existing chat flow:
`POST /api/chatbot-grounded-rewrite`
Request payload shape:
```json
{
"question": "Who was Titus?",
"draftAnswer": "Deterministic answer produced by current retrieval/synthesis.",
"sources": ["Episode 2 - Introduction to Titus"],
"contextChunks": [
{
"title": "Episode 2 - Introduction to Titus",
"sourceLabel": "Episode 2",
"content": "Titus was a Gentile..."
}
]
}
```
If the endpoint fails or is disabled, keep your deterministic answer and existing fallback behavior.
Persistent admin saves:
- Admin updates are written to `data/admin-content.json`.