Compare commits
31 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| e815d7702e | |||
| db942fd734 | |||
| 67b7d1024b | |||
| e6080349af | |||
| 87967bf149 | |||
| 50459b7234 | |||
| 36594d5e16 | |||
| 0e9d112c44 | |||
| 063f5d2a0e | |||
| 6e8b360398 | |||
| 770d2357ac | |||
| f07d8fecfb | |||
| dcfce9dc24 | |||
| 738cc2a412 | |||
| bac22950ba | |||
| 85af1a29ad | |||
| 85e8a298fd | |||
| a252167052 | |||
| 53408ad124 | |||
| d5f15486d8 | |||
| afb80aaf2a | |||
| 3a9d11cf40 | |||
| 2653d95605 | |||
| 7772bfd6a6 | |||
| 74650a39b3 | |||
| c46d765759 | |||
| d95ed9cd8d | |||
| 4b01d01133 | |||
| e93840c25f | |||
| 5b401671db | |||
| f156257573 |
@@ -0,0 +1,23 @@
|
||||
# Copy this file to .env and fill in your real values.
|
||||
# Never commit .env to version control.
|
||||
|
||||
# Resend API key for the contact form
|
||||
# 1. Sign up free at https://resend.com
|
||||
# 2. Go to API Keys and create a new key
|
||||
# 3. Paste it below
|
||||
RESEND_API_KEY=re_your_api_key_here
|
||||
|
||||
# Optional: use a separate key for Contacts/Segments if your main key is send-only
|
||||
RESEND_CONTACTS_API_KEY=
|
||||
|
||||
# Email address that receives contact form submissions
|
||||
RESEND_TO=hello@versebyversewithnate.us
|
||||
|
||||
# Optional: automatically add contact form submitters to a Resend Segment
|
||||
# Create a Segment in Resend and paste its ID here if you want new contacts grouped.
|
||||
RESEND_SEGMENT_ID=
|
||||
|
||||
# "From" address shown on received emails.
|
||||
# During testing you can leave this as-is (uses Resend's shared domain).
|
||||
# For production: verify your own domain at resend.com/domains and change this.
|
||||
RESEND_FROM=Verse by Verse with Nate <hello@versebyversewithnate.us>
|
||||
@@ -17,6 +17,7 @@ dist-ssr
|
||||
# the image default and is preserved here for the Docker build context only.
|
||||
# Uncomment the line below if you do NOT want to track live data in git.
|
||||
# data/admin-content.json
|
||||
data/backups/
|
||||
|
||||
# Editor directories and files
|
||||
.vscode/*
|
||||
|
||||
@@ -50,13 +50,65 @@ docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Admin auth:
|
||||
|
||||
- Set `ADMIN_PASSWORD` on the server/container to protect `/admin` and admin stats/maintenance endpoints.
|
||||
- Without `ADMIN_PASSWORD`, admin login is disabled until configured.
|
||||
|
||||
The app will be available at `http://localhost:4173`.
|
||||
|
||||
## Optional local LLM (Ollama) for grounded rewrites
|
||||
|
||||
You can keep deterministic retrieval as the source-of-truth and optionally rewrite responses with a local model.
|
||||
|
||||
1. Install and run Ollama on your host.
|
||||
2. Pull a small model suited to older hardware, for example:
|
||||
|
||||
```bash
|
||||
ollama pull qwen2.5:3b-instruct
|
||||
```
|
||||
|
||||
3. Start the API with these environment variables:
|
||||
|
||||
```bash
|
||||
CHATBOT_LLM_ENABLED=true
|
||||
CHATBOT_LLM_BASE_URL=http://127.0.0.1:11434
|
||||
CHATBOT_LLM_MODEL=qwen2.5:3b-instruct
|
||||
CHATBOT_LLM_TIMEOUT_MS=25000
|
||||
CHATBOT_LLM_NUM_CTX=2048
|
||||
```
|
||||
|
||||
4. Call the rewrite endpoint from your existing chat flow:
|
||||
|
||||
`POST /api/chatbot-grounded-rewrite`
|
||||
|
||||
Request payload shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"question": "Who was Titus?",
|
||||
"draftAnswer": "Deterministic answer produced by current retrieval/synthesis.",
|
||||
"sources": ["Episode 2 - Introduction to Titus"],
|
||||
"contextChunks": [
|
||||
{
|
||||
"title": "Episode 2 - Introduction to Titus",
|
||||
"sourceLabel": "Episode 2",
|
||||
"content": "Titus was a Gentile..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
If the endpoint fails or is disabled, keep your deterministic answer and existing fallback behavior.
|
||||
|
||||
Persistent admin saves:
|
||||
|
||||
- Admin updates are written to `data/admin-content.json`.
|
||||
- Built-in page hit stats are written to `data/hit-stats.json`.
|
||||
- Detailed visitor analytics are written to `data/visitor-stats.json`.
|
||||
- Backup snapshots are written to `data/backups/`.
|
||||
- `docker-compose.yml` mounts `./data` into the container at `/app/data`.
|
||||
- This keeps your edits after container restarts/rebuilds.
|
||||
- This keeps all admin-managed data after container restarts/rebuilds/updates.
|
||||
|
||||
## Where to edit content
|
||||
|
||||
@@ -80,6 +132,12 @@ Persistent admin saves:
|
||||
- Enter a URL and click **Scan URL metadata** to auto-fill title, summary, domain, category, and icon when available.
|
||||
- Click **Save project** to apply updates instantly.
|
||||
- Saved edits are written to `data/admin-content.json` through the API server.
|
||||
- Built-in stats in `/admin` include page hits plus visitor details (IP, country/state/county/city, returning visitors, and recent visitor log).
|
||||
- Site Stats in `/admin` includes a Bible Questions inbox sourced from contact form submissions marked as Bible Question.
|
||||
- Analytics cookies are consent-based. Visitors can accept or decline tracking from the site banner.
|
||||
- Admin now includes maintenance actions: **Export JSON**, **Backup Now**, **Prune Old Data**, and **Clear Analytics**.
|
||||
- Admin also supports restoring from a backup snapshot from `/admin`.
|
||||
- The server creates startup + daily backup snapshots and retains recent backups automatically.
|
||||
- Use the **Theme** dropdown to switch between Sandstone, Ocean, Midnight, Forest, and Sunset.
|
||||
- Use **Remove project** to delete the selected project from your local Admin data.
|
||||
- Use **Reset project** or **Reset all** to restore defaults from `src/data/projects.ts`.
|
||||
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
BIN
Binary file not shown.
+1
-82
@@ -5,7 +5,7 @@
|
||||
"aboutShowHeading": "Depth. Clarity. Application.",
|
||||
"aboutShowP1": "Verse by Verse with Nate walks through Scripture passage by passage — unpacking the original context, drawing out the meaning, and connecting each verse to how we live today.",
|
||||
"aboutShowP2": "Whether you're in the car, at the gym, or just looking for something to anchor your day, each episode is designed to feed your faith with solid, practical teaching.",
|
||||
"aboutNate": "Nate Emmert is a husband, dad, and lifelong student of the Bible from Nashville, TN. He created Verse by Verse to share the joy of deep Scripture study in a format anyone can follow along with — no seminary required.",
|
||||
"aboutNate": "Nate Emmert is a husband, dad, and lifelong student of the Bible from Lynchburg, Va. He created Verse by Verse to share the joy of deep Scripture study in a format anyone can follow along with — no seminary required.",
|
||||
"seriesLabel": "Now Playing",
|
||||
"seriesTitle": "Study of Titus: Sound Doctrine",
|
||||
"seriesDescription": "A deep-dive into Paul's letter to Titus — unpacking what it means to build a church and a life on sound doctrine.",
|
||||
@@ -17,86 +17,5 @@
|
||||
"customLinks": [],
|
||||
"customBlocks": []
|
||||
},
|
||||
"updatedAt": "2026-04-09T00:00:00.000Z",
|
||||
"_legacy_projects": [
|
||||
{
|
||||
"slug": "verse-by-verse",
|
||||
"title": "Verse by Verse with Nate",
|
||||
"domain": "spotify.com",
|
||||
"url": "https://creators.spotify.com/pod/profile/nmemmert/",
|
||||
"category": "Podcast",
|
||||
"access": "Public",
|
||||
"ownership": "hosted",
|
||||
"status": "Live",
|
||||
"summary": "A verse-by-verse podcast from Nate Emmert focused on Scripture, story, and practical faith application.",
|
||||
"details": "Weekly episodes explore Scripture one verse at a time, mixing theological insight, devotional reflection, and real-life application.",
|
||||
"features": [
|
||||
"Verse-by-verse teaching",
|
||||
"Weekly podcast episodes",
|
||||
"Practical reflection"
|
||||
],
|
||||
"scanSource": "manual",
|
||||
"featured": {
|
||||
"headline": "Podcast teaching shaped around verse-by-verse discovery.",
|
||||
"problem": "Bible study and podcast listening often feel disconnected from each other and from everyday life.",
|
||||
"solution": "Verse by Verse with Nate brings Scripture study and podcast conversation together in a format that is easy to follow and apply.",
|
||||
"stack": [
|
||||
"Podcast hosting",
|
||||
"Spotify",
|
||||
"Verse-by-verse teaching"
|
||||
],
|
||||
"highlights": [
|
||||
"Episode guides",
|
||||
"Scripture focus",
|
||||
"Practical application"
|
||||
],
|
||||
"nextSteps": [
|
||||
"Add episode notes",
|
||||
"Add guest interviews",
|
||||
"Link transcripts and study resources"
|
||||
],
|
||||
"updateNote": "Edit this content in data/admin-content.json for the Verse by Verse podcast project."
|
||||
}
|
||||
}
|
||||
],
|
||||
"siteContent": {
|
||||
"theme": "sand",
|
||||
"homeEyebrow": "Verse by Verse with Nate",
|
||||
"homeTitle": "Podcast episodes, Scripture study, and weekly reflections",
|
||||
"homeIntro": "A podcast experience centered on verse-by-verse conversation, prayerful application, and practical faith rhythms. Explore episodes, show notes, and the latest reflections from Nate.",
|
||||
"projectEyebrow": "Featured Episode",
|
||||
"quickTipsTitle": "Why listeners stay engaged",
|
||||
"quickTipsBody": "Each episode focuses on clear Scripture teaching, practical application, and encouragement you can carry into the week.",
|
||||
"downloadsTitle": "Listen Everywhere",
|
||||
"downloadsIntro": "Use these links to follow the show, share episodes, and stay connected with Verse by Verse with Nate.",
|
||||
"downloads": [
|
||||
{
|
||||
"title": "Spotify Creator Profile",
|
||||
"url": "https://creators.spotify.com/pod/profile/nmemmert/",
|
||||
"description": "Open the official show profile and listen to the latest episodes."
|
||||
}
|
||||
],
|
||||
"showSidebar": false,
|
||||
"showWeatherWidget": false,
|
||||
"showMusicBar": false,
|
||||
"homeSections": [
|
||||
{
|
||||
"id": "summary",
|
||||
"placement": "main"
|
||||
},
|
||||
{
|
||||
"id": "projects",
|
||||
"placement": "main"
|
||||
},
|
||||
{
|
||||
"id": "quickTips",
|
||||
"placement": "main"
|
||||
},
|
||||
{
|
||||
"id": "downloads",
|
||||
"placement": "main"
|
||||
}
|
||||
]
|
||||
},
|
||||
"updatedAt": "2026-04-09T00:00:00.000Z"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,58 @@
|
||||
[
|
||||
{ "query": "who was titus", "expectedEntryId": "cb_who_is_titus_001", "expectedTopK": 3 },
|
||||
{ "query": "who is paul", "expectedEntryId": "881279c7-3ca4-436d-a584-19cc0a5a9bf3", "expectedTopK": 3 },
|
||||
{ "query": "tell me about paul in titus", "expectedEntryId": "881279c7-3ca4-436d-a584-19cc0a5a9bf3", "expectedTopK": 4 },
|
||||
{ "query": "who was saul before his conversion", "expectedEntryId": "167fa083-5438-44a5-ac72-6dd379e937f5", "expectedTopK": 4 },
|
||||
{ "query": "who was titus and why was he in crete", "expectedEntryId": "cb_who_is_titus_001", "expectedTopK": 12 },
|
||||
{ "query": "what bible translation does nate use", "expectedEntryId": "cb_001", "expectedTopK": 1 },
|
||||
{ "query": "does the podcast use bsb", "expectedEntryId": "cb_001", "expectedTopK": 2 },
|
||||
{ "query": "which bible version do you teach from", "expectedEntryId": "cb_001", "expectedTopK": 2 },
|
||||
{ "query": "how can i stay consistent with daily bible reading", "expectedEntryId": "cb_003", "expectedTopK": 2 },
|
||||
{ "query": "help me build a daily reading habit", "expectedEntryId": "cb_003", "expectedTopK": 3 },
|
||||
{ "query": "i keep missing devotion time what should i do", "expectedEntryId": "cb_003", "expectedTopK": 4 },
|
||||
{ "query": "how should christians engage with politics", "expectedEntryId": "cb_christians_politics_001", "expectedTopK": 2 },
|
||||
{ "query": "what does titus say about public life", "expectedEntryId": "cb_christians_politics_001", "expectedTopK": 3 },
|
||||
{ "query": "how can i be peaceable online", "expectedEntryId": "cb_christians_politics_001", "expectedTopK": 4 },
|
||||
{ "query": "give me a summary of titus 3:4-7", "expectedEntryId": "cb_titus347_summary_001", "expectedTopK": 1 },
|
||||
{ "query": "summarize titus 3 4 through 7", "expectedEntryId": "cb_titus347_summary_001", "expectedTopK": 12 },
|
||||
{ "query": "what does titus 3:4-7 teach about salvation", "expectedEntryId": "b68c5659-cc7e-42d2-ab24-a623b7404058", "expectedTopK": 2 },
|
||||
{ "query": "what is the blessed hope", "expectedEntryId": "cb_blessed_hope_001", "expectedTopK": 2 },
|
||||
{ "query": "define blessed hope from titus 2:13", "expectedEntryId": "cb_blessed_hope_001", "expectedTopK": 2 },
|
||||
{ "query": "what does makaria elpis mean", "expectedEntryId": "cb_blessed_hope_001", "expectedTopK": 3 },
|
||||
{ "query": "what does grace train us to do", "expectedEntryId": "cb_grace_trains_001", "expectedTopK": 2 },
|
||||
{ "query": "how does grace teach us to say no to ungodliness", "expectedEntryId": "cb_grace_trains_001", "expectedTopK": 3 },
|
||||
{ "query": "what is paideuo in titus 2", "expectedEntryId": "6be7a644-9276-4797-a7bb-1422b170846c", "expectedTopK": 12 },
|
||||
{ "query": "how do i submit a question to nate", "expectedEntryId": "cb_007", "expectedTopK": 1 },
|
||||
{ "query": "where can i send bible questions", "expectedEntryId": "cb_007", "expectedTopK": 12 },
|
||||
{ "query": "how do i contact nate", "expectedEntryId": "cb_007", "expectedTopK": 2 },
|
||||
{ "query": "what does titus 1 teach about church leadership", "expectedEntryId": "cb_leadership_titus1_001", "expectedTopK": 2 },
|
||||
{ "query": "elder qualifications in titus 1", "expectedEntryId": "cb_leadership_titus1_001", "expectedTopK": 12 },
|
||||
{ "query": "what should an overseer be like", "expectedEntryId": "cb_leadership_titus1_001", "expectedTopK": 3 },
|
||||
{ "query": "what did paul say about elders", "expectedEntryId": "e0ad268f-2826-41a5-b078-90f90aeda21b", "expectedTopK": 4 },
|
||||
{ "query": "episode on faithful leaders", "expectedEntryId": "e0ad268f-2826-41a5-b078-90f90aeda21b", "expectedTopK": 3 },
|
||||
{ "query": "titus 1:5-9 overview", "expectedEntryId": "e0ad268f-2826-41a5-b078-90f90aeda21b", "expectedTopK": 12 },
|
||||
{ "query": "what are false teachers doing in titus", "expectedEntryId": "56626549-7e1e-4564-8c79-2d21b49eb911", "expectedTopK": 3 },
|
||||
{ "query": "what is the circumcision group in titus", "expectedEntryId": "56626549-7e1e-4564-8c79-2d21b49eb911", "expectedTopK": 3 },
|
||||
{ "query": "empty talk and deception in titus 1", "expectedEntryId": "56626549-7e1e-4564-8c79-2d21b49eb911", "expectedTopK": 4 },
|
||||
{ "query": "what does it mean to deny god by your works", "expectedEntryId": "cb_deny_works_001", "expectedTopK": 2 },
|
||||
{ "query": "they profess to know god but deny him by actions", "expectedEntryId": "cb_deny_works_001", "expectedTopK": 3 },
|
||||
{ "query": "detestable disobedient unfit meaning", "expectedEntryId": "cb_deny_works_001", "expectedTopK": 4 },
|
||||
{ "query": "episode about grace training us", "expectedEntryId": "6be7a644-9276-4797-a7bb-1422b170846c", "expectedTopK": 2 },
|
||||
{ "query": "titus 2:11-12 episode", "expectedEntryId": "6be7a644-9276-4797-a7bb-1422b170846c", "expectedTopK": 12 },
|
||||
{ "query": "how should i share faith with skeptical family", "expectedEntryId": "cb_faith_family_001", "expectedTopK": 2 },
|
||||
{ "query": "evangelize skeptical relatives", "expectedEntryId": "cb_faith_family_001", "expectedTopK": 4 },
|
||||
{ "query": "where can i listen to the podcast", "expectedEntryId": "cb_006", "expectedTopK": 2 },
|
||||
{ "query": "how do i find verse by verse with nate on apple podcasts", "expectedEntryId": "cb_006", "expectedTopK": 3 },
|
||||
{ "query": "what is verse by verse with nate podcast", "expectedEntryId": "cb_005", "expectedTopK": 2 },
|
||||
{ "query": "tell me about the podcast", "expectedEntryId": "cb_005", "expectedTopK": 2 },
|
||||
{ "query": "how do i approach a difficult passage", "expectedEntryId": "cb_002", "expectedTopK": 2 },
|
||||
{ "query": "what should i do with confusing bible verses", "expectedEntryId": "cb_002", "expectedTopK": 3 },
|
||||
{ "query": "prayer and quiet time advice", "expectedEntryId": "cb_008", "expectedTopK": 2 },
|
||||
{ "query": "how to start a quiet time", "expectedEntryId": "cb_008", "expectedTopK": 3 },
|
||||
{ "query": "what is episode 9 about", "expectedEntryId": "4e6ec41f-b083-45b9-9581-0dc91f092f66", "expectedTopK": 12 },
|
||||
{ "query": "living between two appearings", "expectedEntryId": "4e6ec41f-b083-45b9-9581-0dc91f092f66", "expectedTopK": 2 },
|
||||
{ "query": "what is episode 12", "expectedEntryId": "b68c5659-cc7e-42d2-ab24-a623b7404058", "expectedTopK": 2 },
|
||||
{ "query": "the gospel in one paragraph", "expectedEntryId": "b68c5659-cc7e-42d2-ab24-a623b7404058", "expectedTopK": 2 },
|
||||
{ "query": "what is episode 14 about", "expectedEntryId": "14ced988-1382-4c8d-a29e-b0f531580af1", "expectedTopK": 3 },
|
||||
{ "query": "grace where it starts and where it ends", "expectedEntryId": "14ced988-1382-4c8d-a29e-b0f531580af1", "expectedTopK": 3 }
|
||||
]
|
||||
@@ -0,0 +1 @@
|
||||
[]
|
||||
+3
-1
@@ -7,6 +7,8 @@ services:
|
||||
- "4173:4173"
|
||||
environment:
|
||||
- PORT=4173
|
||||
- ADMIN_PASSWORD=`generate a random password and set it here`
|
||||
- RESEND_API_KEY=`set your Resend API key here`
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
- /media/ZimaOS-HD/AppData/siteforge/data:/app/data
|
||||
restart: unless-stopped
|
||||
|
||||
+16
-7
@@ -1,17 +1,26 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
DATA_FILE=/app/data/admin-content.json
|
||||
SEED_FILE=/app/data-seed/admin-content.json
|
||||
DATA_DIR=/app/data
|
||||
SEED_DIR=/app/data-seed
|
||||
|
||||
# Ensure the data directory exists (in case the volume was not mounted)
|
||||
mkdir -p /app/data
|
||||
mkdir -p "$DATA_DIR"
|
||||
|
||||
# Only seed from image defaults when no live data file exists yet.
|
||||
# Seed each default JSON file only when missing.
|
||||
# This runs on first boot or a fresh volume, but never overwrites existing data.
|
||||
if [ ! -f "$DATA_FILE" ] && [ -f "$SEED_FILE" ]; then
|
||||
echo "[siteforge] No admin-content.json found. Seeding from image defaults..."
|
||||
cp "$SEED_FILE" "$DATA_FILE"
|
||||
if [ -d "$SEED_DIR" ]; then
|
||||
for seed_file in "$SEED_DIR"/*.json; do
|
||||
[ -f "$seed_file" ] || continue
|
||||
|
||||
base_name=$(basename "$seed_file")
|
||||
target_file="$DATA_DIR/$base_name"
|
||||
|
||||
if [ ! -f "$target_file" ]; then
|
||||
echo "[siteforge] No $base_name found. Seeding from image defaults..."
|
||||
cp "$seed_file" "$target_file"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
exec node server.js
|
||||
|
||||
Generated
+74
-1
@@ -13,7 +13,8 @@
|
||||
"react-dom": "^19.2.4",
|
||||
"react-markdown": "^10.1.0",
|
||||
"react-router-dom": "^7.13.1",
|
||||
"remark-gfm": "^4.0.1"
|
||||
"remark-gfm": "^4.0.1",
|
||||
"resend": "^6.10.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^9.39.4",
|
||||
@@ -863,6 +864,12 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@stablelib/base64": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@stablelib/base64/-/base64-1.0.1.tgz",
|
||||
"integrity": "sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@tybys/wasm-util": {
|
||||
"version": "0.10.1",
|
||||
"resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.1.tgz",
|
||||
@@ -2239,6 +2246,12 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/fast-sha256": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/fast-sha256/-/fast-sha256-1.3.0.tgz",
|
||||
"integrity": "sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==",
|
||||
"license": "Unlicense"
|
||||
},
|
||||
"node_modules/fdir": {
|
||||
"version": "6.5.0",
|
||||
"resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz",
|
||||
@@ -4309,6 +4322,12 @@
|
||||
"url": "https://github.com/sponsors/jonschlinkert"
|
||||
}
|
||||
},
|
||||
"node_modules/postal-mime": {
|
||||
"version": "2.7.4",
|
||||
"resolved": "https://registry.npmjs.org/postal-mime/-/postal-mime-2.7.4.tgz",
|
||||
"integrity": "sha512-0WdnFQYUrPGGTFu1uOqD2s7omwua8xaeYGdO6rb88oD5yJ/4pPHDA4sdWqfD8wQVfCny563n/HQS7zTFft+f/g==",
|
||||
"license": "MIT-0"
|
||||
},
|
||||
"node_modules/postcss": {
|
||||
"version": "8.5.8",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.8.tgz",
|
||||
@@ -4582,6 +4601,27 @@
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/resend": {
|
||||
"version": "6.10.0",
|
||||
"resolved": "https://registry.npmjs.org/resend/-/resend-6.10.0.tgz",
|
||||
"integrity": "sha512-i7CwZpYj4Oho1RxsTpLcCUkO08+HiL4NXrm6jLJ2WzJ89UGI8eROSieLONJA3hnUrf1OYnCyfq5F6POnHUMv1Q==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"postal-mime": "2.7.4",
|
||||
"svix": "1.88.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@react-email/render": "*"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@react-email/render": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/resolve-from": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-4.0.0.tgz",
|
||||
@@ -4866,6 +4906,16 @@
|
||||
"url": "https://github.com/sponsors/wooorm"
|
||||
}
|
||||
},
|
||||
"node_modules/standardwebhooks": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/standardwebhooks/-/standardwebhooks-1.0.0.tgz",
|
||||
"integrity": "sha512-BbHGOQK9olHPMvQNHWul6MYlrRTAOKn03rOe4A8O3CLWhNf4YHBqq2HJKKC+sfqpxiBY52pNeesD6jIiLDz8jg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@stablelib/base64": "^1.0.0",
|
||||
"fast-sha256": "^1.3.0"
|
||||
}
|
||||
},
|
||||
"node_modules/statuses": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
|
||||
@@ -4961,6 +5011,16 @@
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/svix": {
|
||||
"version": "1.88.0",
|
||||
"resolved": "https://registry.npmjs.org/svix/-/svix-1.88.0.tgz",
|
||||
"integrity": "sha512-vm/JrrUd3bVyBE+3L33TIyVSs8gS5fYx7lrISvKlDJXTYX1ACH4REX8P1tHxsSKoZi/rvifM1t0XRc5Vc45THw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"standardwebhooks": "1.0.0",
|
||||
"uuid": "^10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/tinyglobby": {
|
||||
"version": "0.2.15",
|
||||
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.15.tgz",
|
||||
@@ -5246,6 +5306,19 @@
|
||||
"punycode": "^2.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/uuid": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/uuid/-/uuid-10.0.0.tgz",
|
||||
"integrity": "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ==",
|
||||
"funding": [
|
||||
"https://github.com/sponsors/broofa",
|
||||
"https://github.com/sponsors/ctavan"
|
||||
],
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"uuid": "dist/bin/uuid"
|
||||
}
|
||||
},
|
||||
"node_modules/vary": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
|
||||
|
||||
+6
-4
@@ -5,12 +5,13 @@
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"api": "node server.js",
|
||||
"api": "node --env-file=.env server.js",
|
||||
"dev:full": "concurrently \"npm:api\" \"npm:dev\"",
|
||||
"build": "tsc -b && vite build",
|
||||
"start": "node server.js",
|
||||
"start": "node --env-file=.env server.js",
|
||||
"lint": "eslint .",
|
||||
"preview": "vite preview"
|
||||
"preview": "vite preview",
|
||||
"chatbot:eval": "node scripts/evaluate-chatbot.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"express": "^5.2.1",
|
||||
@@ -18,7 +19,8 @@
|
||||
"react-dom": "^19.2.4",
|
||||
"react-markdown": "^10.1.0",
|
||||
"react-router-dom": "^7.13.1",
|
||||
"remark-gfm": "^4.0.1"
|
||||
"remark-gfm": "^4.0.1",
|
||||
"resend": "^6.10.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^9.39.4",
|
||||
|
||||
+19
-1
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 9.3 KiB After Width: | Height: | Size: 1.4 KiB |
@@ -0,0 +1,92 @@
|
||||
import { execSync } from 'child_process'
|
||||
import { readFileSync, writeFileSync } from 'fs'
|
||||
import { randomUUID } from 'crypto'
|
||||
|
||||
const BASE = "/Users/nate.emmert/Documents/github/Siteforge/Verse by Verse with Nate Complete Series"
|
||||
const CHATBOT_FILE = "/Users/nate.emmert/Documents/github/Siteforge/data/chatbot-content.json"
|
||||
|
||||
const FILES = [
|
||||
{ file: `${BASE}/Done/Verse_by_Verse_with_Nate_Episode02.docx`, ep: 2 },
|
||||
{ file: `${BASE}/Done/Verse_by_Verse_with_Nate_Episode03.docx`, ep: 3 },
|
||||
{ file: `${BASE}/Done/Verse_by_Verse_with_Nate_Episode04_Updated.docx`, ep: 4 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode05.docx`, ep: 5 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode06_expanded.docx`, ep: 6 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode07.docx`, ep: 7 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode08.docx`, ep: 8 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode09.docx`, ep: 9 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode10.docx`, ep: 10 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode11.docx`, ep: 11 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode12.docx`, ep: 12 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode13.docx`, ep: 13 },
|
||||
{ file: `${BASE}/Verse_by_Verse_with_Nate_Episode14.docx`, ep: 14 },
|
||||
]
|
||||
|
||||
function extractText(filePath) {
|
||||
const xml = execSync(`unzip -p "${filePath}" word/document.xml 2>/dev/null`, { encoding: 'utf8' })
|
||||
return xml
|
||||
.replace(/<[^>]+>/g, ' ')
|
||||
.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"').replace(/'/g, "'")
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
}
|
||||
|
||||
function parseEpisode(raw, epNum) {
|
||||
// Extract episode subtitle and passage reference from header
|
||||
const headerMatch = raw.match(/EPISODE\s+\d+\s*[—\-\u2013\u2014]+\s*(.+?)\s+(Titus\s+[\d:]+(?:\s*[\-\u2013\u2014]+\s*[\d:]+)?)\s*·/i)
|
||||
const subtitle = headerMatch ? headerMatch[1].trim().replace(/\s+/g, ' ') : ''
|
||||
const passageRef = headerMatch ? headerMatch[2].trim() : 'Titus'
|
||||
const episodeTitle = `Episode ${epNum} — ${subtitle || 'Verse by Verse with Nate'}`
|
||||
|
||||
// Find where the actual teaching content starts
|
||||
let contentStart = raw.indexOf('SEGMENT 1')
|
||||
if (contentStart === -1) contentStart = raw.indexOf('WHO WAS PAUL')
|
||||
if (contentStart === -1) contentStart = raw.indexOf('COLD OPEN')
|
||||
if (contentStart === -1) contentStart = 400
|
||||
|
||||
const rawContent = raw.slice(contentStart, contentStart + 4000)
|
||||
const content = rawContent
|
||||
.replace(/\[[^\]]{0,100}\]/g, '') // remove [stage directions]
|
||||
.replace(/[✝🎙️📖💬🧠💡🔑✅◀▶]/gu, '') // remove emoji
|
||||
.replace(/SEGMENT\s+\d+\s*[—\-\u2013]+\s*/g, '\n\n') // turn SEGMENT headers into breaks
|
||||
.replace(/\s{2,}/g, ' ')
|
||||
.trim()
|
||||
|
||||
// Build keyword list
|
||||
const verseRefs = [...new Set((raw.match(/Titus\s+\d+:\d+/g) || []))].slice(0, 5).map(k => k.toLowerCase())
|
||||
const titleWords = subtitle.toLowerCase().replace(/[^a-z0-9\s]/g, ' ').split(/\s+/).filter(w => w.length > 3)
|
||||
const keywords = [...new Set([
|
||||
'titus', `episode ${epNum}`, passageRef.toLowerCase(),
|
||||
...verseRefs, ...titleWords
|
||||
])].slice(0, 20)
|
||||
|
||||
return {
|
||||
id: randomUUID(),
|
||||
type: 'episode',
|
||||
title: `${episodeTitle} (${passageRef})`,
|
||||
content: content.slice(0, 3900),
|
||||
keywords,
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
}
|
||||
}
|
||||
|
||||
// Load existing entries (keep the 8 hand-written ones)
|
||||
const existing = JSON.parse(readFileSync(CHATBOT_FILE, 'utf8'))
|
||||
// Remove any previously generated episode entries to avoid duplication
|
||||
const baseEntries = existing.filter(e => e.type !== 'episode')
|
||||
|
||||
const newEntries = []
|
||||
for (const { file, ep } of FILES) {
|
||||
try {
|
||||
const raw = extractText(file)
|
||||
const entry = parseEpisode(raw, ep)
|
||||
newEntries.push(entry)
|
||||
console.log(`✓ Ep ${ep}: ${entry.title.slice(0, 80)}`)
|
||||
} catch (err) {
|
||||
console.error(`✗ Ep ${ep}: ${err.message}`)
|
||||
}
|
||||
}
|
||||
|
||||
const combined = [...baseEntries, ...newEntries]
|
||||
writeFileSync(CHATBOT_FILE, JSON.stringify(combined, null, 2), 'utf8')
|
||||
console.log(`\nDone. ${newEntries.length} episode entries added. Total: ${combined.length}`)
|
||||
@@ -0,0 +1,137 @@
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
|
||||
const root = process.cwd()
|
||||
const dataPath = path.join(root, 'data', 'chatbot-content.json')
|
||||
const evalPath = path.join(root, 'data', 'chatbot-eval.json')
|
||||
|
||||
const STOP_WORDS = new Set([
|
||||
'a','an','the','is','are','was','were','be','been','being','have','has','had','do','does','did',
|
||||
'will','would','could','should','may','might','shall','can','i','you','he','she','it','we','they',
|
||||
'me','him','her','us','them','my','your','his','its','our','their','this','that','these','those',
|
||||
'and','but','or','nor','so','yet','for','of','in','on','at','to','from','with','by','about',
|
||||
'what','how','why','when','where','who','which','if','then','than','as','just','not',
|
||||
])
|
||||
|
||||
const TOKEN_ALIASES = {
|
||||
bible: ['translation', 'version', 'scripture', 'bsb', 'berean'],
|
||||
translation: ['version', 'bsb', 'berean', 'bible'],
|
||||
version: ['translation', 'bsb', 'berean', 'bible'],
|
||||
elders: ['elder', 'leadership', 'leaders', 'overseer', 'pastor'],
|
||||
leadership: ['elders', 'elder', 'overseer', 'leaders'],
|
||||
grace: ['salvation', 'saved', 'godliness', 'mercy'],
|
||||
salvation: ['saved', 'grace', 'mercy', 'gospel'],
|
||||
saved: ['salvation', 'grace', 'mercy', 'gospel'],
|
||||
hope: ['blessed', 'appearing', 'return', 'coming'],
|
||||
politics: ['public', 'government', 'authorities'],
|
||||
}
|
||||
|
||||
function tokenize(text) {
|
||||
return text
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9\s]/g, ' ')
|
||||
.split(/\s+/)
|
||||
.filter(token => token.length > 2 && !STOP_WORDS.has(token))
|
||||
}
|
||||
|
||||
function expandTokens(tokens) {
|
||||
const expanded = new Set(tokens)
|
||||
for (const token of tokens) {
|
||||
const aliases = TOKEN_ALIASES[token] ?? []
|
||||
for (const alias of aliases) {
|
||||
for (const aliasToken of tokenize(alias)) expanded.add(aliasToken)
|
||||
}
|
||||
}
|
||||
return [...expanded]
|
||||
}
|
||||
|
||||
function literalTerms(text) {
|
||||
return [...new Set(
|
||||
text
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9:\-\s']/g, ' ')
|
||||
.split(/\s+/)
|
||||
.map(term => term.trim())
|
||||
.filter(term => term.length >= 2 && !STOP_WORDS.has(term)),
|
||||
)]
|
||||
}
|
||||
|
||||
function scoreEntry(entry, query) {
|
||||
const indexText = `${entry.title} ${entry.content} ${entry.keywords.join(' ')}`.toLowerCase()
|
||||
const queryTokens = expandTokens(tokenize(query))
|
||||
const terms = literalTerms(query)
|
||||
|
||||
let score = 0
|
||||
|
||||
for (const token of queryTokens) {
|
||||
if (entry.title.toLowerCase().includes(token)) score += 4
|
||||
else if (indexText.includes(token)) score += 2
|
||||
}
|
||||
|
||||
for (const term of terms) {
|
||||
if (entry.title.toLowerCase().includes(term)) score += 2
|
||||
else if (indexText.includes(term)) score += 1
|
||||
}
|
||||
|
||||
const phrase = query.toLowerCase().replace(/[^a-z0-9:\-\s']/g, ' ').replace(/\s+/g, ' ').trim()
|
||||
if (phrase.length > 6) {
|
||||
if (entry.title.toLowerCase().includes(phrase)) score += 10
|
||||
else if (indexText.includes(phrase)) score += 6
|
||||
}
|
||||
|
||||
return score
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const [contentRaw, evalRaw] = await Promise.all([
|
||||
fs.readFile(dataPath, 'utf8'),
|
||||
fs.readFile(evalPath, 'utf8'),
|
||||
])
|
||||
|
||||
const entries = JSON.parse(contentRaw)
|
||||
const tests = JSON.parse(evalRaw)
|
||||
|
||||
let pass = 0
|
||||
const failures = []
|
||||
|
||||
for (const test of tests) {
|
||||
const ranked = entries
|
||||
.map(entry => ({ entry, score: scoreEntry(entry, test.query) }))
|
||||
.sort((a, b) => b.score - a.score)
|
||||
|
||||
const topK = ranked.slice(0, test.expectedTopK)
|
||||
const hit = topK.some(item => item.entry.id === test.expectedEntryId)
|
||||
|
||||
if (hit) {
|
||||
pass += 1
|
||||
continue
|
||||
}
|
||||
|
||||
failures.push({
|
||||
query: test.query,
|
||||
expectedEntryId: test.expectedEntryId,
|
||||
expectedTopK: test.expectedTopK,
|
||||
actualTop: topK.map(item => ({ id: item.entry.id, title: item.entry.title, score: item.score })),
|
||||
})
|
||||
}
|
||||
|
||||
const total = tests.length
|
||||
const pct = ((pass / total) * 100).toFixed(1)
|
||||
|
||||
console.log(`Chatbot eval: ${pass}/${total} (${pct}%) passed`)
|
||||
|
||||
if (failures.length > 0) {
|
||||
console.log('\nFailures:')
|
||||
for (const failure of failures) {
|
||||
console.log(`- Query: ${failure.query}`)
|
||||
console.log(` Expected: ${failure.expectedEntryId} in top ${failure.expectedTopK}`)
|
||||
console.log(` Actual: ${failure.actualTop.map(item => `${item.id} (${item.score})`).join(', ')}`)
|
||||
}
|
||||
process.exitCode = 1
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(error => {
|
||||
console.error(error)
|
||||
process.exitCode = 1
|
||||
})
|
||||
@@ -0,0 +1,26 @@
|
||||
import { readFileSync, writeFileSync } from 'fs'
|
||||
|
||||
const path = '/Users/nate.emmert/Documents/github/Siteforge/data/chatbot-content.json'
|
||||
const data = JSON.parse(readFileSync(path, 'utf8'))
|
||||
|
||||
const fixes = {
|
||||
2: { title: 'Episode 2 — Introduction to Titus (Background & Overview)', extra: ['introduction', 'background', 'overview', 'crete', 'letter'] },
|
||||
5: { title: 'Episode 5 — The Danger of Empty Words (Titus 1:10–13a)', extra: ['danger', 'empty', 'words', 'false', 'teacher', 'titus 1:10', 'titus 1:13'] },
|
||||
6: { title: 'Episode 6 — Words That Deny What We Claim to Believe (Titus 1:13b–16)', extra: ['deny', 'claim', 'believe', 'titus 1:13', 'titus 1:16'] },
|
||||
14: { title: 'Episode 14 — Grace: Where It Starts and Where It Ends (Titus 3:12–15)', extra: ['grace', 'starts', 'ends', 'review', 'titus 3:12', 'titus 3:15'] },
|
||||
}
|
||||
|
||||
let count = 0
|
||||
for (const entry of data) {
|
||||
const m = entry.title.match(/^Episode (\d+)/)
|
||||
if (!m) continue
|
||||
const ep = Number(m[1])
|
||||
if (fixes[ep]) {
|
||||
entry.title = fixes[ep].title
|
||||
entry.keywords = [...new Set([...entry.keywords, ...fixes[ep].extra])].slice(0, 20)
|
||||
count++
|
||||
}
|
||||
}
|
||||
|
||||
writeFileSync(path, JSON.stringify(data, null, 2), 'utf8')
|
||||
console.log(`Fixed ${count} entries. Total: ${data.length}`)
|
||||
@@ -0,0 +1,194 @@
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { promises as fs } from 'node:fs'
|
||||
import path from 'node:path'
|
||||
|
||||
const ROOT = '/Users/nate.emmert/Documents/github/Siteforge'
|
||||
const DOCS_DIR = path.join(ROOT, 'Verse by Verse with Nate Complete Series')
|
||||
const CHATBOT_FILE = path.join(ROOT, 'data', 'chatbot-content.json')
|
||||
|
||||
const STOP_WORDS = new Set([
|
||||
'the', 'and', 'for', 'that', 'with', 'this', 'from', 'your', 'you', 'are', 'but', 'not', 'have',
|
||||
'has', 'was', 'were', 'his', 'her', 'our', 'their', 'into', 'about', 'what', 'when', 'where',
|
||||
'which', 'will', 'just', 'they', 'them', 'then', 'than', 'how', 'why', 'can', 'all', 'through',
|
||||
])
|
||||
|
||||
function parseEpisodeNumber(filePath) {
|
||||
const match = path.basename(filePath).match(/Episode(\d+)/i)
|
||||
return match ? Number(match[1]) : null
|
||||
}
|
||||
|
||||
function getVariantRank(filePath) {
|
||||
const name = path.basename(filePath).toLowerCase()
|
||||
let score = 0
|
||||
if (name.includes('expanded')) score += 30
|
||||
if (name.includes('updated')) score += 20
|
||||
if (!name.includes('expanded') && !name.includes('updated')) score += 10
|
||||
if (filePath.includes(`${path.sep}Done${path.sep}Old${path.sep}`)) score -= 25
|
||||
return score
|
||||
}
|
||||
|
||||
async function collectDocxFiles(dir) {
|
||||
const out = []
|
||||
const items = await fs.readdir(dir, { withFileTypes: true })
|
||||
for (const item of items) {
|
||||
const fullPath = path.join(dir, item.name)
|
||||
if (item.isDirectory()) {
|
||||
out.push(...await collectDocxFiles(fullPath))
|
||||
continue
|
||||
}
|
||||
if (!item.isFile()) continue
|
||||
if (!item.name.toLowerCase().endsWith('.docx')) continue
|
||||
if (item.name.startsWith('~$')) continue
|
||||
out.push(fullPath)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
function pickBestPerEpisode(docxFiles) {
|
||||
const byEpisode = new Map()
|
||||
|
||||
for (const filePath of docxFiles) {
|
||||
const episode = parseEpisodeNumber(filePath)
|
||||
if (!episode) continue
|
||||
|
||||
const current = byEpisode.get(episode)
|
||||
const next = {
|
||||
filePath,
|
||||
episode,
|
||||
rank: getVariantRank(filePath),
|
||||
}
|
||||
|
||||
if (!current || next.rank > current.rank) {
|
||||
byEpisode.set(episode, next)
|
||||
}
|
||||
}
|
||||
|
||||
return [...byEpisode.values()].sort((a, b) => a.episode - b.episode)
|
||||
}
|
||||
|
||||
function extractDocText(filePath) {
|
||||
const output = execFileSync('textutil', ['-convert', 'txt', '-stdout', filePath], { encoding: 'utf8' })
|
||||
return output
|
||||
}
|
||||
|
||||
function normalizeContent(text) {
|
||||
const lines = text
|
||||
.split(/\r?\n/)
|
||||
.map(line => line.replace(/\s+/g, ' ').trim())
|
||||
.filter(Boolean)
|
||||
|
||||
const filtered = lines.filter(line => {
|
||||
const upper = line.toUpperCase()
|
||||
if (upper === 'VERSE BY VERSE WITH NATE') return false
|
||||
if (upper === 'A JOURNEY THROUGH SCRIPTURE') return false
|
||||
return true
|
||||
})
|
||||
|
||||
return filtered.join(' ').replace(/\s{2,}/g, ' ').trim()
|
||||
}
|
||||
|
||||
function buildKeywords(title, content, existingKeywords = []) {
|
||||
const tokens = `${title} ${content.slice(0, 1600)}`
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9\s:-]/g, ' ')
|
||||
.split(/\s+/)
|
||||
.filter(token => token.length >= 3 && !STOP_WORDS.has(token))
|
||||
|
||||
const counts = new Map()
|
||||
for (const token of tokens) {
|
||||
counts.set(token, (counts.get(token) ?? 0) + 1)
|
||||
}
|
||||
|
||||
const top = [...counts.entries()]
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.slice(0, 20)
|
||||
.map(([token]) => token)
|
||||
|
||||
return [...new Set([...(existingKeywords ?? []), ...top])].slice(0, 25)
|
||||
}
|
||||
|
||||
function getEpisodeFromTitle(title = '') {
|
||||
const match = title.match(/Episode\s+(\d+)/i)
|
||||
return match ? Number(match[1]) : null
|
||||
}
|
||||
|
||||
function getEntryTitleFallback(episodeNumber, rawText, existingTitle) {
|
||||
if (existingTitle && existingTitle.trim()) return existingTitle
|
||||
|
||||
const lineMatch = rawText.match(new RegExp(`EPISODE\\s+${episodeNumber}\\s*[—-]\\s*([^\\n]+)`, 'i'))
|
||||
if (lineMatch) {
|
||||
return `Episode ${episodeNumber} — ${lineMatch[1].trim()}`
|
||||
}
|
||||
return `Episode ${episodeNumber}`
|
||||
}
|
||||
|
||||
async function run() {
|
||||
const raw = await fs.readFile(CHATBOT_FILE, 'utf8')
|
||||
const entries = JSON.parse(raw)
|
||||
|
||||
const docxFiles = await collectDocxFiles(DOCS_DIR)
|
||||
const selected = pickBestPerEpisode(docxFiles)
|
||||
|
||||
const existingByEpisode = new Map()
|
||||
for (const entry of entries) {
|
||||
const episode = getEpisodeFromTitle(entry.title)
|
||||
if (episode) existingByEpisode.set(episode, entry)
|
||||
}
|
||||
|
||||
const now = new Date().toISOString()
|
||||
let updated = 0
|
||||
let added = 0
|
||||
|
||||
for (const item of selected) {
|
||||
const rawText = extractDocText(item.filePath)
|
||||
const content = normalizeContent(rawText)
|
||||
if (!content) continue
|
||||
|
||||
const existing = existingByEpisode.get(item.episode)
|
||||
|
||||
if (existing) {
|
||||
existing.type = 'episode'
|
||||
existing.title = getEntryTitleFallback(item.episode, rawText, existing.title)
|
||||
existing.content = content
|
||||
existing.keywords = buildKeywords(existing.title, content, existing.keywords)
|
||||
existing.updatedAt = now
|
||||
updated += 1
|
||||
continue
|
||||
}
|
||||
|
||||
entries.push({
|
||||
id: randomUUID(),
|
||||
type: 'episode',
|
||||
title: getEntryTitleFallback(item.episode, rawText, ''),
|
||||
content,
|
||||
keywords: buildKeywords(`Episode ${item.episode}`, content, []),
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
added += 1
|
||||
}
|
||||
|
||||
entries.sort((a, b) => {
|
||||
const aEp = getEpisodeFromTitle(a.title)
|
||||
const bEp = getEpisodeFromTitle(b.title)
|
||||
if (aEp && bEp) return aEp - bEp
|
||||
if (aEp && !bEp) return 1
|
||||
if (!aEp && bEp) return -1
|
||||
return 0
|
||||
})
|
||||
|
||||
await fs.writeFile(CHATBOT_FILE, `${JSON.stringify(entries, null, 2)}\n`)
|
||||
|
||||
console.log(`Episodes selected from docs: ${selected.length}`)
|
||||
console.log(`Updated entries: ${updated}`)
|
||||
console.log(`Added entries: ${added}`)
|
||||
for (const item of selected) {
|
||||
console.log(`- Episode ${item.episode}: ${path.relative(ROOT, item.filePath)}`)
|
||||
}
|
||||
}
|
||||
|
||||
run().catch(error => {
|
||||
console.error(error)
|
||||
process.exitCode = 1
|
||||
})
|
||||
+1195
-4
File diff suppressed because it is too large
Load Diff
+1519
-1
File diff suppressed because it is too large
Load Diff
+2174
-6
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user