Podcast: Verse By Verse Ministry studies (browse, play lesson after lesson, handouts, synced place)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaC9HmtHMdRP8dAbN4wAKj

README.md

@@ -63,6 +63,7 @@ e.g. hub.necloud.us. Settings: `SEAFILE_URL`, `ONLYOFFICE_URL` env vars in docke
   - The reader scripts are in `public/vendor/reader/` (epub.js, JSZip, pdf.js, mammoth, the MOBI parser) and load only when a book is opened or added. `node tools/books-test.js` tests the server; `python3 tools/mkbooks.py <dir>` makes a small EPUB, PDF and MOBI to try. **DRM-protected books are refused** (a Kindle file with copy protection is detected on upload and the reason is shown); only DRM-free books work. The MOBI reader was tried on a generated MOBI 6 file, not on real Kindle files, so tell me about any book that opens oddly. Not included: text-to-speech, offline reading.
   **Free audiobooks** (Audiobooks -> Free audiobooks; works without any audiobook sign-in): search **LibriVox**, thousands of public-domain books read aloud by volunteers, by title or author. A book shows its length, language, description and chapters; *Play* (or *Resume*) queues every chapter in the same player as Music so the next one follows by itself, and starting a book puts it on your shelf (*My free audiobooks*, with a progress bar; *Remove from shelf* in the book's sheet). Where you got to is kept on the server per person, so it follows you between devices. The hub searches LibriVox and hands the browser each chapter's address, but only addresses on LibriVox or the Internet Archive, which hosts the recordings; covers come through the hub. `LIBRIVOX_URL` points at a mirror. `node tools/librivox-test.js` tests it against a fake LibriVox; it has not been run against the live site from the build environment.
 - **Radio**: internet radio, with **The Journey FM** (Liberty University, Lynchburg, Virginia; MP3 192 kbps, `http://ic.liberty.edu:8000/WRVL`) always on top, showing what is playing now and the last songs played from the station's own feed. Below it, search thousands of stations from the community [Radio Browser](https://www.radio-browser.info/) directory (free, no key) by name or with quick topic chips (Christian, Worship, Gospel, Talk, News, Classical, Jazz...). The heart saves a station to *My stations* (your own list). Stations play in the same player as Music, so they keep playing as you move between tabs. The hub fetches each stream itself and passes it on, because many stations only offer plain http, which a browser refuses to play inside an https page; it also reads the song title carried in the stream (the player shows it) and fetches station logos. The hub never connects to private, local or link-local addresses for a station. Give it to people in Settings -> People (Radio). `JOURNEYFM_STREAM_URL`, `JOURNEYFM_NOW_URL` and `RADIO_BROWSER_URL` can override the addresses. `node tools/radio-test.js` tests it against fake stations; the real Journey stream could not be reached from the build environment, so if it will not play, open the station's own *Website* button and tell me what changed. Not included: HLS-only stations, recording, alarms.
+- **Verse By Verse** (Podcast -> *Verse By Verse*; needs only the Podcast tab, not Podgrab): free verse-by-verse Bible teaching from [Verse By Verse Ministry International](https://versebyverseministry.org). Browse the ministry's studies (about 80: Old and New Testament books, topical series, single teachings, women's and youth), search them, and open a study to see its lessons with dates and summaries. *Start* / *Continue* plays the study lesson after lesson in the same player as Music, a lesson resumes where you left off (kept on the server per person, so it follows you between devices), and each lesson's handout (PDF) opens from *Notes*. The hub reads the study list from the ministry's category pages and the lessons from each study's own podcast feed (`/bible-studies/<study>/podcast.xml`), keeps what it read for 30 to 60 minutes so the ministry's site is not asked on every visit, and sends the browser straight to the ministry's media server for the audio. Only the ministry's own addresses are ever used. `VBV_URL` can point at a mirror. `node tools/vbv-test.js` tests it against a fake copy of the site; it was also tried against the live site (80 studies, a study's lessons, audio and handout links). If the ministry redesigns its pages, the study list is the first thing that would break, and the screen would say so.
 - **Signal**: your Signal chats inside the app, **on desktop-class screens only** (a window at least 900 px wide with a mouse or trackpad; phones and touch tablets don't
   see it). Besides the Signal tab there is a chat button on **every** screen that opens a panel over whatever you are doing (close it with the x; the
   expand button opens the full page). New messages get your attention with a toast, a chime, the unread count in the browser tab title and on the installed app's icon, and

package.json

@@ -5,7 +5,7 @@
   "main": "server.js",
   "scripts": {
     "start": "node server.js",
-    "test": "for t in smoke tasks push users cal plex player lyrics signal mail mail-imap files podcast abapi abdirect books store librivox radio; do node tools/$t-test.js || exit 1; done",
+    "test": "for t in smoke tasks push users cal plex player lyrics signal mail mail-imap files podcast abapi abdirect books store librivox radio vbv; do node tools/$t-test.js || exit 1; done",
     "audit": "npm audit --omit=dev --audit-level=high"
   },
   "keywords": [],

public/app.js

@@ -200,7 +200,7 @@ function openMore() {
   el.querySelectorAll('a.row').forEach(a => a.addEventListener('click', closeSheet));
 }
 
-const routesOf = () => ({ video: viewVideo, planner: viewPlanner, home: viewHome, mail: viewMail, files: viewFiles, office: viewOffice, audio: args => args[0] === 'classic' ? viewAudio() : args[0] === 'free' ? viewFreeBooks() : viewBooks(args), verse: () => location.replace('#/links'), podcast: args => viewPodcast(args), radio: viewRadio, read: args => viewReader(args), music: viewMusic, tasks: viewTasks, search: viewSearch, calendar: viewCalendar, signal: viewSignal, links: viewLinks, settings: viewSettings, edit: viewEdit });
+const routesOf = () => ({ video: viewVideo, planner: viewPlanner, home: viewHome, mail: viewMail, files: viewFiles, office: viewOffice, audio: args => args[0] === 'classic' ? viewAudio() : args[0] === 'free' ? viewFreeBooks() : viewBooks(args), verse: () => location.replace('#/links'), podcast: args => args[0] === 'vbv' ? viewVbv(args.slice(1)) : viewPodcast(args), radio: viewRadio, read: args => viewReader(args), music: viewMusic, tasks: viewTasks, search: viewSearch, calendar: viewCalendar, signal: viewSignal, links: viewLinks, settings: viewSettings, edit: viewEdit });
 let editor;
 // Menu on a tablet or computer: collapse it to icons (wide screens), or hide it completely. Remembered on this device.
 const navPref = () => { try { return JSON.parse(localStorage.getItem('nav_pref')) || {}; } catch { return {}; } };

public/index.html

@@ -75,6 +75,7 @@
 <script src="/audiobooks-view.js"></script>
 <script src="/librivox-view.js"></script>
 <script src="/radio-view.js"></script>
+<script src="/vbv-view.js"></script>
 <script src="/lyrics-view.js"></script>
 <script src="/settings.js"></script>
 <script src="/files.js"></script>

public/music.js

@@ -13,7 +13,7 @@ const px = {
     if (!r.ok) throw new Error('Plex error ' + r.status);
     return (await r.json()).MediaContainer || {};
   },
-  art: (thumb, w = 300) => thumb && /^\/api\/(podcast|ab|radio)\//.test(thumb) ? thumb : thumb ? `/plex/photo/:/transcode?width=${w}&height=${w}&minSize=1&upscale=1&url=${encodeURIComponent(thumb)}` : 'data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==',
+  art: (thumb, w = 300) => thumb && /^\/api\/(podcast|ab|radio|vbv)\//.test(thumb) ? thumb : thumb ? `/plex/photo/:/transcode?width=${w}&height=${w}&minSize=1&upscale=1&url=${encodeURIComponent(thumb)}` : 'data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==',
   // All music libraries on the server; the chosen one is remembered per server.
   async sections() {
     if (px.secs) return px.secs;

public/player.js

@@ -143,7 +143,7 @@ const toggle = () => { if (pl.a.paused && pl.a.getAttribute('src') == null && cu
 pl.a.preload = 'auto';
 // Tell Plex what is playing, so it shows under Now Playing / Activity in the Plex dashboard (as "NE Hub", under the signed-in Plex account).
 let tlKey = null, tlAt = 0, tlPos = 0, tlDur = 0;
-const tlSend = (state, key, pos, dur) => /^(pod|radio|lv):/.test(String(key)) ? 0 : fetch('/plex/:/timeline?' + new URLSearchParams({ ratingKey: key, key: '/library/metadata/' + key, identifier: 'com.plexapp.plugins.library', state, time: Math.round(pos), duration: Math.round(dur), hasMDE: 1 })).catch(() => {});
+const tlSend = (state, key, pos, dur) => /^(pod|radio|lv|vbv):/.test(String(key)) ? 0 : fetch('/plex/:/timeline?' + new URLSearchParams({ ratingKey: key, key: '/library/metadata/' + key, identifier: 'com.plexapp.plugins.library', state, time: Math.round(pos), duration: Math.round(dur), hasMDE: 1 })).catch(() => {});
 function tlReport(state) {
   const t = cur(); if (!t) return;
   if (tlKey && tlKey !== t.key) tlSend('stopped', tlKey, tlPos, tlDur); // the previous song is over

public/podcast-view.js

@@ -40,7 +40,7 @@ async function viewPodcast(args = []) {
     return await pdHome();
   } catch (e) {
     if (!location.hash.startsWith('#/podcast')) return;
-    v.innerHTML = `<div class="wrap"><h1>Podcast</h1><p class="err">${esc(e.message)}</p><div class="bar"><button class="btn ghost" id="pdretry">Try again</button>${cfg.podcast ? `<a class="btn ghost" href="${esc(cfg.podcast)}" target="_blank" rel="noopener">Open Podgrab</a>` : ''}</div></div>`;
+    v.innerHTML = `<div class="wrap"><h1>Podcast</h1><p class="err">${esc(e.message)}</p><div class="bar"><button class="btn ghost" id="pdretry">Try again</button><a class="btn ghost" href="#/podcast/vbv">Verse By Verse</a>${cfg.podcast ? `<a class="btn ghost" href="${esc(cfg.podcast)}" target="_blank" rel="noopener">Open Podgrab</a>` : ''}</div></div>`;
     $('#pdretry').onclick = () => route();
   }
 }
@@ -48,7 +48,7 @@ async function viewPodcast(args = []) {
 async function pdHome() {
   const list = await pdApi('/podcasts?sort=lastepisode&order=desc'), shows = Array.isArray(list) ? list : (list.podcasts || []);
   if (!location.hash.startsWith('#/podcast')) return;
-  $('#view').innerHTML = `<div class="wrap"><div class="ptop"><h1>Podcast</h1><span class="pspace grow"></span><button class="btn" id="pdadd">${icon('plus')}Add podcast</button></div>
+  $('#view').innerHTML = `<div class="wrap"><div class="ptop"><h1>Podcast</h1><span class="pspace grow"></span><a class="btn ghost" href="#/podcast/vbv">Verse By Verse</a><button class="btn" id="pdadd">${icon('plus')}Add podcast</button></div>
     ${shows.length ? `<div class="vgrid">${shows.map(s => `<a class="vcard" href="#/podcast/show/${encodeURIComponent(pdId(s))}">
       <span class="vimg" style="aspect-ratio:1/1"><img loading="lazy" alt="" src="${pdImg('podcasts', s)}" onerror="this.remove()"></span>
       <b>${esc(pdF(s, 'title'))}</b><small>${esc([pdF(s, 'allEpisodesCount') ? pdF(s, 'allEpisodesCount') + ' episodes' : '', pdF(s, 'isPaused') ? 'paused' : ''].filter(Boolean).join(' · '))}</small></a>`).join('')}</div>`

public/sw.js

@@ -1,5 +1,5 @@
 // App-shell cache only; Seafile/API traffic always goes to the network.
-const CACHE = 'hub-v40';
+const CACHE = 'hub-v41';
 const SHARE = 'hub-share'; // files shared into the app from other apps, held until the person picks a folder
 const SHELL = ['/icon.svg', '/manifest.webmanifest', '/icons/icon-192.png', '/icons/apple-touch-icon.png'];
 

public/vbv-view.js

@@ -0,0 +1,67 @@
+'use strict';
+// ---- Verse By Verse Ministry (Podcast -> Verse By Verse): free verse-by-verse Bible studies from versebyverseministry.org. Pick a study, play a lesson (the rest of the
+// study follows), open its handout. Where you got to in each lesson is kept on the server, so it follows you between devices.
+const vbApi = (p, opts) => jfetch('/api/vbv' + p, opts);
+const vbOpt = (m, o) => ({ method: m, headers: { 'content-type': 'application/json' }, body: JSON.stringify(o) });
+let vbProg = {}, vbQueued = {}, vbLast = 0;
+const vbPlaying = () => { const t = cur(), m = t && /^vbv:([\w-]+):(\d+)$/.exec(String(t.key)); return m ? { slug: m[1], n: +m[2], key: t.key } : null; };
+const vbLoadProg = async () => { try { for (const [k, p] of Object.entries(await vbApi('/progress') || {})) if (!vbProg[k] || p.t > vbProg[k].t) vbProg[k] = p; } catch {} };
+const vbPush = keepalive => {
+  if (!Object.keys(vbQueued).length) return;
+  const body = JSON.stringify(vbQueued); vbQueued = {};
+  fetch('/api/vbv/progress', { method: 'POST', headers: { 'content-type': 'application/json' }, body, keepalive: !!keepalive }).then(r => { if (!r.ok) throw 0; }).catch(() => { vbQueued = { ...JSON.parse(body), ...vbQueued }; });
+};
+const vbSave = keepalive => { const n = vbPlaying(), d = pl.a.duration; if (!n || !isFinite(d) || d <= 0) return; vbProg[n.key] = vbQueued[n.key] = { pos: pl.a.currentTime, dur: d, t: Date.now() }; vbPush(keepalive); };
+pl.a.addEventListener('timeupdate', () => { if (Date.now() - vbLast > 8000) { vbLast = Date.now(); vbSave(); } });
+pl.a.addEventListener('pause', () => vbSave()); pl.a.addEventListener('ended', () => vbSave()); addEventListener('pagehide', () => vbSave(true));
+const vbLeft = key => { const p = vbProg[key]; return p && p.pos > 10 && p.dur && p.pos < p.dur - 30 ? p : null; };
+const vbDone = key => { const p = vbProg[key]; return !!(p && p.dur && p.pos >= p.dur - 30); };
+const vbDate = d => { const t = Date.parse(d); return t > 0 ? new Date(t).toLocaleDateString([], { month: 'short', day: 'numeric', year: 'numeric' }) : ''; };
+const vbCover = (s, big) => `<span class="vimg" style="aspect-ratio:1/1;background:var(--panel2)"><img loading="lazy" alt="" referrerpolicy="no-referrer" src="${esc(s.cover || '')}" onerror="this.remove()"></span>`;
+
+async function viewVbv(args = []) {
+  const v = $('#view'); v.innerHTML = '<div class="wrap"><h1>Verse By Verse</h1><p class="empty">Loading…</p></div>';
+  try { await vbLoadProg(); if (args[0] === 'study' && args[1]) return await vbStudy(args[1]); return await vbHome(); }
+  catch (e) { if (!location.hash.startsWith('#/podcast/vbv')) return; v.innerHTML = `<div class="wrap"><a class="btn ghost" href="#/podcast">${icon('back')}Podcast</a><h1>Verse By Verse</h1><p class="err">${esc(e.message)}</p><div class="bar"><button class="btn ghost" id="vbretry">Try again</button><a class="btn ghost" href="https://versebyverseministry.org" target="_blank" rel="noopener">Open the website</a></div></div>`; $('#vbretry').onclick = () => route(); }
+}
+
+async function vbHome() {
+  const all = await vbApi('/studies'); if (!location.hash.startsWith('#/podcast/vbv')) return;
+  const S = { q: '', cat: '' }, bySlug = new Map(all.studies.map(s => [s.slug, s]));
+  const recent = Object.entries(vbProg).map(([k, p]) => { const m = /^vbv:([\w-]+):(\d+)$/.exec(k); return m && { slug: m[1], n: +m[2], key: k, t: p.t }; }).filter(x => x && bySlug.has(x.slug) && vbLeft(x.key)).sort((a, b) => b.t - a.t)
+    .filter((x, i, a) => a.findIndex(y => y.slug === x.slug) === i).slice(0, 8);
+  const card = s => `<a class="vcard" href="#/podcast/vbv/study/${esc(s.slug)}">${vbCover(s)}<b>${esc(s.title)}</b><small>${esc([s.lessons ? s.lessons + ' lessons' : '', s.teacher].filter(Boolean).join(' · '))}</small></a>`;
+  $('#view').innerHTML = `<div class="wrap"><div class="ptop"><a class="btn ghost" href="#/podcast">${icon('back')}Podcast</a><h1 style="margin:0">Verse By Verse</h1></div>
+    <p class="sub">Free verse-by-verse Bible teaching from <a href="https://versebyverseministry.org" target="_blank" rel="noopener" style="color:var(--accent)">Verse By Verse Ministry International</a>. Each study plays lesson after lesson, and most lessons have a handout.</p>
+    ${recent.length ? `<h2>Continue listening</h2><div class="vrow">${recent.map(x => { const s = bySlug.get(x.slug), p = vbProg[x.key]; return `<a class="vcard wide" style="width:140px;flex:none" href="#/podcast/vbv/study/${esc(x.slug)}">${vbCover(s).replace('</span>', `<i class="vprog"><b style="width:${Math.min(100, p.pos / p.dur * 100)}%"></b></i></span>`)}<b>${esc(s.title)}</b><small>Lesson ${x.n + 1}</small></a>`; }).join('')}</div>` : ''}
+    <input type="search" id="vbq" placeholder="Search studies" autocomplete="off" autocapitalize="off" style="width:100%;margin-top:12px">
+    <div class="seg" id="vbchips"><a href="#" data-c="" class="on">All</a>${all.categories.map(c => `<a href="#" data-c="${esc(c.id)}">${esc(c.name)}</a>`).join('')}</div><div id="vbgrid" class="vgrid"></div></div>`;
+  const draw = () => { const q = S.q.toLowerCase(), list = all.studies.filter(s => (!S.cat || s.categories.includes(S.cat)) && (!q || (s.title + ' ' + s.teacher).toLowerCase().includes(q)));
+    $('#vbgrid').innerHTML = list.map(card).join('') || '<p class="empty">No studies match.</p>'; };
+  $('#vbq').oninput = e => { S.q = e.target.value.trim(); draw(); };
+  $('#vbchips').querySelectorAll('a').forEach(a => a.onclick = e => { e.preventDefault(); S.cat = a.dataset.c; $('#vbchips').querySelectorAll('a').forEach(x => x.classList.toggle('on', x === a)); draw(); });
+  draw();
+}
+
+async function vbStudy(slug) {
+  const s = await vbApi('/study/' + encodeURIComponent(slug)); if (!location.hash.startsWith('#/podcast/vbv')) return;
+  const key = l => `vbv:${slug}:${l.n}`, partial = s.lessons.filter(l => vbLeft(key(l))).sort((a, b) => vbProg[key(b)].t - vbProg[key(a)].t)[0], started = s.lessons.some(l => vbProg[key(l)]);
+  const nextUp = () => partial || s.lessons.find(l => !vbDone(key(l))) || s.lessons[0];
+  const row = l => { const p = vbLeft(key(l)), done = vbDone(key(l)); return `<div class="row vbrow" data-n="${l.n}"><button class="ic" data-play style="background:var(--accent);color:#fff;border:0" aria-label="Play lesson ${l.n + 1}">${icon('play')}</button>
+    <span class="grow"><b${done ? ' style="color:var(--muted)"' : ''}>${l.n + 1}. ${esc(l.title)}</b><small>${esc([vbDate(l.date), done ? 'played' : p ? Math.max(1, Math.round((p.dur - p.pos) / 60)) + ' min left' : ''].filter(Boolean).join(' · '))}</small></span>
+    <a class="btn ghost" data-notes href="/api/vbv/notes/${esc(slug)}/${l.n}" target="_blank" rel="noopener" style="min-height:36px;padding:0 12px" title="Lesson handout (PDF)">Notes</a></div>`; };
+  const resume = nextUp(), rp = vbLeft(key(resume));
+  $('#view').innerHTML = `<div class="wrap"><div class="ptop"><a class="btn ghost" href="#/podcast/vbv">${icon('back')}Studies</a></div>
+    <div style="display:flex;gap:16px;align-items:flex-start"><span style="width:120px;flex:none">${vbCover(s)}</span><div style="min-width:0"><h1 style="margin:0">${esc(s.title)}</h1><p class="sub" style="margin:2px 0 0">${esc(s.teacher)} · ${s.lessons.length} lessons</p>
+      <p class="sub" style="margin:6px 0 0;white-space:normal;display:-webkit-box;-webkit-line-clamp:4;-webkit-box-orient:vertical;overflow:hidden">${esc(s.description)}</p></div></div>
+    <div class="bar"><button class="btn" id="vbgo">${icon('play')}${started ? 'Continue' : 'Start'}${resume.n ? ' · lesson ' + (resume.n + 1) : ''}</button></div><h2>Lessons</h2><div class="list">${s.lessons.map(row).join('')}</div>
+    <p class="sub" style="margin-top:14px">Teaching and handouts © Verse By Verse Ministry International, shared freely at <a href="https://versebyverseministry.org/bible-studies/${esc(slug)}" target="_blank" rel="noopener" style="color:var(--accent)">versebyverseministry.org</a>.</p></div>`;
+  const play = l => { // this lesson (from where you left off in it), then the rest of the study in order
+    const p = vbLeft(key(l)), np = vbPlaying(); if (np && np.key === key(l)) { if (pl.a.paused) pl.a.play().catch(() => {}); return; }
+    const q = s.lessons.slice(l.n).map(x => ({ key: key(x), title: x.title, artist: s.title, album: 'Verse By Verse Ministry', thumb: s.cover, src: `/api/vbv/audio/${encodeURIComponent(slug)}/${x.n}`, nocache: true, dur: 0 }));
+    playQueue(q, 0); if (p) pl.a.addEventListener('loadedmetadata', () => { const t = vbPlaying(); if (t && t.key === key(l)) pl.a.currentTime = p.pos; }, { once: true });
+  };
+  $('#vbgo').onclick = () => play(resume);
+  document.querySelectorAll('.vbrow').forEach(r => { const l = s.lessons[+r.dataset.n]; r.querySelector('[data-play]').onclick = () => play(l);
+    r.querySelector('[data-notes]').onclick = async e => { e.preventDefault(); try { const res = await fetch(e.currentTarget.href, { redirect: 'manual' }); if (res.status === 404) return toast('This lesson has no handout'); window.open(e.currentTarget.href, '_blank', 'noopener'); } catch { window.open(e.currentTarget.href, '_blank', 'noopener'); } }; });
+}

server.js

@@ -38,6 +38,7 @@ const mail = require('./mail');  // Outlook.com mail, through Microsoft Graph (M
 const transcripts = require('./transcripts'); // words for podcasts and audiobooks: the show's own transcript, or speech-to-text (TRANSCRIBE_API_URL)
 const lyrics = require('./lyrics');  // lyrics for the music player, through Lyrica (LYRICS_API_URL)
 const abapi = require('./abapi');  // audiobook player library, audio and progress (AUDIOBOOKS_URL)
+const vbv = require('./vbv');         // Verse By Verse Ministry teaching (free Bible studies)
 const radio = require('./radio');       // internet radio (Radio Browser directory, The Journey FM)
 const librivox = require('./librivox'); // free audiobooks from LibriVox
 const books = require('./books');      // e-book reader: EPUB, PDF and DOCX library with synced progress and notes
@@ -543,6 +544,7 @@ const ACCESS = [
   [/^\/(api\/plex\/|plex\/)/, ['music', 'video']],
   [/^\/api\/(audiobooks|ab)\//, ['audio']],
   [/^\/api\/podcast\//, ['podcast']],
+  [/^\/api\/vbv\//, ['podcast']],
   [/^\/api\/read\//, ['read']],
   [/^\/api\/radio\//, ['radio']],
   [/^\/api\/signal\//, ['signal']],
@@ -639,6 +641,7 @@ async function handleRequest(req, res) {
     if (pathname.startsWith('/api/podcast/')) return podcast.handle(req, res);
     if (pathname.startsWith('/api/read/')) return books.handle(req, res);
     if (pathname.startsWith('/api/radio/')) return radio.handle(req, res);
+    if (pathname.startsWith('/api/vbv/')) return vbv.handle(req, res);
     if (pathname.startsWith('/api/signal/')) return signal.enabled() ? signal.handle(req, res) : forbidden(res, 'Signal is not set up on this server');
     if (pathname.startsWith('/api/mail/')) return mail.handle(req, res);
     if (req.url.startsWith('/api/plex/')) return plexApi(req, res);

tools/vbv-test.js

@@ -0,0 +1,42 @@
+// Verse By Verse Ministry: the study list is read from the ministry's category pages, lessons from a study's podcast feed, and audio and handouts are sent to the ministry's own media addresses only.
+const http = require('http'), assert = require('assert'), fs = require('fs'), os = require('os'), path = require('path');
+process.env.DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'nehub-vbv-'));
+let port; const gif = Buffer.from('R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==', 'base64'), seen = [];
+const card = (slug, title, n, who) => `<li class="study-item -card"><a class="link-wrapper js-study-item" href="/bible-studies/${slug}"><figure><div><img src="http://127.0.0.1:${port}/img/tiny.gif" srcset="http://127.0.0.1:${port}/img/tiny.gif, http://127.0.0.1:${port}/img/small.gif 2x" /></div></figure><div class="content-block"><h3 class="title">${title}</h3><h4 class="lesson-count">${n} Lessons</h4><p class="body-text -serif">${who}</p></div></a></li>`;
+const fake = http.createServer((req, res) => {
+  const u = new URL(req.url, 'http://x'), p = u.pathname; seen.push(p);
+  const H = (t, b) => { res.writeHead(200, { 'content-type': t }); res.end(b); };
+  if (p === '/bible-studies/category/old-testament-books') return H('text/html', `<ul>${card('genesis-2011', 'Genesis', 109, 'Stephen Armstrong')}${card('ruth', 'Ruth &amp; Naomi', 4, 'Wes')}</ul>`);
+  if (p === '/bible-studies/category/new-testament-books') return H('text/html', `<ul>${card('jude', 'Jude', 2, 'Wesley Livingston')}${card('ruth', 'Ruth &amp; Naomi', 4, 'Wes')}</ul>`);
+  if (p.startsWith('/bible-studies/category/')) return H('text/html', '<ul></ul>');
+  if (p === '/bible-studies/jude/podcast.xml') return H('application/xml', `<rss><channel><title>Jude</title><itunes:author>Wesley Livingston</itunes:author><itunes:summary>A study of Jude &amp; more.</itunes:summary><itunes:image href="http://127.0.0.1:${port}/img/cover.gif" />
+    <item><title>1 Jude 1-3</title><link>http://127.0.0.1:${port}/lessons/jude-lesson-1</link><description>Contend for the Faith</description><itunes:subtitle>Opening study</itunes:subtitle><enclosure url="http://127.0.0.1:${port}/media/1.mp3" length="5" type="audio/mpeg" /><pubDate>Sun, 29 Aug 2021 06:00:00 -0500</pubDate></item>
+    <item><title>2 Jude 4-8</title><link>http://127.0.0.1:${port}/lessons/jude-lesson-2</link><description>False teachers</description><enclosure url="https://evil.example.com/2.mp3" length="5" type="audio/mpeg" /></item>
+    <item><title>3 Jude 9-16</title><link>http://127.0.0.1:${port}/lessons/jude-lesson-3</link><description>Woe</description><enclosure url="http://127.0.0.1:${port}/media/3.mp3" length="5" type="audio/mpeg" /></item></channel></rss>`);
+  if (p === '/lessons/jude-lesson-1') return H('text/html', `<a href="http://127.0.0.1:${port}/media/notes1.pdf">Notes</a>`);
+  if (p === '/lessons/jude-lesson-3') return H('text/html', '<p>No handout</p>');
+  if (p.startsWith('/img/')) { res.writeHead(200, { 'content-type': 'image/gif' }); return res.end(gif); }
+  res.writeHead(404); res.end();
+}).listen(0, () => {
+  port = fake.address().port; process.env.VBV_URL = 'http://127.0.0.1:' + port;
+  const vbv = require('../vbv'), users = require('../users'), me = users.create({ name: 'Ann', role: 'admin', handle: 'YQ' }), bob = users.create({ name: 'Bob', role: 'user', tabs: ['podcast'], handle: 'Yg' });
+  const hub = http.createServer((req, res) => users.runAs(req.headers['x-user'] === 'b' ? bob : me, () => vbv.handle(req, res))).listen(0, async () => {
+    const base = 'http://127.0.0.1:' + hub.address().port + '/api/vbv', call = (p, o = {}) => fetch(base + p, { redirect: 'manual', ...o }), j = r => r.json();
+    try {
+      let r = await j(await call('/studies')); assert.deepEqual(r.studies.map(s => s.slug).sort(), ['genesis-2011', 'jude', 'ruth']); const ruth = r.studies.find(s => s.slug === 'ruth');
+      assert.equal(ruth.title, 'Ruth & Naomi'); assert.deepEqual(ruth.categories.sort(), ['new-testament-books', 'old-testament-books'], 'a study in two categories is listed once'); assert.equal(r.studies.find(s => s.slug === 'genesis-2011').lessons, 109); assert.match(r.studies[0].cover, /^\/api\/vbv\/cover\?u=/);
+      assert.deepEqual((await j(await call('/studies?category=old-testament-books&q=gen'))).studies.map(s => s.slug), ['genesis-2011']);
+      r = await j(await call('/study/jude')); assert.equal(r.teacher, 'Wesley Livingston'); assert.equal(r.description, 'A study of Jude & more.'); assert.equal(r.lessons.length, 2, 'a lesson whose audio is on another site is dropped'); assert.equal(r.lessons[0].title, 'Jude 1-3', 'the lesson number in the feed title is not repeated');
+      assert.ok(!JSON.stringify(r).includes('/media/'), 'recording addresses are not sent to the browser'); assert.equal(r.lessons[1].title, 'Jude 9-16');
+      let a = await call('/audio/jude/0'); assert.equal(a.status, 302); assert.equal(a.headers.get('location'), `http://127.0.0.1:${port}/media/1.mp3`); assert.equal((await call('/audio/jude/5')).status, 404);
+      a = await call('/notes/jude/0'); assert.equal(a.status, 302); assert.equal(a.headers.get('location'), `http://127.0.0.1:${port}/media/notes1.pdf`); assert.equal((await call('/notes/jude/1')).status, 404, 'a lesson with no handout says so');
+      assert.equal((await call('/study/..%2Fx')).status, 404); assert.equal((await call('/study/nothere')).status, 404);
+      a = await call('/cover?u=' + encodeURIComponent(`http://127.0.0.1:${port}/img/cover.gif`)); assert.equal(a.headers.get('content-type'), 'image/gif'); assert.equal((await call('/cover?u=' + encodeURIComponent('https://example.com/a.jpg'))).status, 404);
+      const post = o => call('/progress', { method: 'POST', body: JSON.stringify(o) }).then(j);
+      await post({ 'vbv:jude:0': { pos: 300, dur: 3000, t: 10 } }); assert.equal((await post({ 'vbv:jude:0': { pos: 5, dur: 3000, t: 3 } }))['vbv:jude:0'].pos, 300, 'the newest position wins'); assert.deepEqual(await j(await call('/progress', { headers: { 'x-user': 'b' } })), {}, "each person's places are their own");
+      const n = seen.length; await call('/studies'); await call('/study/jude'); assert.equal(seen.length, n, 'the ministry\'s pages are fetched once and kept for a while, not on every visit');
+      console.log('vbv test ok');
+    } catch (err) { console.error(err); process.exitCode = 1; }
+    hub.close(); fake.close();
+  });
+});

vbv.js

@@ -0,0 +1,89 @@
+// Verse By Verse Ministry International (versebyverseministry.org): free verse-by-verse Bible teaching. The ministry publishes every study as a podcast feed
+// (/bible-studies/<study>/podcast.xml) with one MP3 per lesson, and a handout (PDF) on each lesson's page. The hub reads the study list from the ministry's own
+// pages and the feeds, sends the browser straight to each lesson's recording (so the ministry's server does the streaming), and keeps each person's place.
+const users = require('./users');
+const prog = require('./progress')('vbv-progress.json'); // keys vbv:<study>:<lesson number>
+
+const BASE = () => (process.env.VBV_URL || 'https://versebyverseministry.org').replace(/\/+$/, '');
+const HOSTS = () => { const h = new URL(BASE()).hostname; return [h, 'www.' + h, 'media.' + h.replace(/^www\./, ''), 'versebyverseministry.org', 'media.versebyverseministry.org', 'www.versebyverseministry.org']; };
+const allowed = u => { try { const x = new URL(u); return /^https?:$/.test(x.protocol) && HOSTS().includes(x.hostname); } catch { return false; } };
+const UA = 'NE-Hub/1.0 (self-hosted; reads the ministry\'s public pages and feeds)';
+const CATEGORIES = [['new-testament-books', 'New Testament'], ['old-testament-books', 'Old Testament'], ['topical-series', 'Topical'], ['single-teachings', 'Single teachings'], ['womens-teaching', "Women's"], ['youth-resources', 'Youth']];
+const json = (res, code, o) => { res.writeHead(code, { 'content-type': 'application/json', 'cache-control': 'no-store' }); res.end(JSON.stringify(o)); };
+const fail = (code, message) => Object.assign(new Error(message), { code });
+const unent = s => String(s || '').replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, '$1').replace(/&(#x[0-9a-f]+|#\d+|amp|lt|gt|quot|apos|rsquo|lsquo|ldquo|rdquo|mdash|ndash|nbsp);/gi, (m, e) => ({ amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", rsquo: '’', lsquo: '‘', ldquo: '“', rdquo: '”', mdash: '—', ndash: '–', nbsp: ' ' }[e.toLowerCase()] || (e[0] === '#' ? String.fromCodePoint(e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : +e.slice(1)) : m)));
+const plain = s => unent(String(s || '').replace(/<[^>]+>/g, ' ')).replace(/\s+/g, ' ').trim();
+
+async function page(url, max = 3e6) {
+  if (!allowed(url)) throw fail(502, 'Bad address');
+  let r; try { r = await fetch(url, { headers: { 'user-agent': UA, accept: 'text/html,application/xml,*/*;q=0.5' }, redirect: 'follow', signal: AbortSignal.timeout(15000) }); } catch { throw fail(502, 'Could not reach Verse By Verse Ministry'); }
+  if (!allowed(r.url)) throw fail(502, 'Bad address'); if (r.status === 404) throw fail(404, 'Not found'); if (!r.ok) throw fail(502, 'Verse By Verse Ministry answered ' + r.status);
+  const t = await r.text(); if (t.length > max) throw fail(502, 'Too large'); return t;
+}
+const memo = new Map(); // key -> { t, v }
+async function cached(key, ttl, fn) { const c = memo.get(key); if (c && Date.now() - c.t < ttl) return c.v; const v = await fn(); if (memo.size > 300) memo.clear(); memo.set(key, { t: Date.now(), v }); return v; }
+
+const studiesAll = () => cached('studies', 3600e3, async () => {
+  const by = new Map();
+  await Promise.all(CATEGORIES.map(async ([c, name]) => {
+    let h; try { h = await page(`${BASE()}/bible-studies/category/${c}`); } catch { return; }
+    for (const m of h.matchAll(/<li class="study-item[\s\S]*?<\/li>/g)) {
+      const card = m[0], slug = (/href="\/bible-studies\/([\w-]+)"/.exec(card) || [])[1]; if (!slug) continue;
+      const imgs = (/srcset="([^"]+)"/.exec(card) || [])[1], cover = ((imgs || '').split(',').pop() || '').trim().split(' ')[0] || (/<img src="([^"]+)"/.exec(card) || [])[1] || '';
+      const s = by.get(slug) || { slug, title: plain((/<h3[^>]*>([\s\S]*?)<\/h3>/.exec(card) || [])[1]), lessons: parseInt((/(\d+)\s+Lessons?/i.exec(card) || [])[1]) || 0, teacher: plain((/<p class="body-text[^>]*>([\s\S]*?)<\/p>/.exec(card) || [])[1]), cover: allowed(cover) ? cover : '', categories: [] };
+      s.categories.push(c); by.set(slug, s);
+    }
+  }));
+  if (!by.size) throw fail(502, 'Could not read the list of studies');
+  return [...by.values()];
+});
+const feed = slug => cached('feed:' + slug, 1800e3, async () => {
+  if (!/^[\w-]{1,80}$/.test(slug)) throw fail(400, 'Bad study');
+  const x = await page(`${BASE()}/bible-studies/${slug}/podcast.xml`, 5e6), ch = x.replace(/<item>[\s\S]*$/, '');
+  const tag = (s, t) => plain((new RegExp('<' + t + '>([\\s\\S]*?)</' + t + '>', 'i').exec(s) || [])[1]);
+  const items = [...x.matchAll(/<item>([\s\S]*?)<\/item>/g)].map((m, i) => { const it = m[1], enc = /<enclosure[^>]*url="([^"]+)"/.exec(it); return { n: i, title: tag(it, 'title').replace(/^\d+\s+/, '') || 'Lesson ' + (i + 1), description: tag(it, 'description'), summary: tag(it, 'itunes:subtitle'), date: tag(it, 'pubDate'), audio: enc && allowed(unent(enc[1])) ? unent(enc[1]) : '', link: tag(it, 'link') }; }).filter(i => i.audio);
+  if (!items.length) throw fail(404, 'No lessons found');
+  const img = (/<itunes:image href="([^"]+)"/.exec(ch) || [])[1];
+  return { slug, title: tag(ch, 'title'), teacher: tag(ch, 'itunes:author'), description: tag(ch, 'itunes:summary') || tag(ch, 'description'), cover: img && allowed(img) ? img : '', lessons: items };
+});
+const coverUrl = u => u ? '/api/vbv/cover?u=' + encodeURIComponent(u) : '';
+const publicStudy = s => ({ ...s, cover: coverUrl(s.cover), lessons: s.lessons.map(({ audio, link, ...l }) => l) });
+const notesOf = (slug, n, lessonUrl) => cached(`notes:${slug}:${n}`, 6 * 3600e3, async () => {
+  const h = await page(lessonUrl); const pdfs = [...h.matchAll(/https?:\/\/[^"'\s<>]+\.pdf/gi)].map(m => unent(m[0])).filter(allowed); return pdfs[0] || '';
+});
+const readBody = async req => { const c = []; let n = 0; for await (const d of req) { n += d.length; if (n > 1e5) throw fail(413, 'Too large'); c.push(d); } try { return JSON.parse(Buffer.concat(c).toString('utf8') || '{}'); } catch { throw fail(400, 'Bad request'); } };
+
+async function handle(req, res) {
+  const u = new URL(req.url, 'http://x'), p = u.pathname.slice('/api/vbv'.length) || '/';
+  try {
+    let m;
+    if (p === '/studies' && req.method === 'GET') {
+      const q = (u.searchParams.get('q') || '').trim().toLowerCase(), cat = u.searchParams.get('category') || '';
+      const list = (await studiesAll()).filter(s => (!cat || s.categories.includes(cat)) && (!q || (s.title + ' ' + s.teacher).toLowerCase().includes(q)));
+      return json(res, 200, { categories: CATEGORIES.map(([id, name]) => ({ id, name })), studies: list.map(s => ({ ...s, cover: coverUrl(s.cover) })) });
+    }
+    if ((m = /^\/study\/([\w-]+)$/.exec(p)) && req.method === 'GET') return json(res, 200, publicStudy(await feed(m[1])));
+    if ((m = /^\/audio\/([\w-]+)\/(\d+)$/.exec(p)) && req.method === 'GET') { // the recording is on the ministry's media server; the browser is sent straight there (it supports seeking)
+      const l = (await feed(m[1])).lessons[+m[2]]; if (!l) throw fail(404, 'No such lesson');
+      res.writeHead(302, { location: l.audio, 'cache-control': 'private, max-age=600', 'referrer-policy': 'no-referrer' }); return res.end();
+    }
+    if ((m = /^\/notes\/([\w-]+)\/(\d+)$/.exec(p)) && req.method === 'GET') {
+      const l = (await feed(m[1])).lessons[+m[2]]; if (!l) throw fail(404, 'No such lesson');
+      const pdf = l.link && allowed(l.link) ? await notesOf(m[1], m[2], l.link) : ''; if (!pdf) throw fail(404, 'This lesson has no handout');
+      res.writeHead(302, { location: pdf, 'cache-control': 'private, max-age=600', 'referrer-policy': 'no-referrer' }); return res.end();
+    }
+    if (p === '/cover' && req.method === 'GET') {
+      const src = u.searchParams.get('u') || ''; if (!allowed(src)) throw fail(404, 'No cover');
+      const r = await fetch(src, { headers: { 'user-agent': UA }, redirect: 'manual', signal: AbortSignal.timeout(10000) }), type = (r.headers.get('content-type') || '').split(';')[0].trim().toLowerCase();
+      if (!r.ok || !/^image\/(jpeg|png|gif|webp)$/.test(type)) throw fail(404, 'No cover'); const buf = Buffer.from(await r.arrayBuffer()); if (buf.length > 3e6) throw fail(502, 'Too large');
+      res.writeHead(200, { 'content-type': type, 'cache-control': 'private, max-age=604800' }); return res.end(buf);
+    }
+    if (p === '/progress') {
+      if (req.method === 'GET') return json(res, 200, prog.load());
+      if (req.method === 'POST') return json(res, 200, prog.merge(await readBody(req)));
+    }
+    return json(res, 404, { error: 'Not found' });
+  } catch (e) { return json(res, e.code >= 400 && e.code < 600 ? e.code : 502, { error: e.message }); }
+}
+
+module.exports = { handle };