Merge store-fix: Book store reads Project Gutenberg and Standard Ebooks the way they actually work

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaC9HmtHMdRP8dAbN4wAKj

README.md

@@ -59,11 +59,12 @@ e.g. hub.necloud.us. Settings: `SEAFILE_URL`, `ONLYOFFICE_URL` env vars in docke
 - **Books**: an e-book reader for **EPUB, PDF, DOCX, MOBI, AZW and AZW3** (Media group; give it to people in Settings -> People as *Books*). Add files with the button or by dropping them on the page (up to 300 MB each; the server checks a file really is what its name says and does not store the same book twice). Titles, authors and covers are read out of the books in the browser; long-press or right-click a book to edit, download or delete it. Each person has their own shelf in `/data/users/<id>/books/`.
   - **Reading**: EPUB uses epub.js (pages or scroll, two-page spreads on wide screens), PDF uses pdf.js (continuous scroll, zoom, selectable text, dark and sepia tints) DOCX is converted to HTML by mammoth, and MOBI, AZW and AZW3 are read by `@lingo-reader/mobi-parser`; both are shown in a frame that cannot run scripts (scroll only, with contents from the book or its headings). Contents, search inside the book, a progress slider, tap the edges or use the arrow keys, swipe on touch screens, tap the middle to hide the bars. Appearance (theme, text size, font, line spacing, margins, layout, zoom) is remembered on each device.
   - **Synced**: where you got to, bookmarks, highlights (four colours) and notes are kept on the server, so a book opens where you left off on any device; two devices adding notes at once keep both (the newest edit of the same note wins). Continue reading on the library page, sorting and search, and *Copy as text* exports a book's notes.
-  - **Book store** (Books -> Book store): search free, DRM-free books from **Project Gutenberg** (through Gutendex), **Standard Ebooks**, the **Internet Archive** (public-domain EPUBs only; borrow-only items are refused) and any **OPDS catalogue** an admin adds under *Catalogues* (Calibre-Web, Kavita, Komga and so on; an optional username and password are kept encrypted on the server and never sent to the browser). One search covers every source, results show their cover and source, and *Add to library* downloads the book on the server straight onto your shelf. The server only ever contacts a source's own addresses (redirects to other sites are refused), covers come through the hub, and a catalogue book must live on that catalogue's own address. Settings: `GUTENDEX_URL`, `STANDARD_EBOOKS_URL` and `ARCHIVE_URL` can point at mirrors. `node tools/store-test.js` tests it against fake catalogues; it has not been run against the live sites from the build environment, so if one of them changes its feed format, the line under the search box names the source that failed.
+  - **Book store** (Books -> Book store): search free, DRM-free books from **Project Gutenberg** (its own catalogue feed), **Standard Ebooks** (its public website: its OPDS feed is for paying members only, so the hub reads the public search pages and downloads the free EPUB), the **Internet Archive** (public-domain EPUBs only; borrow-only items are refused) and any **OPDS catalogue** an admin adds under *Catalogues* (Calibre-Web, Kavita, Komga and so on; an optional username and password are kept encrypted on the server and never sent to the browser). One search covers every source, results show their cover and source, and *Add to library* downloads the book on the server straight onto your shelf. The server only ever contacts a source's own addresses (redirects to other sites are refused), covers come through the hub, and a catalogue book must live on that catalogue's own address. Settings: `GUTENBERG_URL`, `STANDARD_EBOOKS_URL` and `ARCHIVE_URL` can point at mirrors. `node tools/store-test.js` tests it against fake catalogues; it was also tried against the live Project Gutenberg, Standard Ebooks and Internet Archive (searches, and real EPUBs downloaded); if one of them changes its pages or feed, the line under the search box names the source that failed.
   - The reader scripts are in `public/vendor/reader/` (epub.js, JSZip, pdf.js, mammoth, the MOBI parser) and load only when a book is opened or added. `node tools/books-test.js` tests the server; `python3 tools/mkbooks.py <dir>` makes a small EPUB, PDF and MOBI to try. **DRM-protected books are refused** (a Kindle file with copy protection is detected on upload and the reason is shown); only DRM-free books work. The MOBI reader was tried on a generated MOBI 6 file, not on real Kindle files, so tell me about any book that opens oddly. Not included: text-to-speech, offline reading.
   **Free audiobooks** (Audiobooks -> Free audiobooks; works without any audiobook sign-in): search **LibriVox**, thousands of public-domain books read aloud by volunteers, by title or author. A book shows its length, language, description and chapters; *Play* (or *Resume*) queues every chapter in the same player as Music so the next one follows by itself, and starting a book puts it on your shelf (*My free audiobooks*, with a progress bar; *Remove from shelf* in the book's sheet). Where you got to is kept on the server per person, so it follows you between devices. The hub searches LibriVox and hands the browser each chapter's address, but only addresses on LibriVox or the Internet Archive, which hosts the recordings; covers come through the hub. `LIBRIVOX_URL` points at a mirror. `node tools/librivox-test.js` tests it against a fake LibriVox; it has not been run against the live site from the build environment.
 - **Radio**: internet radio, with **The Journey FM** (Liberty University, Lynchburg, Virginia; MP3 192 kbps, `http://ic.liberty.edu:8000/WRVL`) always on top, showing what is playing now and the last songs played from the station's own feed. Below it, search thousands of stations from the community [Radio Browser](https://www.radio-browser.info/) directory (free, no key) by name or with quick topic chips (Christian, Worship, Gospel, Talk, News, Classical, Jazz...). The heart saves a station to *My stations* (your own list). Stations play in the same player as Music, so they keep playing as you move between tabs. The hub fetches each stream itself and passes it on, because many stations only offer plain http, which a browser refuses to play inside an https page; it also reads the song title carried in the stream (the player shows it) and fetches station logos. The hub never connects to private, local or link-local addresses for a station. Give it to people in Settings -> People (Radio). `JOURNEYFM_STREAM_URL`, `JOURNEYFM_NOW_URL` and `RADIO_BROWSER_URL` can override the addresses. `node tools/radio-test.js` tests it against fake stations; the real Journey stream could not be reached from the build environment, so if it will not play, open the station's own *Website* button and tell me what changed. Not included: HLS-only stations, recording, alarms.
 - **Verse By Verse** (Podcast -> *Verse By Verse*; needs only the Podcast tab, not Podgrab): free verse-by-verse Bible teaching from [Verse By Verse Ministry International](https://versebyverseministry.org). Browse the ministry's studies (about 80: Old and New Testament books, topical series, single teachings, women's and youth), search them, and open a study to see its lessons with dates and summaries. *Start* / *Continue* plays the study lesson after lesson in the same player as Music, a lesson resumes where you left off (kept on the server per person, so it follows you between devices), and each lesson's handout (PDF) opens from *Notes*. The hub reads the study list from the ministry's category pages and the lessons from each study's own podcast feed (`/bible-studies/<study>/podcast.xml`), keeps what it read for 30 to 60 minutes so the ministry's site is not asked on every visit, and sends the browser straight to the ministry's media server for the audio. Only the ministry's own addresses are ever used. `VBV_URL` can point at a mirror. `node tools/vbv-test.js` tests it against a fake copy of the site; it was also tried against the live site (80 studies, a study's lessons, audio and handout links). If the ministry redesigns its pages, the study list is the first thing that would break, and the screen would say so.
+- **Save to Files**: copy things into your own Seafile from where you find them. **Books** (long-press or right-click a book -> *Save to Files*), **free audiobooks** (a book's sheet -> *Save to Files*, one numbered MP3 per chapter in a folder) and **Verse By Verse** (a study's *Save to Files* button saves every lesson and handout, or just the audio or just the handouts; the download button on a lesson saves that one). You pick the library and folder (it remembers the last one); the hub makes a tidy subfolder (`Books`, `Audiobooks/<title>`, `Verse By Verse/<study>`), names files safely and in order, **skips anything already there** (so running it again fills in what is missing) and counts lessons that have no handout instead of failing. The hub does the downloading and uploading itself with the Seafile sign-in already saved for you, so a whole study (several GB) does not pass through your phone and keeps going if you tap *Hide*; a sheet shows progress, *Stop* cancels, and *Open folder* jumps to the result. It needs the Files tab and the tab the content comes from (Books, Audiobooks or Podcast); each person can have two saves running at once; files up to 500 MB. `node tools/savefiles-test.js` tests it against a fake Seafile.
 - **Signal**: your Signal chats inside the app, **on desktop-class screens only** (a window at least 900 px wide with a mouse or trackpad; phones and touch tablets don't
   see it). Besides the Signal tab there is a chat button on **every** screen that opens a panel over whatever you are doing (close it with the x; the
   expand button opens the full page). New messages get your attention with a toast, a chime, the unread count in the browser tab title and on the installed app's icon, and

books.js

@@ -144,4 +144,7 @@ async function handle(req, res) {
   return json(res, 404, { error: 'Not found' });
 }
 
-module.exports = { handle, mergeNotes, looksLike, addBook, index, TYPES };
+/** Where a book's file is on disk (for the signed-in person). */
+const fileOf = id => { const b = index().find(x => x.id === id); return b ? { path: path.join(dir(), id + '.' + b.ext), ext: b.ext, title: b.title, author: b.author } : null; };
+
+module.exports = { handle, mergeNotes, looksLike, addBook, index, TYPES, fileOf };

librivox.js

@@ -70,4 +70,4 @@ async function handle(req, res) {
   } catch (e) { return json(res, e.code >= 400 && e.code < 600 ? e.code : 502, { error: e.message }); }
 }
 
-module.exports = { handle };
+module.exports = { handle, book, allowed };

package.json

@@ -5,7 +5,7 @@
   "main": "server.js",
   "scripts": {
     "start": "node server.js",
-    "test": "for t in smoke tasks push users cal plex player lyrics signal mail mail-imap files podcast abapi abdirect books store librivox radio vbv; do node tools/$t-test.js || exit 1; done",
+    "test": "for t in smoke tasks push users cal plex player lyrics signal mail mail-imap files podcast abapi abdirect books store librivox radio vbv savefiles; do node tools/$t-test.js || exit 1; done",
     "audit": "npm audit --omit=dev --audit-level=high"
   },
   "keywords": [],

public/index.html

@@ -75,6 +75,7 @@
 <script src="/audiobooks-view.js"></script>
 <script src="/librivox-view.js"></script>
 <script src="/radio-view.js"></script>
+<script src="/save-to-files.js"></script>
 <script src="/vbv-view.js"></script>
 <script src="/lyrics-view.js"></script>
 <script src="/settings.js"></script>

public/librivox-view.js

@@ -84,10 +84,11 @@ async function lvSheet(id, after, known = new Map()) {
     el.querySelector('.empty').outerHTML = `<div style="display:flex;gap:14px;align-items:flex-start"><span class="vimg" style="width:96px;flex:none;aspect-ratio:1/1;background:linear-gradient(135deg,hsl(${abHue(b.title)} 55% 40%),hsl(${abHue(b.title) + 40} 55% 28%))"><img alt="" referrerpolicy="no-referrer" src="${esc(b.cover || '')}" onerror="this.remove()"></span>
       <div style="min-width:0"><p class="sub" style="margin:0">${esc([b.language, lvDur(total), b.chapters.length + ' chapters'].filter(Boolean).join(' · '))}</p><p class="sub" style="margin:4px 0 0">${p ? esc(abFmt(p.pos) + ' of ' + abFmt(p.dur) + ' · ' + abFmt(p.dur - p.pos) + ' left') : 'Not started'}</p></div></div>
       ${b.description ? `<p style="margin:10px 0 0;white-space:normal;display:-webkit-box;-webkit-line-clamp:5;-webkit-box-orient:vertical;overflow:hidden">${esc(b.description)}</p>` : ''}
-      <div class="bar"><button class="btn" id="lvp">${icon('play')}${p ? 'Resume' : 'Play'}</button>${p ? '<button class="btn ghost" id="lvs">From the start</button>' : ''}<button class="btn ghost" id="lvc">Chapters</button><button class="btn ghost" id="lvk">${onShelf() ? 'Remove from shelf' : 'Save to shelf'}</button></div><div class="list" id="lvch" style="max-height:40vh;overflow:auto"></div>`;
+      <div class="bar"><button class="btn" id="lvp">${icon('play')}${p ? 'Resume' : 'Play'}</button>${p ? '<button class="btn ghost" id="lvs">From the start</button>' : ''}<button class="btn ghost" id="lvc">Chapters</button>${allowed('files') ? '<button class="btn ghost" id="lvf">Save to Files</button>' : ''}<button class="btn ghost" id="lvk">${onShelf() ? 'Remove from shelf' : 'Save to shelf'}</button></div><div class="list" id="lvch" style="max-height:40vh;overflow:auto"></div>`;
     const go = at => { if (!b.chapters.length) return toast('This book has no audio'); lvPlay(b, at); closeSheet(); if (!onShelf()) lvShelf.push({ id, title: b.title, author: b.author, secs: b.secs, cover: b.cover, added: Date.now() }); if (after) after(); };
     el.querySelector('#lvp').onclick = () => go();
     if (el.querySelector('#lvs')) el.querySelector('#lvs').onclick = () => go(0);
+    if (el.querySelector('#lvf')) el.querySelector('#lvf').onclick = () => { closeSheet(); saveToFiles(b.title, { kind: 'librivox-book', id }); };
     el.querySelector('#lvc').onclick = () => { const offs = lvOffsets(b); el.querySelector('#lvch').innerHTML = b.chapters.map((c, i) => `<button class="row" data-i="${i}"><span class="grow"><b>${esc(c.title)}</b><small>${esc([abFmt(c.secs), c.reader].filter(Boolean).join(' · '))}</small></span></button>`).join(''); el.querySelectorAll('#lvch [data-i]').forEach(x => x.onclick = () => go(offs[x.dataset.i])); };
     el.querySelector('#lvk').onclick = async e => { const btn = e.currentTarget; try { if (onShelf()) { await lvApi('/shelf/' + id, lvOpt('DELETE')); lvShelf = lvShelf.filter(x => x.id !== id); } else { await lvApi('/shelf', lvOpt('POST', { id })); lvShelf.push({ id, title: b.title, author: b.author, secs: b.secs, cover: b.cover, added: Date.now() }); } btn.textContent = onShelf() ? 'Remove from shelf' : 'Save to shelf'; if (after) after(); } catch (x) { toast(x.message); } };
   }

public/reader.js

@@ -82,6 +82,7 @@ function rdBookMenu(b) {
   actionSheet(b.title, b.author || b.ext.toUpperCase(), [
     { icon: 'book', label: 'Open', run: () => { location.hash = '#/read/open/' + b.id; } },
     { icon: 'edit', label: 'Edit title and author', run: () => rdEditBook(b) },
+    ...(allowed('files') ? [{ icon: 'files', label: 'Save to Files', run: () => saveToFiles(b.title, { kind: 'book', id: b.id }) }] : []),
     { icon: 'dl', label: 'Download', run: () => { const a = document.createElement('a'); a.href = `/api/read/book/${b.id}/file`; a.download = b.title + '.' + b.ext; document.body.appendChild(a); a.click(); a.remove(); } },
     { icon: 'trash', label: 'Delete from library', danger: true, run: () => confirmSheet('Delete this book?', `"${b.title}" and your bookmarks, notes and highlights in it are removed.`, 'Delete', async () => {
       try { await rdApi('/book/' + b.id, { method: 'DELETE' }); toast('Deleted'); route(); } catch (e) { toast(e.message); } }) },

public/save-to-files.js

@@ -0,0 +1,37 @@
+'use strict';
+// ---- "Save to Files": copy a book, free audiobook or Verse By Verse lesson into your Seafile. You pick the library and folder; the hub does the downloading and
+// uploading (a whole study keeps going in the background), skips what is already there, and tells you when it is done.
+const sfLoc = () => { try { return JSON.parse(localStorage.getItem('save_loc:' + (ME ? ME.id : ''))); } catch { return null; } };
+async function saveToFiles(title, body) {
+  if (!allowed('files')) return toast('Your account does not have Files');
+  if (!sf.token()) return toast('Open Files and sign in first', { label: 'Files', run: () => { location.hash = '#/files'; } });
+  const dst = await chooseFolder('Save “' + title + '” in…', sfLoc(), { newFolder: true }); if (!dst) return;
+  try { localStorage.setItem('save_loc:' + (ME ? ME.id : ''), JSON.stringify(dst)); } catch {}
+  let job; try { job = await jfetch('/api/save/', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ ...body, repo: dst.repo, dir: dst.dir }) }); } catch (e) { return toast(e.message); }
+  sfProgress(job, dst);
+}
+function sfProgress(job, dst) {
+  let hidden = false, el = null;
+  const where = () => (dst.repoName || 'Files') + (dst.dir === '/' ? '' : dst.dir) + '/' + job.where;
+  const open = () => { location.hash = '#/files/' + encodeURIComponent(dst.repo) + (dst.dir + '/' + job.where).split('/').filter(Boolean).map(s => '/' + encodeURIComponent(s)).join(''); };
+  const draw = () => {
+    if (!el || !document.body.contains(el)) return; const n = job.done + job.skipped + job.missing + job.failed, pct = job.total ? Math.round(n / job.total * 100) : 100;
+    el.querySelector('#sfb').innerHTML = `<p class="sub" style="margin:0 0 8px;word-break:break-word">${esc(where())}</p><div style="height:8px;border-radius:99px;background:var(--panel2);overflow:hidden"><div style="height:100%;width:${pct}%;background:var(--accent);transition:width .3s"></div></div>
+      <p style="margin:10px 0 0">${job.state === 'running' ? `Saving ${Math.min(n + 1, job.total)} of ${job.total}…` : job.state === 'done' ? 'Done' : job.state === 'cancelled' ? 'Stopped' : 'Could not finish'}</p>${job.current && job.state === 'running' ? `<p class="sub" style="margin:2px 0 0;word-break:break-word">${esc(job.current)}</p>` : ''}
+      <p class="sub" style="margin:6px 0 0">${esc(sfSummary(job))}</p>${job.error ? `<div class="err">${esc(job.error)}</div>` : ''}${job.errors.length && !job.error ? `<div class="err" style="font-size:13px">${job.errors.map(esc).join('<br>')}</div>` : ''}`;
+    const b = el.querySelector('#sfx'); b.textContent = job.state === 'running' ? 'Stop' : 'Close'; el.querySelector('#sfh').hidden = job.state !== 'running'; el.querySelector('#sfo').hidden = job.state === 'running' || !job.done && !job.skipped;
+  };
+  el = openSheet('Saving to Files', job.label, '<div id="sfb"></div><div class="bar"><button class="btn ghost" id="sfx" style="flex:1;justify-content:center">Stop</button><button class="btn ghost" id="sfh" style="flex:1;justify-content:center">Hide</button><button class="btn" id="sfo" hidden style="flex:1;justify-content:center">Open folder</button></div>');
+  el.querySelector('#sfx').onclick = async () => { if (job.state === 'running') { try { await fetch('/api/save/job/' + job.id, { method: 'DELETE' }); } catch {} } else closeSheet(); };
+  el.querySelector('#sfh').onclick = () => { hidden = true; closeSheet(); toast('Saving in the background'); };
+  el.querySelector('#sfo').onclick = () => { closeSheet(); open(); };
+  draw();
+  const tick = async () => {
+    try { job = await jfetch('/api/save/job/' + job.id); } catch (e) { if (e.status === 404) return; setTimeout(tick, 4000); return; }
+    draw();
+    if (job.state === 'running') return setTimeout(tick, 1200);
+    if (hidden || !document.body.contains(el)) toast(job.state === 'done' ? `Saved ${job.done} file${job.done === 1 ? '' : 's'} to Files` + (job.failed ? `, ${job.failed} failed` : '') : job.error || 'Saving stopped', job.done || job.skipped ? { label: 'Open', run: open } : undefined);
+  };
+  setTimeout(tick, 800);
+}
+const sfSummary = j => [j.done && `${j.done} saved`, j.skipped && `${j.skipped} already there`, j.missing && `${j.missing} with no handout`, j.failed && `${j.failed} failed`].filter(Boolean).join(' · ');

public/sw.js

@@ -1,5 +1,5 @@
 // App-shell cache only; Seafile/API traffic always goes to the network.
-const CACHE = 'hub-v41';
+const CACHE = 'hub-v44';
 const SHARE = 'hub-share'; // files shared into the app from other apps, held until the person picks a folder
 const SHELL = ['/icon.svg', '/manifest.webmanifest', '/icons/icon-192.png', '/icons/apple-touch-icon.png'];
 

public/vbv-view.js

@@ -49,12 +49,12 @@ async function vbStudy(slug) {
   const nextUp = () => partial || s.lessons.find(l => !vbDone(key(l))) || s.lessons[0];
   const row = l => { const p = vbLeft(key(l)), done = vbDone(key(l)); return `<div class="row vbrow" data-n="${l.n}"><button class="ic" data-play style="background:var(--accent);color:#fff;border:0" aria-label="Play lesson ${l.n + 1}">${icon('play')}</button>
     <span class="grow"><b${done ? ' style="color:var(--muted)"' : ''}>${l.n + 1}. ${esc(l.title)}</b><small>${esc([vbDate(l.date), done ? 'played' : p ? Math.max(1, Math.round((p.dur - p.pos) / 60)) + ' min left' : ''].filter(Boolean).join(' · '))}</small></span>
-    <a class="btn ghost" data-notes href="/api/vbv/notes/${esc(slug)}/${l.n}" target="_blank" rel="noopener" style="min-height:36px;padding:0 12px" title="Lesson handout (PDF)">Notes</a></div>`; };
+    <a class="btn ghost" href="/api/vbv/notes/${esc(slug)}/${l.n}" target="_blank" rel="noopener" style="min-height:36px;padding:0 12px" title="Lesson handout (PDF)">Notes</a>${allowed('files') ? `<button class="ibtn" data-save aria-label="Save to Files" title="Save to Files">${icon('dl')}</button>` : ''}</div>`; };
   const resume = nextUp(), rp = vbLeft(key(resume));
   $('#view').innerHTML = `<div class="wrap"><div class="ptop"><a class="btn ghost" href="#/podcast/vbv">${icon('back')}Studies</a></div>
-    <div style="display:flex;gap:16px;align-items:flex-start"><span style="width:120px;flex:none">${vbCover(s)}</span><div style="min-width:0"><h1 style="margin:0">${esc(s.title)}</h1><p class="sub" style="margin:2px 0 0">${esc(s.teacher)} · ${s.lessons.length} lessons</p>
+    <div style="display:flex;gap:16px;align-items:flex-start"><span style="width:120px;flex:none">${vbCover(s)}</span><div style="min-width:0"><h1 style="margin:0">${esc(s.title)}</h1><p class="sub" style="margin:2px 0 0">${esc([s.teacher, s.lessons.length + ' lessons'].filter(Boolean).join(' · '))}</p>
       <p class="sub" style="margin:6px 0 0;white-space:normal;display:-webkit-box;-webkit-line-clamp:4;-webkit-box-orient:vertical;overflow:hidden">${esc(s.description)}</p></div></div>
-    <div class="bar"><button class="btn" id="vbgo">${icon('play')}${started ? 'Continue' : 'Start'}${resume.n ? ' · lesson ' + (resume.n + 1) : ''}</button></div><h2>Lessons</h2><div class="list">${s.lessons.map(row).join('')}</div>
+    <div class="bar"><button class="btn" id="vbgo">${icon('play')}${started ? 'Continue' : 'Start'}${resume.n ? ' · lesson ' + (resume.n + 1) : ''}</button>${allowed('files') ? `<button class="btn ghost" id="vbsave">${icon('dl')}Save to Files</button>` : ''}</div><h2>Lessons</h2><div class="list">${s.lessons.map(row).join('')}</div>
     <p class="sub" style="margin-top:14px">Teaching and handouts © Verse By Verse Ministry International, shared freely at <a href="https://versebyverseministry.org/bible-studies/${esc(slug)}" target="_blank" rel="noopener" style="color:var(--accent)">versebyverseministry.org</a>.</p></div>`;
   const play = l => { // this lesson (from where you left off in it), then the rest of the study in order
     const p = vbLeft(key(l)), np = vbPlaying(); if (np && np.key === key(l)) { if (pl.a.paused) pl.a.play().catch(() => {}); return; }
@@ -62,6 +62,14 @@ async function vbStudy(slug) {
     playQueue(q, 0); if (p) pl.a.addEventListener('loadedmetadata', () => { const t = vbPlaying(); if (t && t.key === key(l)) pl.a.currentTime = p.pos; }, { once: true });
   };
   $('#vbgo').onclick = () => play(resume);
+  if ($('#vbsave')) $('#vbsave').onclick = () => actionSheet('Save “' + s.title + '” to Files', s.lessons.length + ' lessons. Whatever is already in the folder is skipped.', [
+    { label: 'Lessons and handouts', icon: 'dl', run: () => saveToFiles(s.title, { kind: 'vbv-study', slug, what: 'both' }) },
+    { label: 'Lesson audio only', icon: 'audio', run: () => saveToFiles(s.title, { kind: 'vbv-study', slug, what: 'audio' }) },
+    { label: 'Handouts only', icon: 'doc', run: () => saveToFiles(s.title, { kind: 'vbv-study', slug, what: 'notes' }) }]);
   document.querySelectorAll('.vbrow').forEach(r => { const l = s.lessons[+r.dataset.n]; r.querySelector('[data-play]').onclick = () => play(l);
-    r.querySelector('[data-notes]').onclick = async e => { e.preventDefault(); try { const res = await fetch(e.currentTarget.href, { redirect: 'manual' }); if (res.status === 404) return toast('This lesson has no handout'); window.open(e.currentTarget.href, '_blank', 'noopener'); } catch { window.open(e.currentTarget.href, '_blank', 'noopener'); } }; });
+    const sv = r.querySelector('[data-save]'); if (sv) sv.onclick = () => actionSheet(s.title + ', lesson ' + (l.n + 1), l.title, [
+      { label: 'Lesson and handout', icon: 'dl', run: () => saveToFiles(s.title + ' ' + (l.n + 1), { kind: 'vbv-lesson', slug, n: l.n, what: 'both' }) },
+      { label: 'Lesson audio only', icon: 'audio', run: () => saveToFiles(s.title + ' ' + (l.n + 1), { kind: 'vbv-lesson', slug, n: l.n, what: 'audio' }) },
+      { label: 'Handout only', icon: 'doc', run: () => saveToFiles(s.title + ' ' + (l.n + 1), { kind: 'vbv-notes', slug, n: l.n }) }]);
+  });
 }

savefiles.js

@@ -0,0 +1,111 @@
+// "Save to Files": copy books, free audiobook chapters and Verse By Verse lessons and handouts into the person's own Seafile. The hub does the downloading and the
+// uploading itself (using the Seafile sign-in already saved for that person), so nothing big passes through the phone, and a whole study keeps going in the background
+// while you use the app. Files that are already there are skipped, so running it again just fills in what is missing.
+const fs = require('fs'), crypto = require('crypto');
+const settings = require('./settings'), users = require('./users'), books = require('./books'), vbv = require('./vbv'), librivox = require('./librivox');
+
+const SEAFILE = () => (process.env.SEAFILE_URL || 'https://seafile.necloud.us').replace(/\/$/, '');
+const MAX_FILE = 500 * 1024 * 1024;
+const TABS = { 'vbv-notes': 'podcast', 'vbv-lesson': 'podcast', 'vbv-study': 'podcast', book: 'read', 'librivox-book': 'audio', 'librivox-chapter': 'audio' };
+const json = (res, code, o) => { res.writeHead(code, { 'content-type': 'application/json', 'cache-control': 'no-store' }); res.end(JSON.stringify(o)); };
+const fail = (code, message) => Object.assign(new Error(message), { code });
+const safe = (s, max = 100) => String(s || '').replace(/[\u0000-\u001f\\/:*?"<>|]+/g, ' ').replace(/\s+/g, ' ').replace(/^[. ]+|[. ]+$/g, '').slice(0, max).trim() || 'Untitled';
+const num = (n, of) => String(n + 1).padStart(Math.max(2, String(of).length), '0');
+const readBody = async req => { const c = []; let n = 0; for await (const d of req) { n += d.length; if (n > 1e5) throw fail(413, 'Too large'); c.push(d); } try { return JSON.parse(Buffer.concat(c).toString('utf8') || '{}'); } catch { throw fail(400, 'Bad request'); } };
+
+// ---- Seafile, as the person
+function seafile(token) {
+  const call = async (p, init = {}) => { const r = await fetch(SEAFILE() + p, { ...init, headers: { ...init.headers, Authorization: 'Token ' + token }, signal: AbortSignal.timeout(init.timeout || 30000) });
+    if (r.status === 401) throw fail(401, 'Your Files sign-in has expired. Open Files and sign in again.'); return r; };
+  const enc = encodeURIComponent;
+  return {
+    async list(repo, dir) { const r = await call(`/api2/repos/${enc(repo)}/dir/?p=${enc(dir)}`); if (r.status === 404) return null; if (!r.ok) throw fail(502, 'Files answered ' + r.status); return (await r.json()).map(x => ({ name: x.name, dir: x.type === 'dir' })); },
+    async mkdir(repo, dir) { const r = await call(`/api2/repos/${enc(repo)}/dir/?p=${enc(dir)}`, { method: 'POST', body: new URLSearchParams({ operation: 'mkdir' }), headers: { 'content-type': 'application/x-www-form-urlencoded' } }); if (!r.ok && r.status !== 409) throw fail(r.status === 403 ? 403 : 502, r.status === 403 ? 'You cannot add files to that library' : 'Could not create the folder (' + r.status + ')'); },
+    async upload(repo, dir, name, buf) {
+      const lr = await call(`/api2/repos/${enc(repo)}/upload-link/?p=${enc(dir)}`); if (!lr.ok) throw fail(lr.status === 403 ? 403 : 502, lr.status === 403 ? 'You cannot add files to that library' : 'Files would not accept the upload (' + lr.status + ')');
+      const link = await lr.json(), fd = new FormData(); fd.append('parent_dir', dir); fd.append('replace', '0'); fd.append('file', new Blob([buf]), name);
+      const up = await fetch(new URL(link, SEAFILE()), { method: 'POST', body: fd, signal: AbortSignal.timeout(15 * 60e3) }); if (!up.ok) throw fail(502, 'The upload failed (' + up.status + ')');
+    },
+  };
+}
+async function download(url, allowed, ua) {
+  if (!allowed(url)) throw fail(502, 'That address is not allowed');
+  let r = await fetch(url, { headers: { 'user-agent': ua }, redirect: 'manual', signal: AbortSignal.timeout(10 * 60e3) });
+  for (let i = 0; i < 4 && [301, 302, 303, 307, 308].includes(r.status); i++) { const to = new URL(r.headers.get('location') || '', url).href; if (!allowed(to)) throw fail(502, 'That address is not allowed'); url = to; r = await fetch(url, { headers: { 'user-agent': ua }, redirect: 'manual', signal: AbortSignal.timeout(10 * 60e3) }); }
+  if (!r.ok) throw fail(502, 'The download failed (' + r.status + ')'); if ((+r.headers.get('content-length') || 0) > MAX_FILE) throw fail(413, 'That file is too large');
+  const chunks = []; let n = 0; for await (const c of r.body) { n += c.length; if (n > MAX_FILE) throw fail(413, 'That file is too large'); chunks.push(c); } return Buffer.concat(chunks);
+}
+
+// ---- what to save: a list of files ({ folder, name, load }), worked out before anything starts so the person gets an answer at once
+async function plan(me, b) {
+  const kind = String(b.kind || ''), out = [];
+  if (kind === 'book') {
+    const f = books.fileOf(String(b.id || '')); if (!f) throw fail(404, 'Book not found');
+    out.push({ folder: ['Books'], name: safe(f.author ? `${f.author} - ${f.title}` : f.title, 120) + '.' + f.ext, load: async () => fs.promises.readFile(f.path) });
+    return { label: f.title, out };
+  }
+  if (kind === 'librivox-book' || kind === 'librivox-chapter') {
+    const bk = await librivox.book(String(b.id || '')), only = kind === 'librivox-chapter' ? +b.n : -1;
+    bk.chapters.forEach(c => { if (only >= 0 && c.n !== only) return; out.push({ folder: ['Audiobooks', safe(bk.title)], name: `${num(c.n, bk.chapters.length)} - ${safe(c.title, 80)}.mp3`, load: () => download(c.url, librivox.allowed, 'NE-Hub/1.0') }); });
+    if (!out.length) throw fail(404, 'No chapters to save'); return { label: bk.title, out };
+  }
+  if (kind.startsWith('vbv-')) {
+    const slug = String(b.slug || ''), st = await vbv.studyInfo(slug), folder = ['Verse By Verse', safe(st.title)], what = b.what || 'both';
+    const lessons = kind === 'vbv-study' ? st.lessons : st.lessons.filter(l => l.n === +b.n); if (!lessons.length) throw fail(404, 'No such lesson');
+    const wantAudio = kind === 'vbv-notes' ? false : what !== 'notes', wantNotes = kind === 'vbv-notes' ? true : what !== 'audio';
+    for (const l of lessons) {
+      const base = `${num(l.n, st.lessons.length)} - ${safe(l.title, 80)}`;
+      if (wantAudio) out.push({ folder, name: base + '.mp3', load: async () => download(await vbv.audioUrl(slug, l.n), vbv.allowed, vbv.UA) });
+      if (wantNotes) out.push({ folder, name: base + ' - Notes.pdf', optional: true, load: async () => download(await vbv.notesUrl(slug, l.n), vbv.allowed, vbv.UA) });
+    }
+    return { label: kind === 'vbv-study' ? st.title : `${st.title}, lesson ${+b.n + 1}`, out };
+  }
+  throw fail(400, 'Unknown thing to save');
+}
+
+// ---- jobs
+const jobs = new Map();
+setInterval(() => { for (const [k, j] of jobs) if (Date.now() - j.t > 6 * 3600e3 && j.state !== 'running') jobs.delete(k); }, 600e3).unref();
+const view = j => ({ id: j.id, state: j.state, label: j.label, total: j.total, done: j.done, skipped: j.skipped, missing: j.missing, failed: j.failed, current: j.current, error: j.error, errors: j.errors.slice(0, 5), where: j.where, repo: j.repo, dir: j.dir });
+
+async function run(j, token, items) {
+  const sf = seafile(token), made = new Set(); let existing = new Map();
+  const ensure = async folder => { let p = j.dir === '/' ? '' : j.dir; for (const part of folder) { p += '/' + part; if (made.has(p)) continue; const names = await sf.list(j.repo, p); if (names === null) await sf.mkdir(j.repo, p); made.add(p); } return (p || '/'); };
+  try {
+    for (const it of items) {
+      if (j.cancelled) { j.state = 'cancelled'; return; }
+      j.current = it.name; let target;
+      try {
+        target = await ensure(it.folder); if (!existing.has(target)) existing.set(target, new Set(((await sf.list(j.repo, target)) || []).map(x => x.name.toLowerCase())));
+        if (existing.get(target).has(it.name.toLowerCase())) { j.skipped++; continue; }
+        let buf; try { buf = await it.load(); } catch (e) { if (it.optional && e.code === 404) { j.missing++; continue; } throw e; }
+        try { await sf.upload(j.repo, target, it.name, buf); } catch (e) { if (e.code === 401 || e.code === 403) throw e; await sf.upload(j.repo, target, it.name, buf); } // one more try
+        existing.get(target).add(it.name.toLowerCase()); j.done++;
+      } catch (e) { if (e.code === 401 || e.code === 403) { j.state = 'error'; j.error = e.message; return; } j.failed++; j.errors.push(`${it.name}: ${e.message}`); }
+    }
+    j.state = j.failed && !j.done && !j.skipped ? 'error' : 'done'; if (j.state === 'error') j.error = j.errors[0] || 'Nothing could be saved';
+  } catch (e) { j.state = 'error'; j.error = e.message; } finally { j.current = ''; }
+}
+
+async function handle(req, res) {
+  const u = new URL(req.url, 'http://x'), p = u.pathname.slice('/api/save'.length) || '/', me = users.current();
+  try {
+    let m;
+    if (p === '/' && req.method === 'POST') {
+      const b = await readBody(req), tab = TABS[b.kind]; if (!tab) throw fail(400, 'Unknown thing to save'); if (!users.can(me, tab)) throw fail(403, 'You do not have access to that');
+      const token = settings.get('sf_token'); if (!token) throw fail(401, 'Open Files and sign in first, then try again');
+      const repo = String(b.repo || ''), dir = '/' + String(b.dir || '/').split('/').filter(x => x && x !== '.' && x !== '..').join('/'); if (!/^[\w-]{8,64}$/.test(repo)) throw fail(400, 'Choose a library and folder');
+      if ([...jobs.values()].filter(j => j.user === me.id && j.state === 'running').length >= 2) throw fail(429, 'Two saves are already running. Wait for one to finish.');
+      const { label, out } = await plan(me, b);
+      const j = { id: crypto.randomBytes(6).toString('hex'), user: me.id, t: Date.now(), state: 'running', label, total: out.length, done: 0, skipped: 0, missing: 0, failed: 0, current: '', error: '', errors: [], repo, dir, where: out[0].folder.join('/'), cancelled: false };
+      jobs.set(j.id, j); run(j, token, out); return json(res, 200, view(j));
+    }
+    if ((m = /^\/job\/(\w+)$/.exec(p))) {
+      const j = jobs.get(m[1]); if (!j || j.user !== me.id) throw fail(404, 'Not found');
+      if (req.method === 'DELETE') { j.cancelled = true; return json(res, 200, view(j)); } return json(res, 200, view(j));
+    }
+    return json(res, 404, { error: 'Not found' });
+  } catch (e) { return json(res, e.code >= 400 && e.code < 600 ? e.code : 502, { error: e.message }); }
+}
+
+module.exports = { handle, safe, seafile };

server.js

@@ -38,6 +38,7 @@ const mail = require('./mail');  // Outlook.com mail, through Microsoft Graph (M
 const transcripts = require('./transcripts'); // words for podcasts and audiobooks: the show's own transcript, or speech-to-text (TRANSCRIBE_API_URL)
 const lyrics = require('./lyrics');  // lyrics for the music player, through Lyrica (LYRICS_API_URL)
 const abapi = require('./abapi');  // audiobook player library, audio and progress (AUDIOBOOKS_URL)
+const savefiles = require('./savefiles'); // "Save to Files": copy books, lessons and audiobooks into the person's Seafile
 const vbv = require('./vbv');         // Verse By Verse Ministry teaching (free Bible studies)
 const radio = require('./radio');       // internet radio (Radio Browser directory, The Journey FM)
 const librivox = require('./librivox'); // free audiobooks from LibriVox
@@ -545,6 +546,7 @@ const ACCESS = [
   [/^\/api\/(audiobooks|ab)\//, ['audio']],
   [/^\/api\/podcast\//, ['podcast']],
   [/^\/api\/vbv\//, ['podcast']],
+  [/^\/api\/save\//, ['files']],
   [/^\/api\/read\//, ['read']],
   [/^\/api\/radio\//, ['radio']],
   [/^\/api\/signal\//, ['signal']],
@@ -642,6 +644,7 @@ async function handleRequest(req, res) {
     if (pathname.startsWith('/api/read/')) return books.handle(req, res);
     if (pathname.startsWith('/api/radio/')) return radio.handle(req, res);
     if (pathname.startsWith('/api/vbv/')) return vbv.handle(req, res);
+    if (pathname.startsWith('/api/save/')) return savefiles.handle(req, res);
     if (pathname.startsWith('/api/signal/')) return signal.enabled() ? signal.handle(req, res) : forbidden(res, 'Signal is not set up on this server');
     if (pathname.startsWith('/api/mail/')) return mail.handle(req, res);
     if (req.url.startsWith('/api/plex/')) return plexApi(req, res);

store.js

@@ -1,4 +1,4 @@
-// Book store: free, DRM-free books from Project Gutenberg (Gutendex), Standard Ebooks (OPDS), the Internet Archive (public-domain EPUBs) and any OPDS catalogue an
+// Book store: free, DRM-free books from Project Gutenberg (its own catalogue feed), Standard Ebooks (its public pages), the Internet Archive (public-domain EPUBs) and any OPDS catalogue an
 // admin adds (Calibre-Web, Kavita, ...). The hub server does all the talking to those sites: searching, covers and downloading a chosen book straight onto the
 // signed-in person's shelf (books.js), so the browser never contacts them and no catalogue sign-in reaches the browser.
 const fs = require('fs'), path = require('path'), crypto = require('crypto');
@@ -8,8 +8,8 @@ const UA = 'NE-Hub/1.0 (self-hosted e-book reader)';
 const MAX_BOOK = 300 * 1024 * 1024, MAX_COVER = 3 * 1024 * 1024, MAX_TEXT = 4 * 1024 * 1024;
 const env = (k, d) => (process.env[k] || d).replace(/\/+$/, '');
 const BUILTIN = {
-  gutenberg: { name: 'Project Gutenberg', base: () => env('GUTENDEX_URL', 'https://gutendex.com'), hosts: ['gutendex.com', 'gutenberg.org'] },
+  gutenberg: { name: 'Project Gutenberg', base: () => env('GUTENBERG_URL', 'https://www.gutenberg.org'), hosts: ['gutenberg.org'] }, // its own catalogue feed (fast and official)
-  standardebooks: { name: 'Standard Ebooks', base: () => env('STANDARD_EBOOKS_URL', 'https://standardebooks.org/feeds/opds'), hosts: ['standardebooks.org'], opds: true },
+  standardebooks: { name: 'Standard Ebooks', base: () => env('STANDARD_EBOOKS_URL', 'https://standardebooks.org'), hosts: ['standardebooks.org'] }, // its public pages: the OPDS feed is for paying members only
   archive: { name: 'Internet Archive', base: () => env('ARCHIVE_URL', 'https://archive.org'), hosts: ['archive.org'], needsQuery: true },
 };
 const FORMAT_OF = { 'application/epub+zip': 'epub', 'application/pdf': 'pdf', 'application/x-mobipocket-ebook': 'mobi', 'application/vnd.amazon.mobi8-ebook': 'azw3', 'application/vnd.amazon.ebook': 'azw' };
@@ -34,7 +34,7 @@ const sourceOf = id => {
 
 // ---- talking to a site: only its own addresses are ever fetched (redirects included), a size limit applies, and a catalogue's sign-in goes nowhere else
 const sameSite = (src, u) => { const h = u.hostname; return src.builtin ? src.hosts.some(x => h === x || h.endsWith('.' + x)) : u.origin === new URL(src.base).origin; };
-async function get(src, url, { accept = '*/*', timeout = 10000 } = {}) {
+async function get(src, url, { accept = '*/*', timeout = 15000 } = {}) {
   let u = new URL(url);
   for (let hop = 0; hop < 6; hop++) {
     if (!/^https?:$/.test(u.protocol) || !sameSite(src, u)) throw fail(502, 'That address is not part of ' + src.name);
@@ -57,6 +57,7 @@ const getJson = async (src, url) => { try { return JSON.parse(await getText(src,
 // ---- OPDS (Atom) feeds, read with a small parser (no XML library in the hub)
 const unent = s => String(s || '').replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, '$1').replace(/&(#x[0-9a-f]+|#\d+|amp|lt|gt|quot|apos);/gi, (m, e) => ({ amp: '&', lt: '<', gt: '>', quot: '"', apos: "'" }[e.toLowerCase()] || (e[1] === 'x' || e[1] === 'X' ? String.fromCodePoint(parseInt(e.slice(2), 16)) : e[0] === '#' ? String.fromCodePoint(+e.slice(1)) : m)));
 const tag = (x, t) => { const m = new RegExp('<' + t + '(?:\\s[^>]*)?>([\\s\\S]*?)</' + t + '>', 'i').exec(x); return m ? unent(m[1]).replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim() : ''; };
+const plain = s => unent(String(s || '').replace(/<[^>]+>/g, ' ')).replace(/\s+/g, ' ').trim();
 const attrs = s => { const o = {}; for (const m of s.matchAll(/([\w:-]+)\s*=\s*(?:"([^"]*)"|'([^']*)')/g)) o[m[1].toLowerCase()] = unent(m[2] != null ? m[2] : m[3]); return o; };
 const linksOf = (x, base) => [...x.matchAll(/<link\b([^>]*?)\/?>/gi)].map(m => attrs(m[1])).filter(a => a.href).map(a => ({ ...a, href: new URL(a.href, base).href }));
 function parseFeed(xml, base) {
@@ -78,7 +79,6 @@ async function opdsSearchUrl(src, q) {
   if (atom) return fill(unbrace(atom.href));
   const osd = root.links.find(l => l.rel === 'search' && /opensearchdescription/.test(l.type || ''));
   if (osd) { const x = await getText(src, osd.href, 'application/xml'); for (const m of x.matchAll(/<Url\b([^>]*?)\/?>/gi)) { const a = attrs(m[1]); if (/atom\+xml/.test(a.type || '') && a.template) return fill(unbrace(new URL(a.template.replace(/\{/g, '%7B').replace(/\}/g, '%7D'), osd.href).href)); } }
-  if (src.id === 'standardebooks') return src.base + '/all?query=' + enc;
   throw fail(502, src.name + ' does not offer a search');
 }
 async function opdsSearch(src, q, page) {
@@ -91,17 +91,33 @@ async function opdsSearch(src, q, page) {
   return { items, more: !!feed.links.find(l => l.rel === 'next') };
 }
 
-// ---- Project Gutenberg through Gutendex, and the Internet Archive
+// ---- Project Gutenberg (its own OPDS catalogue), Standard Ebooks (its public pages), and the Internet Archive
-const tidyName = n => { const m = /^([^,]+),\s*(.+)$/.exec(n || ''); return m ? (m[2] + ' ' + m[1]).replace(/\s*\(.*?\)/g, '').trim() : String(n || ''); };
+const entriesOf = xml => [...xml.matchAll(/<entry[\s>][\s\S]*?<\/entry>/gi)].map(m => m[0]);
 async function gutenbergSearch(src, q, page) {
-  const j = await getJson(src, `${src.base}/books/?${q ? 'search=' + encodeURIComponent(q) + '&' : ''}page=${page}`);
+  const url = `${src.base}/ebooks/search.opds/?${q ? 'query=' + encodeURIComponent(q) : 'sort_order=downloads'}${page > 1 ? '&start_index=' + ((page - 1) * 25 + 1) : ''}`;
-  const items = (j.results || []).filter(b => (b.formats || {})['application/epub+zip']).map(b => ({ ref: String(b.id), format: 'epub', title: b.title, author: (b.authors || []).map(a => tidyName(a.name)).join(', '), summary: (b.subjects || []).slice(0, 3).join(' · '), cover: (b.formats || {})['image/jpeg'] || '' }));
+  const xml = await getText(src, url, 'application/atom+xml'), items = [], seen = new Set();
-  return { items, more: !!j.next };
+  for (const e of entriesOf(xml)) { const id = (/\/ebooks\/(\d+)\.opds/.exec(tag0(e, 'id')) || [])[1]; if (!id || seen.has(id)) continue; seen.add(id); items.push({ ref: id, format: 'epub', title: tag(e, 'title'), author: tag(e, 'content'), summary: '', cover: `${src.base}/cache/epub/${id}/pg${id}.cover.medium.jpg` }); }
+  return { items, more: /<link[^>]+rel="next"/i.test(xml.replace(/<entry[\s>][\s\S]*?<\/entry>/gi, '')) };
 }
+const tag0 = (x, t) => (new RegExp('<' + t + '>([^<]*)</' + t + '>', 'i').exec(x) || [])[1] || '';
 async function gutenbergResolve(src, ref) {
   if (!/^\d{1,9}$/.test(ref)) throw fail(400, 'Bad book');
-  const b = await getJson(src, `${src.base}/books/${ref}`), url = (b.formats || {})['application/epub+zip']; if (!url) throw fail(404, 'No EPUB for that book');
+  return { url: `${src.base}/ebooks/${ref}.epub3.images`, format: 'epub' }; // (the address Project Gutenberg itself gives for the EPUB with pictures)
-  return { url, format: 'epub' };
+}
+async function standardSearch(src, q, page) {
+  const html = await getText(src, `${src.base}/ebooks?${q ? 'query=' + encodeURIComponent(q) + '&' : ''}page=${page}&per-page=24`, 'text/html, application/xhtml+xml'), items = [];
+  for (const m of html.matchAll(/<li typeof="schema:Book" about="(\/ebooks\/[^"]+)">([\s\S]*?)<\/li>/g)) {
+    const card = m[2], title = plain((/<p><a [^>]*property="schema:url"><span property="schema:name">([\s\S]*?)<\/span>/.exec(card) || [])[1]);
+    const who = [...(/class="author"[\s\S]*?<\/p>/.exec(card) || [''])[0].matchAll(/<span property="schema:name">([\s\S]*?)<\/span>/g)].map(x => plain(x[1])).join(', '), img = (/<img src="([^"]+)"/.exec(card) || [])[1] || '';
+    if (title) items.push({ ref: m[1], format: 'epub', title, author: who, summary: '', cover: img ? new URL(unent(img), src.base).href : '' });
+  }
+  return { items, more: new RegExp('href="[^"]*[?&;]page=' + (page + 1) + '[^0-9]').test(html) };
+}
+async function standardResolve(src, ref) {
+  if (!/^\/ebooks\/[\w\/.-]+$/.test(ref) || ref.includes('..')) throw fail(400, 'Bad book');
+  const html = await getText(src, src.base + ref, 'text/html'), links = [...html.matchAll(/href="([^"]*\/downloads\/[^"]+\.epub)"/g)].map(m => unent(m[1])).filter(h => !/kepub|_advanced/.test(h));
+  const href = links[0] || `${ref}/downloads/${ref.replace(/^\/ebooks\//, '').split('/').join('_')}.epub`;
+  return { url: new URL(href, src.base).href + '?source=download', format: 'epub' }; // (without this the site answers with a "your download has started" page)
 }
 async function archiveSearch(src, q, page) {
   const query = `(${q}) AND mediatype:texts AND format:EPUB AND -access-restricted-item:true`;
@@ -119,6 +135,7 @@ async function archiveResolve(src, ref) {
 
 async function searchOne(src, q, page) {
   if (src.id === 'gutenberg') return gutenbergSearch(src, q, page);
+  if (src.id === 'standardebooks') return standardSearch(src, q, page);
   if (src.id === 'archive') return q ? archiveSearch(src, q, page) : { items: [], more: false };
   return opdsSearch(src, q, page);
 }
@@ -161,6 +178,7 @@ async function handle(req, res, p) {
       const b = await readBody(req), src = sourceOf(String(b.source || '')), ref = String(b.ref || '');
       let target;
       if (src.id === 'gutenberg') target = await gutenbergResolve(src, ref);
+      else if (src.id === 'standardebooks') target = await standardResolve(src, ref);
       else if (src.id === 'archive') target = await archiveResolve(src, ref);
       else { if (!sameSite(src, new URL(ref, src.base))) throw fail(400, 'Bad book address'); target = { url: ref, format: String(b.format || 'epub') }; }
       const ext = books.TYPES[target.format] ? target.format : 'epub';

tools/savefiles-test.js

@@ -0,0 +1,48 @@
+// "Save to Files": books, free audiobook chapters and Verse By Verse lessons are copied into the person's Seafile (made folders, skipped files already there, handouts that do not exist).
+const http = require('http'), assert = require('assert'), fs = require('fs'), os = require('os'), path = require('path'), cp = require('child_process');
+process.env.DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'nehub-save-'));
+const fx = fs.mkdtempSync(path.join(os.tmpdir(), 'nehub-savefx-')); cp.execFileSync('python3', [path.join(__dirname, 'mkbooks.py'), fx]); const epub = fs.readFileSync(path.join(fx, 'test.epub'));
+const tree = new Map([['/', new Set()]]), files = [], seen = []; let port; // fake Seafile: folders and uploaded files
+const fake = http.createServer(async (req, res) => {
+  const u = new URL(req.url, 'http://x'), p = u.pathname, J = (o, c = 200) => { res.writeHead(c, { 'content-type': 'application/json' }); res.end(JSON.stringify(o)); }, T = (t, b = '') => { res.writeHead(200, { 'content-type': t }); res.end(b); };
+  seen.push(req.method + ' ' + p + (u.searchParams.get('p') ? ' ' + u.searchParams.get('p') : ''));
+  if (p.startsWith('/api2/repos/') || p.startsWith('/seafhttp/')) { if (p.startsWith('/api2/') && req.headers.authorization !== 'Token SFTOKEN') { res.writeHead(401); return res.end(); } }
+  if (/^\/api2\/repos\/R{8}\/dir\/$/.test(p)) { const d = u.searchParams.get('p');
+    if (req.method === 'POST') { const par = d.replace(/\/[^/]+$/, '') || '/'; if (!tree.has(par)) { res.writeHead(404); return res.end(); } tree.set(d, new Set()); tree.get(par).add('D:' + d.split('/').pop()); return J('success', 201); }
+    if (!tree.has(d)) { res.writeHead(404); return res.end('{}'); } return J([...tree.get(d)].map(n => n.startsWith('D:') ? { name: n.slice(2), type: 'dir' } : { name: n, type: 'file' })); }
+  if (/^\/api2\/repos\/R{8}\/upload-link\/$/.test(p)) return J(`http://127.0.0.1:${port}/seafhttp/upload-api/tok?dir=${encodeURIComponent(u.searchParams.get('p'))}`);
+  if (p === '/seafhttp/upload-api/tok') { const b = Buffer.concat(await req.toArray()); const name = /filename="([^"]+)"/.exec(b.toString('latin1'))[1]; const d = u.searchParams.get('dir'); if (!tree.has(d)) { res.writeHead(404); return res.end(); }
+    if (name === 'fail.mp3') { res.writeHead(500); return res.end(); } tree.get(d).add(name); files.push(d + '/' + name + ' ' + b.length); return T('text/plain', 'id'); }
+  // sources: Verse By Verse, LibriVox
+  if (p === '/bible-studies/jude/podcast.xml') return T('application/xml', `<rss><channel><title>Jude: Faith</title><itunes:author>Wes</itunes:author><itunes:image href="http://127.0.0.1:${port}/c.gif"/>` + [1, 2, 3].map(n => `<item><title>${n} Lesson ${n}</title><link>http://127.0.0.1:${port}/lessons/jude-${n}</link><enclosure url="http://127.0.0.1:${port}/media/${n}.mp3" length="5" type="audio/mpeg"/></item>`).join('') + '</channel></rss>');
+  if (/^\/lessons\/jude-[12]$/.test(p)) return T('text/html', `<a href="http://127.0.0.1:${port}/media/notes${p.slice(-1)}.pdf">n</a>`); if (p === '/lessons/jude-3') return T('text/html', 'none');
+  if (/^\/media\/\d\.mp3$/.test(p)) return T('audio/mpeg', 'AUDIO' + p); if (/^\/media\/notes\d\.pdf$/.test(p)) return T('application/pdf', '%PDF-1.4 notes');
+  if (p === '/api/feed/audiobooks/') { const base = { id: '9', title: 'Old: Story?', totaltimesecs: 60, authors: [{ first_name: 'A', last_name: 'B' }] }; return J({ books: [{ ...base, sections: [1, 2].map(n => ({ title: 'Ch ' + n + '/x', listen_url: `http://127.0.0.1:${port}/media/${n}.mp3`, playtime: '30' })) }] }); }
+  res.writeHead(404); res.end();
+}).listen(0, () => {
+  port = fake.address().port; Object.assign(process.env, { SEAFILE_URL: 'http://127.0.0.1:' + port, VBV_URL: 'http://127.0.0.1:' + port, LIBRIVOX_URL: 'http://127.0.0.1:' + port }); tree.set('/Lib', new Set());
+  const users = require('../users'), books = require('../books'), save = require('../savefiles');
+  const ann = users.create({ name: 'Ann', role: 'admin', handle: 'YQ' }), bob = users.create({ name: 'Bob', role: 'user', tabs: ['files'], handle: 'Yg' }), noTok = users.create({ name: 'Cy', role: 'admin', handle: 'Yw' });
+  const as = (u, f) => users.runAs(u, f); as(ann, () => fs.writeFileSync(path.join(users.dir(), 'settings.json'), JSON.stringify({ sf_token: 'SFTOKEN' }))); as(bob, () => fs.writeFileSync(path.join(users.dir(), 'settings.json'), JSON.stringify({ sf_token: 'SFTOKEN' })));
+  const bookId = as(ann, () => books.addBook(epub, 'epub', { title: 'My Book', author: 'Some Author' }).id);
+  const hub = http.createServer((req, res) => as({ b: bob, c: noTok }[req.headers['x-user']] || ann, () => save.handle(req, res))).listen(0, async () => {
+    const base = 'http://127.0.0.1:' + hub.address().port + '/api/save', R = 'RRRRRRRR', j = r => r.json(), post = (b, h = {}) => fetch(base + '/', { method: 'POST', body: JSON.stringify({ repo: R, dir: '/Lib', ...b }), headers: h });
+    const finish = async id => { for (let i = 0; i < 100; i++) { const s = await j(await fetch(base + '/job/' + id)); if (s.state !== 'running') return s; await new Promise(ok => setTimeout(ok, 50)); } throw new Error('job did not finish'); };
+    try {
+      let r = await post({ kind: 'book', id: bookId }); let s = await finish((await j(r)).id); assert.equal(s.state, 'done'); assert.equal(s.done, 1); assert.ok(files.some(f => f.startsWith('/Lib/Books/Some Author - My Book.epub ')), 'the book is in a Books folder, named by author and title');
+      r = await post({ kind: 'book', id: bookId }); s = await finish((await j(r)).id); assert.equal(s.skipped, 1); assert.equal(s.done, 0, 'running it again skips what is already there');
+      r = await post({ kind: 'vbv-lesson', slug: 'jude', n: 0, what: 'both' }); s = await finish((await j(r)).id); assert.equal(s.done, 2); assert.ok(files.some(f => f.startsWith('/Lib/Verse By Verse/Jude Faith/01 - Lesson 1.mp3')), 'folders are made and names are safe'); assert.ok(files.some(f => f.includes('01 - Lesson 1 - Notes.pdf')));
+      r = await post({ kind: 'vbv-study', slug: 'jude', what: 'both' }); s = await finish((await j(r)).id); assert.equal(s.total, 6); assert.equal(s.skipped, 2, 'lesson 1 was already saved'); assert.equal(s.missing, 1, 'a lesson with no handout is counted, not failed'); assert.equal(s.failed, 0); assert.equal(s.done, 3);
+      assert.ok(files.some(f => f.includes('03 - Lesson 3.mp3')) && !files.some(f => f.includes('03 - Lesson 3 - Notes')));
+      r = await post({ kind: 'librivox-book', id: '9', dir: '/Lib/Sub' }); assert.equal((await finish((await j(r)).id)).state, 'error', 'a folder that does not exist in Files is an error, not a crash');
+      r = await post({ kind: 'librivox-book', id: '9' }); s = await finish((await j(r)).id); assert.equal(s.done, 2); assert.ok(files.some(f => f.startsWith('/Lib/Audiobooks/Old Story/01 - Ch 1 x.mp3')), 'odd characters in names are removed');
+      r = await post({ kind: 'book', id: bookId }, { 'x-user': 'b' }); assert.equal(r.status, 403, 'a person without the Books tab cannot save books');
+      r = await post({ kind: 'book', id: bookId }, { 'x-user': 'c' }); assert.equal(r.status, 401, 'no Files sign-in saved: told to sign in'); assert.match((await j(r)).error, /sign in/);
+      assert.equal((await post({ kind: 'nope' })).status, 400); assert.equal((await post({ kind: 'book', id: bookId, repo: '../x' })).status, 400); assert.equal((await post({ kind: 'book', id: 'zzz' })).status, 404);
+      r = await post({ kind: 'book', id: bookId, dir: '/Lib/../../etc' }); s = await finish((await j(r)).id); assert.ok(!s.dir.includes('..'), 'a folder path cannot climb out');
+      assert.equal((await fetch(base + '/job/' + s.id, { headers: { 'x-user': 'b' } })).status, 404, "another person's save is not visible");
+      console.log('savefiles test ok');
+    } catch (err) { console.error(err); process.exitCode = 1; }
+    hub.close(); fake.close();
+  });
+});

tools/store-test.js

@@ -6,25 +6,28 @@ const epub = fs.readFileSync(path.join(fx, 'test.epub')), pdf = fs.readFileSync(
 const seen = []; let port;
 const feed = (body, links = '') => `<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom">${links}${body}</feed>`;
 const acq = (id, title, author, href, type = 'application/epub+zip') => `<entry><title>${title}</title><author><name>${author}</name></author><id>${id}</id><summary>A &amp; B</summary><link rel="http://opds-spec.org/image" href="/img/${id}.gif" type="image/gif"/><link rel="http://opds-spec.org/acquisition/open-access" href="${href}" type="${type}"/></entry>`;
+const seCard = (slug, title, who) => `<li typeof="schema:Book" about="/ebooks/${slug}"><div><a href="/ebooks/${slug}"><picture><img src="/img/${slug.replace(/\W/g, '')}.gif" alt="" property="schema:image"/></picture></a></div><p><a href="/ebooks/${slug}" property="schema:url"><span property="schema:name">${title}</span></a></p><p class="author" typeof="schema:Person"><a href="https://x/y" property="schema:url"><span property="schema:name">${who}</span></a></p></li>`;
 const fake = http.createServer((req, res) => {
-  const u = new URL(req.url, 'http://x'), p = u.pathname; seen.push(req.method + ' ' + p + u.search + (req.headers.authorization ? ' [auth]' : ''));
+  const u = new URL(req.url, 'http://x'), p = u.pathname; seen.push(p + u.search);
-  const J = o => { res.writeHead(200, { 'content-type': 'application/json' }); res.end(JSON.stringify(o)); }, X = x => { res.writeHead(200, { 'content-type': 'application/atom+xml' }); res.end(x); };
+  const J = o => { res.writeHead(200, { 'content-type': 'application/json' }); res.end(JSON.stringify(o)); }, X = x => { res.writeHead(200, { 'content-type': 'application/atom+xml' }); res.end(x); }, HT = x => { res.writeHead(200, { 'content-type': 'application/xhtml+xml' }); res.end(x); };
-  if (p === '/gut/books/' ) return J({ next: u.searchParams.get('page') === '1' ? 'x' : null, results: [{ id: 11, title: 'Alice', authors: [{ name: 'Carroll, Lewis' }], subjects: ['Fantasy'], formats: { 'application/epub+zip': `http://127.0.0.1:${port}/gut/file/11.epub`, 'image/jpeg': `http://127.0.0.1:${port}/img/11.gif` } }, { id: 12, title: 'No epub', authors: [], formats: {} }] });
+  // Project Gutenberg's catalogue feed
-  if (p === '/gut/books/11') return J({ id: 11, formats: { 'application/epub+zip': `http://127.0.0.1:${port}/gut/file/11.epub` } });
+  if (p === '/ebooks/search.opds/') { const more = u.searchParams.get('start_index') ? '' : '<link rel="next" type="application/atom+xml" href="/ebooks/search.opds/?query=x&amp;start_index=26"/>';
-  if (p === '/gut/file/11.epub') { res.writeHead(200, { 'content-type': 'application/epub+zip' }); return res.end(epub); }
+    return X(feed(`<entry><title>Authors</title><id>https://x/ebooks/author/1.opds</id></entry><entry><title>Alice in Wonderland</title><id>https://www.gutenberg.org/ebooks/11.opds</id><content type="text">Lewis Carroll</content></entry><entry><title>Alice in Wonderland</title><id>https://www.gutenberg.org/ebooks/11.opds</id><content>Lewis Carroll</content></entry><entry><title>Dracula &amp; Co</title><id>https://www.gutenberg.org/ebooks/345.opds</id><content>Bram Stoker</content></entry>`, more)); }
-  if (p === '/gut/books/66') return J({ id: 66, formats: { 'application/epub+zip': `http://127.0.0.1:${port}/gut/evil` } });
+  if (p === '/ebooks/11.epub3.images') { res.writeHead(200, { 'content-type': 'application/epub+zip' }); return res.end(epub); }
-  if (p === '/gut/evil') { res.writeHead(302, { location: `http://localhost:${port}/secret` }); return res.end(); }
+  if (p === '/ebooks/66.epub3.images') { res.writeHead(302, { location: `http://localhost:${port}/secret` }); return res.end(); }
-  if (p === '/img/11.gif' || p.startsWith('/img/')) { res.writeHead(200, { 'content-type': 'image/gif' }); return res.end(gif); }
+  if (p.startsWith('/cache/epub/') || p.startsWith('/img/')) { res.writeHead(200, { 'content-type': 'image/gif' }); return res.end(gif); }
+  // Standard Ebooks' public pages (their OPDS feed is for members only)
+  if (p === '/ebooks') { const q = u.searchParams.get('query') || ''; return HT(`<html><body><ol class="ebooks-list grid">${seCard('bram-stoker/dracula', 'Dracula ' + q, 'Bram Stoker')}${seCard('john-bunyan/the-pilgrims-progress', 'The Pilgrim&#8217;s Progress', 'John Bunyan')}</ol><nav><a href="/ebooks?page=2">2</a></nav></body></html>`); }
+  if (p === '/ebooks/bram-stoker/dracula') return HT(`<a href="/ebooks/bram-stoker/dracula/downloads/bram-stoker_dracula.kepub.epub">k</a><a href="/ebooks/bram-stoker/dracula/downloads/bram-stoker_dracula_advanced.epub">a</a><a href="/ebooks/bram-stoker/dracula/downloads/bram-stoker_dracula.epub?source=download">e</a>`);
+  if (p === '/ebooks/bram-stoker/dracula/downloads/bram-stoker_dracula.epub') { if (u.searchParams.get('source') !== 'download') { res.writeHead(200, { 'content-type': 'text/html' }); return res.end('<html>Your Download Has Started!</html>'); } res.writeHead(200, { 'content-type': 'application/epub+zip' }); return res.end(Buffer.concat([epub, Buffer.from('se')])); }
+  if (p === '/feeds/opds') { res.writeHead(401); return res.end('members only'); }
+  // the Internet Archive
   if (p === '/ia/advancedsearch.php') return J({ response: { numFound: 1, docs: [{ identifier: 'moby1', title: 'Moby', creator: ['Melville'] }, { identifier: '../bad', title: 'x' }] } });
   if (p === '/ia/metadata/moby1') return J({ metadata: {}, files: [{ name: 'moby1.pdf', format: 'Text PDF' }, { name: 'moby1.epub', format: 'EPUB' }] });
   if (p === '/ia/metadata/locked') return J({ metadata: { 'access-restricted-item': 'true' }, files: [{ name: 'x.epub', format: 'EPUB' }] });
   if (p === '/ia/download/moby1/moby1.epub') { res.writeHead(200, { 'content-type': 'application/epub+zip' }); return res.end(Buffer.concat([epub, Buffer.from('ia')])); }
-  if (p === '/se/opds') return X(feed('<entry><title>New releases</title><id>n</id><link rel="http://opds-spec.org/sort/new" href="/se/opds/new" type="application/atom+xml"/></entry>', '<link rel="search" href="/se/opds/all?query={searchTerms}" type="application/atom+xml"/>'));
+  // an OPDS catalogue that needs a sign-in (Calibre-Web style)
-  if (p === '/se/opds/all') return X(feed(acq('se1', 'Dracula ' + u.searchParams.get('query'), 'Bram Stoker', '/se/dl/dracula.epub')));
+  if (p.startsWith('/cal/')) {
-  if (p === '/se/opds/new') return X(feed(acq('se2', 'Newest Book', 'Someone', '/se/dl/new.epub'), '<link rel="next" href="/se/opds/new2" type="application/atom+xml"/>'));
-  if (p === '/se/opds/new2') return X(feed(acq('se3', 'Page Two', 'Someone', '/se/dl/new.epub')));
-  if (p === '/se/dl/dracula.epub' || p === '/se/dl/new.epub') { res.writeHead(200, { 'content-type': 'application/epub+zip' }); return res.end(epub); }
-  if (p.startsWith('/cal/')) { // Calibre-Web style: needs a sign-in; search is described by an OpenSearch file
     if (req.headers.authorization !== 'Basic ' + Buffer.from('reader:pw').toString('base64')) { res.writeHead(401); return res.end(); }
     if (p === '/cal/opds') return X(feed('', '<link rel="search" href="/cal/opds/osd" type="application/opensearchdescription+xml"/>'));
     if (p === '/cal/opds/osd') return X('<OpenSearchDescription><Url type="application/atom+xml" template="/cal/opds/search/{searchTerms}"/></OpenSearchDescription>');
@@ -34,7 +37,7 @@ const fake = http.createServer((req, res) => {
   res.writeHead(404); res.end('no');
 }).listen(0, async () => {
   port = fake.address().port; const B = 'http://127.0.0.1:' + port;
-  Object.assign(process.env, { GUTENDEX_URL: B + '/gut', ARCHIVE_URL: B + '/ia', STANDARD_EBOOKS_URL: B + '/se/opds' });
+  Object.assign(process.env, { GUTENBERG_URL: B, ARCHIVE_URL: B + '/ia', STANDARD_EBOOKS_URL: B });
   const books = require('../books'), users = require('../users');
   const people = { a: users.create({ name: 'Ann', role: 'admin', handle: 'YQ' }), b: users.create({ name: 'Bob', role: 'user', tabs: ['read'], handle: 'Yg' }) };
   const hub = http.createServer((req, res) => users.runAs(req.headers['x-user'] === 'b' ? people.b : people.a, () => books.handle(req, res))).listen(0, async () => {
@@ -42,24 +45,26 @@ const fake = http.createServer((req, res) => {
     try {
       let r = await call('/sources'); assert.deepEqual((await r.json()).map(s => s.id), ['gutenberg', 'standardebooks', 'archive']);
       r = await call('/search?q=alice'); let j = await r.json();
-      assert.deepEqual(j.items.map(i => i.source), ['gutenberg', 'standardebooks', 'archive'], 'one result from each catalogue in turn, books with no EPUB left out');
+      assert.deepEqual(j.items.map(i => i.source), ['gutenberg', 'standardebooks', 'archive', 'gutenberg', 'standardebooks'].slice(0, j.items.length), 'one result from each catalogue in turn'); assert.deepEqual(j.errors, {});
-      assert.equal(j.items[0].author, 'Lewis Carroll'); assert.equal(j.items[1].summary, 'A & B'); assert.match(j.items[0].cover, /^\/api\/read\/store\/cover\?s=gutenberg/); assert.equal(j.more, true);
+      const g = j.items.filter(i => i.source === 'gutenberg'); assert.deepEqual(g.map(i => i.ref), ['11', '345'], 'catalogue entries that are not books, and repeats, are left out'); assert.equal(g[0].author, 'Lewis Carroll'); assert.equal(g[1].title, 'Dracula & Co'); assert.match(g[0].cover, /^\/api\/read\/store\/cover\?s=gutenberg/); assert.equal(j.more, true);
-      assert.ok(seen.includes('GET /se/opds/all?query=alice'), 'the search link in the catalogue is used'); assert.ok(!j.items.some(i => i.ref === '../bad'), 'odd identifiers are dropped');
+      const se = j.items.filter(i => i.source === 'standardebooks'); assert.equal(se[0].title, 'Dracula alice'); assert.equal(se[0].author, 'Bram Stoker'); assert.equal(se[1].title, 'The Pilgrim’s Progress'); assert.equal(se[0].ref, '/ebooks/bram-stoker/dracula');
-      r = await call('/search?q=&source=standardebooks&page=2'); j = await r.json(); assert.deepEqual(j.items.map(i => i.title), ['Page Two'], 'no search words: the newest list, and its next pages');
+      assert.ok(!j.items.some(i => i.ref === '../bad'), 'odd identifiers are dropped'); assert.ok(!seen.some(x => x.startsWith('/feeds/opds')), 'the members-only feed is never asked for');
+      r = await call('/search?q=&source=gutenberg'); j = await r.json(); assert.ok(seen.some(x => x.includes('sort_order=downloads')), 'no search words: the most downloaded books'); assert.equal(j.items.length, 2);
+      r = await call('/search?q=x&source=gutenberg&page=2'); assert.ok(seen.some(x => x.includes('start_index=26')), 'later pages ask for the next 25'); assert.equal((await r.json()).more, false);
       r = await call('/search?q=x&source=nope'); j = await r.json(); assert.ok(j.errors.nope, 'a broken catalogue is reported and does not stop the rest');
-      r = await call(j.items ? '/cover?s=gutenberg&u=' + encodeURIComponent(B + '/img/11.gif') : ''); assert.equal(r.headers.get('content-type'), 'image/gif');
+      r = await call('/cover?s=gutenberg&u=' + encodeURIComponent(B + '/img/11.gif')); assert.equal(r.headers.get('content-type'), 'image/gif');
       assert.equal((await call('/cover?s=gutenberg&u=' + encodeURIComponent('http://localhost:' + port + '/img/11.gif'))).status, 502, 'covers only come from the catalogue\'s own site');
       // adding
-      r = await post('/add', { source: 'gutenberg', ref: '11', title: 'Alice', author: 'Lewis Carroll', cover: B + '/img/11.gif' }); let book = await r.json(); assert.equal(r.status, 200); assert.equal(book.ext, 'epub'); assert.equal(book.author, 'Lewis Carroll'); assert.equal(book.cover, true);
+      r = await post('/add', { source: 'gutenberg', ref: '11', title: 'Alice', author: 'Lewis Carroll', cover: B + '/cache/epub/11/pg11.cover.medium.jpg' }); let book = await r.json(); assert.equal(r.status, 200); assert.equal(book.ext, 'epub'); assert.equal(book.author, 'Lewis Carroll'); assert.equal(book.cover, true);
+      assert.ok(seen.includes('/ebooks/11.epub3.images'), 'the EPUB with pictures is fetched');
       r = await post('/add', { source: 'gutenberg', ref: '11', title: 'Alice' }); assert.equal((await r.json()).duplicate, true, 'adding the same book twice does not store it twice');
       assert.equal((await (await fetch(base.replace('/store', '/list'))).json()).length, 1);
       assert.equal((await post('/add', { source: 'gutenberg', ref: '11' }, { 'x-user': 'b' })).status, 200); assert.equal((await (await fetch(base.replace('/store', '/list'), { headers: { 'x-user': 'b' } })).json()).length, 1, "each person's shelf is their own");
-      r = await post('/add', { source: 'archive', ref: 'moby1', title: 'Moby', author: 'Melville' }); assert.equal((await r.json()).title, 'Moby'); assert.ok(seen.includes('GET /ia/download/moby1/moby1.epub'), 'the EPUB is chosen over the PDF');
+      r = await post('/add', { source: 'standardebooks', ref: '/ebooks/bram-stoker/dracula', title: 'Dracula', author: 'Bram Stoker' }); assert.equal(r.status, 200, 'the real EPUB is fetched, not the "download has started" page'); assert.ok(seen.some(x => x.includes('bram-stoker_dracula.epub?source=download')) && !seen.some(x => x.includes('kepub') && x.includes('source')), 'the plain EPUB (not the Kobo or advanced one) is chosen');
+      r = await post('/add', { source: 'standardebooks', ref: '/ebooks/../etc/passwd' }); assert.equal(r.status, 400);
+      r = await post('/add', { source: 'archive', ref: 'moby1', title: 'Moby', author: 'Melville' }); assert.equal((await r.json()).title, 'Moby'); assert.ok(seen.includes('/ia/download/moby1/moby1.epub'), 'the EPUB is chosen over the PDF');
       r = await post('/add', { source: 'archive', ref: 'locked' }); assert.equal(r.status, 403); assert.match((await r.json()).error, /DRM/);
-      r = await post('/add', { source: 'gutenberg', ref: '66' }); assert.equal(r.status, 502, 'a download that redirects off the site is refused'); assert.ok(!seen.some(x => x.includes('/secret')));
       r = await post('/add', { source: 'gutenberg', ref: '../x' }); assert.equal(r.status, 400);
-      r = await post('/add', { source: 'standardebooks', ref: 'http://localhost:' + port + '/se/dl/dracula.epub' }); assert.equal(r.status, 400, 'a catalogue book must live on the catalogue\'s own address');
-      r = await post('/add', { source: 'standardebooks', ref: B + '/se/dl/dracula.epub', title: 'Dracula', format: 'epub' }); assert.equal(r.status, 200);
       // custom catalogue
       assert.equal((await post('/sources', { name: 'Mine', url: B + '/cal/opds' }, { 'x-user': 'b' })).status, 403, 'only an admin adds catalogues');
       assert.equal((await post('/sources', { name: 'Bad', url: B + '/cal/opds', user: 'reader', pass: 'wrong' })).status, 502, 'a catalogue that refuses the sign-in is not added');

tools/vbv-test.js

@@ -13,6 +13,10 @@ const fake = http.createServer((req, res) => {
     <item><title>1 Jude 1-3</title><link>http://127.0.0.1:${port}/lessons/jude-lesson-1</link><description>Contend for the Faith</description><itunes:subtitle>Opening study</itunes:subtitle><enclosure url="http://127.0.0.1:${port}/media/1.mp3" length="5" type="audio/mpeg" /><pubDate>Sun, 29 Aug 2021 06:00:00 -0500</pubDate></item>
     <item><title>2 Jude 4-8</title><link>http://127.0.0.1:${port}/lessons/jude-lesson-2</link><description>False teachers</description><enclosure url="https://evil.example.com/2.mp3" length="5" type="audio/mpeg" /></item>
     <item><title>3 Jude 9-16</title><link>http://127.0.0.1:${port}/lessons/jude-lesson-3</link><description>Woe</description><enclosure url="http://127.0.0.1:${port}/media/3.mp3" length="5" type="audio/mpeg" /></item></channel></rss>`);
+  if (p === '/bible-studies/empty/podcast.xml') return H('application/xml', '<rss><channel><title>Empty</title></channel></rss>');
+  if (p === '/bible-studies/empty') return H('text/html', `<html><head><title>Empty Study | Ministry</title><meta property="og:image" content="http://127.0.0.1:${port}/img/cover.gif"></head><body><h1>Empty Study</h1><ul class="lesson-list"><li class="lesson -x"><a class="lesson-link" href="/lessons/empty-1"><span class="lesson-number">1A</span><p class="body-text -light">Chapter 1:1-5</p></a></li><li class="lesson"><a class="lesson-link" href="/lessons/empty-2"><span class="lesson-number">1B</span><p class="body-text -light">Chapter 1:6-9</p></a></li></ul></body></html>`);
+  if (p === '/lessons/empty-1') return H('text/html', `<audio src="http://127.0.0.1:${port}/media/e1.mp3"></audio>`);
+  if (p === '/lessons/empty-2') return H('text/html', '<video src="x.mp4"></video>');
   if (p === '/lessons/jude-lesson-1') return H('text/html', `<a href="http://127.0.0.1:${port}/media/notes1.pdf">Notes</a>`);
   if (p === '/lessons/jude-lesson-3') return H('text/html', '<p>No handout</p>');
   if (p.startsWith('/img/')) { res.writeHead(200, { 'content-type': 'image/gif' }); return res.end(gif); }
@@ -30,6 +34,9 @@ const fake = http.createServer((req, res) => {
       assert.ok(!JSON.stringify(r).includes('/media/'), 'recording addresses are not sent to the browser'); assert.equal(r.lessons[1].title, 'Jude 9-16');
       let a = await call('/audio/jude/0'); assert.equal(a.status, 302); assert.equal(a.headers.get('location'), `http://127.0.0.1:${port}/media/1.mp3`); assert.equal((await call('/audio/jude/5')).status, 404);
       a = await call('/notes/jude/0'); assert.equal(a.status, 302); assert.equal(a.headers.get('location'), `http://127.0.0.1:${port}/media/notes1.pdf`); assert.equal((await call('/notes/jude/1')).status, 404, 'a lesson with no handout says so');
+      r = await j(await call('/study/empty')); assert.equal(r.title, 'Empty Study'); assert.deepEqual(r.lessons.map(l => l.title), ['1A \u2013 Chapter 1:1-5', '1B \u2013 Chapter 1:6-9'], 'a study with an empty feed is read from its page');
+      a = await call('/audio/empty/0'); assert.equal(a.headers.get('location'), `http://127.0.0.1:${port}/media/e1.mp3`, "its recordings are found on each lesson's page when played"); assert.equal((await call('/audio/empty/1')).status, 404, 'a lesson with no recording says so');
+      a = await call('/notes/jude/1', { headers: { accept: 'text/html' } }); assert.equal(a.status, 404); assert.match(await a.text(), /no handout/, 'opened in a tab, a missing handout is explained in words');
       assert.equal((await call('/study/..%2Fx')).status, 404); assert.equal((await call('/study/nothere')).status, 404);
       a = await call('/cover?u=' + encodeURIComponent(`http://127.0.0.1:${port}/img/cover.gif`)); assert.equal(a.headers.get('content-type'), 'image/gif'); assert.equal((await call('/cover?u=' + encodeURIComponent('https://example.com/a.jpg'))).status, 404);
       const post = o => call('/progress', { method: 'POST', body: JSON.stringify(o) }).then(j);

vbv.js

@@ -39,18 +39,36 @@ const studiesAll = () => cached('studies', 3600e3, async () => {
 });
 const feed = slug => cached('feed:' + slug, 1800e3, async () => {
   if (!/^[\w-]{1,80}$/.test(slug)) throw fail(400, 'Bad study');
-  const x = await page(`${BASE()}/bible-studies/${slug}/podcast.xml`, 5e6), ch = x.replace(/<item>[\s\S]*$/, '');
+  let x = ''; try { x = await page(`${BASE()}/bible-studies/${slug}/podcast.xml`, 5e6); } catch (e) { if (e.code !== 404) throw e; }
+  const ch = x.replace(/<item>[\s\S]*$/, '');
   const tag = (s, t) => plain((new RegExp('<' + t + '>([\\s\\S]*?)</' + t + '>', 'i').exec(s) || [])[1]);
   const items = [...x.matchAll(/<item>([\s\S]*?)<\/item>/g)].map((m, i) => { const it = m[1], enc = /<enclosure[^>]*url="([^"]+)"/.exec(it); return { n: i, title: tag(it, 'title').replace(/^\d+\s+/, '') || 'Lesson ' + (i + 1), description: tag(it, 'description'), summary: tag(it, 'itunes:subtitle'), date: tag(it, 'pubDate'), audio: enc && allowed(unent(enc[1])) ? unent(enc[1]) : '', link: tag(it, 'link') }; }).filter(i => i.audio);
+  if (items.length < 3) { // a few studies have an empty or partial feed; their lessons are all on the study's own page
+    let alt = null; try { alt = await fromPage(slug); } catch (e) { if (!items.length) throw e; }
+    if (alt && alt.lessons.length > items.length) return alt;
+  }
   if (!items.length) throw fail(404, 'No lessons found');
   const img = (/<itunes:image href="([^"]+)"/.exec(ch) || [])[1];
   return { slug, title: tag(ch, 'title'), teacher: tag(ch, 'itunes:author'), description: tag(ch, 'itunes:summary') || tag(ch, 'description'), cover: img && allowed(img) ? img : '', lessons: items };
 });
+/** A study whose feed lists nothing: read its lesson list from its page; each lesson's recording is found when it is played. */
+async function fromPage(slug) {
+  const h = await page(`${BASE()}/bible-studies/${slug}`), meta = n => plain((new RegExp('<meta[^>]+(?:property|name)="' + n + '"[^>]+content="([^"]*)"', 'i').exec(h) || [])[1]);
+  const lessons = [...h.matchAll(/<li class="lesson[^"]*"[\s\S]*?<\/li>/g)].map(m => { const b = m[0], href = (/href="(\/lessons\/[\w-]+)"/.exec(b) || [])[1]; if (!href) return null;
+    const num = plain((/<span class="lesson-number">([\s\S]*?)<\/span>/.exec(b) || [])[1]), sub = plain((/<p class="body-text[^>]*>([\s\S]*?)<\/p>/.exec(b) || [])[1]);
+    return { title: [num, sub].filter(Boolean).join(' \u2013 '), description: sub, summary: '', date: '', audio: '', link: BASE() + href }; }).filter(Boolean).map((l, n) => ({ ...l, n, title: l.title || 'Lesson ' + (n + 1) }));
+  if (!lessons.length) throw fail(404, 'No lessons found for this study');
+  const cover = meta('og:image');
+  return { slug, title: plain((/<h1[^>]*>([\s\S]*?)<\/h1>/.exec(h) || [])[1]) || plain((/<title>([^<|]*)/.exec(h) || [])[1]), teacher: '', description: meta('og:description') || meta('description'), cover: allowed(cover) ? cover : '', lessons };
+}
+const lessonAudio = link => cached('audio:' + link, 6 * 3600e3, async () => { const h = await page(link); return [...h.matchAll(/https?:\/\/[^"'\s<>]+\.mp3/gi)].map(m => unent(m[0])).find(allowed) || ''; });
 const coverUrl = u => u ? '/api/vbv/cover?u=' + encodeURIComponent(u) : '';
 const publicStudy = s => ({ ...s, cover: coverUrl(s.cover), lessons: s.lessons.map(({ audio, link, ...l }) => l) });
 const notesOf = (slug, n, lessonUrl) => cached(`notes:${slug}:${n}`, 6 * 3600e3, async () => {
   const h = await page(lessonUrl); const pdfs = [...h.matchAll(/https?:\/\/[^"'\s<>]+\.pdf/gi)].map(m => unent(m[0])).filter(allowed); return pdfs[0] || '';
 });
+const audioUrl = async (slug, n) => { const l = (await feed(slug)).lessons[n]; if (!l) throw fail(404, 'No such lesson'); const a = l.audio || (l.link && allowed(l.link) ? await lessonAudio(l.link) : ''); if (!a) throw fail(404, 'This lesson has no audio'); return a; };
+const notesUrl = async (slug, n) => { const l = (await feed(slug)).lessons[n]; if (!l) throw fail(404, 'No such lesson'); const pdf = l.link && allowed(l.link) ? await notesOf(slug, n, l.link) : ''; if (!pdf) throw fail(404, 'This lesson has no handout'); return pdf; };
 const readBody = async req => { const c = []; let n = 0; for await (const d of req) { n += d.length; if (n > 1e5) throw fail(413, 'Too large'); c.push(d); } try { return JSON.parse(Buffer.concat(c).toString('utf8') || '{}'); } catch { throw fail(400, 'Bad request'); } };
 
 async function handle(req, res) {
@@ -64,12 +82,11 @@ async function handle(req, res) {
     }
     if ((m = /^\/study\/([\w-]+)$/.exec(p)) && req.method === 'GET') return json(res, 200, publicStudy(await feed(m[1])));
     if ((m = /^\/audio\/([\w-]+)\/(\d+)$/.exec(p)) && req.method === 'GET') { // the recording is on the ministry's media server; the browser is sent straight there (it supports seeking)
-      const l = (await feed(m[1])).lessons[+m[2]]; if (!l) throw fail(404, 'No such lesson');
+      const audio = await audioUrl(m[1], +m[2]);
-      res.writeHead(302, { location: l.audio, 'cache-control': 'private, max-age=600', 'referrer-policy': 'no-referrer' }); return res.end();
+      res.writeHead(302, { location: audio, 'cache-control': 'private, max-age=600', 'referrer-policy': 'no-referrer' }); return res.end();
     }
     if ((m = /^\/notes\/([\w-]+)\/(\d+)$/.exec(p)) && req.method === 'GET') {
-      const l = (await feed(m[1])).lessons[+m[2]]; if (!l) throw fail(404, 'No such lesson');
+      const pdf = await notesUrl(m[1], +m[2]);
-      const pdf = l.link && allowed(l.link) ? await notesOf(m[1], m[2], l.link) : ''; if (!pdf) throw fail(404, 'This lesson has no handout');
       res.writeHead(302, { location: pdf, 'cache-control': 'private, max-age=600', 'referrer-policy': 'no-referrer' }); return res.end();
     }
     if (p === '/cover' && req.method === 'GET') {
@@ -83,7 +100,14 @@ async function handle(req, res) {
       if (req.method === 'POST') return json(res, 200, prog.merge(await readBody(req)));
     }
     return json(res, 404, { error: 'Not found' });
-  } catch (e) { return json(res, e.code >= 400 && e.code < 600 ? e.code : 502, { error: e.message }); }
+  } catch (e) {
+    const code = e.code >= 400 && e.code < 600 ? e.code : 502;
+    if (/^\/notes\//.test(p) && /text\/html/.test(req.headers.accept || '')) { // opened in a new tab: say it in words, with a way back
+      res.writeHead(code, { 'content-type': 'text/html; charset=utf-8', 'cache-control': 'no-store' });
+      return res.end(`<!doctype html><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Lesson handout</title><body style="font:17px/1.5 system-ui;max-width:32em;margin:15vh auto;padding:0 20px;text-align:center"><h2>${code === 404 ? 'This lesson has no handout' : 'The handout could not be loaded'}</h2><p style="color:#666">${code === 404 ? 'Not every lesson comes with notes.' : 'Verse By Verse Ministry did not answer. Try again in a moment.'}</p><p><a href="javascript:history.length>1?history.back():window.close()">Go back</a></p>`);
+    }
+    return json(res, code, { error: e.message });
+  }
 }
 
-module.exports = { handle };
+module.exports = { handle, studyInfo: feed, audioUrl, notesUrl, allowed, UA };