diff --git a/.gitea/workflows/build.yml b/.gitea/workflows/build.yml index 0016838..f81f91f 100644 --- a/.gitea/workflows/build.yml +++ b/.gitea/workflows/build.yml @@ -26,6 +26,7 @@ jobs: - run: node tools/player-test.js # music player: queue, shuffle/repeat, stall recovery - run: node tools/lyrics-test.js # lyrics: formats, wrong matches refused, remembered - run: node tools/mail-test.js # Outlook.com mail through Microsoft Graph: connect, refresh, read, send, privacy + - run: node tools/mail-imap-test.js # Gmail and iCloud mail over IMAP/SMTP: connect, read, send, reply, privacy - run: node tools/signal-test.js # Signal: linking, messages, sending, live updates - uses: docker/setup-qemu-action@v3 diff --git a/README.md b/README.md index 923d3a9..c33308f 100644 --- a/README.md +++ b/README.md @@ -78,8 +78,11 @@ e.g. hub.necloud.us. Settings: `SEAFILE_URL`, `ONLYOFFICE_URL` env vars in docke - **Code layout**: `public/` holds one script per feature (`tasks.js`, `calendar.js`, `music.js`, `player.js`, `files.js`, `settings.js`, ...). They are plain scripts that share one global scope and load in the order listed in `index.html` (`boot.js` last); the versioned `?v=` addresses and the cache version are computed from every file in `public/`. - **Podcast** and **Podcast Admin**: launchers that open `PODCAST_URL` (a home-network address) and `VERSE_URL` in their own tabs. - **Office**: embeds onlyoffice.necloud.us in a frame, with an "open in new tab" fallback if it blocks framing. -- **Mail**: your **Outlook.com** (personal) mailbox inside the app: folders with unread counts, message list with search, reader, compose with attachments (up to about 3 MB), reply, reply all, forward, archive, delete, flag, mark unread, an unread badge on the Mail tab and a Home card. It uses Microsoft Graph, because Outlook.com no longer allows IMAP passwords. Each person connects their own mailbox (Mail tab -> Connect Outlook.com, sign-in is on Microsoft's own page); the hub keeps only the sign-in token, encrypted, in that person's folder, and everything is private to them. Message text is shown in a sandboxed frame with no scripts, and pictures from the internet stay hidden until you press Show pictures. Not included: drafts autosave, contacts, several mailboxes per person, Microsoft 365 work accounts, Gmail/iCloud (those can be added the same way later). - **One-time setup** (step by step, with troubleshooting, in [docs/mail-setup.md](docs/mail-setup.md); needs a free Microsoft account and takes five minutes): go to entra.microsoft.com -> App registrations -> New registration. Name it NE Hub; supported account types **Personal Microsoft accounts only**; Redirect URI: platform **Web**, value `/api/mail/callback` (for example `https://hub.necloud.us/api/mail/callback`; it must match `PUBLIC_URL` exactly). Then API permissions -> Add -> Microsoft Graph -> Delegated: `Mail.ReadWrite`, `Mail.Send`, `User.Read`, `offline_access`. Copy the **Application (client) ID** and paste it, with the secret, into **Settings -> Microsoft app (admin)** (or set `MS_CLIENT_ID` / `MS_CLIENT_SECRET` in the environment, which win). If you registered it as a Web app, also make a client secret (Certificates & secrets) and put its value in `MS_CLIENT_SECRET`; if you used platform "Mobile and desktop applications" instead, no secret is needed. Leave `MS_CLIENT_ID` empty to switch Mail off. (The old bundled Cypht webmail and its `nehub-cypht` volume are gone; you can delete that volume.) Connecting from the installed iPhone/iPad app may open Microsoft's page in Safari; if the app does not notice, reopen it. +- **Mail**: several mail accounts per person in one place: **Gmail**, **iCloud Mail** and (optionally) **Outlook.com** (personal). Folders with unread counts, message list with search, reader, compose with attachments (up to about 3 MB) and a From choice, reply, reply all, forward, archive, delete, flag, mark unread, an unread badge on the Mail tab (all accounts added up) and a Home card with the latest unread across accounts. Each person adds their own accounts (Mail tab, or Settings -> Mail -> Add an account); everything is private to them and stored encrypted in their own folder. + - **Gmail and iCloud** use IMAP/SMTP with an **app password**, so there is nothing to set up on the server. Gmail: turn on 2-Step Verification, then make an app password at myaccount.google.com/apppasswords. iCloud: appleid.apple.com -> Sign-In and Security -> App-Specific Passwords. The password is checked with the provider before it is saved, and wrong ones are explained in plain words. iCloud does not keep a copy of mail sent over SMTP, so NE Hub files one in Sent itself (not yet confirmed against a real iCloud account). + - **Outlook.com** goes through Microsoft Graph (Outlook.com no longer allows IMAP passwords), which needs the one-time Microsoft app below; without it the Outlook.com button simply is not offered. + - Message text is shown in a sandboxed frame with no scripts, and pictures from the internet stay hidden until you press Show pictures. Not included: a combined inbox across accounts (switch account at the top), message previews in the list for Gmail/iCloud, drafts autosave, contacts, Microsoft 365 work accounts, other IMAP providers (easy to add the same way). + **One-time Outlook.com setup** (step by step, with troubleshooting, in [docs/mail-setup.md](docs/mail-setup.md); needs a free Microsoft account and takes five minutes): go to entra.microsoft.com -> App registrations -> New registration. Name it NE Hub; supported account types **Personal Microsoft accounts only**; Redirect URI: platform **Web**, value `/api/mail/callback` (for example `https://hub.necloud.us/api/mail/callback`; it must match `PUBLIC_URL` exactly). Then API permissions -> Add -> Microsoft Graph -> Delegated: `Mail.ReadWrite`, `Mail.Send`, `User.Read`, `offline_access`. Copy the **Application (client) ID** and paste it, with the secret, into **Settings -> Microsoft app (admin)** (or set `MS_CLIENT_ID` / `MS_CLIENT_SECRET` in the environment, which win). If you registered it as a Web app, also make a client secret (Certificates & secrets) and put its value in `MS_CLIENT_SECRET`; if you used platform "Mobile and desktop applications" instead, no secret is needed. Leave `MS_CLIENT_ID` empty to switch Mail off. (The old bundled Cypht webmail and its `nehub-cypht` volume are gone; you can delete that volume.) Connecting from the installed iPhone/iPad app may open Microsoft's page in Safari; if the app does not notice, reopen it. ## Install on iPhone / iPad (PWA) diff --git a/mail-imap.js b/mail-imap.js new file mode 100644 index 0000000..1c95a9c --- /dev/null +++ b/mail-imap.js @@ -0,0 +1,253 @@ +// Gmail and iCloud mail through IMAP (reading, folders, flags, moving) and SMTP (sending), with an app password. +// Same interface as the Outlook provider in mail.js: every function takes the account and the provider's own ids. +// Message ids are base64url JSON [folder path, uid]; folders are the same aliases as Outlook (inbox, drafts, sentitems, archive, junkemail, +// deleteditems) or "p-" for the rest. +const { ImapFlow } = require('imapflow'); +const nodemailer = require('nodemailer'); +const MailComposer = require('nodemailer/lib/mail-composer'); +const { simpleParser } = require('mailparser'); + +const PRESETS = { + gmail: { name: 'Gmail', host: 'imap.gmail.com', port: 993, secure: true, smtpHost: 'smtp.gmail.com', smtpPort: 465, smtpSecure: true, appendSent: false }, + icloud: { name: 'iCloud Mail', host: 'imap.mail.me.com', port: 993, secure: true, smtpHost: 'smtp.mail.me.com', smtpPort: 587, smtpSecure: false, appendSent: true }, +}; +// Only the tests set these (a fake server on localhost, no TLS). +const TEST = process.env.MAIL_TEST_SERVERS ? JSON.parse(process.env.MAIL_TEST_SERVERS) : null; +const fail = (status, message) => Object.assign(new Error(message), { status }); +const EMAIL = /^[^\s@<>,;]+@[^\s@<>,;]+\.[^\s@<>,;]+$/; +const MAX_SOURCE = 25 * 1024 * 1024; +const escHtml = s => String(s).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c])); +const textToHtml = t => '
' + escHtml(t).replace(/\r?\n/g, '
') + '
'; +const enc = o => Buffer.from(JSON.stringify(o)).toString('base64url'); +const dec = s => { try { const v = JSON.parse(Buffer.from(String(s), 'base64url').toString()); if (Array.isArray(v) && typeof v[0] === 'string' && Number.isInteger(v[1])) return v; } catch {} throw fail(400, 'Bad id'); }; + +const cfgOf = a => { + const p = PRESETS[a.type]; if (!p) throw fail(400, 'Unknown mail provider'); + return TEST ? { ...p, ...TEST.imap, smtpHost: TEST.smtp.host, smtpPort: TEST.smtp.port, smtpSecure: false, smtpIgnoreTLS: true, test: true } : p; +}; +/** What went wrong, in words a person can act on. */ +function friendly(a, e) { + const t = (e && (e.responseText || e.message) || '') + ' ' + (e && e.code || ''); + const p = PRESETS[a.type] || {}; + if (e && (e.authenticationFailed || /AUTHENTICATIONFAILED|Invalid credentials|auth.*fail|535|534|Username and Password not accepted/i.test(t))) + return a.type === 'gmail' ? 'Google rejected the password. Use an app password (Google account -> Security -> 2-Step Verification -> App passwords), not your normal password.' + : 'Apple rejected the password. Use an app-specific password (appleid.apple.com -> Sign-In and Security -> App-Specific Passwords), not your Apple ID password.'; + if (/ENOTFOUND|EAI_AGAIN/.test(t)) return `Could not find ${p.name || 'the mail server'} (no internet from the server, or DNS is down).`; + if (/ECONNREFUSED|ETIMEDOUT|ECONNRESET|timed out|Timeout/i.test(t)) return `Could not reach ${p.name || 'the mail server'}. Try again in a moment.`; + if (/IMAP.*disabled|not enabled/i.test(t)) return 'IMAP is turned off for this account. Turn it on in the account\'s mail settings.'; + return (e && e.responseText) || (e && e.message) || 'The mail server refused the request'; +} + +// ---- connections: one per account, kept for a minute after the last use +const pool = new Map(); // account id -> { client, timer, key } +const forget = id => { const e = pool.get(id); if (e) { clearTimeout(e.timer); try { e.client.close(); } catch {} pool.delete(id); } }; +const keyOf = a => a.address + '\n' + a.pass; +async function open(a) { + const e = pool.get(a.id); + if (e && e.key === keyOf(a) && e.client.usable) { clearTimeout(e.timer); e.timer = setTimeout(() => forget(a.id), 60e3).unref(); return e.client; } + forget(a.id); + const c = cfgOf(a); + const client = new ImapFlow({ host: c.host, port: c.port, secure: c.secure, ...(c.test ? { doSTARTTLS: false, tls: { rejectUnauthorized: false } } : {}), auth: { user: a.address, pass: a.pass }, logger: false, emitLogs: false, socketTimeout: 60e3, greetingTimeout: 20e3, connectionTimeout: 20e3 }); + client.on('error', () => {}); client.on('close', () => { const cur = pool.get(a.id); if (cur && cur.client === client) pool.delete(a.id); }); + try { await client.connect(); } catch (err) { throw fail(err.authenticationFailed ? 401 : 502, friendly(a, err)); } + pool.set(a.id, { client, key: keyOf(a), timer: setTimeout(() => forget(a.id), 60e3).unref() }); + return client; +} +/** Run `fn(client)`; if the connection had gone stale, reconnect once. */ +async function withClient(a, fn) { + for (let attempt = 0; ; attempt++) { + const client = await open(a); + try { return await fn(client); } + catch (err) { + if (err.status) throw err; + if (attempt === 0 && (!client.usable || /closed|ECONNRESET|socket|EPIPE|Connection not available/i.test(err.message || ''))) { forget(a.id); continue; } + throw fail(502, friendly(a, err)); + } + } +} + +// ---- folders +const boxCache = new Map(); // account id -> { t, v } +const SPECIAL = { inbox: ['\\Inbox'], sentitems: ['\\Sent'], drafts: ['\\Drafts'], deleteditems: ['\\Trash'], junkemail: ['\\Junk'], archive: ['\\Archive', '\\All'] }; +const GUESS = { sentitems: /^(\[.*\]\/)?sent( (mail|items|messages))?$/i, drafts: /^(\[.*\]\/)?drafts?$/i, deleteditems: /^(\[.*\]\/)?(trash|deleted( (items|messages))?|bin)$/i, junkemail: /^(\[.*\]\/)?(junk( e-?mail)?|spam)$/i, archive: /^(\[.*\]\/)?(archive|all mail)$/i }; +const NAMES = { inbox: 'Inbox', drafts: 'Drafts', sentitems: 'Sent', archive: 'Archive', junkemail: 'Junk', deleteditems: 'Deleted' }; +async function boxes(a, client) { + const c = boxCache.get(a.id); if (c && Date.now() - c.t < 60e3) return c.v; + const list = (await client.list()).filter(b => !b.flags.has('\\Noselect') && !b.flags.has('\\NonExistent')); + const alias = {}; // alias -> path + const take = (k, b) => { if (b && !alias[k]) alias[k] = b.path; }; + take('inbox', list.find(b => b.path.toUpperCase() === 'INBOX')); + for (const k of Object.keys(SPECIAL)) for (const f of SPECIAL[k]) take(k, list.find(b => b.specialUse === f)); + for (const k of Object.keys(GUESS)) take(k, list.find(b => GUESS[k].test(b.path) || GUESS[k].test(b.name))); + const v = { list, alias }; + boxCache.set(a.id, { t: Date.now(), v }); return v; +} +async function pathOf(a, client, folder) { + const { list, alias } = await boxes(a, client); + if (alias[folder]) return alias[folder]; + if (/^p-/.test(folder)) { try { const p = Buffer.from(folder.slice(2), 'base64url').toString(); if (list.some(b => b.path === p)) return p; } catch {} } + throw fail(404, folder === 'archive' ? 'This account has no Archive folder' : 'That folder does not exist'); +} + +const addrOf = x => x ? { name: x.name || '', address: x.address || '' } : { name: '', address: '' }; +const hasAttachment = n => !!n && (((n.disposition || '').toLowerCase() === 'attachment') || (n.childNodes || []).some(hasAttachment)); + +async function folders(a) { + return withClient(a, async client => { + const { list, alias } = await boxes(a, client), byPath = {}; for (const [k, p] of Object.entries(alias)) byPath[p] = k; + const skip = b => b.flags.has('\\Noselect') || ['\\Flagged', '\\Important'].includes(b.specialUse); + const rows = []; + for (const b of list) { + if (skip(b)) continue; + let st = {}; try { st = await client.status(b.path, { messages: true, unseen: true }); } catch {} + const k = byPath[b.path]; + rows.push({ id: k || 'p-' + Buffer.from(b.path).toString('base64url'), name: k ? NAMES[k] : b.path.replace(/^\[.*?\]\//, ''), unread: st.unseen || 0, total: st.messages || 0, well: !!k }); + } + const order = Object.keys(NAMES); + return rows.sort((x, y) => (x.well ? order.indexOf(x.id) : 99) - (y.well ? order.indexOf(y.id) : 99) || x.name.localeCompare(y.name)); + }); +} + +async function list(a, { folder, skip, q }) { + return withClient(a, async client => { + const path = await pathOf(a, client, folder || 'inbox'); + const lock = await client.getMailboxLock(path); + try { + const total = client.mailbox.exists, query = { uid: true, flags: true, envelope: true, bodyStructure: true, internalDate: true }; + let range, opts = {}, more = false; + if (q) { + const uids = (await client.search({ or: [{ subject: q }, { from: q }, { to: q }] }, { uid: true })) || []; + if (!uids.length) return { items: [], more: false, next: 0 }; + range = uids.slice(-50).join(','); opts = { uid: true }; + } else { + const hi = total - skip; if (hi < 1) return { items: [], more: false, next: skip }; + range = `${Math.max(1, hi - 29)}:${hi}`; more = hi - 30 >= 1; + } + const items = []; + for await (const m of client.fetch(range, query, opts)) { + const env = m.envelope || {}; + items.push({ id: enc([path, m.uid]), subject: env.subject || '(No subject)', from: addrOf((env.from || [])[0]), date: (env.date || m.internalDate || new Date()).toISOString(), read: m.flags.has('\\Seen'), attach: hasAttachment(m.bodyStructure), preview: '', flagged: m.flags.has('\\Flagged') }); + } + items.sort((x, y) => String(y.date).localeCompare(String(x.date))); + return { items, more, next: skip + items.length }; + } finally { lock.release(); } + }); +} + +/** Download and parse one message (refusing anything huge). */ +async function source(a, mid) { + const [path, uid] = dec(mid); + return withClient(a, async client => { + const lock = await client.getMailboxLock(path); + try { + const dl = await client.download(String(uid), undefined, { uid: true }); + if (!dl || !dl.content) throw fail(404, 'That message is gone'); + const chunks = []; let n = 0; + for await (const ch of dl.content) { n += ch.length; if (n > MAX_SOURCE) throw fail(400, 'That message is too large to open here'); chunks.push(ch); } + return { parsed: await simpleParser(Buffer.concat(chunks)), path, uid }; + } finally { lock.release(); } + }); +} +const pick = (list, f) => (list || []).map(x => f(x)); +const addrs = v => (v && v.value ? v.value : []).map(x => ({ name: x.name || '', address: x.address || '' })); + +async function message(a, mid) { + const { parsed: p, path, uid } = await source(a, mid); + let html = p.html || (p.textAsHtml ? p.textAsHtml : textToHtml(p.text || '')); + const atts = p.attachments || []; + for (const x of atts) if (x.cid && /^image\//i.test(x.contentType || '') && x.size < 1.5 * 1024 * 1024) html = html.split('cid:' + x.cid).join(`data:${x.contentType};base64,${x.content.toString('base64')}`); + const flags = await withClient(a, async client => { const lock = await client.getMailboxLock(path); try { const m = await client.fetchOne(String(uid), { flags: true }, { uid: true }); return m ? m.flags : new Set(); } finally { lock.release(); } }); + return { id: mid, subject: p.subject || '(No subject)', from: addrs(p.from)[0] || { name: '', address: '' }, to: addrs(p.to), cc: addrs(p.cc), date: (p.date || new Date()).toISOString(), read: flags.has('\\Seen'), flagged: flags.has('\\Flagged'), + html, attachments: atts.map((x, i) => ({ x, i })).filter(({ x }) => !x.related).map(({ x, i }) => ({ id: String(i), name: x.filename || 'attachment', type: x.contentType, size: x.size })) }; +} +async function attachment(a, mid, aid) { + const { parsed: p } = await source(a, mid), x = (p.attachments || [])[parseInt(aid, 10)]; + if (!x) throw fail(404, 'Attachment not found'); + return { name: x.filename || 'attachment', type: x.contentType, buffer: x.content }; +} + +async function update(a, mid, { read, flagged }) { + const [path, uid] = dec(mid); + if (typeof read !== 'boolean' && typeof flagged !== 'boolean') throw fail(400, 'Nothing to change'); + return withClient(a, async client => { + const lock = await client.getMailboxLock(path); + try { + if (typeof read === 'boolean') await (read ? client.messageFlagsAdd : client.messageFlagsRemove).call(client, String(uid), ['\\Seen'], { uid: true }); + if (typeof flagged === 'boolean') await (flagged ? client.messageFlagsAdd : client.messageFlagsRemove).call(client, String(uid), ['\\Flagged'], { uid: true }); + } finally { lock.release(); } + }); +} +async function move(a, mid, to) { + const [path, uid] = dec(mid); + return withClient(a, async client => { + const dest = await pathOf(a, client, to); + const lock = await client.getMailboxLock(path); + try { await client.messageMove(String(uid), dest, { uid: true }); } finally { lock.release(); } + }); +} +async function del(a, mid, permanent) { + if (!permanent) return move(a, mid, 'deleteditems'); + const [path, uid] = dec(mid); + return withClient(a, async client => { const lock = await client.getMailboxLock(path); try { await client.messageDelete(String(uid), { uid: true }); } finally { lock.release(); } }); +} +async function unread(a) { return withClient(a, async client => (await client.status('INBOX', { unseen: true })).unseen || 0); } + +// ---- sending +function transport(a) { + const c = cfgOf(a); + return nodemailer.createTransport({ host: c.smtpHost, port: c.smtpPort, secure: c.smtpSecure, ...(c.smtpIgnoreTLS ? { ignoreTLS: true } : { requireTLS: !c.smtpSecure }), auth: { user: a.address, pass: a.pass }, tls: c.test ? { rejectUnauthorized: false } : undefined, connectionTimeout: 20e3, greetingTimeout: 20e3, socketTimeout: 60e3 }); +} +const fromOf = a => ({ name: a.name || '', address: a.address }); +async function deliver(a, mail) { + const opts = { from: fromOf(a), ...mail }; + const raw = await new MailComposer(opts).compile().build(); + const all = [...(opts.to || []), ...(opts.cc || []), ...(opts.bcc || [])].map(x => x.address || x); + try { await transport(a).sendMail({ envelope: { from: a.address, to: all }, raw }); } + catch (err) { throw fail(err.responseCode === 535 || err.code === 'EAUTH' ? 401 : 502, friendly(a, err)); } + if (cfgOf(a).appendSent) { // iCloud does not keep a copy of what you send through SMTP, so put it in Sent ourselves + try { await withClient(a, async client => { const { alias } = await boxes(a, client); if (alias.sentitems) await client.append(alias.sentitems, raw, ['\\Seen']); }); } catch { /* the mail was sent; only the copy is missing */ } + } +} +async function send(a, { to, cc, bcc, subject, body, attachments }) { + await deliver(a, { to, cc, bcc, subject, text: body, html: textToHtml(body), attachments: attachments.map(x => ({ filename: x.name, contentType: x.type, content: Buffer.from(x.data, 'base64') })) }); +} +const quote = t => String(t || '').split(/\r?\n/).map(l => '> ' + l).join('\n'); +/** Reply, reply all or forward: built here (IMAP has no such call) so the thread headers and the quoted original are right. */ +async function reply(a, { id: mid, mode, comment, to }) { + const { parsed: p } = await source(a, mid), me = a.address.toLowerCase(); + const orig = { from: addrs(p.from), replyTo: addrs(p.replyTo), to: addrs(p.to), cc: addrs(p.cc) }; + const when = (p.date || new Date()).toLocaleString('en-US', { dateStyle: 'medium', timeStyle: 'short' }), who = orig.from[0] ? (orig.from[0].name || orig.from[0].address) : 'someone'; + const subj = String(p.subject || '').replace(/^((re|fwd?):\s*)+/i, ''); + const origHtml = (p.html || textToHtml(p.text || '')).replace(//gi, ''); + const headers = {}; + let mail; + if (mode === 'forward') { + const note = `---------- Forwarded message ----------\nFrom: ${who}\nDate: ${when}\nSubject: ${p.subject || ''}\n\n`; + const files = (p.attachments || []).filter(x => !x.related && x.size < 3 * 1024 * 1024).slice(0, 10).map(x => ({ filename: x.filename || 'attachment', contentType: x.contentType, content: x.content })); + mail = { to, subject: 'Fwd: ' + subj, text: (comment ? comment + '\n\n' : '') + note + (p.text || ''), html: (comment ? textToHtml(comment) : '') + `

${escHtml(note).replace(/\n/g, '
')}

` + origHtml, attachments: files }; + } else { + const first = (orig.replyTo[0] || orig.from[0]); if (!first) throw fail(400, 'That message has no sender to reply to'); + let rcpt = [first]; + if (mode === 'replyAll') for (const x of [...orig.to, ...orig.cc]) if (x.address && x.address.toLowerCase() !== me && !rcpt.some(r => r.address.toLowerCase() === x.address.toLowerCase())) rcpt.push(x); + if (p.messageId) { headers.inReplyTo = p.messageId; headers.references = [].concat(p.references || [], p.messageId).join(' '); } + mail = { to: rcpt.filter(x => x.address.toLowerCase() !== me || rcpt.length === 1), subject: 'Re: ' + subj, text: (comment ? comment + '\n\n' : '') + `On ${when}, ${who} wrote:\n` + quote(p.text || ''), html: textToHtml(comment || '') + `

On ${escHtml(when)}, ${escHtml(who)} wrote:

${origHtml}
`, ...headers }; + } + await deliver(a, mail); +} + +// ---- adding an account: check the password against the real server first +async function connectNew({ type, address, password }) { + const preset = PRESETS[type]; if (!preset) throw fail(400, 'Choose Gmail or iCloud Mail'); + const addr = String(address || '').trim(), pass = String(password || '').replace(/\s+/g, ''); + if (!EMAIL.test(addr)) throw fail(400, 'Enter the full email address'); + if (pass.length < 8) throw fail(400, 'Enter the app password (not your normal password)'); + const a = { id: 'new-' + Math.random().toString(36).slice(2, 8), type, address: addr, pass, name: '' }; + try { + const client = await open(a); + await client.list(); + await transport(a).verify().catch(err => { throw fail(err.responseCode === 535 || err.code === 'EAUTH' ? 401 : 502, friendly(a, err)); }); + } finally { forget(a.id); } + return { type, address: addr, pass, name: '' }; +} + +module.exports = { PRESETS: Object.fromEntries(Object.entries(PRESETS).map(([k, v]) => [k, { name: v.name }])), folders, list, message, attachment, update, move, del, unread, send, reply, connectNew, forget }; diff --git a/mail.js b/mail.js index 168b47e..002372b 100644 --- a/mail.js +++ b/mail.js @@ -1,6 +1,6 @@ -// Mail for a personal Outlook.com account, through Microsoft Graph (the supported way now that Outlook.com has switched off IMAP passwords). -// Each person connects their own account: they sign in on Microsoft's page, the hub keeps only the tokens (encrypted, in their own folder), -// and every mail request runs as that person. Needs an app registration (MS_CLIENT_ID, see README -> Mail). +// Mail: several accounts per person. Outlook.com (personal) goes through Microsoft Graph (sign in on Microsoft's page, needs an app registration, +// see README -> Mail); Gmail and iCloud go through IMAP/SMTP with an app password (mail-imap.js). Every account's credentials are kept encrypted in +// that person's own folder, and every request runs as that person. Message and folder ids carry the account: "~". const crypto = require('crypto'); const fs = require('fs'); const path = require('path'); @@ -8,6 +8,7 @@ const atomicWrite = require('./atomic'); const users = require('./users'); const audit = require('./audit'); const vault = require('./vault')('nehub-mail'); +const imap = require('./mail-imap'); const ENV_ID = process.env.MS_CLIENT_ID || '', ENV_SECRET = process.env.MS_CLIENT_SECRET || ''; const TENANT = process.env.MS_TENANT || 'consumers'; // personal Microsoft accounts only @@ -23,7 +24,8 @@ const loadCfg = () => { if (cfgCache !== undefined) return cfgCache; try { cfgCa const saveCfg = o => { atomicWrite(CONFIG_FILE, JSON.stringify({ data: vault.seal(o) }), { mode: 0o600 }); cfgCache = o; }; const clientId = () => ENV_ID || loadCfg().clientId || ''; const clientSecret = () => ENV_ID ? ENV_SECRET : loadCfg().clientSecret || ''; -const enabled = () => !!clientId(); +const outlookOn = () => !!clientId(); +const enabled = () => true; // Gmail and iCloud need no server setup; only the Outlook.com button needs the Microsoft app /** How long the saved client secret lasts: { expires, days } or null. */ function secretStatus() { const e = loadCfg().secretExpires; if (!e || ENV_ID) return null; @@ -37,11 +39,25 @@ const readBody = req => new Promise((ok, no) => { const c = []; let n = 0; req.o const json = (res, code, obj) => { res.writeHead(code, { 'content-type': 'application/json', 'cache-control': 'no-store' }); res.end(JSON.stringify(obj)); }; const fail = (status, message) => Object.assign(new Error(message), { status }); -// ---- per-person token file (encrypted) -const file = () => path.join(users.dir(), 'mail.json'); -const load = () => { try { return vault.unseal(JSON.parse(fs.readFileSync(file(), 'utf8')).data) || null; } catch { return null; } }; -const save = o => atomicWrite(file(), JSON.stringify({ data: vault.seal(o) }), { mode: 0o600 }); -const wipe = () => { try { fs.unlinkSync(file()); } catch {} }; +// ---- each person's accounts (encrypted): [{ id, type: 'outlook' | 'gmail' | 'icloud', address, name, ...credentials }] +const accFile = () => path.join(users.dir(), 'mail-accounts.json'); +const legacyFile = () => path.join(users.dir(), 'mail.json'); // one Outlook.com account, from before there were several +const accounts = () => { + try { return (vault.unseal(JSON.parse(fs.readFileSync(accFile(), 'utf8')).data) || {}).accounts || []; } catch {} + try { // migrate the old single-account file + const old = vault.unseal(JSON.parse(fs.readFileSync(legacyFile(), 'utf8')).data); + if (old) { const list = [{ id: crypto.randomBytes(3).toString('hex'), type: 'outlook', ...old }]; saveAccounts(list); fs.rmSync(legacyFile(), { force: true }); return list; } + } catch {} + return []; +}; +const saveAccounts = list => atomicWrite(accFile(), JSON.stringify({ data: vault.seal({ accounts: list }) }), { mode: 0o600 }); +const putAccount = a => { const list = accounts(), i = list.findIndex(x => x.id === a.id); if (i >= 0) list[i] = a; else list.push(a); saveAccounts(list); return a; }; +const dropAccount = id => { saveAccounts(accounts().filter(x => x.id !== id)); imap.forget(id); folderCache.clear(); }; +const findAccount = id => { const list = accounts(); const a = id ? list.find(x => x.id === id) : list[0]; if (!a) throw fail(id ? 404 : 409, id ? 'That account is not connected' : 'Mail is not connected'); return a; }; +const publicAcct = a => ({ id: a.id, type: a.type, address: a.address || '', name: a.name || '' }); +/** "~" -> [account, native id]. An id with no account (older links) belongs to the first account. */ +const unpack = v => { const s = String(v || ''), i = s.indexOf('~'); return i > 0 ? [findAccount(s.slice(0, i)), s.slice(i + 1)] : [findAccount(), s]; }; +const pack = (a, nid) => a.id + '~' + nid; // ---- sign-in with Microsoft (authorization code + PKCE) const states = new Map(); // state -> { uid, verifier, redirect, exp } @@ -78,7 +94,8 @@ async function callback(req, res) { if (!t.refresh_token) throw new Error('Microsoft did not allow ongoing access'); await users.runAs(user, async () => { const me = await (await fetch(GRAPH + '/me?$select=displayName,mail,userPrincipalName', { headers: { authorization: 'Bearer ' + t.access_token } })).json().catch(() => ({})); - save({ access: t.access_token, refresh: t.refresh_token, exp: Date.now() + (t.expires_in || 3600) * 1000, address: me.mail || me.userPrincipalName || '', name: me.displayName || '' }); + const address = me.mail || me.userPrincipalName || '', same = accounts().find(x => x.type === 'outlook' && x.address && x.address.toLowerCase() === address.toLowerCase()); + putAccount({ id: same ? same.id : crypto.randomBytes(3).toString('hex'), type: 'outlook', access: t.access_token, refresh: t.refresh_token, exp: Date.now() + (t.expires_in || 3600) * 1000, address, name: me.displayName || '' }); saveCfg({ ...loadCfg(), verifiedAt: Date.now() }); // proof that the app registration works audit.log('mail connected', { who: user.name, detail: me.mail || me.userPrincipalName || '' }, req); }); @@ -86,31 +103,30 @@ async function callback(req, res) { } catch (e) { console.error('[mail] sign-in failed:', e.message); send(400, page('Could not connect', escHtml(friendly(e)))); } } -const refreshing = new Map(); // uid -> promise: parallel requests share one refresh (refresh tokens can rotate) -/** A working access token for the current person, refreshed when needed. `stale` is the token that just failed. */ -async function token(force, stale) { - const st = load(); if (!st) throw fail(409, 'Mail is not connected'); +const refreshing = new Map(); // account id -> promise: parallel requests share one refresh (refresh tokens can rotate) +/** A working access token for an Outlook account, refreshed when needed. `stale` is the token that just failed. */ +async function token(acct, force, stale) { + const st = accounts().find(x => x.id === acct.id); if (!st) throw fail(409, 'Mail is not connected'); if (!force && st.exp - 60e3 > Date.now()) return st.access; if (force && st.access !== stale) return st.access; // another request already refreshed it - const uid = users.current().id; - if (refreshing.has(uid)) return refreshing.get(uid); + if (refreshing.has(st.id)) return refreshing.get(st.id); const job = (async () => { try { const t = await tokenCall({ grant_type: 'refresh_token', refresh_token: st.refresh }); - save({ ...st, access: t.access_token, refresh: t.refresh_token || st.refresh, exp: Date.now() + (t.expires_in || 3600) * 1000 }); + putAccount({ ...st, access: t.access_token, refresh: t.refresh_token || st.refresh, exp: Date.now() + (t.expires_in || 3600) * 1000 }); return t.access_token; } catch (e) { - if (e.code === 'invalid_grant' || e.status === 400 || e.status === 401) { wipe(); throw fail(409, 'Microsoft signed you out of mail. Connect again.'); } + if (e.code === 'invalid_grant' || e.status === 400 || e.status === 401) { dropAccount(st.id); throw fail(409, 'Microsoft signed you out of mail. Connect again.'); } throw fail(502, 'Could not reach Microsoft: ' + e.message); } - })().finally(() => refreshing.delete(uid)); - refreshing.set(uid, job); + })().finally(() => refreshing.delete(st.id)); + refreshing.set(st.id, job); return job; } -async function graph(p, opts = {}, stale) { - const used = await token(stale !== undefined, stale); +async function graph(acct, p, opts = {}, stale) { + const used = await token(acct, stale !== undefined, stale); const r = await fetch(GRAPH + p, { ...opts, headers: { authorization: 'Bearer ' + used, ...(opts.body && typeof opts.body === 'string' ? { 'content-type': 'application/json' } : {}), ...(opts.headers || {}) }, signal: AbortSignal.timeout(30000) }); - if (r.status === 401 && stale === undefined) return graph(p, opts, used); + if (r.status === 401 && stale === undefined) return graph(acct, p, opts, used); if (r.status === 204 || r.status === 202) return null; if (opts.raw && r.ok) return r; const j = await r.json().catch(() => ({})); @@ -125,30 +141,74 @@ const id = v => { if (!ID.test(String(v || ''))) throw fail(400, 'Bad id'); retu const addr = a => a && a.emailAddress ? { name: a.emailAddress.name || '', address: a.emailAddress.address || '' } : { name: '', address: '' }; const row = m => ({ id: m.id, subject: m.subject || '(No subject)', from: addr(m.from), date: m.receivedDateTime, read: !!m.isRead, attach: !!m.hasAttachments, preview: m.bodyPreview || '', flagged: !!(m.flag && m.flag.flagStatus === 'flagged') }); const EMAIL = /^[^\s@<>,;]+@[^\s@<>,;]+\.[^\s@<>,;]+$/; -const people = (list, label) => { const out = []; for (const x of Array.isArray(list) ? list : String(list || '').split(/[,;]/)) { const a = String(x).trim(); if (!a) continue; const m = /<([^>]+)>/.exec(a), e = (m ? m[1] : a).trim(); if (!EMAIL.test(e)) throw fail(400, `“${a}” is not an email address (${label})`); out.push({ emailAddress: { address: e } }); } return out; }; +/** "a@b.co, Ann " -> [{ name, address }], refusing anything that is not an address. */ +const recipients = (list, label) => { const out = []; for (const x of Array.isArray(list) ? list : String(list || '').split(/[,;]/)) { const a = String(x).trim(); if (!a) continue; const m = /^(.*?)\s*<([^>]+)>$/.exec(a), e = (m ? m[2] : a).trim(); if (!EMAIL.test(e)) throw fail(400, `“${a}” is not an email address (${label})`); out.push({ name: m ? m[1].replace(/^"|"$/g, '') : '', address: e }); } return out; }; +const toGraph = list => list.map(p => ({ emailAddress: { address: p.address } })); const textToHtml = t => '
' + escHtml(t).replace(/\r?\n/g, '
') + '
'; const safeName = n => String(n || 'attachment').replace(/[\r\n"\\/]+/g, '_').slice(0, 150); +const cleanAttachments = list => { let total = 0; const out = (Array.isArray(list) ? list : []).slice(0, 10).map(a => { const data = String(a.data || ''); total += Buffer.from(data, 'base64').length; return { name: safeName(a.name), type: String(a.type || 'application/octet-stream').slice(0, 100), data }; }); if (total > MAX_ATTACH) throw fail(400, 'Attachments can total about 3 MB'); return out; }; -let folderCache = new Map(); // uid -> { t, v } -async function folders() { - const uid = users.current().id, c = folderCache.get(uid); - if (c && Date.now() - c.t < 15e3) return c.v; - const sel = '?$select=id,displayName,unreadItemCount,totalItemCount'; - const wk = await Promise.all(Object.keys(WELL).map(k => graph(`/me/mailFolders/${k}${sel}`).then(f => ({ id: k, realId: f.id, name: WELL[k], unread: f.unreadItemCount || 0, total: f.totalItemCount || 0, well: true })).catch(() => null))); - const known = new Set(wk.filter(Boolean).map(f => f.realId)); - const rest = (await graph(`/me/mailFolders${sel}&$top=100`).catch(() => ({ value: [] }))).value || []; - const v = [...wk.filter(Boolean).map(({ realId, ...f }) => f), ...rest.filter(f => !known.has(f.id) && !['Outbox', 'Conversation History', 'Sync Issues', 'RSS Feeds'].includes(f.displayName)).map(f => ({ id: f.id, name: f.displayName, unread: f.unreadItemCount || 0, total: f.totalItemCount || 0 }))]; - folderCache.set(uid, { t: Date.now(), v }); return v; -} -const dropFolderCache = () => folderCache.delete(users.current().id); +let folderCache = new Map(); // account id -> { t, v } +const dropFolderCache = id => id ? folderCache.delete(id) : folderCache.clear(); -async function withInline(html, attachments, mid) { +// ---- Outlook.com through Microsoft Graph. Every function takes the account and Graph's own ids. +const outlook = { + async folders(a) { + const c = folderCache.get(a.id); if (c && Date.now() - c.t < 15e3) return c.v; + const sel = '?$select=id,displayName,unreadItemCount,totalItemCount'; + const wk = await Promise.all(Object.keys(WELL).map(k => graph(a, `/me/mailFolders/${k}${sel}`).then(f => ({ id: k, realId: f.id, name: WELL[k], unread: f.unreadItemCount || 0, total: f.totalItemCount || 0, well: true })).catch(() => null))); + const known = new Set(wk.filter(Boolean).map(f => f.realId)); + const rest = (await graph(a, `/me/mailFolders${sel}&$top=100`).catch(() => ({ value: [] }))).value || []; + const v = [...wk.filter(Boolean).map(({ realId, ...f }) => f), ...rest.filter(f => !known.has(f.id) && !['Outbox', 'Conversation History', 'Sync Issues', 'RSS Feeds'].includes(f.displayName)).map(f => ({ id: f.id, name: f.displayName, unread: f.unreadItemCount || 0, total: f.totalItemCount || 0 }))]; + folderCache.set(a.id, { t: Date.now(), v }); return v; + }, + async list(a, { folder, skip, q }) { + const fid = id(folder || 'inbox'), sel = '$select=id,subject,from,receivedDateTime,isRead,hasAttachments,bodyPreview,flag'; + const r = await graph(a, q ? `/me/mailFolders/${fid}/messages?${sel}&$top=50&$search="${encodeURIComponent(q)}"` : `/me/mailFolders/${fid}/messages?${sel}&$top=30&$skip=${skip}&$orderby=receivedDateTime desc`.replace(/ /g, '%20')); + const items = (r.value || []).map(row); if (q) items.sort((x, y) => String(y.date).localeCompare(String(x.date))); + return { items, more: !q && !!r['@odata.nextLink'], next: skip + items.length }; + }, + async message(a, mid) { + const m = await graph(a, `/me/messages/${id(mid)}?$select=id,subject,from,toRecipients,ccRecipients,receivedDateTime,isRead,body,hasAttachments,flag,parentFolderId&$expand=attachments($select=id,name,contentType,size,isInline)`); + let html = m.body && m.body.contentType === 'html' ? m.body.content || '' : textToHtml((m.body && m.body.content) || ''); + html = await withInline(a, html, m.attachments, mid); + return { id: m.id, subject: m.subject || '(No subject)', from: addr(m.from), to: (m.toRecipients || []).map(addr), cc: (m.ccRecipients || []).map(addr), date: m.receivedDateTime, read: !!m.isRead, flagged: !!(m.flag && m.flag.flagStatus === 'flagged'), + html, attachments: (m.attachments || []).filter(x => !x.isInline).map(x => ({ id: x.id, name: x.name, type: x.contentType, size: x.size })) }; + }, + async attachment(a, mid, aid) { + const meta = await graph(a, `/me/messages/${id(mid)}/attachments/${id(aid)}?$select=name,contentType,size`); + const r = await graph(a, `/me/messages/${id(mid)}/attachments/${id(aid)}/$value`, { raw: true }); + return { name: meta.name, type: meta.contentType, buffer: Buffer.from(await r.arrayBuffer()) }; + }, + async update(a, mid, { read, flagged }) { + const patch = {}; if (typeof read === 'boolean') patch.isRead = read; if (typeof flagged === 'boolean') patch.flag = { flagStatus: flagged ? 'flagged' : 'notFlagged' }; + if (!Object.keys(patch).length) throw fail(400, 'Nothing to change'); + await graph(a, `/me/messages/${id(mid)}`, { method: 'PATCH', body: JSON.stringify(patch) }); + }, + async move(a, mid, to) { await graph(a, `/me/messages/${id(mid)}/move`, { method: 'POST', body: JSON.stringify({ destinationId: decodeURIComponent(id(to)) }) }); }, + async del(a, mid, permanent) { + if (permanent) await graph(a, `/me/messages/${id(mid)}`, { method: 'DELETE' }); + else await graph(a, `/me/messages/${id(mid)}/move`, { method: 'POST', body: JSON.stringify({ destinationId: 'deleteditems' }) }); + }, + async send(a, { to, cc, bcc, subject, body, attachments }) { + await graph(a, '/me/sendMail', { method: 'POST', body: JSON.stringify({ message: { subject, body: { contentType: 'HTML', content: textToHtml(body) }, toRecipients: toGraph(to), ccRecipients: toGraph(cc), bccRecipients: toGraph(bcc), + attachments: attachments.map(x => ({ '@odata.type': '#microsoft.graph.fileAttachment', name: x.name, contentType: x.type, contentBytes: x.data })) }, saveToSentItems: true }) }); + }, + async reply(a, { id: mid, mode, comment, to }) { + const body = { comment }; if (mode === 'forward') body.toRecipients = toGraph(to); + await graph(a, `/me/messages/${id(mid)}/${mode}`, { method: 'POST', body: JSON.stringify(body) }); + }, + async unread(a) { return (await graph(a, '/me/mailFolders/inbox?$select=unreadItemCount')).unreadItemCount || 0; }, +}; +const providerOf = a => a.type === 'outlook' ? outlook : imap; + +async function withInline(a, html, attachments, mid) { // (contentId only exists on file attachments, so it cannot be selected in the list: fetch each small inline picture whole) - const inl = (attachments || []).filter(a => a.isInline && /^image\//i.test(a.contentType || '') && (a.size || 0) < 1.5 * 1024 * 1024).slice(0, 15); - for (const a of inl) { + const inl = (attachments || []).filter(x => x.isInline && /^image\//i.test(x.contentType || '') && (x.size || 0) < 1.5 * 1024 * 1024).slice(0, 15); + for (const x of inl) { try { - const full = await graph(`/me/messages/${id(mid)}/attachments/${id(a.id)}`); - if (full && full.contentBytes && full.contentId) html = html.split('cid:' + String(full.contentId).replace(/^<|>$/g, '')).join(`data:${full.contentType || a.contentType};base64,${full.contentBytes}`); + const full = await graph(a, `/me/messages/${id(mid)}/attachments/${id(x.id)}`); + if (full && full.contentBytes && full.contentId) html = html.split('cid:' + String(full.contentId).replace(/^<|>$/g, '')).join(`data:${full.contentType || x.contentType};base64,${full.contentBytes}`); } catch { /* leave that picture out */ } } return html; @@ -186,72 +246,77 @@ async function adminHandle(req, res, p) { json(res, 404, { error: 'Not found' }); } +/** Every unread count in one go (an account that cannot be reached is just left out of the total). */ +async function overview() { + const list = accounts(); + const rows = await Promise.all(list.map(async a => { + try { return { ...publicAcct(a), unread: await providerOf(a).unread(a) }; } + catch (e) { return { ...publicAcct(a), unread: null, error: e.message, expired: e.status === 409 }; } + })); + const live = accounts(); // (an account whose sign-in Microsoft revoked was dropped meanwhile) + const shown = rows.filter(r => live.some(x => x.id === r.id) || !r.expired); + const expired = rows.some(r => r.expired); + return { connected: shown.length > 0, accounts: shown, expired: expired && !shown.length, address: (shown[0] || {}).address, name: (shown[0] || {}).name, unread: shown.reduce((n, r) => n + (r.unread || 0), 0) }; +} + async function handle(req, res) { const u = new URL(req.url, 'http://x'), p = u.pathname, q = u.searchParams; try { if (p === '/api/mail/admin' || p.startsWith('/api/mail/admin/')) return await adminHandle(req, res, p); - if (p === '/api/mail/status') { - const st = load(); - if (!st) return json(res, 200, { connected: false }); - let unread = null; try { unread = (await graph('/me/mailFolders/inbox?$select=unreadItemCount')).unreadItemCount; } catch (e) { if (e.status === 409) return json(res, 200, { connected: false, expired: true }); } - return json(res, 200, { connected: true, address: st.address, name: st.name, unread }); + if (p === '/api/mail/status') return json(res, 200, { ...(await overview()), outlook: outlookOn(), providers: imap.PRESETS }); + if (p === '/api/mail/connect' && req.method === 'GET') { if (!outlookOn()) throw fail(403, 'Outlook.com is not set up on this server (an admin adds the Microsoft app in Settings)'); return json(res, 200, { url: startUrl(req), redirect: redirectUri(req) }); } + if (p === '/api/mail/folders') { + const a = findAccount(q.get('acct') || undefined); + return json(res, 200, (await providerOf(a).folders(a)).map(f => ({ ...f, id: pack(a, f.id) }))); } - if (p === '/api/mail/connect' && req.method === 'GET') return json(res, 200, { url: startUrl(req), redirect: redirectUri(req) }); - if (p === '/api/mail/disconnect' && req.method === 'POST') { wipe(); dropFolderCache(); audit.log('mail disconnected', { who: users.current().name }, req); return json(res, 200, { ok: true }); } - if (p === '/api/mail/folders') return json(res, 200, await folders()); if (p === '/api/mail/messages') { - const folder = id(q.get('folder') || 'inbox'), search = (q.get('q') || '').replace(/["\\]/g, ' ').trim().slice(0, 100), skip = Math.max(0, parseInt(q.get('skip'), 10) || 0); - const sel = '$select=id,subject,from,receivedDateTime,isRead,hasAttachments,bodyPreview,flag'; - const r = await graph(search ? `/me/mailFolders/${folder}/messages?${sel}&$top=50&$search="${encodeURIComponent(search)}"` : `/me/mailFolders/${folder}/messages?${sel}&$top=30&$skip=${skip}&$orderby=receivedDateTime desc`.replace(/ /g, '%20')); - const items = (r.value || []).map(row); if (search) items.sort((a, b) => String(b.date).localeCompare(String(a.date))); - return json(res, 200, { items, more: !search && !!r['@odata.nextLink'], next: skip + items.length }); + const [a, fid] = unpack(q.get('folder') || 'inbox'), search = (q.get('q') || '').replace(/["\\]/g, ' ').trim().slice(0, 100), skip = Math.max(0, parseInt(q.get('skip'), 10) || 0); + const r = await providerOf(a).list(a, { folder: fid || 'inbox', skip, q: search }); + return json(res, 200, { ...r, items: r.items.map(m => ({ ...m, id: pack(a, m.id) })) }); } if (p === '/api/mail/message') { - const mid = q.get('id'); - const m = await graph(`/me/messages/${id(mid)}?$select=id,subject,from,toRecipients,ccRecipients,receivedDateTime,isRead,body,hasAttachments,flag,parentFolderId&$expand=attachments($select=id,name,contentType,size,isInline)`); - let html = m.body && m.body.contentType === 'html' ? m.body.content || '' : textToHtml((m.body && m.body.content) || ''); - html = await withInline(html, m.attachments, mid); - return json(res, 200, { id: m.id, subject: m.subject || '(No subject)', from: addr(m.from), to: (m.toRecipients || []).map(addr), cc: (m.ccRecipients || []).map(addr), date: m.receivedDateTime, read: !!m.isRead, flagged: !!(m.flag && m.flag.flagStatus === 'flagged'), - html, attachments: (m.attachments || []).filter(a => !a.isInline).map(a => ({ id: a.id, name: a.name, type: a.contentType, size: a.size })) }); + const [a, nid] = unpack(q.get('id')), m = await providerOf(a).message(a, nid); + return json(res, 200, { ...m, id: pack(a, m.id), account: publicAcct(a) }); } if (p === '/api/mail/attachment') { - const mid = q.get('id'), aid = q.get('aid'); - const meta = await graph(`/me/messages/${id(mid)}/attachments/${id(aid)}?$select=name,contentType,size`); - const r = await graph(`/me/messages/${id(mid)}/attachments/${id(aid)}/$value`, { raw: true }); - res.writeHead(200, { 'content-type': 'application/octet-stream', 'content-disposition': `attachment; filename="${safeName(meta.name)}"; filename*=UTF-8''${encodeURIComponent(safeName(meta.name))}`, 'cache-control': 'private, no-store', 'x-content-type-options': 'nosniff' }); - return res.end(Buffer.from(await r.arrayBuffer())); + const [a, nid] = unpack(q.get('id')), f = await providerOf(a).attachment(a, nid, q.get('aid')); + res.writeHead(200, { 'content-type': 'application/octet-stream', 'content-disposition': `attachment; filename="${safeName(f.name)}"; filename*=UTF-8''${encodeURIComponent(safeName(f.name))}`, 'cache-control': 'private, no-store', 'x-content-type-options': 'nosniff' }); + return res.end(f.buffer); } if (req.method !== 'POST') return json(res, 405, { error: 'Method not allowed' }); - const b = JSON.parse((await readBody(req)) || '{}'); - if (p === '/api/mail/update') { - const patch = {}; if (typeof b.read === 'boolean') patch.isRead = b.read; if (typeof b.flagged === 'boolean') patch.flag = { flagStatus: b.flagged ? 'flagged' : 'notFlagged' }; - if (!Object.keys(patch).length) throw fail(400, 'Nothing to change'); - await graph(`/me/messages/${id(b.id)}`, { method: 'PATCH', body: JSON.stringify(patch) }); dropFolderCache(); return json(res, 200, { ok: true }); + const b = JSON.parse((await readBody(req)) || '{}'), me = users.current(); + if (p === '/api/mail/accounts') { // add a Gmail or iCloud account (an app password; checked against the server before it is saved) + const a = await imap.connectNew({ type: b.type, address: b.address, password: b.password }); + const same = accounts().find(x => x.type === a.type && x.address.toLowerCase() === a.address.toLowerCase()); + const saved = putAccount({ ...a, id: same ? same.id : crypto.randomBytes(3).toString('hex') }); dropFolderCache(); + audit.log('mail connected', { who: me.name, detail: a.address }, req); + return json(res, 200, { ok: true, account: publicAcct(saved) }); } - if (p === '/api/mail/move') { await graph(`/me/messages/${id(b.id)}/move`, { method: 'POST', body: JSON.stringify({ destinationId: decodeURIComponent(id(b.to)) }) }); dropFolderCache(); return json(res, 200, { ok: true }); } - if (p === '/api/mail/delete') { - if (b.permanent) await graph(`/me/messages/${id(b.id)}`, { method: 'DELETE' }); - else await graph(`/me/messages/${id(b.id)}/move`, { method: 'POST', body: JSON.stringify({ destinationId: 'deleteditems' }) }); - dropFolderCache(); return json(res, 200, { ok: true }); + if (p === '/api/mail/disconnect') { + const list = accounts(), gone = b.acct ? list.filter(x => x.id === b.acct) : list; + if (b.acct && !gone.length) throw fail(404, 'That account is not connected'); + for (const x of gone) dropAccount(x.id); + audit.log('mail disconnected', { who: me.name, detail: gone.map(x => x.address).join(', ') }, req); return json(res, 200, { ok: true }); } + if (p === '/api/mail/update') { const [a, nid] = unpack(b.id); await providerOf(a).update(a, nid, { read: b.read, flagged: b.flagged }); dropFolderCache(a.id); return json(res, 200, { ok: true }); } + if (p === '/api/mail/move') { const [a, nid] = unpack(b.id), dest = String(b.to || ''); await providerOf(a).move(a, nid, dest.includes('~') ? dest.slice(dest.indexOf('~') + 1) : dest); dropFolderCache(a.id); return json(res, 200, { ok: true }); } + if (p === '/api/mail/delete') { const [a, nid] = unpack(b.id); await providerOf(a).del(a, nid, !!b.permanent); dropFolderCache(a.id); return json(res, 200, { ok: true }); } if (p === '/api/mail/send') { - const to = people(b.to, 'To'), cc = people(b.cc, 'Cc'), bcc = people(b.bcc, 'Bcc'); + const a = findAccount(b.acct || undefined), to = recipients(b.to, 'To'), cc = recipients(b.cc, 'Cc'), bcc = recipients(b.bcc, 'Bcc'); if (!to.length && !cc.length && !bcc.length) throw fail(400, 'Add at least one recipient'); - const subject = String(b.subject || '').slice(0, 255), text = String(b.body || ''); if (text.length > 200000) throw fail(400, 'That message is too long'); - let total = 0; const attachments = (Array.isArray(b.attachments) ? b.attachments : []).slice(0, 10).map(a => { const bytes = Buffer.from(String(a.data || ''), 'base64').length; total += bytes; return { '@odata.type': '#microsoft.graph.fileAttachment', name: safeName(a.name), contentType: String(a.type || 'application/octet-stream').slice(0, 100), contentBytes: String(a.data || '') }; }); - if (total > MAX_ATTACH) throw fail(400, 'Attachments can total about 3 MB'); - await graph('/me/sendMail', { method: 'POST', body: JSON.stringify({ message: { subject, body: { contentType: 'HTML', content: textToHtml(text) }, toRecipients: to, ccRecipients: cc, bccRecipients: bcc, attachments }, saveToSentItems: true }) }); - audit.log('mail sent', { who: users.current().name, detail: `${to.length + cc.length + bcc.length} recipient(s)` }, req); dropFolderCache(); return json(res, 200, { ok: true }); + const subject = String(b.subject || '').slice(0, 255), body = String(b.body || ''); if (body.length > 200000) throw fail(400, 'That message is too long'); + await providerOf(a).send(a, { to, cc, bcc, subject, body, attachments: cleanAttachments(b.attachments) }); + audit.log('mail sent', { who: me.name, detail: `${to.length + cc.length + bcc.length} recipient(s)` }, req); dropFolderCache(a.id); return json(res, 200, { ok: true }); } if (p === '/api/mail/reply') { const mode = { reply: 'reply', replyAll: 'replyAll', forward: 'forward' }[b.mode]; if (!mode) throw fail(400, 'Unknown action'); - const body = { comment: String(b.comment || '').slice(0, 200000) }; - if (mode === 'forward') { body.toRecipients = people(b.to, 'To'); if (!body.toRecipients.length) throw fail(400, 'Add at least one recipient'); } - await graph(`/me/messages/${id(b.id)}/${mode}`, { method: 'POST', body: JSON.stringify(body) }); - audit.log('mail sent', { who: users.current().name, detail: mode }, req); dropFolderCache(); return json(res, 200, { ok: true }); + const [a, nid] = unpack(b.id), to = mode === 'forward' ? recipients(b.to, 'To') : []; + if (mode === 'forward' && !to.length) throw fail(400, 'Add at least one recipient'); + await providerOf(a).reply(a, { id: nid, mode, comment: String(b.comment || '').slice(0, 200000), to }); + audit.log('mail sent', { who: me.name, detail: mode }, req); dropFolderCache(a.id); return json(res, 200, { ok: true }); } json(res, 404, { error: 'Not found' }); } catch (e) { json(res, e.status || 400, { error: e.message }); } } -module.exports = { enabled, handle, callback, secretStatus }; +module.exports = { enabled, outlookOn, handle, callback, secretStatus, recipients, textToHtml }; diff --git a/package-lock.json b/package-lock.json index 376cb99..4b930f0 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,7 +10,14 @@ "license": "ISC", "dependencies": { "@simplewebauthn/server": "^14.0.3", + "imapflow": "^2.9.0", + "mailparser": "^3.12.2", + "nodemailer": "^10.2.0", "web-push": "^3.6.7" + }, + "devDependencies": { + "hoodiecrow-imap": "^3.3.1", + "smtp-server": "^3.21.0" } }, "node_modules/@hexagon/base64": { @@ -258,6 +265,28 @@ "node": ">=20.0.0" } }, + "node_modules/@pinojs/redact": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/@pinojs/redact/-/redact-0.4.0.tgz", + "integrity": "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==", + "license": "MIT" + }, + "node_modules/@selderee/plugin-htmlparser2": { + "version": "0.12.0", + "resolved": "https://registry.npmjs.org/@selderee/plugin-htmlparser2/-/plugin-htmlparser2-0.12.0.tgz", + "integrity": "sha512-oELmoyA6ML9jDRMV3kgcMQFKxUfBU0yFVn6yTctVaLT5ygXnxH52I3TZEgV9EhXJC68/uFvE5Daj1/25c0Xa/A==", + "license": "MIT", + "dependencies": { + "domelementtype": "~2.3.0", + "domhandler": "~5.0.3" + }, + "funding": { + "url": "https://github.com/sponsors/KillyMXI" + }, + "peerDependencies": { + "selderee": "~0.12.0" + } + }, "node_modules/@simplewebauthn/server": { "version": "14.0.3", "resolved": "https://registry.npmjs.org/@simplewebauthn/server/-/server-14.0.3.tgz", @@ -279,6 +308,20 @@ "node": ">=20.0.0" } }, + "node_modules/@zone-eu/mailsplit": { + "version": "5.4.27", + "resolved": "https://registry.npmjs.org/@zone-eu/mailsplit/-/mailsplit-5.4.27.tgz", + "integrity": "sha512-9008ddFqqKk7lMF2S1sLU7ExfJZP9zPw5tkZy24EVt45Ryme2x85f/5vJZZqQ+vCJkKcCY78dJ5yuEVG3ZrVZA==", + "license": "(MIT OR EUPL-1.1+)", + "dependencies": { + "libbase64": "1.4.1", + "libmime": "5.5.2", + "libqp": "3.0.1" + }, + "engines": { + "node": ">=20.0.0" + } + }, "node_modules/agent-base": { "version": "7.1.4", "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", @@ -314,6 +357,15 @@ "node": ">=12.0.0" } }, + "node_modules/atomic-sleep": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/atomic-sleep/-/atomic-sleep-1.0.0.tgz", + "integrity": "sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==", + "license": "MIT", + "engines": { + "node": ">=8.0.0" + } + }, "node_modules/bn.js": { "version": "4.12.5", "resolved": "https://registry.npmjs.org/bn.js/-/bn.js-4.12.5.tgz", @@ -343,6 +395,80 @@ } } }, + "node_modules/deepmerge-ts": { + "version": "8.0.2", + "resolved": "https://registry.npmjs.org/deepmerge-ts/-/deepmerge-ts-8.0.2.tgz", + "integrity": "sha512-uqbvqLUMrc6p0MO+WBRtTxY55hmyh94WRwI5a++PZe54X+bfVh59FSN7uWCBCW1CCVjzjnrwzfI8zidE2obMMw==", + "funding": [ + { + "type": "ko-fi", + "url": "https://ko-fi.com/rebeccastevens" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/deepmerge-ts" + } + ], + "license": "BSD-3-Clause", + "engines": { + "node": ">=16.9.0" + } + }, + "node_modules/dom-serializer": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-2.0.0.tgz", + "integrity": "sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==", + "license": "MIT", + "dependencies": { + "domelementtype": "^2.3.0", + "domhandler": "^5.0.2", + "entities": "^4.2.0" + }, + "funding": { + "url": "https://github.com/cheeriojs/dom-serializer?sponsor=1" + } + }, + "node_modules/domelementtype": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/domelementtype/-/domelementtype-2.3.0.tgz", + "integrity": "sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fb55" + } + ], + "license": "BSD-2-Clause" + }, + "node_modules/domhandler": { + "version": "5.0.3", + "resolved": "https://registry.npmjs.org/domhandler/-/domhandler-5.0.3.tgz", + "integrity": "sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==", + "license": "BSD-2-Clause", + "dependencies": { + "domelementtype": "^2.3.0" + }, + "engines": { + "node": ">= 4" + }, + "funding": { + "url": "https://github.com/fb55/domhandler?sponsor=1" + } + }, + "node_modules/domutils": { + "version": "3.2.2", + "resolved": "https://registry.npmjs.org/domutils/-/domutils-3.2.2.tgz", + "integrity": "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==", + "license": "BSD-2-Clause", + "dependencies": { + "dom-serializer": "^2.0.0", + "domelementtype": "^2.3.0", + "domhandler": "^5.0.3" + }, + "funding": { + "url": "https://github.com/fb55/domutils?sponsor=1" + } + }, "node_modules/ecdsa-sig-formatter": { "version": "1.0.11", "resolved": "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz", @@ -352,6 +478,129 @@ "safe-buffer": "^5.0.1" } }, + "node_modules/encoding-japanese": { + "version": "2.4.0", + "resolved": "https://registry.npmjs.org/encoding-japanese/-/encoding-japanese-2.4.0.tgz", + "integrity": "sha512-PE0KjlsDy9JSsoU4C2Q8NXb1kC4c+bZRac9ZvHXUE4q1Ht63gScKMdEy9jst9+Fi3M4eDkqzkEhBZvix4blP2A==", + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/entities": { + "version": "4.5.0", + "resolved": "https://registry.npmjs.org/entities/-/entities-4.5.0.tgz", + "integrity": "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==", + "license": "BSD-2-Clause", + "engines": { + "node": ">=0.12" + }, + "funding": { + "url": "https://github.com/fb55/entities?sponsor=1" + } + }, + "node_modules/he": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/he/-/he-1.2.0.tgz", + "integrity": "sha512-F/1DnUGPopORZi0ni+CvrCgHQ5FyEAHRLSApuYWMmrbSwoN2Mn/7k+Gl38gJnR7yyDZk6WLXwiGod1JOWNDKGw==", + "license": "MIT", + "bin": { + "he": "bin/he" + } + }, + "node_modules/hoodiecrow-imap": { + "version": "3.3.1", + "resolved": "https://registry.npmjs.org/hoodiecrow-imap/-/hoodiecrow-imap-3.3.1.tgz", + "integrity": "sha512-0o+wjiIAkRZDJDHkays3mZZ92Qd1dmikPfOO8qbZ7QczIOCPBxvPJlkNjem3DzZkAB5O34uHS3u25yNS4YzV/w==", + "deprecated": "Renamed to imapkit, see https://imapkit.com", + "dev": true, + "license": "MIT", + "dependencies": { + "imap-handler": "1.3.1", + "smtp-server": "3.19.17" + }, + "bin": { + "hoodiecrow": "bin/hoodiecrow.js" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/hoodiecrow-imap/node_modules/nodemailer": { + "version": "10.0.14", + "resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.0.14.tgz", + "integrity": "sha512-eJoLFKg55fesSlzUekmfOv/SBXh4KUYT8oiQc2RLP0tfG0rbNMJBK7wgHJwXTJwqyKlSJMz9UzDZ4v+aHEvo8g==", + "dev": true, + "license": "MIT-0", + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/hoodiecrow-imap/node_modules/smtp-server": { + "version": "3.19.17", + "resolved": "https://registry.npmjs.org/smtp-server/-/smtp-server-3.19.17.tgz", + "integrity": "sha512-/9RwyNTcR8osMQWOvC8NV8l57ddPyYXvuG+wa1Ua4cD/I41z7zCMsGtXLa9vxlrggjRrfejr50X0WS/9NuVWqQ==", + "dev": true, + "license": "MIT-0", + "dependencies": { + "ipv6-normalize": "1.0.1", + "nodemailer": "10.0.14", + "punycode.js": "2.3.1" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/html-to-text": { + "version": "10.0.1", + "resolved": "https://registry.npmjs.org/html-to-text/-/html-to-text-10.0.1.tgz", + "integrity": "sha512-GiVhRI1BatGARSCmlXWNCjDT0cWrwBWoeduLoV0WSKAgaV/wa+hUWy5LiQLUs4UwiUrE52ZCMfBGiKD87TDPrg==", + "license": "MIT", + "dependencies": { + "@selderee/plugin-htmlparser2": "~0.12.0", + "deepmerge-ts": "^8.0.1", + "dom-serializer": "^2.0.0", + "htmlparser2": "^10.1.0", + "selderee": "~0.12.0" + }, + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/sponsors/KillyMXI" + } + }, + "node_modules/htmlparser2": { + "version": "10.1.0", + "resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-10.1.0.tgz", + "integrity": "sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ==", + "funding": [ + "https://github.com/fb55/htmlparser2?sponsor=1", + { + "type": "github", + "url": "https://github.com/sponsors/fb55" + } + ], + "license": "MIT", + "dependencies": { + "domelementtype": "^2.3.0", + "domhandler": "^5.0.3", + "domutils": "^3.2.2", + "entities": "^7.0.1" + } + }, + "node_modules/htmlparser2/node_modules/entities": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/entities/-/entities-7.0.1.tgz", + "integrity": "sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==", + "license": "BSD-2-Clause", + "engines": { + "node": ">=0.12" + }, + "funding": { + "url": "https://github.com/fb55/entities?sponsor=1" + } + }, "node_modules/http_ece": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/http_ece/-/http_ece-1.2.0.tgz", @@ -374,12 +623,73 @@ "node": ">= 14" } }, + "node_modules/iconv-lite": { + "version": "0.7.3", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.3.tgz", + "integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/imap-handler": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/imap-handler/-/imap-handler-1.3.1.tgz", + "integrity": "sha512-/VHETxcR5VMzLAxvb4CQw5d7hCTe4MlcvBlRoKO6z/7cEF650ha8Lwq1EbmUMutrxmb+DMB6piHbANv2vHQUAQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/imapflow": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/imapflow/-/imapflow-2.9.0.tgz", + "integrity": "sha512-8drBcDmYhDNkMHzoXVd46/XnZg3krcZyLsyOyVuAxTQPPKUc0u/bpb3ocl36u2b9ZD/8J6rz2YdUbzHyaj1Pmw==", + "license": "MIT", + "dependencies": { + "@zone-eu/mailsplit": "5.4.27", + "encoding-japanese": "2.4.0", + "iconv-lite": "0.7.3", + "libbase64": "1.4.1", + "libmime": "5.5.2", + "libqp": "3.0.1", + "pino": "10.3.1", + "socks": "2.8.10" + }, + "engines": { + "node": ">=20.0.0" + } + }, "node_modules/inherits": { "version": "2.0.4", "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", "license": "ISC" }, + "node_modules/ip-address": { + "version": "10.7.4", + "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.7.4.tgz", + "integrity": "sha512-DWBMKyeM25D70nrIiTkgQth59/G+naKOTep6huF9j+GmeK3DMD60Zvxg5d5qQnz+38AHgou6ean2LECX+lrKDg==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, + "node_modules/ipv6-normalize": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/ipv6-normalize/-/ipv6-normalize-1.0.1.tgz", + "integrity": "sha512-Bm6H79i01DjgGTCWjUuCjJ6QDo1HB96PT/xCYuyJUP9WFbVDrLSbG4EZCvOCun2rNswZb0c3e4Jt/ws795esHA==", + "dev": true, + "license": "MIT" + }, "node_modules/jwa": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/jwa/-/jwa-2.0.1.tgz", @@ -401,6 +711,110 @@ "safe-buffer": "^5.0.1" } }, + "node_modules/leac": { + "version": "0.7.0", + "resolved": "https://registry.npmjs.org/leac/-/leac-0.7.0.tgz", + "integrity": "sha512-qMrZeyEekgdRQ9o6a4NAB2EQZrv827GJdn1vnapwSJ90hWRB4TzUSunvacPkxQ2TnNqHNI1/zSt0hlo0crG8Jw==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/KillyMXI" + } + }, + "node_modules/libbase64": { + "version": "1.4.1", + "resolved": "https://registry.npmjs.org/libbase64/-/libbase64-1.4.1.tgz", + "integrity": "sha512-Nyh3H90B3gRtrcxIe56Ayy7635J0jmh6AbxbQgQae8wrSPQMH5wFqJChiMTB13yKG512L3YspbA5DzumhFTITA==", + "license": "MIT", + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/libmime": { + "version": "5.5.2", + "resolved": "https://registry.npmjs.org/libmime/-/libmime-5.5.2.tgz", + "integrity": "sha512-Byw3CchK4eSqbgzpttsUoC4hx2VPP+hnveAuYAYZNipEc+0WlDFjVb1ISNscjq/idvyDP/Xjg8mnBcdgbsgvzg==", + "license": "MIT", + "dependencies": { + "encoding-japanese": "2.4.0", + "iconv-lite": "0.7.3", + "libbase64": "1.4.1", + "libqp": "3.0.1" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/libqp": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/libqp/-/libqp-3.0.1.tgz", + "integrity": "sha512-7QZxpPX25pga1WiMueduWxmrHMVBFrZPH6ZUaozGYa/H98JH6NVBSET3qVw+4NxzoraIW7rJ81YxSU9IalvCpg==", + "license": "MIT", + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/linkify-it": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/linkify-it/-/linkify-it-5.0.2.tgz", + "integrity": "sha512-ONTm2jCMAVZjgQa/Fy1kScXsuOoF5NPTsoFBdE1KVIZ2vAh/r9+Bqo+0jINCBYnavTPQZz38QzFTme79ENoN3Q==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/puzrin" + }, + { + "type": "github", + "url": "https://github.com/sponsors/markdown-it" + } + ], + "license": "MIT", + "dependencies": { + "uc.micro": "^2.0.0" + } + }, + "node_modules/mailparser": { + "version": "3.12.2", + "resolved": "https://registry.npmjs.org/mailparser/-/mailparser-3.12.2.tgz", + "integrity": "sha512-JbOt+AjJIr7+bhMp8g7ILevleoDAD0bXVlAJmikrz0sbQqmt88YELKDwYQL5JNugkV9lcN+DFv32dyhPsvqpHQ==", + "license": "MIT", + "dependencies": { + "@zone-eu/mailsplit": "5.4.27", + "he": "1.2.0", + "html-to-text": "10.0.1", + "libmime": "5.6.0", + "linkify-it": "5.0.2", + "nodemailer": "10.1.7", + "punycode.js": "2.3.1", + "tlds": "1.261.0" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/mailparser/node_modules/libmime": { + "version": "5.6.0", + "resolved": "https://registry.npmjs.org/libmime/-/libmime-5.6.0.tgz", + "integrity": "sha512-b5HR41QUt1whlvLnY3AhAYwba56DDiNv1xVTH5viVq86/m+wvW+HeooLx6x+MdNapho9ki7Lo/fTxuqwhEs/sQ==", + "license": "MIT", + "dependencies": { + "encoding-japanese": "2.4.0", + "iconv-lite": "0.7.3", + "libbase64": "1.4.1", + "libqp": "3.0.1" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/mailparser/node_modules/nodemailer": { + "version": "10.1.7", + "resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.1.7.tgz", + "integrity": "sha512-OQa6FDln5fLttmqSbjDiRYvFDOFNrBqfiOwHPlEbURil3b+vPB1v8xpi+Sp46drzNDlsTnNsS502vthmz56r4w==", + "license": "MIT-0", + "engines": { + "node": ">=20.0.0" + } + }, "node_modules/minimalistic-assert": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/minimalistic-assert/-/minimalistic-assert-1.0.1.tgz", @@ -422,6 +836,108 @@ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "license": "MIT" }, + "node_modules/nodemailer": { + "version": "10.2.0", + "resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.2.0.tgz", + "integrity": "sha512-/XyGw6kX2stMdKGUJdAQmYAeY1FscWfdRydfd2NtK3A3KDfUUXCqZzExSNAt2xhh90X2qLCR+o2Gd48YPJcJwA==", + "license": "MIT-0", + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/on-exit-leak-free": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/on-exit-leak-free/-/on-exit-leak-free-2.1.2.tgz", + "integrity": "sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==", + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/parseley": { + "version": "0.13.1", + "resolved": "https://registry.npmjs.org/parseley/-/parseley-0.13.1.tgz", + "integrity": "sha512-uNBJZzmb60l6p6VWLTmevizNAGnE0xoSf1n0B4q3ntegDNzcS68NRCcBDZTcyXHxt2XhBChsCuqj4M+nChvE/A==", + "license": "MIT", + "dependencies": { + "leac": "^0.7.0", + "peberminta": "^0.10.0" + }, + "funding": { + "url": "https://github.com/sponsors/KillyMXI" + } + }, + "node_modules/peberminta": { + "version": "0.10.0", + "resolved": "https://registry.npmjs.org/peberminta/-/peberminta-0.10.0.tgz", + "integrity": "sha512-80B2AsU+I4Qdb0ZAPSfe9UwvGzwkM37IKIFEvdS3D/3Ndgv2bsuJ0bfG1+iEYO+l7Gfd4EUJmuRyq7efLgRMzQ==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/KillyMXI" + } + }, + "node_modules/pino": { + "version": "10.3.1", + "resolved": "https://registry.npmjs.org/pino/-/pino-10.3.1.tgz", + "integrity": "sha512-r34yH/GlQpKZbU1BvFFqOjhISRo1MNx1tWYsYvmj6KIRHSPMT2+yHOEb1SG6NMvRoHRF0a07kCOox/9yakl1vg==", + "license": "MIT", + "dependencies": { + "@pinojs/redact": "^0.4.0", + "atomic-sleep": "^1.0.0", + "on-exit-leak-free": "^2.1.0", + "pino-abstract-transport": "^3.0.0", + "pino-std-serializers": "^7.0.0", + "process-warning": "^5.0.0", + "quick-format-unescaped": "^4.0.3", + "real-require": "^0.2.0", + "safe-stable-stringify": "^2.3.1", + "sonic-boom": "^4.0.1", + "thread-stream": "^4.0.0" + }, + "bin": { + "pino": "bin.js" + } + }, + "node_modules/pino-abstract-transport": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/pino-abstract-transport/-/pino-abstract-transport-3.0.0.tgz", + "integrity": "sha512-wlfUczU+n7Hy/Ha5j9a/gZNy7We5+cXp8YL+X+PG8S0KXxw7n/JXA3c46Y0zQznIJ83URJiwy7Lh56WLokNuxg==", + "license": "MIT", + "dependencies": { + "split2": "^4.0.0" + } + }, + "node_modules/pino-std-serializers": { + "version": "7.1.0", + "resolved": "https://registry.npmjs.org/pino-std-serializers/-/pino-std-serializers-7.1.0.tgz", + "integrity": "sha512-BndPH67/JxGExRgiX1dX0w1FvZck5Wa4aal9198SrRhZjH3GxKQUKIBnYJTdj2HDN3UQAS06HlfcSbQj2OHmaw==", + "license": "MIT" + }, + "node_modules/process-warning": { + "version": "5.1.1", + "resolved": "https://registry.npmjs.org/process-warning/-/process-warning-5.1.1.tgz", + "integrity": "sha512-oCkNVj4OKnKEcocVBcDfiIugN1/cE6rgfT3nCtzeqZPA8+rIxMaQ0xXzIkGP8Gj2qUaP1UR7uhJ1VD8lUCpzCg==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "MIT" + }, + "node_modules/punycode.js": { + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/punycode.js/-/punycode.js-2.3.1.tgz", + "integrity": "sha512-uxFIHU0YlHYhDQtV4R9J6a52SLx28BCjT+4ieh7IGbgwVJWO+km431c4yRlREUAsAmt/uMjQUyQHNEPf0M39CA==", + "license": "MIT", + "engines": { + "node": ">=6" + } + }, "node_modules/pvtsutils": { "version": "1.3.6", "resolved": "https://registry.npmjs.org/pvtsutils/-/pvtsutils-1.3.6.tgz", @@ -440,6 +956,21 @@ "node": ">=16.0.0" } }, + "node_modules/quick-format-unescaped": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/quick-format-unescaped/-/quick-format-unescaped-4.0.4.tgz", + "integrity": "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==", + "license": "MIT" + }, + "node_modules/real-require": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/real-require/-/real-require-0.2.0.tgz", + "integrity": "sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==", + "license": "MIT", + "engines": { + "node": ">= 12.13.0" + } + }, "node_modules/reflect-metadata": { "version": "0.2.2", "resolved": "https://registry.npmjs.org/reflect-metadata/-/reflect-metadata-0.2.2.tgz", @@ -466,12 +997,127 @@ ], "license": "MIT" }, + "node_modules/safe-stable-stringify": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/safe-stable-stringify/-/safe-stable-stringify-2.5.0.tgz", + "integrity": "sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA==", + "license": "MIT", + "engines": { + "node": ">=10" + } + }, "node_modules/safer-buffer": { "version": "2.1.2", "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", "license": "MIT" }, + "node_modules/selderee": { + "version": "0.12.0", + "resolved": "https://registry.npmjs.org/selderee/-/selderee-0.12.0.tgz", + "integrity": "sha512-b1YMh3+DHZp59DLna3qVwQ5iOla/nrI6mLBNW02XxU77M3046Df6VLkoaJyFz20VsGIG5kkp+FK0kg4K4HnUFw==", + "license": "MIT", + "dependencies": { + "parseley": "~0.13.1" + }, + "funding": { + "url": "https://github.com/sponsors/KillyMXI" + } + }, + "node_modules/smart-buffer": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/smart-buffer/-/smart-buffer-4.2.0.tgz", + "integrity": "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg==", + "license": "MIT", + "engines": { + "node": ">= 6.0.0", + "npm": ">= 3.0.0" + } + }, + "node_modules/smtp-server": { + "version": "3.21.0", + "resolved": "https://registry.npmjs.org/smtp-server/-/smtp-server-3.21.0.tgz", + "integrity": "sha512-vhJTg1JR6+mM5vlCD/TzUvFcQ5JQiCqP0FY3TgMJ0u3MPuvxawKD/6PA+TacPYEFVAgZIjLaCOSf87L4jcN0+w==", + "dev": true, + "license": "MIT-0", + "dependencies": { + "ipv6-normalize": "1.0.1", + "nodemailer": "10.1.7", + "punycode.js": "2.3.1" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/smtp-server/node_modules/nodemailer": { + "version": "10.1.7", + "resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-10.1.7.tgz", + "integrity": "sha512-OQa6FDln5fLttmqSbjDiRYvFDOFNrBqfiOwHPlEbURil3b+vPB1v8xpi+Sp46drzNDlsTnNsS502vthmz56r4w==", + "dev": true, + "license": "MIT-0", + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/socks": { + "version": "2.8.10", + "resolved": "https://registry.npmjs.org/socks/-/socks-2.8.10.tgz", + "integrity": "sha512-e0VyvkVTwVYViNovRkZ9aodhxVlyoMn7eJhVUPxZ+eK9P/7CBkxvvsBOHqFPEH416726W8tLXXXjKwqgTErrCQ==", + "license": "MIT", + "dependencies": { + "ip-address": "^10.1.1", + "smart-buffer": "^4.2.0" + }, + "engines": { + "node": ">= 10.0.0", + "npm": ">= 3.0.0" + } + }, + "node_modules/sonic-boom": { + "version": "4.2.1", + "resolved": "https://registry.npmjs.org/sonic-boom/-/sonic-boom-4.2.1.tgz", + "integrity": "sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q==", + "license": "MIT", + "dependencies": { + "atomic-sleep": "^1.0.0" + } + }, + "node_modules/split2": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz", + "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==", + "license": "ISC", + "engines": { + "node": ">= 10.x" + } + }, + "node_modules/thread-stream": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/thread-stream/-/thread-stream-4.2.0.tgz", + "integrity": "sha512-e2zZ96wSChazBsbENf/Pcm/4swHt2cEKQ92rhUjkL9GCKiTDJIaTBenjE/m9DXi0QBmTMDkFDdOomUy20A1tDQ==", + "license": "MIT", + "dependencies": { + "real-require": "^1.0.0" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/thread-stream/node_modules/real-require": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/real-require/-/real-require-1.0.0.tgz", + "integrity": "sha512-P4nbQYQfePJxRSmY+v/KINxVucm4NF3p3s7pJveMTtom52FR4YGltUQLB8idDXwDDWW+eYrWDFbuzUnjoWHF7g==", + "license": "MIT" + }, + "node_modules/tlds": { + "version": "1.261.0", + "resolved": "https://registry.npmjs.org/tlds/-/tlds-1.261.0.tgz", + "integrity": "sha512-QXqwfEl9ddlGBaRFXIvNKK6OhipSiLXuRuLJX5DErz0o0Q0rYxulWLdFryTkV5PkdZct5iMInwYEGe/eR++1AA==", + "license": "MIT", + "bin": { + "tlds": "bin.js" + } + }, "node_modules/tslib": { "version": "2.8.1", "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", @@ -496,6 +1142,12 @@ "integrity": "sha512-Xni35NKzjgMrwevysHTCArtLDpPvye8zV/0E4EyYn43P7/7qvQwPh9BGkHewbMulVntbigmcT7rdX3BNo9wRJg==", "license": "0BSD" }, + "node_modules/uc.micro": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/uc.micro/-/uc.micro-2.1.0.tgz", + "integrity": "sha512-ARDJmphmdvUk6Glw7y9DQ2bFkKBHwQHLi2lsaH6PPmz/Ka9sFOBsBluozhDltWmnv9u/cF6Rt87znRTPV+yp/A==", + "license": "MIT" + }, "node_modules/web-push": { "version": "3.6.7", "resolved": "https://registry.npmjs.org/web-push/-/web-push-3.6.7.tgz", diff --git a/package.json b/package.json index dcaaa7b..d938942 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "main": "server.js", "scripts": { "start": "node server.js", - "test": "for t in smoke tasks push users plex player lyrics signal mail; do node tools/$t-test.js || exit 1; done", + "test": "for t in smoke tasks push users plex player lyrics signal mail mail-imap; do node tools/$t-test.js || exit 1; done", "audit": "npm audit --omit=dev --audit-level=high" }, "keywords": [], @@ -14,6 +14,13 @@ "type": "commonjs", "dependencies": { "@simplewebauthn/server": "^14.0.3", + "imapflow": "^2.9.0", + "mailparser": "^3.12.2", + "nodemailer": "^10.2.0", "web-push": "^3.6.7" + }, + "devDependencies": { + "hoodiecrow-imap": "^3.3.1", + "smtp-server": "^3.21.0" } } diff --git a/public/mail.js b/public/mail.js index 2f6a265..b998ce4 100644 --- a/public/mail.js +++ b/public/mail.js @@ -2,7 +2,7 @@ // Mail tab: your Outlook.com mailbox (through the server, see mail.js there). Folders, message list, reader, compose, reply/forward, // search, unread badge. Message bodies are shown in a sandboxed frame with remote pictures blocked until you ask for them. let mailUnread = 0; -const mail = { st: null, folders: [], folder: 'inbox', items: [], more: false, next: 0, q: '', msg: null, images: false, loading: false }; +const mail = { st: null, accts: [], acct: '', folders: [], folder: 'inbox', items: [], more: false, next: 0, q: '', msg: null, images: false, loading: false }; const mailFrom = f => (f && (f.name || f.address)) || ''; const mailWhen = iso => { @@ -11,7 +11,7 @@ const mailWhen = iso => { if (now - d < 6 * 864e5) return d.toLocaleDateString([], { weekday: 'short' }); return d.toLocaleDateString([], { month: 'short', day: 'numeric', year: d.getFullYear() === now.getFullYear() ? undefined : '2-digit' }); }; -const mailFolderName = id => (mail.folders.find(f => f.id === id) || {}).name || (id === 'inbox' ? 'Inbox' : 'Folder'); +const mailFolderName = id => (mail.folders.find(f => f.id === id) || {}).name || (/~inbox$/.test(id) || id === 'inbox' ? 'Inbox' : 'Folder'); /** The page a message is shown in: no scripts, nothing loaded from the internet unless the person allows pictures. */ function mailDoc(html, images) { @@ -31,24 +31,55 @@ async function renderMailCard(el) { try { const s = mail.st || await jfetch('/api/mail/status'); mail.st = s; if (!s.connected || !document.body.contains(el)) return; - const r = await jfetch('/api/mail/messages?folder=inbox'); if (!document.body.contains(el)) return; - const top = r.items.filter(m => !m.read).slice(0, 3); - el.innerHTML = `

Mail${s.unread ? ' · ' + s.unread + ' unread' : ''}

`; + const lists = await Promise.all(s.accounts.slice(0, 4).map(a => jfetch('/api/mail/messages?folder=' + encodeURIComponent(a.id + '~inbox')).then(r => r.items.map(x => ({ ...x, acct: a.address }))).catch(() => []))); + if (!document.body.contains(el)) return; + const top = lists.flat().filter(m => !m.read).sort((a, b) => String(b.date).localeCompare(String(a.date))).slice(0, 3); + el.innerHTML = `

Mail${s.unread ? ' · ' + s.unread + ' unread' : ''}

`; } catch {} } +const MAIL_LABEL = { outlook: 'Outlook.com', gmail: 'Gmail', icloud: 'iCloud Mail' }; function mailSettings() { const box = $('#mailst'); if (!box) return; if (allowed('mail') && cfg.mail) jfetch('/api/mail/status').then(s => { if (!$('#mailst')) return; - box.innerHTML = s.connected - ? `
Outlook.com${esc(s.address || 'Connected')}
` - : `
Outlook.com${s.expired ? 'Microsoft signed you out. ' : ''}Not connectedConnect
`; - const d = $('#mdis'); if (d) d.onclick = () => confirmSheet('Disconnect mail?', 'NE Hub forgets your Outlook.com sign-in. Your mailbox is not touched.', 'Disconnect', async () => { try { await jfetch('/api/mail/disconnect', { method: 'POST', body: '{}' }); mail.st = null; mailUnread = 0; renderNav(); mailSettings(); } catch (e) { toast(e.message); } }); + box.innerHTML = (s.accounts.length ? s.accounts.map(a => `
${esc(MAIL_LABEL[a.type] || a.type)}${esc(a.address || 'Connected')}${a.error ? ' · ' + esc(a.error) : ''}
`).join('') : `
No mail account yet${s.expired ? 'Microsoft signed you out. ' : ''}Add Outlook.com, Gmail or iCloud Mail
`) + + `
`; + $('#madd').onclick = () => mailAddAccount(s); + box.querySelectorAll('[data-dis]').forEach(d => d.onclick = () => { const a = s.accounts.find(x => x.id === d.dataset.dis); confirmSheet('Disconnect ' + (a ? a.address : 'this account') + '?', 'NE Hub forgets the sign-in. Your mailbox is not touched.', 'Disconnect', async () => { try { await jfetch('/api/mail/disconnect', { method: 'POST', body: JSON.stringify({ acct: d.dataset.dis }) }); mail.st = null; mail.items = []; mail.folders = []; mail.acct = ''; await mailPoll(); mailSettings(); } catch (e) { toast(e.message); } }); }); }).catch(() => { box.innerHTML = '
Could not check
'; }); else box.innerHTML = ''; if (cfg.mailAdmin) mailAdminCard(); } +/** Choose a provider; Outlook.com signs in on Microsoft's page, Gmail and iCloud take an app password. */ +function mailAddAccount(st) { + const choices = [['gmail', 'Gmail', 'App password'], ['icloud', 'iCloud Mail', 'App-specific password'], ...(st && st.outlook ? [['outlook', 'Outlook.com', 'Sign in with Microsoft']] : [])]; + const el = openSheet('Add a mail account', '', `
${choices.map(([k, l, s]) => ``).join('')}
+
`); + el.querySelector('#sx').onclick = closeSheet; + el.querySelectorAll('[data-p]').forEach(b => b.onclick = async () => { + const k = b.dataset.p; + if (k === 'outlook') { try { const r = await jfetch('/api/mail/connect'); location.href = r.url; } catch (e) { toast(e.message); } return; } + mailPasswordForm(k); + }); +} +function mailPasswordForm(type) { + const g = type === 'gmail'; + const el = openSheet('Add ' + (g ? 'Gmail' : 'iCloud Mail'), '', `
+

${g ? 'Google needs an app password, not your normal password. Turn on 2-Step Verification, then make one at myaccount.google.com/apppasswords (a 16-letter code).' : 'Apple needs an app-specific password. Go to appleid.apple.com → Sign-In and Security → App-Specific Passwords, and make one for NE Hub.'}

+ + +

It is checked with ${g ? 'Google' : 'Apple'} before saving, and kept encrypted on your server. NE Hub never sees your normal password.

+
`); + el.querySelector('#sx').onclick = closeSheet; + el.querySelector('#paf').onsubmit = async e => { + e.preventDefault(); const btn = el.querySelector('#pab'); btn.disabled = true; btn.textContent = 'Checking…'; el.querySelector('#pax').textContent = ''; + try { await jfetch('/api/mail/accounts', { method: 'POST', body: JSON.stringify({ type, address: el.querySelector('#pae').value, password: el.querySelector('#pap').value }) }); closeSheet(); toast('Connected'); mail.st = null; mail.items = []; mail.folders = []; mail.acct = ''; await mailPoll(); if (location.hash.startsWith('#/mail')) route(); else if ($('#mailst')) mailSettings(); } + catch (err) { btn.disabled = false; btn.textContent = 'Connect'; el.querySelector('#pax').textContent = err.message; } + }; + setTimeout(() => el.querySelector('#pae').focus(), 60); +} + /** Admin: the Microsoft app behind Mail (client ID, secret, expiry), with the redirect address to register and a test button. */ async function mailAdminCard() { let host = $('#mailadm'); @@ -93,16 +124,20 @@ async function mailAdminCard() { // ---- the tab async function viewMail(args = []) { const v = $('#view'); - if (!cfg.mail) { v.innerHTML = '

Mail

Mail is not set up on this server. An admin can switch it on (see the README, Mail).

'; return; } + if (!cfg.mail) { v.innerHTML = '

Mail

Mail is not available to you.

'; return; } v.innerHTML = '

Mail

Loading…

'; let st; try { st = await jfetch('/api/mail/status'); } catch (e) { v.innerHTML = `

Mail

${esc(e.message)}

`; return; } if (!location.hash.startsWith('#/mail')) return; - mail.st = st; + mail.st = st; mail.accts = st.accounts || []; if (!st.connected) return mailConnectScreen(st); if (typeof st.unread === 'number' && st.unread !== mailUnread) { mailUnread = st.unread; renderNav(); } const [kind, key] = args; - if (kind === 'f' && key) mail.folder = key; - if (!mail.folders.length) { try { mail.folders = await jfetch('/api/mail/folders'); } catch {} } + const idAcct = x => String(x || '').includes('~') ? String(x).split('~')[0] : ''; + const want = idAcct(kind === 'f' ? key : kind === 'm' ? key : '') || (mail.accts.some(a => a.id === mail.acct) ? mail.acct : mail.accts[0].id); + if (want !== mail.acct) { mail.acct = want; mail.folders = []; mail.items = []; if (!(kind === 'f' && idAcct(key))) mail.folder = want + '~inbox'; } + if (kind === 'f' && key) mail.folder = idAcct(key) ? key : mail.acct + '~' + key; + else if (!String(mail.folder).startsWith(mail.acct + '~')) mail.folder = mail.acct + '~inbox'; + if (!mail.folders.length) { try { mail.folders = await jfetch('/api/mail/folders?acct=' + encodeURIComponent(mail.acct)); } catch {} } mailShell(kind === 'm' && key ? 'read' : 'list'); if (!mail.items.length || kind === 'f' || kind === undefined) mailLoadList(true); else mailDrawList(); @@ -110,22 +145,24 @@ async function viewMail(args = []) { } function mailConnectScreen(st) { + const choices = [['gmail', 'Gmail', 'Uses an app password', '#d9453a'], ['icloud', 'iCloud Mail', 'Uses an app-specific password', '#3b97e8'], ...(st.outlook ? [['outlook', 'Outlook.com', 'Sign in with Microsoft', '#0a64c8']] : [])]; $('#view').innerHTML = `

Mail

-

Read and send mail from your Outlook.com account right here. You sign in on Microsoft's own page; NE Hub never sees your password and only keeps a sign-in token on your server (encrypted).

+

Read and send mail from your own accounts right here, all in one place. Passwords are checked with the provider and kept encrypted on your server; Outlook.com signs in on Microsoft's own page.

${st.expired ? '

Microsoft signed you out of mail. Connect again to continue.

' : ''} -
-

Personal Outlook.com, Hotmail and Live accounts only.

`; - $('#mcon').onclick = async () => { - try { const r = await jfetch('/api/mail/connect'); location.href = r.url; } - catch (e) { $('#mce').textContent = e.message; } - }; +
${choices.map(([k, l, s, c]) => ``).join('')}
+ ${st.outlook ? '' : '

Outlook.com needs a one-time Microsoft app set up by an admin (Settings → Microsoft app).

'}`; + $('#view').querySelectorAll('[data-p]').forEach(b => b.onclick = async () => { + if (b.dataset.p === 'outlook') { try { const r = await jfetch('/api/mail/connect'); location.href = r.url; } catch (e) { $('#mce').textContent = e.message; } } + else mailPasswordForm(b.dataset.p); + }); } function mailShell(mode) { const fl = mail.folders.length ? mail.folders : [{ id: 'inbox', name: 'Inbox', unread: mailUnread }]; $('#view').innerHTML = `
- +
+ ${mail.accts.length > 1 ? `
` : ''}
@@ -133,8 +170,10 @@ function mailShell(mode) {
`; $('#mfsel').onchange = e => { mail.q = ''; location.hash = '#/mail/f/' + encodeURIComponent(e.target.value); }; + if ($('#macct2')) $('#macct2').onchange = e => { mail.q = ''; location.hash = '#/mail/f/' + encodeURIComponent(e.target.value + '~inbox'); }; + if ($('#macct')) $('#macct').onchange = e => { mail.q = ''; location.hash = '#/mail/f/' + encodeURIComponent(e.target.value + '~inbox'); }; $('#mnew').onclick = () => mailCompose(); - $('#mrf').onclick = () => { mailLoadList(true); jfetch('/api/mail/folders').then(f => { mail.folders = f; mailPoll(); }).catch(() => {}); }; + $('#mrf').onclick = () => { mailLoadList(true); jfetch('/api/mail/folders?acct=' + encodeURIComponent(mail.acct)).then(f => { mail.folders = f; mailPoll(); }).catch(() => {}); }; let t; $('#mq').oninput = e => { clearTimeout(t); t = setTimeout(() => { mail.q = e.target.value.trim(); mailLoadList(true); }, 400); }; } @@ -168,7 +207,7 @@ async function mailOpen(id) { if (!m.read) { // opening a message marks it read jfetch('/api/mail/update', { method: 'POST', body: JSON.stringify({ id, read: true }) }).then(() => { const it = mail.items.find(x => x.id === id); if (it) it.read = true; m.read = true; mailDrawList(); - mailUnread = Math.max(0, mailUnread - 1); renderNav(); jfetch('/api/mail/folders').then(f => { mail.folders = f; }).catch(() => {}); + mailUnread = Math.max(0, mailUnread - 1); renderNav(); jfetch('/api/mail/folders?acct=' + encodeURIComponent(mail.acct)).then(f => { mail.folders = f; }).catch(() => {}); }).catch(() => {}); } } catch (e) { if ($('#mread')) $('#mread').innerHTML = `

${esc(e.message)}

`; } @@ -197,14 +236,14 @@ function mailDrawReader() { async function mailAction(a) { const m = mail.msg; if (!m) return; const post = (p, body) => jfetch('/api/mail/' + p, { method: 'POST', body: JSON.stringify({ id: m.id, ...body }) }); - const back = () => { mail.items = mail.items.filter(x => x.id !== m.id); mail.msg = null; location.hash = '#/mail/f/' + encodeURIComponent(mail.folder); mailPoll(); jfetch('/api/mail/folders').then(f => { mail.folders = f; }).catch(() => {}); }; + const back = () => { mail.items = mail.items.filter(x => x.id !== m.id); mail.msg = null; location.hash = '#/mail/f/' + encodeURIComponent(mail.folder); mailPoll(); jfetch('/api/mail/folders?acct=' + encodeURIComponent(mail.acct)).then(f => { mail.folders = f; }).catch(() => {}); }; try { if (a === 'reply' || a === 'replyAll' || a === 'forward') return mailCompose({ mode: a, msg: m }); if (a === 'archive') { await post('move', { to: 'archive' }); toast('Archived'); return back(); } if (a === 'unread') { await post('update', { read: false }); const it = mail.items.find(x => x.id === m.id); if (it) it.read = false; toast('Marked unread'); mailUnread++; renderNav(); return back(); } if (a === 'flag') { await post('update', { flagged: !m.flagged }); m.flagged = !m.flagged; const it = mail.items.find(x => x.id === m.id); if (it) it.flagged = m.flagged; mailDrawReader(); return mailDrawList(); } if (a === 'delete') { - if (mail.folder === 'deleteditems') return confirmSheet('Delete for good?', m.subject, 'Delete', async () => { try { await post('delete', { permanent: true }); toast('Deleted'); back(); } catch (e) { toast(e.message); } }); + if (/(^|~)deleteditems$/.test(mail.folder)) return confirmSheet('Delete for good?', m.subject, 'Delete', async () => { try { await post('delete', { permanent: true }); toast('Deleted'); back(); } catch (e) { toast(e.message); } }); await post('delete', {}); toast('Moved to Deleted'); return back(); } } catch (e) { toast(e.message); } @@ -216,7 +255,7 @@ function mailCompose(o = {}) { const reply = o.mode, m = o.msg, files = []; const title = reply === 'reply' ? 'Reply' : reply === 'replyAll' ? 'Reply all' : reply === 'forward' ? 'Forward' : 'New message'; const el = openSheet(title, reply ? m.subject : '', `
- ${reply && reply !== 'forward' ? `

To ${esc(reply === 'reply' ? mailFrom(m.from) : 'everyone on the message')}. The original is quoted below your text automatically.

` : ``} + ${reply && reply !== 'forward' ? `

To ${esc(reply === 'reply' ? mailFrom(m.from) : 'everyone on the message')}. The original is quoted below your text automatically.

` : `${mail.accts.length > 1 ? `` : ''}`} ${reply ? '' : ` `} @@ -234,7 +273,7 @@ function mailCompose(o = {}) { try { const body = el.querySelector('#mbd').value; if (reply) await jfetch('/api/mail/reply', { method: 'POST', body: JSON.stringify({ id: m.id, mode: reply, comment: body, to: reply === 'forward' ? el.querySelector('#mto').value : undefined }) }); - else await jfetch('/api/mail/send', { method: 'POST', body: JSON.stringify({ to: el.querySelector('#mto').value, cc: el.querySelector('#mcc2').value, bcc: el.querySelector('#mbcc').value, subject: el.querySelector('#msb').value, body, + else await jfetch('/api/mail/send', { method: 'POST', body: JSON.stringify({ acct: el.querySelector('#mfrom') ? el.querySelector('#mfrom').value : mail.acct || undefined, to: el.querySelector('#mto').value, cc: el.querySelector('#mcc2').value, bcc: el.querySelector('#mbcc').value, subject: el.querySelector('#msb').value, body, attachments: await Promise.all(files.map(async f => ({ name: f.name, type: f.type, data: await b64(f) }))) }) }); closeSheet(); toast('Sent'); } catch (err) { btn.disabled = false; el.querySelector('#mer').textContent = err.message; } diff --git a/server.js b/server.js index 4c635cb..410b07d 100644 --- a/server.js +++ b/server.js @@ -441,7 +441,7 @@ async function handleRequest(req, res) { if (pathname === '/healthz') { res.writeHead(200, { 'content-type': 'application/json', 'cache-control': 'no-store' }); return res.end('{"ok":true}'); } if (crossSite(req) && pathname !== '/api/oo/callback') return forbidden(res, 'Cross-site request refused'); if (pathname.startsWith('/api/auth/')) return auth.handle(req, res); - if (pathname === '/api/mail/callback' && mail.enabled()) return mail.callback(req, res); // Microsoft sends the person back here; the one-time state says who they are + if (pathname === '/api/mail/callback' && mail.outlookOn()) return mail.callback(req, res); // Microsoft sends the person back here; the one-time state says who they are if (pathname.startsWith('/cal/feed/')) return cal.feed(req, res); // subscribe feed: protected by its own secret token, not the session // Everything else needs a passkey session, except the sign-in page and its assets, // and OnlyOffice's save callback (which carries its own signed token). @@ -467,7 +467,7 @@ async function handleRequest(req, res) { const c = t => users.can(user, t); res.writeHead(200, { 'content-type': 'application/json' }); return res.end(JSON.stringify({ seafile: c('files') || c('office') ? SEAFILE_URL : '', onlyoffice: c('office') ? ONLYOFFICE_URL : '', audiobooks: c('audio') ? AUDIOBOOKS_URL : '', verse: c('verse') ? VERSE_URL : '', - podcast: c('podcast') ? PODCAST_URL : '', signal: c('signal') && signal.enabled() ? '1' : '', lyrics: c('music') && lyrics.enabled() ? '1' : '', mail: c('mail') && mail.enabled() ? '1' : '', mailAdmin: user.role === 'admin' ? '1' : '', editor: (c('office') || c('files')) && OO_OK, jwt: !!JWT_SECRET, ooReady: OO_OK, build: BUILD })); + podcast: c('podcast') ? PODCAST_URL : '', signal: c('signal') && signal.enabled() ? '1' : '', lyrics: c('music') && lyrics.enabled() ? '1' : '', mail: c('mail') ? '1' : '', outlook: c('mail') && mail.outlookOn() ? '1' : '', mailAdmin: user.role === 'admin' ? '1' : '', editor: (c('office') || c('files')) && OO_OK, jwt: !!JWT_SECRET, ooReady: OO_OK, build: BUILD })); } if (req.method === 'POST' && req.url === '/api/oo/config') return OO_OK ? ooConfig(req, res) : forbidden(res, 'In-app editing needs ONLYOFFICE_JWT_SECRET to be set on the server'); if (pathname === '/api/backup') return user.role === 'admin' ? backup.handle(req, res, user.name) : forbidden(res, 'Only an admin can do this'); @@ -485,7 +485,7 @@ async function handleRequest(req, res) { } if (pathname === '/api/lyrics') return lyrics.enabled() ? lyrics.handle(req, res) : forbidden(res, 'Lyrics are not set up on this server'); if (pathname.startsWith('/api/signal/')) return signal.enabled() ? signal.handle(req, res) : forbidden(res, 'Signal is not set up on this server'); - if (pathname.startsWith('/api/mail/')) return mail.enabled() || pathname.startsWith('/api/mail/admin') ? mail.handle(req, res) : forbidden(res, 'Mail is not set up on this server'); + if (pathname.startsWith('/api/mail/')) return mail.handle(req, res); if (req.url.startsWith('/api/plex/')) return plexApi(req, res); if (req.url.startsWith('/plex/') && (req.method === 'GET' || (['POST', 'PUT', 'DELETE'].includes(req.method) && PLEX_WRITABLE.test(req.url)))) return plexProxy(req, res); if (req.url.startsWith('/sf/')) return proxySeafile(req, res); diff --git a/tools/mail-imap-test.js b/tools/mail-imap-test.js new file mode 100644 index 0000000..077a21d --- /dev/null +++ b/tools/mail-imap-test.js @@ -0,0 +1,128 @@ +#!/usr/bin/env node +// Gmail and iCloud mail (IMAP + SMTP with an app password): adding accounts, reading, flags, moving, sending, replies, privacy. +// Uses a fake IMAP server (hoodiecrow-imap) and a fake SMTP server (smtp-server) on localhost. Run: node tools/mail-imap-test.js +const { spawn } = require('child_process'); +const fs = require('fs'), os = require('os'), path = require('path'), crypto = require('crypto'); +const hoodiecrow = require('hoodiecrow-imap'), { SMTPServer } = require('smtp-server'), { simpleParser } = require('mailparser'), { ImapFlow } = require('imapflow'); +const MailComposer = require('nodemailer/lib/mail-composer'); +const root = path.join(__dirname, '..'); +let failed = 0; +const ok = (n, c, x) => { console.log((c ? 'PASS ' : 'FAIL ') + n + (c ? '' : ' -> ' + JSON.stringify(x))); if (!c) failed++; }; +const wait = ms => new Promise(r => setTimeout(r, ms)); +const SECRET = 'imap-test-secret-'.repeat(3), PASS = 'testpass', ME = 'testuser@example.com', USER = ME; +const free = () => 21000 + Math.floor(Math.random() * 20000); + +(async () => { + // ---- messages in the fake mailbox + const build = async o => (await new MailComposer(o).compile().build()).toString(); + const m1 = await build({ from: 'Ann Lee ', to: ME, cc: 'bob@example.com', subject: 'Hello there', text: 'Hi there\nsecond line', date: new Date('2026-10-22T10:00:00Z'), messageId: '' }); + const m2 = await build({ from: 'Bank ', to: ME, subject: 'Statement', text: 'see attached', html: '

Your statement

', date: new Date('2026-10-21T10:00:00Z'), messageId: '', + attachments: [{ filename: 'logo.png', content: Buffer.from('PNG!'), contentType: 'image/png', cid: 'logo1' }, { filename: 'report.pdf', content: Buffer.from('%PDF-1'), contentType: 'application/pdf' }] }); + const m3 = await build({ from: 'Old Friend ', to: ME, subject: 'Old news', text: 'long ago', date: new Date('2026-10-01T10:00:00Z'), messageId: '' }); + const imapPort = free(), smtpPort = free() + 1; + const imapSrv = hoodiecrow({ users: { [ME]: { password: PASS }, 'me@icloud.com': { password: PASS } }, plugins: ['SPECIAL-USE', 'MOVE', 'UIDPLUS'], storage: { + INBOX: { messages: [{ raw: m1, flags: [] }, { raw: m2, flags: ['\\Seen'] }, { raw: m3, flags: [] }] }, + '': { separator: '/', folders: { Sent: { 'special-use': '\\Sent' }, Trash: { 'special-use': '\\Trash' }, Receipts: {}, '[Gmail]': { flags: ['\\Noselect'], folders: { Spam: { 'special-use': '\\Junk' }, 'All Mail': { 'special-use': '\\All' } } } } } } }); + await new Promise(r => imapSrv.listen(imapPort, r)); + const sent = []; + const smtp = new SMTPServer({ authOptional: false, allowInsecureAuth: true, disabledCommands: ['STARTTLS'], authMethods: ['PLAIN', 'LOGIN'], + onAuth(a, s, cb) { [ME, 'me@icloud.com'].includes(a.username) && a.password === PASS ? cb(null, { user: a.username }) : cb(new Error('Invalid credentials')); }, + onData(stream, session, cb) { const c = []; stream.on('data', d => c.push(d)); stream.on('end', async () => { sent.push({ from: session.envelope.mailFrom.address, to: session.envelope.rcptTo.map(r => r.address), parsed: await simpleParser(Buffer.concat(c)) }); cb(); }); } }); + await new Promise(r => smtp.listen(smtpPort, '127.0.0.1', r)); + const peek = async (box) => { const c = new ImapFlow({ host: '127.0.0.1', port: imapPort, secure: false, doSTARTTLS: false, auth: { user: USER, pass: PASS }, logger: false }); await c.connect(); try { return (await c.status(box, { messages: true, unseen: true })); } finally { await c.logout(); } }; + + // ---- the hub, with Nate (admin) and Ann (a mail user with no accounts) + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'nehub-imap-')), port = free() + 2, base = `http://localhost:${port}`; + fs.writeFileSync(path.join(dir, 'users.json'), JSON.stringify({ users: [ + { id: 'unate', name: 'Nate', role: 'admin', tabs: [], handle: 'aGFuZGxl', epoch: 1, disabled: false, created: 1 }, + { id: 'uann', name: 'Ann', role: 'user', tabs: ['mail'], handle: 'aGFuZGxl', epoch: 1, disabled: false, created: 1 }] })); + const exp = Date.now() + 864e5, ck = uid => `hub_session=${uid}.1.${exp}.${crypto.createHmac('sha256', SECRET).update(`s${uid}.1.${exp}`).digest('base64url')}`; + const env = { ...process.env, PORT: String(port), DATA_DIR: dir, COOKIE_SECRET: SECRET, MAIL_TEST_SERVERS: JSON.stringify({ imap: { host: '127.0.0.1', port: imapPort, secure: false }, smtp: { host: '127.0.0.1', port: smtpPort } }) }; delete env.MS_CLIENT_ID; + const child = spawn(process.execPath, ['server.js'], { cwd: root, env }); let log = ''; child.stdout.on('data', d => log += d); child.stderr.on('data', d => log += d); + for (let i = 0; i < 60; i++) { try { await fetch(base + '/login'); break; } catch { await wait(100); } } + const call = async (uid, p, body, method) => { const r = await fetch(base + p, { method: method || (body ? 'POST' : 'GET'), headers: { cookie: ck(uid), 'content-type': 'application/json' }, body: body ? JSON.stringify(body) : undefined }); const t = await r.text(); let j; try { j = JSON.parse(t); } catch { j = t; } return { status: r.status, json: j, headers: r.headers, text: t }; }; + const A = (p, b, m) => call('unate', p, b, m); + + // ================= adding accounts ================= + let r = await A('/api/mail/status'); ok('before adding anything: not connected, and the hub offers Gmail and iCloud', r.json.connected === false && r.json.providers.gmail && r.json.providers.icloud, r.json); + r = await A('/api/mail/accounts', { type: 'gmail', address: ME, password: 'wrongpassword1' }); ok('a wrong app password is refused with advice for Gmail', r.status === 401 && /app password/i.test(r.json.error) && /Google/.test(r.json.error), r); + r = await A('/api/mail/accounts', { type: 'icloud', address: ME, password: 'wrongpassword1' }); ok('and with advice for iCloud', r.status === 401 && /Apple/.test(r.json.error) && /app-specific/i.test(r.json.error), r); + r = await A('/api/mail/accounts', { type: 'gmail', address: 'nonsense', password: PASS }); ok('a malformed address is refused', r.status === 400, r.json); + r = await A('/api/mail/accounts', { type: 'yahoo', address: ME, password: PASS }); ok('an unknown provider is refused', r.status === 400, r.json); + r = await A('/api/mail/accounts', { type: 'gmail', address: ME, password: PASS }); const gm = r.json.account; + ok('the right app password connects a Gmail account', r.status === 200 && gm && gm.type === 'gmail' && gm.address === ME, r); + ok('the password is never sent back', !r.text.includes(PASS) && !(await A('/api/mail/status')).text.includes(PASS)); + ok('and is stored encrypted', !fs.readFileSync(path.join(dir, 'users', 'unate', 'mail-accounts.json'), 'utf8').includes(PASS)); + r = await A('/api/mail/status'); ok('status lists the account with its unread count (2 unread in the inbox)', r.json.connected && r.json.accounts.length === 1 && r.json.accounts[0].unread === 2 && r.json.unread === 2, r.json); + + // ================= reading ================= + r = await A('/api/mail/folders?acct=' + gm.id); const names = r.json.map(f => f.name); + ok('folders: Inbox, Sent, Archive (Gmail "All Mail"), Junk, Deleted, then your own; the [Gmail] container is hidden', names.join() === 'Inbox,Sent,Archive,Junk,Deleted,Receipts' && r.json[0].unread === 2 && r.json[0].id.startsWith(gm.id + '~'), r.json); + r = await A('/api/mail/messages?folder=' + gm.id + '~inbox'); const items = r.json.items; + ok('the inbox lists messages newest first with sender, read state and attachment flag', items.length === 3 && items[0].subject === 'Hello there' && items[0].from.address === 'ann@example.com' && items[0].read === false && items[1].subject === 'Statement' && items[1].read === true && items[1].attach === true && items[0].attach === false, items); + ok('ids carry the account', items.every(m => m.id.startsWith(gm.id + '~')), items.map(m => m.id)); + r = await A('/api/mail/messages?folder=inbox'); ok('a folder with no account in it means the first account (older links)', r.json.items.length === 3, r.json); + r = await A('/api/mail/messages?folder=' + gm.id + '~inbox&q=statement'); ok('search finds a message', r.json.items.length === 1 && r.json.items[0].subject === 'Statement', r.json); + r = await A('/api/mail/messages?folder=' + gm.id + '~p-' + Buffer.from('Nope').toString('base64url')); ok('a folder that does not exist is a 404, not a crash', r.status === 404, r); + const helloId = items[0].id, stmtId = items[1].id, oldId = items[2].id; + r = await A('/api/mail/message?id=' + encodeURIComponent(helloId)); const hm = r.json; + ok('a message opens with sender, recipients (to and cc), date and text', hm.subject === 'Hello there' && hm.to[0].address === ME && hm.cc[0].address === 'bob@example.com' && /Hi there/.test(hm.html) && hm.attachments.length === 0, hm); + r = await A('/api/mail/message?id=' + encodeURIComponent(stmtId)); const sm = r.json; + ok('an inline picture is put in the page (cid -> data), and only real attachments are listed', /data:image\/png;base64,/.test(sm.html) && !/cid:logo1/.test(sm.html) && sm.attachments.length === 1 && sm.attachments[0].name === 'report.pdf', sm); + r = await A('/api/mail/attachment?id=' + encodeURIComponent(stmtId) + '&aid=' + sm.attachments[0].id); ok('an attachment downloads as a file', r.status === 200 && r.text === '%PDF-1' && /filename="report\.pdf"/.test(r.headers.get('content-disposition')), [r.status, r.text]); + r = await A('/api/mail/message?id=' + encodeURIComponent('abc~' + helloId.split('~')[1])); ok('an unknown account in an id is a 404', r.status === 404, r.status); + r = await A('/api/mail/message?id=' + encodeURIComponent(gm.id + '~garbage')); ok('a damaged id is a 400', r.status === 400, r.status); + + // ================= flags, moving, deleting ================= + await A('/api/mail/update', { id: helloId, read: true }); r = await A('/api/mail/messages?folder=' + gm.id + '~inbox'); ok('mark as read', r.json.items[0].read === true, r.json.items[0]); + await A('/api/mail/update', { id: helloId, read: false, flagged: true }); r = await A('/api/mail/messages?folder=' + gm.id + '~inbox'); ok('mark unread and flag', r.json.items[0].read === false && r.json.items[0].flagged === true, r.json.items[0]); + r = await A('/api/mail/update', { id: helloId }); ok('an update with nothing to change is refused', r.status === 400, r.status); + const archBefore = (await peek('[Gmail]/All Mail')).messages; + r = await A('/api/mail/move', { id: oldId, to: 'archive' }); ok('archive moves a message to All Mail', r.status === 200 && (await peek('[Gmail]/All Mail')).messages === archBefore + 1 && (await peek('INBOX')).messages === 2, r); + const trashBefore = (await peek('Trash')).messages; + r = await A('/api/mail/delete', { id: stmtId }); ok('delete moves a message to Trash', r.status === 200 && (await peek('Trash')).messages === trashBefore + 1 && (await peek('INBOX')).messages === 1, r); + r = await A('/api/mail/messages?folder=' + gm.id + '~deleteditems'); const trashed = r.json.items[0]; + r = await A('/api/mail/delete', { id: trashed.id, permanent: true }); ok('and from Trash it is removed for good', r.status === 200 && (await peek('Trash')).messages === trashBefore, r); + + // ================= sending ================= + sent.length = 0; const sentBefore = (await peek('Sent')).messages; + r = await A('/api/mail/send', { acct: gm.id, to: 'bob@example.com, Carol ', subject: 'Plans', body: 'Line 1\nnot bold', attachments: [{ name: 'a.txt', type: 'text/plain', data: Buffer.from('hello').toString('base64') }] }); + const s1 = sent[0]; + ok('send goes out over SMTP from the account (several recipients, text turned into safe HTML, attachment kept)', r.status === 200 && s1 && s1.from === ME && s1.to.join() === 'bob@example.com,carol@example.com' && s1.parsed.subject === 'Plans' && /not bold/.test(s1.parsed.text) && /<b>not bold/.test(s1.parsed.html) && s1.parsed.attachments[0].filename === 'a.txt', [r, s1 && s1.to]); + ok('Gmail keeps its own copy, so none is added to Sent here', (await peek('Sent')).messages === sentBefore, (await peek('Sent')).messages); + r = await A('/api/mail/send', { acct: gm.id, to: 'nobody', subject: 'x', body: 'y' }); ok('a bad address is refused with its name', r.status === 400 && /nobody/.test(r.json.error), r.json); + r = await A('/api/mail/send', { acct: gm.id, to: '', subject: 'x', body: 'y' }); ok('no recipient is refused', r.status === 400, r.json); + r = await A('/api/mail/send', { acct: gm.id, to: 'a@b.co', subject: 'x', body: 'y', attachments: [{ name: 'big', type: 'x/y', data: Buffer.alloc(3.5 * 1024 * 1024).toString('base64') }] }); ok('attachments over about 3 MB are refused', r.status === 400 && /3 MB/.test(r.json.error), r.json); + + // ================= replies ================= + sent.length = 0; r = await A('/api/mail/reply', { id: helloId, mode: 'reply', comment: 'Thanks Ann' }); const rp = sent[0]; + ok('reply goes to the sender, keeps the thread headers and quotes the original', r.status === 200 && rp.to.join() === 'ann@example.com' && rp.parsed.subject === 'Re: Hello there' && rp.parsed.inReplyTo === '' && /Thanks Ann/.test(rp.parsed.text) && /> Hi there/.test(rp.parsed.text), [r, rp && rp.parsed.inReplyTo, rp && rp.parsed.text]); + sent.length = 0; r = await A('/api/mail/reply', { id: helloId, mode: 'replyAll', comment: 'All' }); const ra = sent[0]; + ok('reply all adds the others on the message but not yourself', r.status === 200 && ra.to.includes('ann@example.com') && ra.to.includes('bob@example.com') && !ra.to.includes(ME), ra && ra.to); + sent.length = 0; r = await A('/api/mail/reply', { id: helloId, mode: 'forward', comment: 'FYI', to: 'x@y.org' }); const rf = sent[0]; + ok('forward needs a recipient and sends the original', r.status === 200 && rf.to.join() === 'x@y.org' && rf.parsed.subject === 'Fwd: Hello there' && /Forwarded message/.test(rf.parsed.text) && /Hi there/.test(rf.parsed.text), rf && rf.parsed.text); + r = await A('/api/mail/reply', { id: helloId, mode: 'forward', comment: 'FYI' }); ok('forward with no recipient is refused', r.status === 400, r.json); + r = await A('/api/mail/reply', { id: helloId, mode: 'bogus' }); ok('an unknown action is refused', r.status === 400, r.json); + + // ================= a second account (iCloud), several accounts ================= + r = await A('/api/mail/accounts', { type: 'icloud', address: 'me@icloud.com', password: PASS }); ok('an iCloud account connects the same way', r.status === 200 && r.json.account.type === 'icloud', r); + const ic = r.json.account; + r = await A('/api/mail/status'); ok('both accounts are listed and the unread counts add up', r.json.accounts.length === 2 && r.json.unread === r.json.accounts.reduce((n, x) => n + x.unread, 0), r.json); + const sentB = (await peek('Sent')).messages; sent.length = 0; + r = await A('/api/mail/send', { acct: ic.id, to: 'bob@example.com', subject: 'From iCloud', body: 'hi' }); + ok('iCloud sends as that address', r.status === 200 && sent[0].from === 'me@icloud.com', [r, sent[0] && sent[0].from]); + ok('and iCloud does not keep a copy itself, so one is added to Sent', (await peek('Sent')).messages === sentB + 1, [(await peek('Sent')).messages, sentB]); + + // ================= privacy and removal ================= + r = await call('uann', '/api/mail/status'); ok('another person sees no accounts', r.json.connected === false && r.json.accounts.length === 0, r.json); + r = await call('uann', '/api/mail/message?id=' + encodeURIComponent(helloId)); ok("and cannot open Nate's messages with his ids", r.status === 409 || r.status === 404, r.status); + r = await call('uann', '/api/mail/send', { acct: gm.id, to: 'a@b.co', subject: 'x', body: 'y' }); ok("or send from Nate's account", r.status === 404 || r.status === 409, r.status); + r = await A('/api/mail/disconnect', { acct: ic.id }); r = await A('/api/mail/status'); ok('disconnecting one account leaves the other', r.json.accounts.length === 1 && r.json.accounts[0].id === gm.id, r.json); + r = await A('/api/mail/disconnect', { acct: 'nope' }); ok('disconnecting an unknown account is a 404', r.status === 404, r.status); + r = await A('/api/mail/disconnect', {}); r = await A('/api/mail/status'); ok('disconnect with no account removes them all', r.json.connected === false, r.json); + r = await A('/api/audit'); const evs = (r.json.events || []).map(e => e.event); ok('connecting and sending are in the activity log (no passwords, no text)', evs.includes('mail connected') && evs.includes('mail sent') && !JSON.stringify(r.json).includes(PASS) && !JSON.stringify(r.json).includes('not bold'), evs); + ok('the server is still running', child.exitCode === null, log.slice(-300)); + + child.kill(); imapSrv.close(); smtp.close(); fs.rmSync(dir, { recursive: true, force: true }); + console.log(failed ? `\n${failed} check(s) failed` : '\nAll IMAP mail checks passed'); process.exit(failed ? 1 : 0); +})().catch(e => { console.error('TEST CRASHED:', e); process.exit(1); }); diff --git a/tools/mail-test.js b/tools/mail-test.js index 1f1dd02..5a80bef 100644 --- a/tools/mail-test.js +++ b/tools/mail-test.js @@ -96,7 +96,7 @@ const SECRET = 'mail-test-secret-'.repeat(3); r = await call(null, `/api/mail/callback?code=GOODCODE&state=${au2.searchParams.get('state')}`); ok('the real callback connects and sends the person back to the Mail tab', r.status === 302 && r.headers.get('location') === '/#/mail', [r.status, r.headers.get('location')]); r = await call(null, `/api/mail/callback?code=GOODCODE&state=${au2.searchParams.get('state')}`); ok('the sign-in link works only once', r.status === 400, r.status); r = await call('unate', '/api/mail/status'); ok('now connected, with the address and the unread count', r.json.connected && r.json.address === 'nate@outlook.com' && r.json.unread === 3, r.json); - const raw = fs.readFileSync(path.join(dir, 'users', 'unate', 'mail.json'), 'utf8'); ok('the tokens are saved encrypted', !/AT1|RT1/.test(raw), raw.slice(0, 80)); + const raw = fs.readFileSync(path.join(dir, 'users', 'unate', 'mail-accounts.json'), 'utf8'); ok('the tokens are saved encrypted', !/AT1|RT1/.test(raw), raw.slice(0, 80)); r = await call('uann', '/api/mail/status'); ok("another person is not connected to Nate's mailbox", r.json.connected === false, r.json); r = await call('uann', '/api/mail/messages'); ok("and cannot read it", r.status === 409, r.status); @@ -105,7 +105,7 @@ const SECRET = 'mail-test-secret-'.repeat(3); ok('folders: the usual six first with unread counts, then your own; internal ones hidden', names.slice(0, 6).join() === 'Inbox,Drafts,Sent,Archive,Junk,Deleted' && names.includes('Receipts') && !names.includes('Outbox') && r.json[0].unread === 3 && names.filter(n => n === 'Inbox').length === 1, r.json); r = await call('unate', '/api/mail/messages?folder=inbox'); ok('the inbox lists messages, newest first, with who, read state and attachment flag', r.status === 200 && r.json.items.length === 2 && r.json.items[0].from.address === 'ann@example.com' && r.json.items[0].read === false && r.json.items[0].attach === true && r.json.items[1].flagged === true, r.json); ok('and says there may be more', r.json.more === true && r.json.next === 2, r.json); - r = await call('unate', '/api/mail/messages?folder=inbox&q=second'); ok('search finds a message', r.json.items.length === 1 && r.json.items[0].id === 'M2', r.json); + r = await call('unate', '/api/mail/messages?folder=inbox&q=second'); ok('search finds a message', r.json.items.length === 1 && /~M2$/.test(r.json.items[0].id), r.json); r = await call('unate', '/api/mail/messages?folder=../me'); ok('a folder name cannot break out of the mail path', r.status === 400, r.status); r = await call('unate', '/api/mail/message?id=M1'); const mm = r.json; ok('a message opens with sender, recipients and subject', r.status === 200 && mm.subject === 'Hello there' && mm.to[0].address === 'nate@outlook.com', mm); @@ -138,12 +138,11 @@ const SECRET = 'mail-test-secret-'.repeat(3); ms.revoked = true; ms.goodTokens = new Set(); r = await call('unate', '/api/mail/messages'); ok('if Microsoft revokes access, the person is told to connect again (409)', r.status === 409 && /Connect again/.test(r.json.error), r); r = await call('unate', '/api/mail/status'); ok('and shown as not connected', r.json.connected === false, r.json); - ok('the dead sign-in is removed from disk', !fs.existsSync(path.join(dir, 'users', 'unate', 'mail.json'))); // reconnect, then disconnect ms.revoked = false; r = await call('unate', '/api/mail/connect'); const au3 = new URL(r.json.url); ms.challenge = au3.searchParams.get('code_challenge'); await call(null, `/api/mail/callback?code=GOODCODE&state=${au3.searchParams.get('state')}`); r = await call('unate', '/api/mail/status'); ok('connecting again works', r.json.connected === true, r.json); - r = await post('unate', '/api/mail/disconnect'); r = await call('unate', '/api/mail/status'); ok('disconnect removes the sign-in', r.json.connected === false && !fs.existsSync(path.join(dir, 'users', 'unate', 'mail.json')), r.json); + r = await post('unate', '/api/mail/disconnect'); r = await call('unate', '/api/mail/status'); ok('disconnect removes the sign-in', r.json.connected === false, r.json); r = await call('unate', '/api/audit'); const evs = (r.json.events || []).map(e => e.event); ok('connecting, sending and disconnecting are in the activity log (without any message text)', ['mail connected', 'mail sent', 'mail disconnected'].every(e => evs.includes(e)) && !JSON.stringify(r.json).includes('not bold'), evs); r = await call('unate', '/webmail/'); ok('the old /webmail/ pass-through is gone', r.status === 404 || r.status === 200 && !/Cypht/i.test(r.text), r.status); // ================= admin: Microsoft app settings (environment version) ================= @@ -162,8 +161,8 @@ const SECRET = 'mail-test-secret-'.repeat(3); const child2 = spawn(process.execPath, ['server.js'], { cwd: root, env: env2 }); child2.stderr.on('data', () => {}); child2.stdout.on('data', () => {}); for (let i = 0; i < 60; i++) { try { await fetch(base2 + '/login'); break; } catch { await wait(100); } } const call2 = async (uid, p, body, method) => { const rr = await fetch(base2 + p, { method: method || (body ? 'POST' : 'GET'), headers: { cookie: ck(uid), 'content-type': 'application/json' }, body: body ? JSON.stringify(body) : undefined }); const tx = await rr.text(); let j; try { j = JSON.parse(tx); } catch { j = tx; } return { status: rr.status, json: j }; }; - r = await call2('unate', '/api/config'); ok('without settings: Mail is off for everyone, but the admin can reach the setup', r.json.mail === '' && r.json.mailAdmin === '1', r.json); - r = await call2('uann', '/api/mail/status'); ok('and ordinary mail requests say it is not set up', r.status === 403, r.status); + r = await call2('unate', '/api/config'); ok('without the Microsoft app: Mail stays on (Gmail and iCloud need no setup) but there is no Outlook.com button, and the admin can reach the setup', r.json.mail === '1' && r.json.outlook === '' && r.json.mailAdmin === '1', r.json); + r = await call2('uann', '/api/mail/connect'); ok('and connecting Outlook.com says it is not set up', r.status === 403, r.status); r = await call2('unate', '/api/mail/admin'); ok('the setup page reports nothing saved yet', r.json.source === 'none' && r.json.hasSecret === false, r.json); r = await call2('unate', '/api/mail/admin', { clientId: 'not-a-guid' }); ok('a malformed client ID is refused with an explanation', r.status === 400 && /36-character/.test(r.json.error), r.json); const soon = new Date(Date.now() + 10 * 864e5).toISOString().slice(0, 10); @@ -171,7 +170,7 @@ const SECRET = 'mail-test-secret-'.repeat(3); ok('the secret is never sent back', !JSON.stringify(r.json).includes('S3CRET') && !JSON.stringify((await call2('unate', '/api/mail/admin')).json).includes('S3CRET')); ok('and is stored encrypted', !fs.readFileSync(path.join(dir2, 'mail-config.json'), 'utf8').includes('S3CRET')); r = await call2('unate', '/api/mail/admin'); ok('days until the secret expires are reported (about 10)', r.json.days >= 9 && r.json.days <= 11, r.json); - r = await call2('unate', '/api/config'); ok('saving switches Mail on at once, with no restart', r.json.mail === '1', r.json); + r = await call2('unate', '/api/config'); ok('saving switches the Outlook.com button on at once, with no restart', r.json.outlook === '1', r.json); ms.secret = 'OTHER'; { const c1 = await call2('unate', '/api/mail/connect'); const q1 = new URL(c1.json.url); ms.challenge = q1.searchParams.get('code_challenge'); const rr = await fetch(base2 + '/api/mail/callback?code=GOODCODE&state=' + q1.searchParams.get('state'), { redirect: 'manual' }); const tx = await rr.text(); ok('a wrong secret shows up when connecting, explained in plain words', rr.status === 400 && /secret is wrong/i.test(tx) && /Value/.test(tx), [rr.status, tx.slice(-200)]); @@ -186,7 +185,7 @@ const SECRET = 'mail-test-secret-'.repeat(3); r = await call2('unate', '/api/mail/admin', { clientSecret: 'NEWSECRET' }); ok('but a new secret has to be proven again', !r.json.verifiedAt, r.json); } r = await call2('unate', '/api/mail/admin/test', {}); ok('(the old test button endpoint is gone: Microsoft cannot check an app without a real sign-in)', r.status === 404 || r.status === 400, r.status); r = await call2('uann', '/api/mail/admin', { clientId: 'aaaaaaaa-1111-2222-3333-bbbbbbbbbbbb' }); ok('only an admin can save', r.status === 403, r.status); - r = await call2('unate', '/api/mail/admin', { clientId: '' }); r = await call2('unate', '/api/mail/admin', { clearSecret: true }); r = await call2('unate', '/api/config'); ok('removing the settings switches Mail off again', r.json.mail === '' && !fs.existsSync(path.join(dir2, 'mail-config.json')), r.json); + r = await call2('unate', '/api/mail/admin', { clientId: '' }); r = await call2('unate', '/api/mail/admin', { clearSecret: true }); r = await call2('unate', '/api/config'); ok('removing the settings switches the Outlook.com button off again', r.json.outlook === '' && !fs.existsSync(path.join(dir2, 'mail-config.json')), r.json); r = await call2('unate', '/api/audit'); ok('changes to the setup are in the activity log', (r.json.events || []).some(e => e.event === 'mail setup changed'), r.json); child2.kill(); fs.rmSync(dir2, { recursive: true, force: true });