Add K-LOVE support and Podman-friendly container deployment
This commit is contained in:
1 parent
c0ebe118cb
commit
18f777e3e7
19 files changed
+2560
-682
No files matched your search
@@ -0,0 +1,11 @@
|
||||
.git
|
||||
.gitignore
|
||||
.venv
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
*.log
|
||||
*.db
|
||||
nohup.out
|
||||
icon.png
|
||||
@@ -11,6 +11,7 @@ __pycache__/
|
||||
|
||||
# Logs
|
||||
playlist_log.txt
|
||||
app_log.txt
|
||||
page.html
|
||||
|
||||
# Selenium/ChromeDriver
|
||||
@@ -28,3 +29,7 @@ Thumbs.db
|
||||
|
||||
# Environment variables (if you add a .env file later)
|
||||
.env
|
||||
|
||||
# Debug / local runtime outputs
|
||||
debug_scrapes/
|
||||
test_live_klove.py
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN mkdir -p /data
|
||||
|
||||
# System deps for headless Chromium + Selenium
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
chromium \
|
||||
chromium-driver \
|
||||
ca-certificates \
|
||||
fonts-liberation \
|
||||
libasound2 \
|
||||
libatk-bridge2.0-0 \
|
||||
libatk1.0-0 \
|
||||
libc6 \
|
||||
libcairo2 \
|
||||
libcups2 \
|
||||
libdbus-1-3 \
|
||||
libdrm2 \
|
||||
libexpat1 \
|
||||
libfontconfig1 \
|
||||
libgbm1 \
|
||||
libglib2.0-0 \
|
||||
libgtk-3-0 \
|
||||
libnspr4 \
|
||||
libnss3 \
|
||||
libpango-1.0-0 \
|
||||
libx11-6 \
|
||||
libx11-xcb1 \
|
||||
libxcb1 \
|
||||
libxcomposite1 \
|
||||
libxdamage1 \
|
||||
libxext6 \
|
||||
libxfixes3 \
|
||||
libxrandr2 \
|
||||
libxrender1 \
|
||||
libxss1 \
|
||||
libxtst6 \
|
||||
xdg-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY requirements.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
# Override if your distro path differs
|
||||
ENV CHROME_BINARY=/usr/bin/chromium \
|
||||
CHROMEDRIVER_PATH=/usr/bin/chromedriver \
|
||||
JOURNEYFM_DATA_DIR=/data \
|
||||
JOURNEYFM_CONTAINER=1
|
||||
|
||||
VOLUME ["/data"]
|
||||
|
||||
CMD ["python", "run_container.py"]
|
||||
@@ -7,6 +7,7 @@ Desktop app + script that pulls recently played songs from Journey FM / Spirit F
|
||||
- Scrapes recently played songs from:
|
||||
- `https://www.myjourneyfm.com/recently-played/`
|
||||
- `https://spiritfm.com/ajax/now_playing_history.txt`
|
||||
- `https://www.klove.com/music/songs`
|
||||
- Matches tracks in your Plex Music library with title/artist normalization.
|
||||
- Creates or updates your configured Plex playlist.
|
||||
- Avoids adding duplicates already in the target playlist.
|
||||
@@ -93,17 +94,19 @@ python journey_fm_app.py
|
||||
|
||||
Use the GUI setup wizard (first launch) or edit `config.json` directly.
|
||||
|
||||
The Plex token is now stored in your OS credential store via `keyring` when available.
|
||||
On Windows this means the app no longer needs to persist `PLEX_TOKEN` in `config.json`.
|
||||
|
||||
Example `config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"PLEX_TOKEN": "your-plex-token",
|
||||
"SERVER_IP": "192.168.1.100",
|
||||
"PLAYLIST_NAME": "Journey FM Recently Played",
|
||||
"AUTO_UPDATE": true,
|
||||
"UPDATE_INTERVAL": 15,
|
||||
"UPDATE_UNIT": "Minutes",
|
||||
"SELECTED_STATIONS": "journey_fm,spirit_fm"
|
||||
"SELECTED_STATIONS": "journey_fm,spirit_fm,klove"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -112,6 +115,10 @@ You can also provide runtime env vars:
|
||||
- `PLEX_TOKEN`
|
||||
- `SERVER_IP`
|
||||
- `PLAYLIST_NAME`
|
||||
- `AUTO_UPDATE`
|
||||
- `UPDATE_INTERVAL`
|
||||
- `UPDATE_UNIT`
|
||||
- `SELECTED_STATIONS`
|
||||
|
||||
Optional browser override:
|
||||
|
||||
@@ -127,20 +134,75 @@ Run one update cycle from CLI:
|
||||
python main.py
|
||||
```
|
||||
|
||||
## Container Deployment
|
||||
|
||||
Container mode is intended for scheduled/headless playlist updates. The container now runs as a long-lived loop and sleeps between syncs instead of relying on restart churn.
|
||||
|
||||
1. Create a `.env` file in the project root:
|
||||
|
||||
```env
|
||||
PLEX_TOKEN=your-plex-token
|
||||
SERVER_IP=192.168.1.100
|
||||
PLAYLIST_NAME=Journey FM Recently Played
|
||||
SELECTED_STATIONS=journey_fm,spirit_fm,klove
|
||||
UPDATE_INTERVAL=15
|
||||
UPDATE_UNIT=Minutes
|
||||
CONTAINER_RUN_MODE=loop
|
||||
```
|
||||
|
||||
2. Build and run with Docker Compose:
|
||||
|
||||
```bash
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
Podman equivalent:
|
||||
|
||||
```bash
|
||||
podman compose up -d --build
|
||||
```
|
||||
|
||||
3. Trigger a one-off run and review logs:
|
||||
|
||||
```bash
|
||||
docker-compose run --rm -e CONTAINER_RUN_MODE=once journey-fm-playlist
|
||||
docker-compose logs -f journey-fm-playlist
|
||||
```
|
||||
|
||||
Podman equivalent:
|
||||
|
||||
```bash
|
||||
podman compose run --rm -e CONTAINER_RUN_MODE=once journey-fm-playlist
|
||||
podman compose logs -f journey-fm-playlist
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- The container uses Chromium + chromedriver in headless mode.
|
||||
- Runtime state is stored in the named volume mounted at `/data`, so `config.json`, `playlist_history.db`, `playlist_log.txt`, `amazon_buy_list.txt`, and `debug_scrapes/` persist without bind-mounting the repo.
|
||||
- GUI (`journey_fm_app.py`) is still intended to run natively on your desktop.
|
||||
- In container mode, environment variables are preferred over local config or keyring storage.
|
||||
- The Compose file is Podman-friendly because it uses a named volume instead of a source bind mount.
|
||||
|
||||
## GUI Notes
|
||||
|
||||
- Minimize-to-tray on close when tray is available.
|
||||
- Tray menu supports Show / Update Now / Quit.
|
||||
- Auto-update runs on configurable interval.
|
||||
- Works on X11 and Wayland Linux sessions.
|
||||
- Updated visual style includes a dashboard hero section, richer status chips, and a refreshed control layout.
|
||||
- Setup now includes Plex connection testing, keyring-backed token storage, playlist browsing with item counts, music-only filtering, and typed creation of new playlist names.
|
||||
|
||||
## Output Files
|
||||
|
||||
- `config.json` - runtime settings shared by GUI and CLI
|
||||
- `config.json` - non-secret runtime settings shared by GUI and CLI
|
||||
- `playlist_history.db` - update history and statistics data
|
||||
- `playlist_log.txt` - GUI update logs
|
||||
- `amazon_buy_list.txt` - missing songs with Amazon links
|
||||
- `playlist_export.csv` - exported playlist from GUI action
|
||||
- `debug_scrapes/` - captured raw station responses for scraper debugging
|
||||
|
||||
In container mode these files live under `/data` inside the container volume.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -153,15 +215,18 @@ python main.py
|
||||
## Core Files
|
||||
|
||||
- `journey_fm_app.py` - desktop GUI and tray app
|
||||
- `main.py` - scraping + Plex update pipeline
|
||||
- `main.py` - CLI entrypoint for a single update run
|
||||
- `run_container.py` - loop/once runner used by container deployments
|
||||
- `journeyfm/` - shared service modules (config, scraper, Plex, history, orchestration)
|
||||
- `requirements.txt` - Python dependencies
|
||||
- `install.sh` - Linux setup helper
|
||||
|
||||
## Pre-Release Checklist
|
||||
|
||||
- Verify `config.json` has valid `PLEX_TOKEN`, `SERVER_IP`, and `PLAYLIST_NAME`.
|
||||
- Verify `config.json` has valid `SERVER_IP`, `PLAYLIST_NAME`, and station/schedule settings.
|
||||
- Run one CLI update: `python main.py` and confirm songs/history are written.
|
||||
- Run GUI update: `python journey_fm_app.py` and verify status/log output updates.
|
||||
- Confirm tray behavior (minimize to tray, Show/Update/Quit actions).
|
||||
- On Windows/Linux, confirm Chrome/Chromium detection or set `CHROME_BINARY`.
|
||||
- Export playlist once and verify `playlist_export.csv` is created.
|
||||
- Export playlist once and verify `playlist_export.csv` is created.
|
||||
- For container mode, run `docker-compose run --rm -e CONTAINER_RUN_MODE=once journey-fm-playlist` and verify logs/history output.
|
||||
@@ -0,0 +1,24 @@
|
||||
services:
|
||||
journey-fm-playlist:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: journey-fm-playlist
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
PLEX_TOKEN: ${PLEX_TOKEN}
|
||||
SERVER_IP: ${SERVER_IP}
|
||||
PLAYLIST_NAME: ${PLAYLIST_NAME:-Journey FM Recently Played}
|
||||
SELECTED_STATIONS: ${SELECTED_STATIONS:-journey_fm,spirit_fm}
|
||||
AUTO_UPDATE: ${AUTO_UPDATE:-true}
|
||||
UPDATE_INTERVAL: ${UPDATE_INTERVAL:-15}
|
||||
UPDATE_UNIT: ${UPDATE_UNIT:-Minutes}
|
||||
CONTAINER_RUN_MODE: ${CONTAINER_RUN_MODE:-loop}
|
||||
CHROME_BINARY: /usr/bin/chromium
|
||||
CHROMEDRIVER_PATH: /usr/bin/chromedriver
|
||||
JOURNEYFM_DATA_DIR: /data
|
||||
volumes:
|
||||
- journey-fm-data:/data
|
||||
|
||||
volumes:
|
||||
journey-fm-data:
|
||||
+1100
-149
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1 @@
|
||||
"""Shared services for Journey FM Playlist Creator."""
|
||||
@@ -0,0 +1,170 @@
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from journeyfm.paths import data_path
|
||||
|
||||
try:
|
||||
import keyring
|
||||
except Exception:
|
||||
keyring = None
|
||||
|
||||
CONFIG_PATH = data_path("config.json")
|
||||
KEYRING_SERVICE = "JourneyFMPlaylistCreator"
|
||||
DEFAULT_CONFIG = {
|
||||
"PLEX_TOKEN": "",
|
||||
"SERVER_IP": "",
|
||||
"PLAYLIST_NAME": "Journey FM Recently Played",
|
||||
"AUTO_UPDATE": False,
|
||||
"UPDATE_INTERVAL": 15,
|
||||
"UPDATE_UNIT": "Minutes",
|
||||
"SELECTED_STATIONS": ["journey_fm", "spirit_fm"],
|
||||
}
|
||||
NON_SECRET_KEYS = {
|
||||
"SERVER_IP",
|
||||
"PLAYLIST_NAME",
|
||||
"AUTO_UPDATE",
|
||||
"UPDATE_INTERVAL",
|
||||
"UPDATE_UNIT",
|
||||
"SELECTED_STATIONS",
|
||||
}
|
||||
|
||||
|
||||
def is_containerized():
|
||||
return os.path.exists("/.dockerenv") or os.getenv("JOURNEYFM_CONTAINER") == "1"
|
||||
|
||||
|
||||
def _normalize_selected_stations(value):
|
||||
if isinstance(value, str):
|
||||
return [item.strip() for item in value.split(",") if item.strip()]
|
||||
if isinstance(value, list):
|
||||
return [str(item).strip() for item in value if str(item).strip()]
|
||||
return list(DEFAULT_CONFIG["SELECTED_STATIONS"])
|
||||
|
||||
|
||||
def _read_config_file(config_path=CONFIG_PATH):
|
||||
if not Path(config_path).exists():
|
||||
return {}
|
||||
try:
|
||||
with open(config_path, "r", encoding="utf-8") as file_handle:
|
||||
return json.load(file_handle)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _write_config_file(config, config_path=CONFIG_PATH):
|
||||
with open(config_path, "w", encoding="utf-8") as file_handle:
|
||||
json.dump(config, file_handle, indent=2)
|
||||
|
||||
|
||||
def get_secret(key):
|
||||
env_value = os.getenv(key, "").strip()
|
||||
if env_value:
|
||||
return env_value
|
||||
if not is_containerized() and keyring is not None:
|
||||
try:
|
||||
stored = keyring.get_password(KEYRING_SERVICE, key)
|
||||
if stored:
|
||||
return stored
|
||||
except Exception:
|
||||
pass
|
||||
file_config = _read_config_file()
|
||||
return str(file_config.get(key, "")).strip()
|
||||
|
||||
|
||||
def set_secret(key, value):
|
||||
if not value:
|
||||
delete_secret(key)
|
||||
return
|
||||
if not is_containerized() and keyring is not None:
|
||||
try:
|
||||
keyring.set_password(KEYRING_SERVICE, key, value)
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
file_config = _read_config_file()
|
||||
file_config[key] = value
|
||||
_write_config_file(file_config)
|
||||
|
||||
|
||||
def delete_secret(key):
|
||||
if not is_containerized() and keyring is not None:
|
||||
try:
|
||||
keyring.delete_password(KEYRING_SERVICE, key)
|
||||
except Exception:
|
||||
pass
|
||||
file_config = _read_config_file()
|
||||
if key in file_config:
|
||||
del file_config[key]
|
||||
_write_config_file(file_config)
|
||||
|
||||
|
||||
def migrate_legacy_secrets(config_path=CONFIG_PATH):
|
||||
file_config = _read_config_file(config_path)
|
||||
token = str(file_config.get("PLEX_TOKEN", "")).strip()
|
||||
if token and not is_containerized() and keyring is not None:
|
||||
try:
|
||||
keyring.set_password(KEYRING_SERVICE, "PLEX_TOKEN", token)
|
||||
del file_config["PLEX_TOKEN"]
|
||||
_write_config_file(file_config, config_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def load_runtime_config(config_path=CONFIG_PATH):
|
||||
migrate_legacy_secrets(config_path)
|
||||
config = dict(DEFAULT_CONFIG)
|
||||
file_config = _read_config_file(config_path)
|
||||
config.update({key: value for key, value in file_config.items() if key in NON_SECRET_KEYS})
|
||||
config["PLEX_TOKEN"] = get_secret("PLEX_TOKEN")
|
||||
|
||||
env_server = os.getenv("SERVER_IP", "").strip()
|
||||
env_playlist = os.getenv("PLAYLIST_NAME", "").strip()
|
||||
env_stations = os.getenv("SELECTED_STATIONS", "").strip()
|
||||
env_auto_update = os.getenv("AUTO_UPDATE", "").strip()
|
||||
env_interval = os.getenv("UPDATE_INTERVAL", "").strip()
|
||||
env_unit = os.getenv("UPDATE_UNIT", "").strip()
|
||||
|
||||
if env_server:
|
||||
config["SERVER_IP"] = env_server
|
||||
if env_playlist:
|
||||
config["PLAYLIST_NAME"] = env_playlist
|
||||
if env_stations:
|
||||
config["SELECTED_STATIONS"] = _normalize_selected_stations(env_stations)
|
||||
else:
|
||||
config["SELECTED_STATIONS"] = _normalize_selected_stations(config.get("SELECTED_STATIONS"))
|
||||
if env_auto_update:
|
||||
config["AUTO_UPDATE"] = env_auto_update.lower() in {"1", "true", "yes", "on"}
|
||||
if env_interval:
|
||||
try:
|
||||
config["UPDATE_INTERVAL"] = int(env_interval)
|
||||
except ValueError:
|
||||
pass
|
||||
if env_unit:
|
||||
config["UPDATE_UNIT"] = env_unit
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def save_runtime_config(config, config_path=CONFIG_PATH):
|
||||
config = dict(config)
|
||||
token = str(config.pop("PLEX_TOKEN", "")).strip()
|
||||
if token:
|
||||
set_secret("PLEX_TOKEN", token)
|
||||
|
||||
persisted = {}
|
||||
for key in NON_SECRET_KEYS:
|
||||
if key not in config:
|
||||
continue
|
||||
value = config[key]
|
||||
if isinstance(value, list):
|
||||
persisted[key] = ",".join(value)
|
||||
else:
|
||||
persisted[key] = value
|
||||
_write_config_file(persisted, config_path)
|
||||
|
||||
|
||||
def get_display_config(config_path=CONFIG_PATH):
|
||||
config = load_runtime_config(config_path)
|
||||
config["PLEX_TOKEN"] = get_secret("PLEX_TOKEN")
|
||||
return config
|
||||
@@ -0,0 +1,73 @@
|
||||
import json
|
||||
import sqlite3
|
||||
from datetime import datetime
|
||||
|
||||
from journeyfm.paths import data_path
|
||||
|
||||
HISTORY_COLUMNS = {
|
||||
"status": "TEXT DEFAULT 'success'",
|
||||
"scraped_count": "INTEGER DEFAULT 0",
|
||||
"matched_count": "INTEGER DEFAULT 0",
|
||||
"duplicate_count": "INTEGER DEFAULT 0",
|
||||
"skipped_count": "INTEGER DEFAULT 0",
|
||||
"station_breakdown": "TEXT DEFAULT '[]'",
|
||||
"skipped_songs": "TEXT DEFAULT '[]'",
|
||||
"error_message": "TEXT DEFAULT ''",
|
||||
}
|
||||
|
||||
|
||||
def init_history_db(db_path=None):
|
||||
db_path = db_path or data_path("playlist_history.db")
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS history (
|
||||
id INTEGER PRIMARY KEY,
|
||||
date TEXT,
|
||||
added_count INTEGER,
|
||||
added_songs TEXT,
|
||||
missing_count INTEGER,
|
||||
missing_songs TEXT
|
||||
)
|
||||
"""
|
||||
)
|
||||
cursor.execute("PRAGMA table_info(history)")
|
||||
existing_columns = {row[1] for row in cursor.fetchall()}
|
||||
for column_name, definition in HISTORY_COLUMNS.items():
|
||||
if column_name not in existing_columns:
|
||||
cursor.execute(f"ALTER TABLE history ADD COLUMN {column_name} {definition}")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
|
||||
def save_history_entry(result, db_path=None):
|
||||
db_path = db_path or data_path("playlist_history.db")
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO history (
|
||||
date, added_count, added_songs, missing_count, missing_songs,
|
||||
status, scraped_count, matched_count, duplicate_count, skipped_count,
|
||||
station_breakdown, skipped_songs, error_message
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
datetime.now().isoformat(),
|
||||
result.get("added_count", 0),
|
||||
json.dumps(result.get("added_songs", [])),
|
||||
result.get("missing_count", 0),
|
||||
json.dumps(result.get("missing_songs", [])),
|
||||
result.get("status", "success"),
|
||||
result.get("scraped_count", 0),
|
||||
result.get("matched_count", 0),
|
||||
result.get("duplicate_count", 0),
|
||||
result.get("skipped_count", 0),
|
||||
json.dumps(result.get("station_breakdown", [])),
|
||||
json.dumps(result.get("skipped_songs", [])),
|
||||
result.get("error_message", ""),
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
@@ -0,0 +1,16 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def get_data_dir() -> Path:
|
||||
data_dir = os.getenv("JOURNEYFM_DATA_DIR", "").strip()
|
||||
if data_dir:
|
||||
path = Path(data_dir)
|
||||
else:
|
||||
path = Path(".")
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def data_path(*parts: str) -> Path:
|
||||
return get_data_dir().joinpath(*parts)
|
||||
@@ -0,0 +1,273 @@
|
||||
import urllib.parse
|
||||
|
||||
from plexapi.myplex import MyPlexAccount
|
||||
from plexapi.server import PlexServer
|
||||
|
||||
|
||||
class PlexConnectionError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def normalize_server_target(server_ip):
|
||||
target = (server_ip or "").strip()
|
||||
if not target:
|
||||
return ""
|
||||
if "://" not in target:
|
||||
target = f"http://{target}"
|
||||
parsed = urllib.parse.urlparse(target)
|
||||
return (parsed.hostname or "").strip().lower()
|
||||
|
||||
|
||||
def resolve_plex_server_url(token, server_ip):
|
||||
target_host = normalize_server_target(server_ip)
|
||||
if not token:
|
||||
raise PlexConnectionError("Missing Plex token")
|
||||
if not target_host:
|
||||
raise PlexConnectionError("Missing Plex server address")
|
||||
|
||||
try:
|
||||
account = MyPlexAccount(token=token)
|
||||
except Exception as exc:
|
||||
raise PlexConnectionError(f"Failed to authenticate with Plex: {exc}") from exc
|
||||
|
||||
for resource in account.resources():
|
||||
for connection in resource.connections:
|
||||
connection_host = normalize_server_target(connection.uri)
|
||||
address = (connection.address or "").strip().lower()
|
||||
if connection_host == target_host or target_host == address or target_host in address:
|
||||
return connection.uri
|
||||
|
||||
raise PlexConnectionError(
|
||||
f"Server not found at {server_ip}. Verify the configured address and that Plex is reachable."
|
||||
)
|
||||
|
||||
|
||||
def connect_to_plex_server(token, server_ip):
|
||||
server_url = resolve_plex_server_url(token, server_ip)
|
||||
try:
|
||||
return PlexServer(server_url, token)
|
||||
except Exception as exc:
|
||||
raise PlexConnectionError(f"Failed to connect to Plex server at {server_url}: {exc}") from exc
|
||||
|
||||
|
||||
def playlist_item_count(playlist):
|
||||
leaf_count = getattr(playlist, "leafCount", None)
|
||||
if isinstance(leaf_count, int):
|
||||
return leaf_count
|
||||
try:
|
||||
return len(playlist.items())
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def fetch_playlists(token, server_ip, music_only=False):
|
||||
plex = connect_to_plex_server(token, server_ip)
|
||||
return fetch_playlists_for_plex(plex, music_only)
|
||||
|
||||
|
||||
def fetch_playlists_for_plex(plex, music_only=False):
|
||||
playlists = []
|
||||
for playlist in plex.playlists():
|
||||
playlist_type = getattr(playlist, "playlistType", "unknown")
|
||||
if music_only and playlist_type != "audio":
|
||||
continue
|
||||
playlists.append(
|
||||
{
|
||||
"title": playlist.title,
|
||||
"playlist_type": playlist_type,
|
||||
"item_count": playlist_item_count(playlist),
|
||||
}
|
||||
)
|
||||
playlists.sort(key=lambda item: (item["title"].lower(), item["playlist_type"]))
|
||||
return playlists
|
||||
|
||||
|
||||
def validate_playlist_target(token, server_ip, playlist_name, music_only=False):
|
||||
plex = connect_to_plex_server(token, server_ip)
|
||||
playlist_name = (playlist_name or "").strip()
|
||||
if not playlist_name:
|
||||
raise PlexConnectionError("Playlist name is required")
|
||||
|
||||
playlists = fetch_playlists_for_plex(plex, music_only=False)
|
||||
for playlist in playlists:
|
||||
if playlist["title"].strip().lower() == playlist_name.lower():
|
||||
if music_only and playlist["playlist_type"] != "audio":
|
||||
raise PlexConnectionError("Selected playlist is not a music playlist")
|
||||
return {
|
||||
"exists": True,
|
||||
"playlist": playlist,
|
||||
"message": f"Ready to write to existing playlist '{playlist_name}'",
|
||||
}
|
||||
|
||||
try:
|
||||
plex.library.section("Music")
|
||||
except Exception as exc:
|
||||
raise PlexConnectionError(f"Music library is not accessible on this server: {exc}") from exc
|
||||
|
||||
return {
|
||||
"exists": False,
|
||||
"playlist": {
|
||||
"title": playlist_name,
|
||||
"playlist_type": "audio",
|
||||
"item_count": 0,
|
||||
},
|
||||
"message": f"Playlist '{playlist_name}' will be created on first sync",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_artist(name):
|
||||
"""Return a simplified artist token for fuzzy comparison."""
|
||||
import re
|
||||
name = name.lower()
|
||||
# Drop featured artists — everything after feat/ft/with/w/
|
||||
name = re.sub(r"\s+(feat\.?|ft\.?|featuring|w/|with)\s+.*", "", name)
|
||||
# Drop parenthetical / bracketed suffixes
|
||||
name = re.sub(r"[\(\[].*?[\)\]]", "", name)
|
||||
# Expand & / + to "and"
|
||||
name = re.sub(r"\s*[&+]\s*", " and ", name)
|
||||
# Collapse articles at start: "the foo" → "foo"
|
||||
name = re.sub(r"^(the|a|an)\s+", "", name)
|
||||
# Remove common punctuation that Plex sometimes strips
|
||||
name = re.sub(r"['\.\-,]", "", name)
|
||||
# Collapse whitespace
|
||||
name = re.sub(r"\s+", " ", name).strip()
|
||||
return name
|
||||
|
||||
|
||||
def _artists_match(search_artist, track_artist_raw):
|
||||
"""
|
||||
Return True if the station artist string plausibly refers to the same
|
||||
artist stored in Plex, using several normalisation / containment passes.
|
||||
"""
|
||||
sa = _normalize_artist(search_artist)
|
||||
ta = _normalize_artist(track_artist_raw)
|
||||
|
||||
if not sa or not ta:
|
||||
return False
|
||||
|
||||
# Exact normalised match
|
||||
if sa == ta:
|
||||
return True
|
||||
|
||||
# One fully contains the other (e.g. "Brandon Lake" in "Brandon Lake & Maverick")
|
||||
if sa in ta or ta in sa:
|
||||
return True
|
||||
|
||||
# Token-set overlap: every token in the shorter name appears in the longer
|
||||
sa_tokens = set(sa.split())
|
||||
ta_tokens = set(ta.split())
|
||||
if sa_tokens and ta_tokens:
|
||||
shorter = sa_tokens if len(sa_tokens) <= len(ta_tokens) else ta_tokens
|
||||
longer = ta_tokens if len(sa_tokens) <= len(ta_tokens) else sa_tokens
|
||||
# All tokens of the shorter name found in the longer (handles "for king country" ↔ "for king and country")
|
||||
if shorter <= longer:
|
||||
return True
|
||||
# At least >60 % of tokens match (handles minor word-count diffs)
|
||||
overlap = len(shorter & longer)
|
||||
if overlap / len(shorter) >= 0.6:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def create_or_update_playlist(plex, songs, playlist_name, dry_run=False):
|
||||
import re
|
||||
music_library = plex.library.section("Music")
|
||||
tracks = []
|
||||
seen_keys = set()
|
||||
missing = []
|
||||
skipped = []
|
||||
added_songs = []
|
||||
duplicate_songs = []
|
||||
|
||||
for song in songs:
|
||||
clean_title = re.sub(r"\([^)]*\)", "", song["title"]).strip()
|
||||
clean_title = re.sub(r"\s+", " ", clean_title)
|
||||
clean_artist = re.sub(r"\([^)]*\)", "", song["artist"]).strip()
|
||||
clean_artist = re.sub(r"\s+", " ", clean_artist)
|
||||
|
||||
if not clean_title or len(clean_title) < 3 or clean_title.lower() in {"by", "recently played", "now playing:", "search"}:
|
||||
skipped.append({"title": song["title"], "artist": song["artist"], "reason": "invalid-title"})
|
||||
continue
|
||||
if re.match(r"^[!?\s]+$", clean_title):
|
||||
skipped.append({"title": song["title"], "artist": song["artist"], "reason": "punctuation-only-title"})
|
||||
continue
|
||||
|
||||
search_title = re.sub(r"[!?]", "", clean_title).strip()
|
||||
search_artist = re.sub(r"[!?]", "", clean_artist).strip()
|
||||
# Expand & / + in title for Plex search (artist expansion handled by _artists_match)
|
||||
search_title = re.sub(r"\s*[&+]\s*", " and ", search_title, flags=re.IGNORECASE)
|
||||
|
||||
results = music_library.searchTracks(title=search_title)
|
||||
if not results:
|
||||
# Secondary search: try the raw title in case Plex stores it differently
|
||||
results = music_library.searchTracks(title=clean_title)
|
||||
if not results:
|
||||
missing.append({"title": song["title"], "artist": song["artist"], "reason": "not-found"})
|
||||
continue
|
||||
|
||||
matched = False
|
||||
for track in results:
|
||||
try:
|
||||
track_artist_raw = track.artist().title
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
if _artists_match(search_artist, track_artist_raw):
|
||||
if track.ratingKey not in seen_keys:
|
||||
tracks.append((track, song))
|
||||
seen_keys.add(track.ratingKey)
|
||||
matched = True
|
||||
break
|
||||
|
||||
if not matched:
|
||||
missing.append({"title": song["title"], "artist": song["artist"], "reason": "artist-mismatch"})
|
||||
|
||||
matched_count = len(tracks)
|
||||
added_count = 0
|
||||
if tracks:
|
||||
try:
|
||||
try:
|
||||
existing = plex.playlist(playlist_name)
|
||||
existing_keys = {item.ratingKey for item in existing.items()}
|
||||
new_tracks = []
|
||||
for track, song in tracks:
|
||||
if track.ratingKey in existing_keys:
|
||||
duplicate_songs.append({"title": track.title, "artist": track.artist().title, "reason": "already-in-playlist"})
|
||||
continue
|
||||
new_tracks.append((track, song))
|
||||
|
||||
if new_tracks:
|
||||
if not dry_run:
|
||||
existing.addItems([item[0] for item in new_tracks])
|
||||
for track, song in new_tracks:
|
||||
track.addLabel("Journey FM")
|
||||
track.rate(5)
|
||||
added_songs.append(f"{track.title} by {track.artist().title} ({song['source']})")
|
||||
else:
|
||||
for track, song in new_tracks:
|
||||
added_songs.append(f"{track.title} by {track.artist().title} ({song['source']})")
|
||||
added_count = len(new_tracks)
|
||||
except Exception:
|
||||
if not dry_run:
|
||||
plex.createPlaylist(playlist_name, [item[0] for item in tracks])
|
||||
for track, song in tracks:
|
||||
track.addLabel("Journey FM")
|
||||
track.rate(5)
|
||||
added_songs.append(f"{track.title} by {track.artist().title} ({song['source']})")
|
||||
else:
|
||||
for track, song in tracks:
|
||||
added_songs.append(f"{track.title} by {track.artist().title} ({song['source']})")
|
||||
added_count = len(tracks)
|
||||
except Exception as exc:
|
||||
raise PlexConnectionError(f"Failed updating playlist '{playlist_name}': {exc}") from exc
|
||||
|
||||
return {
|
||||
"matched_count": matched_count,
|
||||
"added_count": added_count,
|
||||
"added_songs": added_songs,
|
||||
"missing_songs": missing,
|
||||
"duplicate_count": len(duplicate_songs),
|
||||
"duplicate_songs": duplicate_songs,
|
||||
"skipped_songs": skipped,
|
||||
}
|
||||
@@ -0,0 +1,452 @@
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
from selenium import webdriver
|
||||
from selenium.webdriver.chrome.options import Options
|
||||
from selenium.webdriver.chrome.service import Service
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from webdriver_manager.chrome import ChromeDriverManager
|
||||
import requests
|
||||
|
||||
from journeyfm.paths import data_path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
DEBUG_SCRAPE_DIR = data_path("debug_scrapes")
|
||||
|
||||
|
||||
def detect_chrome_binary():
|
||||
env_binary = os.getenv("CHROME_BINARY", "").strip()
|
||||
if env_binary and os.path.exists(env_binary):
|
||||
return env_binary
|
||||
|
||||
candidates = []
|
||||
if os.name == "nt":
|
||||
local_app_data = os.getenv("LOCALAPPDATA", "")
|
||||
program_files = os.getenv("PROGRAMFILES", "")
|
||||
program_files_x86 = os.getenv("PROGRAMFILES(X86)", "")
|
||||
candidates.extend([
|
||||
os.path.join(local_app_data, "Google", "Chrome", "Application", "chrome.exe"),
|
||||
os.path.join(program_files, "Google", "Chrome", "Application", "chrome.exe"),
|
||||
os.path.join(program_files_x86, "Google", "Chrome", "Application", "chrome.exe"),
|
||||
os.path.join(program_files, "Chromium", "Application", "chrome.exe"),
|
||||
os.path.join(program_files_x86, "Chromium", "Application", "chrome.exe"),
|
||||
])
|
||||
else:
|
||||
candidates.extend([
|
||||
"/usr/bin/google-chrome",
|
||||
"/usr/bin/google-chrome-stable",
|
||||
"/usr/bin/chromium",
|
||||
"/usr/bin/chromium-browser",
|
||||
"/opt/google/chrome/chrome",
|
||||
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
|
||||
"/Applications/Chromium.app/Contents/MacOS/Chromium",
|
||||
])
|
||||
|
||||
for binary in candidates:
|
||||
if binary and os.path.exists(binary):
|
||||
return binary
|
||||
|
||||
for name in ["google-chrome", "google-chrome-stable", "chromium", "chromium-browser", "chrome"]:
|
||||
found = shutil.which(name)
|
||||
if found:
|
||||
return found
|
||||
return None
|
||||
|
||||
|
||||
def detect_chromedriver_path():
|
||||
env_driver = os.getenv("CHROMEDRIVER_PATH", "").strip()
|
||||
if env_driver and os.path.exists(env_driver):
|
||||
return env_driver
|
||||
for candidate in ["/usr/bin/chromedriver", "/usr/local/bin/chromedriver"]:
|
||||
if os.path.exists(candidate):
|
||||
return candidate
|
||||
return shutil.which("chromedriver")
|
||||
|
||||
|
||||
def _write_debug_payload(station_key, payload, extension):
|
||||
DEBUG_SCRAPE_DIR.mkdir(exist_ok=True)
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
file_path = DEBUG_SCRAPE_DIR / f"{station_key}_{timestamp}.{extension}"
|
||||
with open(file_path, "w", encoding="utf-8") as file_handle:
|
||||
file_handle.write(payload)
|
||||
latest_path = DEBUG_SCRAPE_DIR / f"{station_key}_latest.{extension}"
|
||||
with open(latest_path, "w", encoding="utf-8") as file_handle:
|
||||
file_handle.write(payload)
|
||||
return str(file_path)
|
||||
|
||||
|
||||
class JourneyFMScraper:
|
||||
station_key = "journey_fm"
|
||||
display_name = "Journey FM"
|
||||
url = "https://www.myjourneyfm.com/recently-played/"
|
||||
|
||||
SELECTOR_PATTERNS = [
|
||||
{
|
||||
"name": "rp-item",
|
||||
"container": ("div", "rp-item"),
|
||||
"title": ("h5", "song-title"),
|
||||
"artist": ("p", "song-artist"),
|
||||
},
|
||||
{
|
||||
"name": "top-rp-list",
|
||||
"container": ("div", "song-item"),
|
||||
"title": ("div", "title-artist"),
|
||||
"artist": ("span", None),
|
||||
},
|
||||
]
|
||||
|
||||
@classmethod
|
||||
def _extract_with_pattern(cls, soup, pattern):
|
||||
songs = []
|
||||
container_tag, container_class = pattern["container"]
|
||||
title_tag, title_class = pattern["title"]
|
||||
artist_tag, artist_class = pattern["artist"]
|
||||
|
||||
for item in soup.find_all(container_tag, class_=container_class):
|
||||
if title_class:
|
||||
title_elem = item.find(title_tag, class_=title_class)
|
||||
else:
|
||||
title_elem = item.find(title_tag)
|
||||
if not title_elem:
|
||||
continue
|
||||
|
||||
if pattern["name"] == "top-rp-list":
|
||||
paragraph = title_elem.find("p")
|
||||
if not paragraph:
|
||||
continue
|
||||
combined = paragraph.get_text(" ", strip=True)
|
||||
parts = re.split(r"\s+by\s+", combined, maxsplit=1, flags=re.IGNORECASE)
|
||||
if len(parts) != 2:
|
||||
continue
|
||||
title = parts[0].strip()
|
||||
artist = parts[1].strip()
|
||||
else:
|
||||
if artist_class:
|
||||
artist_elem = item.find(artist_tag, class_=artist_class)
|
||||
else:
|
||||
artist_elem = item.find(artist_tag)
|
||||
if not artist_elem:
|
||||
continue
|
||||
title = title_elem.get_text(" ", strip=True).strip()
|
||||
artist = artist_elem.get_text(" ", strip=True).strip()
|
||||
|
||||
if title and artist:
|
||||
songs.append({"title": title, "artist": artist, "source": cls.display_name})
|
||||
|
||||
return songs
|
||||
|
||||
@classmethod
|
||||
def parse_html_with_telemetry(cls, html):
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
for pattern in cls.SELECTOR_PATTERNS:
|
||||
songs = cls._extract_with_pattern(soup, pattern)
|
||||
if songs:
|
||||
return songs, pattern["name"]
|
||||
return [], "none"
|
||||
|
||||
@staticmethod
|
||||
def parse_html(html):
|
||||
songs, _pattern = JourneyFMScraper.parse_html_with_telemetry(html)
|
||||
return songs
|
||||
|
||||
@classmethod
|
||||
def scrape(cls, driver):
|
||||
driver.get(cls.url)
|
||||
WebDriverWait(driver, 15).until(
|
||||
lambda _driver: (
|
||||
bool(_driver.find_elements(By.CSS_SELECTOR, "div.rp-item"))
|
||||
or bool(_driver.find_elements(By.CSS_SELECTOR, "div.song-item"))
|
||||
)
|
||||
)
|
||||
try:
|
||||
more_button = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "moreSongs")))
|
||||
more_button.click()
|
||||
time.sleep(2)
|
||||
except Exception:
|
||||
pass
|
||||
html = driver.page_source
|
||||
raw_path = _write_debug_payload(cls.station_key, html, "html")
|
||||
songs, parse_pattern = cls.parse_html_with_telemetry(html)
|
||||
return songs, raw_path, parse_pattern
|
||||
|
||||
@classmethod
|
||||
def scrape_without_driver(cls):
|
||||
response = requests.get(cls.url, timeout=20)
|
||||
response.raise_for_status()
|
||||
html = response.text
|
||||
raw_path = _write_debug_payload(cls.station_key, html, "html")
|
||||
songs, parse_pattern = cls.parse_html_with_telemetry(html)
|
||||
return songs, raw_path, f"{parse_pattern}-requests-fallback"
|
||||
|
||||
|
||||
class SpiritFMScraper:
|
||||
station_key = "spirit_fm"
|
||||
display_name = "Spirit FM"
|
||||
url = "https://spiritfm.com/ajax/now_playing_history.txt"
|
||||
|
||||
@staticmethod
|
||||
def parse_text(text):
|
||||
songs = []
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
match = re.match(r"^\w+\s+\d+:\d+[AP]M\s+(.+?)\s+-\s+(.+)$", line)
|
||||
if not match:
|
||||
continue
|
||||
artist = match.group(1).strip().replace("&", "&")
|
||||
title = match.group(2).strip().replace("&", "&")
|
||||
if title and artist:
|
||||
songs.append({"title": title, "artist": artist, "source": SpiritFMScraper.display_name})
|
||||
return songs
|
||||
|
||||
@classmethod
|
||||
def scrape(cls, driver):
|
||||
driver.get(cls.url)
|
||||
time.sleep(2)
|
||||
text = BeautifulSoup(driver.page_source, "html.parser").get_text()
|
||||
raw_path = _write_debug_payload(cls.station_key, text, "txt")
|
||||
return cls.parse_text(text), raw_path, "plain-text-regex"
|
||||
|
||||
@classmethod
|
||||
def scrape_without_driver(cls):
|
||||
response = requests.get(cls.url, timeout=20)
|
||||
response.raise_for_status()
|
||||
text = response.text
|
||||
raw_path = _write_debug_payload(cls.station_key, text, "txt")
|
||||
return cls.parse_text(text), raw_path, "plain-text-regex-requests-fallback"
|
||||
|
||||
|
||||
class KLOVEScraper:
|
||||
station_key = "klove"
|
||||
display_name = "K-LOVE"
|
||||
url = "https://www.klove.com/music/songs"
|
||||
# K-LOVE is a JS-rendered React app; Selenium is the primary path.
|
||||
# The requests fallback may still return partial SSR content worth parsing.
|
||||
_LOAD_SELECTOR = "a[href*='/music/artists/']"
|
||||
|
||||
@staticmethod
|
||||
def parse_html(html):
|
||||
songs, _pattern = KLOVEScraper.parse_html_with_telemetry(html)
|
||||
return songs
|
||||
|
||||
@staticmethod
|
||||
def parse_html_with_telemetry(html):
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
songs = []
|
||||
seen = set()
|
||||
|
||||
# Collect every anchor that points at /music/artists/{artist}/{song}
|
||||
# The page renders each card as two anchors (art + title) — use hrefs
|
||||
# as a dedup key so we only process each song once.
|
||||
processed_hrefs = set()
|
||||
for anchor in soup.find_all("a", href=re.compile(r"/music/artists/[^/]+/[^/]+")):
|
||||
href = anchor.get("href", "").strip().rstrip("/")
|
||||
parts = href.strip("/").split("/")
|
||||
# Expect: music / artists / artist-slug / song-slug
|
||||
if len(parts) < 4:
|
||||
continue
|
||||
|
||||
title_text = anchor.get_text(" ", strip=True)
|
||||
if not title_text or len(title_text) < 2:
|
||||
# Image-only anchor (cover art link) — skip it; the title anchor
|
||||
# for this same href will appear later in the DOM with real text.
|
||||
continue
|
||||
|
||||
# Now that we have a text-bearing anchor, dedup by href
|
||||
if href in processed_hrefs:
|
||||
continue
|
||||
processed_hrefs.add(href)
|
||||
|
||||
# Skip navigation / album-only links (e.g. artist page without song)
|
||||
song_slug = parts[-1]
|
||||
artist_slug = parts[-2] if len(parts) >= 4 else ""
|
||||
|
||||
# Find "By {artist}" text in nearby DOM — check siblings and parent
|
||||
artist_text = ""
|
||||
parent = anchor.parent
|
||||
if parent:
|
||||
full_text = parent.get_text(" ", strip=True)
|
||||
by_match = re.search(r"\bBy\s+(.+)", full_text)
|
||||
if by_match:
|
||||
artist_text = by_match.group(1).strip()
|
||||
|
||||
if not artist_text and parent and parent.parent:
|
||||
grandparent_text = parent.parent.get_text(" ", strip=True)
|
||||
by_match = re.search(r"\bBy\s+(.+?)(?:\bBy\b|$)", grandparent_text)
|
||||
if by_match:
|
||||
artist_text = by_match.group(1).strip()
|
||||
|
||||
if not artist_text:
|
||||
# Derive artist from URL slug as last resort
|
||||
artist_text = " ".join(w.capitalize() for w in artist_slug.split("-"))
|
||||
|
||||
# Strip leftover boilerplate that sometimes bleeds in
|
||||
artist_text = re.sub(r"\s*(feat\.|ft\.|featuring|Play Sample|Image).*$", "",
|
||||
artist_text, flags=re.IGNORECASE).strip()
|
||||
|
||||
if title_text and artist_text:
|
||||
key = (title_text.lower(), artist_text.lower())
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
songs.append({"title": title_text, "artist": artist_text,
|
||||
"source": KLOVEScraper.display_name})
|
||||
|
||||
if songs:
|
||||
return songs, "klove-anchor-by"
|
||||
return [], "none"
|
||||
|
||||
@classmethod
|
||||
def scrape(cls, driver):
|
||||
driver.get(cls.url)
|
||||
# Wait for song anchor links to appear (JS-rendered)
|
||||
WebDriverWait(driver, 20).until(
|
||||
EC.presence_of_element_located((By.CSS_SELECTOR, cls._LOAD_SELECTOR))
|
||||
)
|
||||
time.sleep(2) # Allow lazy-load tiles to settle
|
||||
html = driver.page_source
|
||||
raw_path = _write_debug_payload(cls.station_key, html, "html")
|
||||
songs, parse_pattern = cls.parse_html_with_telemetry(html)
|
||||
return songs, raw_path, parse_pattern
|
||||
|
||||
@classmethod
|
||||
def scrape_without_driver(cls):
|
||||
headers = {
|
||||
"User-Agent": (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/120.0.0.0 Safari/537.36"
|
||||
)
|
||||
}
|
||||
response = requests.get(cls.url, headers=headers, timeout=20)
|
||||
response.raise_for_status()
|
||||
html = response.text
|
||||
raw_path = _write_debug_payload(cls.station_key, html, "html")
|
||||
songs, parse_pattern = cls.parse_html_with_telemetry(html)
|
||||
return songs, raw_path, f"{parse_pattern}-requests-fallback"
|
||||
|
||||
|
||||
SCRAPERS = {
|
||||
JourneyFMScraper.station_key: JourneyFMScraper,
|
||||
SpiritFMScraper.station_key: SpiritFMScraper,
|
||||
KLOVEScraper.station_key: KLOVEScraper,
|
||||
}
|
||||
|
||||
|
||||
def build_driver():
|
||||
options = Options()
|
||||
options.add_argument("--headless=new")
|
||||
options.add_argument("--no-sandbox")
|
||||
options.add_argument("--disable-dev-shm-usage")
|
||||
options.add_argument("--disable-gpu")
|
||||
options.add_argument("--window-size=1920,1080")
|
||||
options.add_argument("--disable-blink-features=AutomationControlled")
|
||||
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
|
||||
options.add_experimental_option("excludeSwitches", ["enable-automation"])
|
||||
options.add_experimental_option("useAutomationExtension", False)
|
||||
|
||||
chrome_binary = detect_chrome_binary()
|
||||
if chrome_binary:
|
||||
options.binary_location = chrome_binary
|
||||
chromedriver_path = detect_chromedriver_path()
|
||||
if chromedriver_path:
|
||||
service = Service(chromedriver_path)
|
||||
else:
|
||||
service = Service(ChromeDriverManager().install())
|
||||
return webdriver.Chrome(service=service, options=options)
|
||||
|
||||
|
||||
def scrape_recently_played(selected_stations=None):
|
||||
selected_stations = selected_stations or ["journey_fm", "spirit_fm"]
|
||||
driver = None
|
||||
driver_start_error = ""
|
||||
all_songs = []
|
||||
seen = set()
|
||||
station_results = []
|
||||
|
||||
try:
|
||||
try:
|
||||
driver = build_driver()
|
||||
except Exception as exc:
|
||||
driver = None
|
||||
driver_start_error = str(exc)
|
||||
logger.warning("Browser scraper unavailable; using fallback mode: %s", exc)
|
||||
for station_key in selected_stations:
|
||||
scraper_cls = SCRAPERS.get(station_key)
|
||||
if scraper_cls is None:
|
||||
station_results.append({
|
||||
"station": station_key,
|
||||
"display_name": station_key,
|
||||
"success": False,
|
||||
"error": "unsupported-station",
|
||||
"scraped_count": 0,
|
||||
"raw_path": "",
|
||||
})
|
||||
continue
|
||||
try:
|
||||
if driver is not None:
|
||||
try:
|
||||
songs, raw_path, parse_pattern = scraper_cls.scrape(driver)
|
||||
except Exception as scrape_error:
|
||||
logger.warning(
|
||||
"Selenium scrape failed for %s, retrying fallback mode: %s",
|
||||
scraper_cls.display_name,
|
||||
scrape_error,
|
||||
)
|
||||
songs, raw_path, parse_pattern = scraper_cls.scrape_without_driver()
|
||||
else:
|
||||
songs, raw_path, parse_pattern = scraper_cls.scrape_without_driver()
|
||||
raw_payload_bytes = 0
|
||||
if raw_path and os.path.exists(raw_path):
|
||||
try:
|
||||
raw_payload_bytes = os.path.getsize(raw_path)
|
||||
except Exception:
|
||||
raw_payload_bytes = 0
|
||||
added_for_station = 0
|
||||
for song in songs:
|
||||
dedupe_key = (song["title"].lower(), song["artist"].lower())
|
||||
if dedupe_key in seen:
|
||||
continue
|
||||
seen.add(dedupe_key)
|
||||
all_songs.append(song)
|
||||
added_for_station += 1
|
||||
logger.info(" Found: %s by %s", song["title"], song["artist"])
|
||||
station_results.append({
|
||||
"station": station_key,
|
||||
"display_name": scraper_cls.display_name,
|
||||
"success": True,
|
||||
"error": "",
|
||||
"scraped_count": added_for_station,
|
||||
"raw_path": raw_path,
|
||||
"raw_payload_bytes": raw_payload_bytes,
|
||||
"parse_pattern": parse_pattern,
|
||||
})
|
||||
except Exception as exc:
|
||||
combined_error = str(exc)
|
||||
if driver is None and driver_start_error:
|
||||
combined_error = f"{combined_error}; browser init: {driver_start_error}"
|
||||
station_results.append({
|
||||
"station": station_key,
|
||||
"display_name": scraper_cls.display_name,
|
||||
"success": False,
|
||||
"error": combined_error,
|
||||
"scraped_count": 0,
|
||||
"raw_path": "",
|
||||
"raw_payload_bytes": 0,
|
||||
"parse_pattern": "error",
|
||||
})
|
||||
logger.error("Error scraping %s: %s", scraper_cls.display_name, exc)
|
||||
finally:
|
||||
if driver is not None:
|
||||
driver.quit()
|
||||
|
||||
logger.info("Total unique songs found: %s", len(all_songs))
|
||||
return {"songs": all_songs, "station_results": station_results}
|
||||
@@ -0,0 +1,138 @@
|
||||
import logging
|
||||
import os
|
||||
import urllib.parse
|
||||
|
||||
from journeyfm.config_store import load_runtime_config
|
||||
from journeyfm.history_service import init_history_db, save_history_entry
|
||||
from journeyfm.paths import data_path
|
||||
from journeyfm.plex_service import PlexConnectionError, connect_to_plex_server, create_or_update_playlist
|
||||
from journeyfm.scraper_service import scrape_recently_played
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def update_buy_list(missing_songs, buy_list_path=None):
|
||||
buy_list_path = buy_list_path or data_path("amazon_buy_list.txt")
|
||||
if not missing_songs:
|
||||
logger.info("No new songs to add to buy list.")
|
||||
return []
|
||||
|
||||
existing_songs = set()
|
||||
if os.path.exists(buy_list_path):
|
||||
try:
|
||||
with open(buy_list_path, "r", encoding="utf-8") as file_handle:
|
||||
lines = file_handle.read().splitlines()
|
||||
for line in lines:
|
||||
if line.strip() and not line.startswith("http") and " - " in line:
|
||||
artist, title = line.split(" - ", 1)
|
||||
existing_songs.add((artist.strip(), title.strip()))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
new_missing = []
|
||||
for song in missing_songs:
|
||||
key = (song["artist"], song["title"])
|
||||
if key in existing_songs:
|
||||
continue
|
||||
existing_songs.add(key)
|
||||
new_missing.append(song)
|
||||
|
||||
if not new_missing:
|
||||
logger.info("No new songs to add to buy list.")
|
||||
return []
|
||||
|
||||
file_exists = os.path.exists(buy_list_path)
|
||||
file_empty = not file_exists or os.path.getsize(buy_list_path) == 0
|
||||
with open(buy_list_path, "a", encoding="utf-8") as file_handle:
|
||||
if file_empty:
|
||||
file_handle.write("Songs not in your library - Amazon search links:\n\n")
|
||||
for song in new_missing:
|
||||
query = urllib.parse.quote(f"{song['artist']} {song['title']}")
|
||||
file_handle.write(f"{song['artist']} - {song['title']}\n")
|
||||
file_handle.write(f"https://www.amazon.com/s?k={query}&i=digital-music\n\n")
|
||||
|
||||
logger.info("Buy list updated.")
|
||||
return new_missing
|
||||
|
||||
|
||||
def format_result_summary(result):
|
||||
station_bits = []
|
||||
for station in result.get("station_breakdown", []):
|
||||
if station.get("success"):
|
||||
pattern = station.get("parse_pattern", "unknown")
|
||||
payload_bytes = int(station.get("raw_payload_bytes", 0) or 0)
|
||||
payload_kib = payload_bytes / 1024.0
|
||||
station_bits.append(
|
||||
f"{station['display_name']}: {station['scraped_count']} [{pattern}, {payload_kib:.1f} KiB]"
|
||||
)
|
||||
else:
|
||||
station_bits.append(f"{station['display_name']}: failed")
|
||||
station_summary = ", ".join(station_bits) or "No station data"
|
||||
|
||||
lines = [
|
||||
f"Scraped {result.get('scraped_count', 0)} songs ({station_summary})",
|
||||
f"Matched {result.get('matched_count', 0)} in Plex",
|
||||
f"Added {result.get('added_count', 0)} to playlist",
|
||||
f"Skipped {result.get('skipped_count', 0)} invalid entries",
|
||||
f"Suppressed {result.get('duplicate_count', 0)} duplicates already in playlist",
|
||||
f"Missing {result.get('missing_count', 0)} from Plex library",
|
||||
]
|
||||
if result.get("status") == "error":
|
||||
lines.append(f"Error: {result.get('error_message', 'Unknown error')}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def run_update_job(config=None, dry_run=False, persist_history=True, write_buy_list=True):
|
||||
config = config or load_runtime_config()
|
||||
result = {
|
||||
"status": "success",
|
||||
"error_message": "",
|
||||
"station_breakdown": [],
|
||||
"scraped_count": 0,
|
||||
"matched_count": 0,
|
||||
"added_count": 0,
|
||||
"added_songs": [],
|
||||
"missing_count": 0,
|
||||
"missing_songs": [],
|
||||
"duplicate_count": 0,
|
||||
"duplicate_songs": [],
|
||||
"skipped_count": 0,
|
||||
"skipped_songs": [],
|
||||
}
|
||||
|
||||
token = config.get("PLEX_TOKEN", "").strip()
|
||||
server_ip = config.get("SERVER_IP", "").strip()
|
||||
playlist_name = config.get("PLAYLIST_NAME", "Journey FM Recently Played").strip()
|
||||
selected_stations = config.get("SELECTED_STATIONS", ["journey_fm", "spirit_fm"])
|
||||
|
||||
if not token or not server_ip:
|
||||
result["status"] = "error"
|
||||
result["error_message"] = "Missing configuration: set Plex token and server IP before running updates"
|
||||
return result
|
||||
|
||||
if persist_history:
|
||||
init_history_db()
|
||||
|
||||
scrape_result = scrape_recently_played(selected_stations)
|
||||
songs = scrape_result["songs"]
|
||||
result["station_breakdown"] = scrape_result["station_results"]
|
||||
result["scraped_count"] = len(songs)
|
||||
|
||||
try:
|
||||
plex = connect_to_plex_server(token, server_ip)
|
||||
playlist_result = create_or_update_playlist(plex, songs, playlist_name, dry_run=dry_run)
|
||||
result.update(playlist_result)
|
||||
result["missing_count"] = len(result.get("missing_songs", []))
|
||||
result["skipped_count"] = len(result.get("skipped_songs", []))
|
||||
if write_buy_list and not dry_run:
|
||||
update_buy_list(result.get("missing_songs", []))
|
||||
except PlexConnectionError as exc:
|
||||
result["status"] = "error"
|
||||
result["error_message"] = str(exc)
|
||||
except Exception as exc:
|
||||
result["status"] = "error"
|
||||
result["error_message"] = f"Unexpected update failure: {exc}"
|
||||
|
||||
if persist_history:
|
||||
save_history_entry(result)
|
||||
return result
|
||||
@@ -1,46 +1,23 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Journey FM Playlist Creator
|
||||
"""Journey FM Playlist Creator CLI entrypoint."""
|
||||
|
||||
This script scrapes recently played songs from myjourneyfm.com,
|
||||
checks if they exist in the local music library,
|
||||
creates a playlist of matching songs, and imports it into Plex.
|
||||
"""
|
||||
|
||||
import time
|
||||
import sys
|
||||
from selenium import webdriver
|
||||
from selenium.webdriver.chrome.options import Options
|
||||
from selenium.webdriver.chrome.service import Service
|
||||
from webdriver_manager.chrome import ChromeDriverManager
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
from bs4 import BeautifulSoup
|
||||
import os
|
||||
from plexapi.myplex import MyPlexAccount
|
||||
from mutagen.easyid3 import EasyID3
|
||||
import re
|
||||
from datetime import datetime
|
||||
import json
|
||||
import subprocess
|
||||
import urllib.parse
|
||||
import sqlite3
|
||||
import logging
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
from journeyfm.config_store import load_runtime_config
|
||||
from journeyfm.history_service import init_history_db
|
||||
from journeyfm.paths import data_path
|
||||
from journeyfm.plex_service import connect_to_plex_server
|
||||
from journeyfm.update_service import format_result_summary, run_update_job
|
||||
|
||||
# Configuration (defaults, can be overridden by env vars or config.json)
|
||||
PLEX_TOKEN = os.getenv('PLEX_TOKEN', '').strip()
|
||||
SERVER_IP = os.getenv('SERVER_IP', '').strip()
|
||||
PLAYLIST_NAME = os.getenv('PLAYLIST_NAME', 'Journey FM Holiday').strip()
|
||||
|
||||
def configure_logging():
|
||||
"""Configure logging with safe handlers for pythonw/GUI environments."""
|
||||
"""Configure logging with quiet defaults for CLI and GUI entrypoints."""
|
||||
log_format = '%(asctime)s - %(levelname)s - %(message)s'
|
||||
handlers = []
|
||||
|
||||
try:
|
||||
handlers.append(logging.FileHandler('playlist_log.txt', encoding='utf-8'))
|
||||
handlers.append(logging.FileHandler(data_path('playlist_log.txt'), encoding='utf-8'))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -49,494 +26,34 @@ def configure_logging():
|
||||
handlers.append(logging.StreamHandler(stream))
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format=log_format, handlers=handlers, force=True)
|
||||
for noisy_logger in ('webdriver_manager', 'matplotlib', 'urllib3', 'plexapi'):
|
||||
for noisy_logger in ('webdriver_manager', 'WDM', 'matplotlib', 'urllib3', 'plexapi'):
|
||||
logging.getLogger(noisy_logger).setLevel(logging.WARNING)
|
||||
|
||||
|
||||
configure_logging()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
def detect_chrome_binary():
|
||||
"""Detect a Chrome/Chromium executable across common OS locations."""
|
||||
env_binary = os.getenv('CHROME_BINARY', '').strip()
|
||||
if env_binary and os.path.exists(env_binary):
|
||||
return env_binary
|
||||
|
||||
candidates = []
|
||||
|
||||
if os.name == 'nt':
|
||||
local_app_data = os.getenv('LOCALAPPDATA', '')
|
||||
program_files = os.getenv('PROGRAMFILES', '')
|
||||
program_files_x86 = os.getenv('PROGRAMFILES(X86)', '')
|
||||
candidates.extend([
|
||||
os.path.join(local_app_data, 'Google', 'Chrome', 'Application', 'chrome.exe'),
|
||||
os.path.join(program_files, 'Google', 'Chrome', 'Application', 'chrome.exe'),
|
||||
os.path.join(program_files_x86, 'Google', 'Chrome', 'Application', 'chrome.exe'),
|
||||
os.path.join(program_files, 'Chromium', 'Application', 'chrome.exe'),
|
||||
os.path.join(program_files_x86, 'Chromium', 'Application', 'chrome.exe')
|
||||
])
|
||||
elif os.name == 'posix':
|
||||
candidates.extend([
|
||||
'/usr/bin/google-chrome',
|
||||
'/usr/bin/google-chrome-stable',
|
||||
'/usr/bin/chromium',
|
||||
'/usr/bin/chromium-browser',
|
||||
'/opt/google/chrome/chrome',
|
||||
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
|
||||
'/Applications/Chromium.app/Contents/MacOS/Chromium'
|
||||
])
|
||||
|
||||
for binary in candidates:
|
||||
if binary and os.path.exists(binary):
|
||||
return binary
|
||||
|
||||
for name in ['google-chrome', 'google-chrome-stable', 'chromium', 'chromium-browser', 'chrome']:
|
||||
found = shutil.which(name)
|
||||
if found:
|
||||
return found
|
||||
|
||||
return None
|
||||
|
||||
def scrape_recently_played(selected_stations=None):
|
||||
"""Scrape recently played songs from Journey FM and Spirit FM"""
|
||||
if selected_stations is None:
|
||||
selected_stations = ['journey_fm', 'spirit_fm']
|
||||
|
||||
options = Options()
|
||||
options.add_argument("--headless=new")
|
||||
options.add_argument("--no-sandbox")
|
||||
options.add_argument("--disable-dev-shm-usage")
|
||||
options.add_argument("--disable-gpu")
|
||||
options.add_argument("--window-size=1920,1080")
|
||||
options.add_argument("--disable-blink-features=AutomationControlled")
|
||||
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
|
||||
options.add_experimental_option("excludeSwitches", ["enable-automation"])
|
||||
options.add_experimental_option('useAutomationExtension', False)
|
||||
chrome_binary = detect_chrome_binary()
|
||||
if chrome_binary:
|
||||
options.binary_location = chrome_binary
|
||||
else:
|
||||
logger.warning("No Chrome/Chromium binary detected; Selenium will use its default browser resolution")
|
||||
|
||||
service = Service(ChromeDriverManager().install())
|
||||
driver = None
|
||||
|
||||
try:
|
||||
driver = webdriver.Chrome(service=service, options=options)
|
||||
except Exception as e:
|
||||
logger.error("Failed to start Chrome: %s", e)
|
||||
return []
|
||||
|
||||
all_songs = []
|
||||
seen = set()
|
||||
|
||||
# Journey FM - uses <strong> for title, normal text for artist
|
||||
if 'journey_fm' in selected_stations:
|
||||
try:
|
||||
logger.info("Scraping Journey FM...")
|
||||
driver.get('https://www.myjourneyfm.com/recently-played/')
|
||||
WebDriverWait(driver, 15).until(
|
||||
EC.presence_of_element_located((By.CSS_SELECTOR, 'div.rp-item'))
|
||||
)
|
||||
|
||||
# Click "View More" button if exists
|
||||
try:
|
||||
more_button = WebDriverWait(driver, 5).until(
|
||||
EC.element_to_be_clickable((By.ID, "moreSongs"))
|
||||
)
|
||||
more_button.click()
|
||||
time.sleep(2)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
soup = BeautifulSoup(driver.page_source, 'html.parser')
|
||||
|
||||
# Find all song items using the proper class structure
|
||||
song_items = soup.find_all('div', class_='rp-item')
|
||||
|
||||
for item in song_items:
|
||||
try:
|
||||
title_elem = item.find('h5', class_='song-title')
|
||||
artist_elem = item.find('p', class_='song-artist')
|
||||
|
||||
if title_elem and artist_elem:
|
||||
title = title_elem.get_text().strip()
|
||||
artist = artist_elem.get_text().strip()
|
||||
|
||||
if title and artist:
|
||||
key = (title.lower(), artist.lower())
|
||||
if key not in seen:
|
||||
all_songs.append({'title': title, 'artist': artist, 'source': 'Journey FM'})
|
||||
seen.add(key)
|
||||
logger.info(" Found: %s by %s", title, artist)
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Error scraping Journey FM: %s", e)
|
||||
|
||||
# Spirit FM - loads songs from iframe text file
|
||||
if 'spirit_fm' in selected_stations:
|
||||
try:
|
||||
logger.info("Scraping Spirit FM...")
|
||||
driver.get('https://spiritfm.com/ajax/now_playing_history.txt')
|
||||
time.sleep(2)
|
||||
|
||||
# Parse the plain text format: "Mon 01:45PM Artist - Title"
|
||||
soup = BeautifulSoup(driver.page_source, 'html.parser')
|
||||
text_content = soup.get_text()
|
||||
|
||||
# Split into lines and parse each
|
||||
for line in text_content.split('\n'):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
# Match pattern: Day HH:MM(AM/PM) Artist - Title
|
||||
match = re.match(r'^\w+\s+\d+:\d+[AP]M\s+(.+?)\s+-\s+(.+)$', line)
|
||||
if match:
|
||||
artist = match.group(1).strip()
|
||||
title = match.group(2).strip()
|
||||
|
||||
# Clean up HTML entities
|
||||
artist = artist.replace('&', '&')
|
||||
title = title.replace('&', '&')
|
||||
|
||||
if title and artist:
|
||||
key = (title.lower(), artist.lower())
|
||||
if key not in seen:
|
||||
all_songs.append({'title': title, 'artist': artist, 'source': 'Spirit FM'})
|
||||
seen.add(key)
|
||||
logger.info(" Found: %s by %s", title, artist)
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Error scraping Spirit FM: %s", e)
|
||||
|
||||
if driver:
|
||||
driver.quit()
|
||||
|
||||
logger.info("Total unique songs found: %s", len(all_songs))
|
||||
return all_songs
|
||||
|
||||
def normalize_string(s):
|
||||
"""Normalize string for comparison"""
|
||||
return re.sub(r'[^\w\s]', '', s).lower().strip()
|
||||
|
||||
def find_song_in_library(artist, title, library_path):
|
||||
"""Check if song exists in local library"""
|
||||
for root, dirs, files in os.walk(library_path):
|
||||
for file in files:
|
||||
if file.endswith(('.mp3', '.flac', '.m4a')): # Add more formats if needed
|
||||
try:
|
||||
audio = EasyID3(os.path.join(root, file))
|
||||
file_artist = normalize_string(audio.get('artist', [''])[0])
|
||||
file_title = normalize_string(audio.get('title', [''])[0])
|
||||
if normalize_string(artist) == file_artist and normalize_string(title) == file_title:
|
||||
return os.path.join(root, file)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
def create_playlist_in_plex(plex, songs, playlist_name):
|
||||
"""Create playlist in Plex and return added count, added songs, and missing songs"""
|
||||
# Assuming Plex has a music library named 'Music'
|
||||
music_library = plex.library.section('Music')
|
||||
tracks = []
|
||||
seen_keys = set() # Track which ratingKeys we've already added to prevent duplicates within this run
|
||||
missing = []
|
||||
added_songs = []
|
||||
|
||||
for song in songs:
|
||||
# Clean title and artist by removing text in parentheses and normalizing spaces
|
||||
clean_title = re.sub(r'\([^)]*\)', '', song['title']).strip()
|
||||
clean_title = re.sub(r'\s+', ' ', clean_title) # Replace multiple spaces with single space
|
||||
clean_artist = re.sub(r'\([^)]*\)', '', song['artist']).strip()
|
||||
clean_artist = re.sub(r'\s+', ' ', clean_artist) # Replace multiple spaces with single space
|
||||
|
||||
# Skip invalid entries
|
||||
if not clean_title or len(clean_title) < 3 or clean_title.lower() in ['by', 'recently played', 'now playing:', 'search']:
|
||||
continue
|
||||
|
||||
# Skip entries that are ONLY punctuation (but allow titles with punctuation in them)
|
||||
if re.match(r'^[!?\\\\s]+$', clean_title):
|
||||
continue
|
||||
|
||||
# Remove only ! and ? for searching, but keep apostrophes
|
||||
search_title = re.sub(r'[!?]', '', clean_title).strip()
|
||||
search_artist = re.sub(r'[!?]', '', clean_artist).strip()
|
||||
|
||||
# Normalize & and + to "and" for both title and artist
|
||||
search_title = re.sub(r'\s*[&+]\s*', ' and ', search_title, flags=re.IGNORECASE)
|
||||
search_artist = re.sub(r'\s*[&+]\s*', ' and ', search_artist, flags=re.IGNORECASE)
|
||||
|
||||
# Search for tracks by cleaned title
|
||||
results = music_library.searchTracks(title=search_title)
|
||||
if results:
|
||||
# Try to match by artist as well
|
||||
matched = False
|
||||
for track in results:
|
||||
track_artist = track.artist().title.lower()
|
||||
search_artist_lower = search_artist.lower()
|
||||
|
||||
# Clean featured artists from search (e.g., "W/ Emerson Day", "feat. Someone")
|
||||
search_artist_main = re.sub(r'\s+(w/|feat\.|featuring|ft\.).*', '', search_artist_lower).strip()
|
||||
|
||||
# Normalize & and + in track artist as well
|
||||
track_artist = re.sub(r'\s*[&+]\s*', ' and ', track_artist)
|
||||
|
||||
# Check if artist matches (case-insensitive, partial match)
|
||||
# Match if main artist name is in the track artist or vice versa
|
||||
if (search_artist_main in track_artist or track_artist in search_artist_main or
|
||||
search_artist_lower in track_artist or track_artist in search_artist_lower):
|
||||
# Only add if we haven't already added this track in this run
|
||||
if track.ratingKey not in seen_keys:
|
||||
tracks.append((track, song))
|
||||
seen_keys.add(track.ratingKey)
|
||||
matched = True
|
||||
break
|
||||
# If no artist match, skip this track (don't add wrong songs)
|
||||
if not matched:
|
||||
missing.append(song)
|
||||
|
||||
added = 0
|
||||
if tracks:
|
||||
try:
|
||||
# Try to find existing playlist
|
||||
try:
|
||||
existing = plex.playlist(playlist_name)
|
||||
# Get existing track rating keys to avoid duplicates
|
||||
existing_keys = {item.ratingKey for item in existing.items()}
|
||||
# Filter out tracks already in playlist
|
||||
new_tracks = [track_song for track_song in tracks if track_song[0].ratingKey not in existing_keys]
|
||||
|
||||
if new_tracks:
|
||||
existing.addItems([ts[0] for ts in new_tracks])
|
||||
for track, song in new_tracks:
|
||||
# Auto-tag added songs
|
||||
track.addLabel("Journey FM")
|
||||
track.rate(5)
|
||||
added = len(new_tracks)
|
||||
added_songs.extend([f"{track.title} by {track.artist().title} ({song['source']})" for track, song in new_tracks])
|
||||
else:
|
||||
added = 0
|
||||
except Exception:
|
||||
# Playlist doesn't exist, create it
|
||||
plex.createPlaylist(playlist_name, [ts[0] for ts in tracks])
|
||||
for track, song in tracks:
|
||||
# Auto-tag added songs
|
||||
track.addLabel("Journey FM")
|
||||
track.rate(5)
|
||||
added = len(tracks)
|
||||
added_songs.extend([f"{track.title} by {track.artist().title} ({song['source']})" for track, song in tracks])
|
||||
except Exception:
|
||||
added = 0
|
||||
added_songs = []
|
||||
else:
|
||||
logger.info("No matching tracks found in Plex.")
|
||||
added = 0
|
||||
added_songs = []
|
||||
|
||||
return added, added_songs, missing
|
||||
|
||||
def prompt_for_config():
|
||||
"""Prompt user for configuration variables"""
|
||||
print("Configuration not found. Please provide the following:")
|
||||
plex_token = input("Enter your Plex token: ").strip()
|
||||
server_ip = input("Enter your Plex server IP: ").strip()
|
||||
playlist_name = input("Enter the playlist name: ").strip()
|
||||
config = {
|
||||
'PLEX_TOKEN': plex_token,
|
||||
'SERVER_IP': server_ip,
|
||||
'PLAYLIST_NAME': playlist_name
|
||||
}
|
||||
with open('config.json', 'w') as f:
|
||||
json.dump(config, f)
|
||||
return config
|
||||
|
||||
def load_config():
|
||||
"""Load runtime configuration from config.json and environment variables."""
|
||||
config = {
|
||||
'PLEX_TOKEN': PLEX_TOKEN,
|
||||
'SERVER_IP': SERVER_IP,
|
||||
'PLAYLIST_NAME': PLAYLIST_NAME,
|
||||
'SELECTED_STATIONS': ['journey_fm', 'spirit_fm']
|
||||
}
|
||||
"""Compatibility wrapper for existing callers."""
|
||||
return load_runtime_config()
|
||||
|
||||
if not os.path.exists('config.json'):
|
||||
return config
|
||||
|
||||
try:
|
||||
with open('config.json', 'r') as f:
|
||||
file_config = json.load(f)
|
||||
except Exception as e:
|
||||
logger.error("Unable to parse config.json: %s", e)
|
||||
return config
|
||||
|
||||
config['PLEX_TOKEN'] = file_config.get('PLEX_TOKEN', config['PLEX_TOKEN'])
|
||||
config['SERVER_IP'] = file_config.get('SERVER_IP', config['SERVER_IP'])
|
||||
config['PLAYLIST_NAME'] = file_config.get('PLAYLIST_NAME', config['PLAYLIST_NAME'])
|
||||
|
||||
selected_stations = file_config.get('SELECTED_STATIONS')
|
||||
if isinstance(selected_stations, str) and selected_stations.strip():
|
||||
config['SELECTED_STATIONS'] = [s.strip() for s in selected_stations.split(',') if s.strip()]
|
||||
elif isinstance(selected_stations, list) and selected_stations:
|
||||
config['SELECTED_STATIONS'] = [str(s).strip() for s in selected_stations if str(s).strip()]
|
||||
|
||||
return config
|
||||
|
||||
def init_history_db():
|
||||
"""Initialize the history database"""
|
||||
conn = sqlite3.connect('playlist_history.db')
|
||||
c = conn.cursor()
|
||||
c.execute('''CREATE TABLE IF NOT EXISTS history (
|
||||
id INTEGER PRIMARY KEY,
|
||||
date TEXT,
|
||||
added_count INTEGER,
|
||||
added_songs TEXT,
|
||||
missing_count INTEGER,
|
||||
missing_songs TEXT
|
||||
)''')
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
def save_history_entry(added_count, added_songs, missing_count, missing_songs):
|
||||
"""Save a history entry to the database"""
|
||||
conn = sqlite3.connect('playlist_history.db')
|
||||
c = conn.cursor()
|
||||
date = datetime.now().isoformat()
|
||||
c.execute('INSERT INTO history (date, added_count, added_songs, missing_count, missing_songs) VALUES (?, ?, ?, ?, ?)',
|
||||
(date, added_count, json.dumps(added_songs), missing_count, json.dumps(missing_songs)))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
def setup_scheduler(params):
|
||||
"""Run the setup_scheduler.ps1 with parameters"""
|
||||
cmd = ['powershell.exe', '-File', 'setup_scheduler.ps1']
|
||||
for k, v in params.items():
|
||||
cmd.extend(['-' + k, str(v)])
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
logger.info("Scheduler setup completed.")
|
||||
except subprocess.CalledProcessError as e:
|
||||
logger.error("Scheduler setup failed: %s", e)
|
||||
|
||||
def main():
|
||||
global PLEX_TOKEN, SERVER_IP, PLAYLIST_NAME
|
||||
config = load_config()
|
||||
PLEX_TOKEN = config['PLEX_TOKEN']
|
||||
SERVER_IP = config['SERVER_IP']
|
||||
PLAYLIST_NAME = config['PLAYLIST_NAME']
|
||||
selected_stations = config['SELECTED_STATIONS']
|
||||
|
||||
if not PLEX_TOKEN or not SERVER_IP:
|
||||
logger.error("Missing configuration: set PLEX_TOKEN and SERVER_IP in config.json or environment variables")
|
||||
return
|
||||
|
||||
# Log start time
|
||||
start_time = datetime.now()
|
||||
|
||||
# Initialize history database
|
||||
"""Run one update cycle and print a concise result summary."""
|
||||
init_history_db()
|
||||
|
||||
# Scrape songs with selected stations
|
||||
songs = scrape_recently_played(selected_stations)
|
||||
|
||||
# Connect to Plex via MyPlexAccount
|
||||
try:
|
||||
account = MyPlexAccount(token=PLEX_TOKEN)
|
||||
resources = account.resources()
|
||||
except Exception as e:
|
||||
logger.error("Failed to authenticate with Plex: %s", e)
|
||||
return
|
||||
|
||||
# Find the server by IP
|
||||
server_resource = None
|
||||
for r in resources:
|
||||
for conn in r.connections:
|
||||
if SERVER_IP in conn.address:
|
||||
server_resource = r
|
||||
break
|
||||
if server_resource:
|
||||
break
|
||||
|
||||
if not server_resource:
|
||||
logger.error("Server not found at %s. Verify SERVER_IP and that Plex is reachable.", SERVER_IP)
|
||||
return
|
||||
|
||||
try:
|
||||
plex = server_resource.connect()
|
||||
except Exception as e:
|
||||
logger.error("Failed to connect to Plex server: %s", e)
|
||||
return
|
||||
|
||||
# Find matching songs in local library (optional, since Plex search might suffice)
|
||||
# But to filter only those in local library, perhaps skip or adjust
|
||||
# For now, assume all scraped songs are to be added if found in Plex
|
||||
|
||||
# Create playlist
|
||||
if songs:
|
||||
added, added_songs, missing = create_playlist_in_plex(plex, songs, PLAYLIST_NAME)
|
||||
else:
|
||||
logger.info("No songs found.")
|
||||
added, added_songs, missing = 0, [], []
|
||||
|
||||
# Save to history
|
||||
save_history_entry(added, added_songs, len(missing), missing)
|
||||
|
||||
# Create Amazon buy list for missing songs
|
||||
if missing:
|
||||
|
||||
# Read existing buy list
|
||||
existing_songs = set()
|
||||
if os.path.exists('amazon_buy_list.txt'):
|
||||
try:
|
||||
with open('amazon_buy_list.txt', 'r') as f:
|
||||
content = f.read()
|
||||
lines = content.split('\n')
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
if lines[i].startswith('Songs not in your library'):
|
||||
i += 2 # Skip header
|
||||
continue
|
||||
if lines[i].strip() and not lines[i].startswith('http'):
|
||||
if ' - ' in lines[i]:
|
||||
artist, title = lines[i].split(' - ', 1)
|
||||
existing_songs.add((artist.strip(), title.strip()))
|
||||
i += 1
|
||||
except Exception as e:
|
||||
logger.warning("Failed reading amazon_buy_list.txt: %s", e)
|
||||
|
||||
# Filter new missing songs
|
||||
new_missing = []
|
||||
for song in missing:
|
||||
key = (song['artist'], song['title'])
|
||||
if key not in existing_songs:
|
||||
new_missing.append(song)
|
||||
existing_songs.add(key)
|
||||
|
||||
if new_missing:
|
||||
with open('amazon_buy_list.txt', 'a') as f: # Append mode
|
||||
if not os.path.exists('amazon_buy_list.txt') or os.path.getsize('amazon_buy_list.txt') == 0:
|
||||
f.write("Songs not in your library - Amazon search links:\n\n")
|
||||
for song in new_missing:
|
||||
artist = song['artist']
|
||||
title = song['title']
|
||||
# Create Amazon search URL
|
||||
query = urllib.parse.quote(f"{artist} {title}")
|
||||
url = f"https://www.amazon.com/s?k={query}&i=digital-music"
|
||||
f.write(f"{artist} - {title}\n{url}\n\n")
|
||||
else:
|
||||
logger.info("No new songs to add to buy list.")
|
||||
|
||||
logger.info("Buy list updated.")
|
||||
|
||||
# Log completion
|
||||
end_time = datetime.now()
|
||||
duration = (end_time - start_time).total_seconds()
|
||||
result = run_update_job(load_runtime_config())
|
||||
summary = format_result_summary(result)
|
||||
print(summary)
|
||||
if result.get('status') == 'error':
|
||||
logger.error(summary)
|
||||
return result
|
||||
|
||||
|
||||
def update_playlist():
|
||||
"""Function to update playlist, callable from GUI"""
|
||||
main()
|
||||
"""Compatibility wrapper used by the GUI."""
|
||||
return main()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+2
-1
@@ -5,4 +5,5 @@ mutagen
|
||||
selenium
|
||||
webdriver-manager
|
||||
PySide6
|
||||
matplotlib
|
||||
matplotlib
|
||||
keyring
|
||||
@@ -0,0 +1,38 @@
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
|
||||
from journeyfm.config_store import load_runtime_config
|
||||
from journeyfm.update_service import format_result_summary, run_update_job
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_sleep_seconds():
|
||||
interval = os.getenv("UPDATE_INTERVAL", "15").strip()
|
||||
unit = os.getenv("UPDATE_UNIT", "Minutes").strip().lower()
|
||||
try:
|
||||
interval_value = max(1, int(interval))
|
||||
except ValueError:
|
||||
interval_value = 15
|
||||
if unit.startswith("hour"):
|
||||
return interval_value * 3600
|
||||
return interval_value * 60
|
||||
|
||||
|
||||
def main():
|
||||
os.environ.setdefault("JOURNEYFM_CONTAINER", "1")
|
||||
run_mode = os.getenv("CONTAINER_RUN_MODE", "loop").strip().lower()
|
||||
while True:
|
||||
result = run_update_job(load_runtime_config())
|
||||
logger.info(format_result_summary(result))
|
||||
if run_mode == "once":
|
||||
break
|
||||
sleep_seconds = get_sleep_seconds()
|
||||
logger.info("Sleeping %s seconds before next container sync", sleep_seconds)
|
||||
time.sleep(sleep_seconds)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+56
-19
@@ -1,23 +1,60 @@
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
url = 'https://www.myjourneyfm.com/recently-played/'
|
||||
response = requests.get(url)
|
||||
soup = BeautifulSoup(response.content, 'html.parser')
|
||||
from journeyfm.scraper_service import JourneyFMScraper, SpiritFMScraper, KLOVEScraper
|
||||
|
||||
# Try to find song elements
|
||||
print("Song items:")
|
||||
song_items = soup.find_all('div', class_='song-item')
|
||||
for item in song_items:
|
||||
print(item.prettify())
|
||||
|
||||
print("\nScripts with 'song' or 'played':")
|
||||
scripts = soup.find_all('script')
|
||||
for script in scripts:
|
||||
if script.string and ('song' in script.string.lower() or 'played' in script.string.lower()):
|
||||
print(script.string[:2000])
|
||||
class ScraperParserTests(unittest.TestCase):
|
||||
def fixture_text(self, name):
|
||||
fixture_path = Path("tests") / "fixtures" / name
|
||||
return fixture_path.read_text(encoding="utf-8")
|
||||
|
||||
print("\nAll links:")
|
||||
for a in soup.find_all('a'):
|
||||
if 'recently' in a.get('href', '').lower():
|
||||
print(a)
|
||||
def test_parse_journey_fm_html(self):
|
||||
html = """
|
||||
<div class='rp-item'>
|
||||
<h5 class='song-title'>No Survivors</h5>
|
||||
<p class='song-artist'>Jeremy Camp</p>
|
||||
</div>
|
||||
<div class='rp-item'>
|
||||
<h5 class='song-title'>Take It All Back</h5>
|
||||
<p class='song-artist'>Tauren Wells</p>
|
||||
</div>
|
||||
"""
|
||||
songs = JourneyFMScraper.parse_html(html)
|
||||
self.assertEqual(2, len(songs))
|
||||
self.assertEqual('No Survivors', songs[0]['title'])
|
||||
self.assertEqual('Jeremy Camp', songs[0]['artist'])
|
||||
|
||||
def test_parse_journey_fm_current_markup_fixture(self):
|
||||
html = self.fixture_text("journeyfm_recent_markup.html")
|
||||
songs, parse_pattern = JourneyFMScraper.parse_html_with_telemetry(html)
|
||||
self.assertEqual("top-rp-list", parse_pattern)
|
||||
self.assertEqual(2, len(songs))
|
||||
self.assertEqual('Have Your Way', songs[0]['title'])
|
||||
self.assertEqual('Katy Nichole', songs[0]['artist'])
|
||||
|
||||
def test_parse_spirit_fm_text(self):
|
||||
text = """
|
||||
Mon 01:45PM Brandon Lake - Hard Fought Hallelujah
|
||||
Mon 01:49PM Danny Gokey - Love God Love People
|
||||
"""
|
||||
songs = SpiritFMScraper.parse_text(text)
|
||||
self.assertEqual(2, len(songs))
|
||||
self.assertEqual('Brandon Lake', songs[0]['artist'])
|
||||
self.assertEqual('Hard Fought Hallelujah', songs[0]['title'])
|
||||
|
||||
def test_parse_klove_fixture(self):
|
||||
html = self.fixture_text("klove_recent_markup.html")
|
||||
songs, parse_pattern = KLOVEScraper.parse_html_with_telemetry(html)
|
||||
self.assertEqual("klove-anchor-by", parse_pattern)
|
||||
self.assertGreaterEqual(len(songs), 3)
|
||||
# Verify first song title and artist
|
||||
titles = [s['title'] for s in songs]
|
||||
artists = [s['artist'] for s in songs]
|
||||
self.assertIn('Testimony', titles)
|
||||
self.assertIn('Katy Nichole', artists)
|
||||
self.assertIn('TobyMac', artists)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
<div class="top-rp-list" tabindex="0">
|
||||
<div class="song-item">
|
||||
<div class="title-artist">
|
||||
<p>Have Your Way <span> by </span> Katy Nichole</p>
|
||||
</div>
|
||||
<p class="song-time">8:39 AM Today</p>
|
||||
</div>
|
||||
<div class="song-item">
|
||||
<div class="title-artist">
|
||||
<p>Jesus Does <span> by </span> We The Kingdom</p>
|
||||
</div>
|
||||
<p class="song-time">8:31 AM Today</p>
|
||||
</div>
|
||||
</div>
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<body>
|
||||
<!-- Minimal representative fixture matching K-LOVE's live page structure -->
|
||||
<div class="song-card">
|
||||
<a href="/music/artists/katy-nichole/testimony">
|
||||
<img alt="Testimony" />
|
||||
</a>
|
||||
<div class="song-info">
|
||||
<a href="/music/artists/katy-nichole/testimony">Testimony</a>
|
||||
<span>By Katy Nichole</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="song-card">
|
||||
<a href="/music/artists/phil-wickham/i-believe">
|
||||
<img alt="I Believe" />
|
||||
</a>
|
||||
<div class="song-info">
|
||||
<a href="/music/artists/phil-wickham/i-believe">I Believe</a>
|
||||
<span>By Phil Wickham</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="song-card">
|
||||
<a href="/music/artists/tobymac/cornerstone-feat-zach-williams">
|
||||
<img alt="Cornerstone feat. Zach Williams" />
|
||||
</a>
|
||||
<div class="song-info">
|
||||
<a href="/music/artists/tobymac/cornerstone-feat-zach-williams">Cornerstone feat. Zach Williams</a>
|
||||
<span>By TobyMac</span>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in new issue
Block a user