vid2sub

vid2sub: design

Status: written 2026-10-14 from a brainstorming session with Dan and a five-reviewer design council the same day (§27). Every decision is dated and attributed in the decision log (§28). Things that could not be verified while designing are listed in §29; each one is checked in the milestone that first depends on it.

vid2sub turns a video, a local file or a YouTube URL, into subtitle files: it extracts the audio, transcribes it with timestamps in the video’s language, optionally translates it, and writes SRT (and optionally ASS) next to the video. A browser UI plays the video with the subtitles and edits them. All the work runs on the user’s own machine, or on models the user runs themselves, such as Dan’s DGX Spark; OpenAI is an opt-in fallback.

Contents

1. Purpose and scope
2. Users, languages and platforms
3. Architecture
4. Repository layout and packaging
5. Configuration
6. Security model
7. Privacy
8. Media pipeline
9. Transcription engines
10. Detect, suggest, confirm
11. Model catalog
12. Building cues
13. Translation
14. The sidecar and data model
15. Subtitle files
16. Glossary
17. Jobs and the queue
18. CLI
19. The local web app
20. The editor
21. The Pages editor
22. Testing
23. CI, releases and distribution
24. Development conventions
25. Milestones
26. Items for other repos
27. Council review, 2026-10-14
28. Decision log
29. Verify during the build

---

1. Purpose and scope

Goal (Dan, 2026-10-14). Better closed captions, or a good starting point for them, for YouTube videos and TV shows. Point the app at a video; the user’s machine downloads it if needed, extracts the audio, transcribes it into subtitles in the video’s language (auto-detected, overridable), optionally translates them, and writes SRT or ASS.

Principles.

• The user’s machine does the work. There is no vid2sub server. vid2sub is a Python CLI application with a local web app over the same core (§3.1). Both run commands on the user’s own computer, against models on that computer, on a machine they run (a DGX Spark reached through an SSH tunnel), or, only when they opt in for a run, a cloud API.
• The app stores no state of its own. Settings live in a file the user writes (§5); keys live in the user’s environment; results live next to the video (§14, §15). Caches are disposable (§8.9).
• Never overwrite an artifact. The sidecar is append-only; subtitle files are projections of it and are only ever rewritten when vid2sub wrote them (§14, §15). This is the rule Dan’s audio-router and recordings follow.
• Cross-platform. macOS, Windows and Linux, each with README setup that covers the modelling parts (§2.3, §23.5).

In scope for v1. Everything in §3–§24, delivered in the order of §25.

Not in v1 (the Later list in §25 says when each might come): the Pages UI driving a local engine (“B”, §3.3); importing subtitle tracks embedded in MKV files; embedding YouTube’s player; speaker labels; a vid2sub compare model comparison; dedicated translation models (TranslateGemma, HY-MT); mlx-whisper, Parakeet and local Qwen3-ASR on CUDA as in-process engines; a version-history UI; sidecar compaction.

2. Users, languages and platforms

2.1 Users

Dan first, on a Mac with the DGX Spark at home. The app and README are written so another person can install it on their own machine with their own models or an OpenAI key.

2.2 Languages (Dan, 2026-10-14)

• Input: Mandarin, Cantonese, Japanese, English.
• Output: mainly English. English in → English captions out is a real case too.
• Not one model for everything. Each source language gets the model that is best at it (§11), suggested after the audio is sampled and the user can override (§10).

2.3 Platforms

Platform	Supported	Local transcription engine in v1
macOS 14+ on Apple Silicon	yes	mlx-qwen3-asr in-process (§9.4)
Windows 10/11 x64	yes	faster-whisper (CPU, or NVIDIA through the CUDA 12 cuBLAS runtime)
Linux x64 / aarch64	yes	faster-whisper (CPU or NVIDIA)
Intel Macs	no	onnxruntime, needed by the VAD (§8.6), has no Intel macOS wheel

Any platform can use an OpenAI-compatible endpoint instead (§9.3): the Spark, OpenAI, a local whisper.cpp server, or mlx-qwen3-asr serve.

Python ≥3.12. The floor comes from the engines’ wheels, not from Pyodide (recordings needs 3.14 and still ships Pyodide’s 3.12); the pure core is tested on 3.12 and 3.14 (§22).

3. Architecture

3.1 Three ways to run it

flowchart LR
  subgraph user["User's machine"]
    term["Terminal<br/>vid2sub run · detect · translate"]
    browser["Browser<br/>React UI"]
    web["vid2sub serve<br/>FastAPI + Shiny on 127.0.0.1"]
    core["vid2sub core library<br/>media · engines · subs · translate"]
    tools["ffmpeg · ffprobe · yt-dlp + deno"]
    local["In-process engines<br/>mlx-qwen3-asr · faster-whisper"]
    files[("Video folder<br/>video · .vid2sub.jsonl · .xx.srt")]
  end
  spark["DGX Spark via SSH tunnel<br/>127.0.0.1:9100/v1<br/>whisper.cpp · Gemma 4"]
  cloud["OpenAI<br/>opt-in per run"]
  pages["GitHub Pages<br/>Shinylive editor"]
  term --> core
  browser <-->|"websocket + token cookie"| web
  web --> core
  core --> tools
  core --> local
  core <--> files
  core -->|"chunks, cue text"| spark
  core -.->|"only with allow_cloud"| cloud
  pages -->|"File System Access"| files

• The CLI (vid2sub run, detect, translate, …; §18): a Python command-line application for the terminal and for batch work, such as a season, a folder tree, a list of files, a script or a scheduled job. It needs no browser and no Node. It is installed as a uv tool, or editable from a clone (§23.5). Dan, 2026-10-14: vid2sub is a CLI application, not only a web app.
• The local web app (vid2sub serve): the same core behind a Python engine on 127.0.0.1 that serves the React UI, runs jobs, and reads and writes files next to videos.
• The Pages editor (chendaniely.github.io/vid2sub/): the same UI and the same pure Python subtitle code running in the browser through Shinylive. It opens a folder on the user’s disk, plays the video, edits and saves subtitles. It cannot generate subtitles (§21).

Generating subtitles happens only in the core, through the CLI or vid2sub serve; the two share every behaviour (§18.3).

3.2 Core library with a CLI and the app on top (Dan, 2026-10-14)

All behaviour lives in vid2sub’s core: plain functions and small classes for probing, extracting, chunking, transcribing, building cues, translating and writing. The CLI (§18) and the web app (§19) call the same core. The subtitle model (vid2sub.subs) and the catalog (vid2sub.catalog) are pure: standard library, pydantic, pysubs2, opencc-purepy and charset-normalizer only, no I/O beyond what callers hand them, so they run unchanged in Pyodide. An import-graph test enforces it (§22).

3.3 “B”, later: the Pages UI driving the local engine

Dan wants B if it can be made to work (2026-10-14). It is not in v1, but v1 is shaped for it:

• The Shiny server talks to a Backend interface whose requests and responses are versioned, JSON-serialisable types keyed by item id. LocalBackend (the engine) and BrowserBackend (Pages) implement it. B adds a small versioned JSON API over LocalBackend; it never exposes the Shiny websocket cross-origin (§6.10).
• B’s own security rules are decided now (§6.10).

4. Repository layout and packaging

4.1 Layout

pyproject.toml            one distributable package: vid2sub
src/vid2sub/
  subs/                   PURE — model.py (Word, Cue, Track…), build.py (cues from words, §12),
                          edit.py (split/merge/nudge/insert/delete), lint.py, text.py (EAW width,
                          kinsoku, canonical text), formats.py (SRT/ASS via pysubs2, §15),
                          sidecar.py (records, heads, merge, §14), naming.py (§15.1),
                          scripts.py (OpenCC Traditional/Simplified), translate_shape.py
                          (batch schemas + partition validator, §13)
  catalog.py              PURE — the model catalog (§11)
  parse/                  PURE — engine response parsers → normalised words (§9.2)
  media/                  probe.py, identity.py, decode.py, vad.py (Silero v6 ONNX), chunk.py,
                          preview.py, youtube.py, tools.py (resolving and running ffmpeg/yt-dlp)
  engines/                base.py (Engine protocol), endpoint.py, qwen_mlx.py,
                          faster_whisper.py, clients.py (the one client factory, §7.2)
  translate/              chatlas provider factory, batching, staleness
  pipeline.py             one job = ordered stages, progress events, cancellation
  jobs.py                 the in-memory queue (§17)
  config.py               vid2sub.toml loading and validation (§5)
  security.py             launch token, safe paths and writes, subprocess env (§6)
  cli.py                  `vid2sub …` (§18)
  web/
    app.py                FastAPI shell: /auth, /healthz, /media/{token}; Shiny mounted at /
    guards.py             HostGuard, WriteGuard (copied from recordings), TokenGuard, CSP
    server.py             the shinyreact server (ReactApp(server) is called in this file)
    backend.py            Backend protocol, LocalBackend, BrowserBackend
    www/                  ui.js + ui.css (build output), fonts
frontend/                 Vite + React 19 + TypeScript + Tailwind v4 + shadcn (radix-nova)
pages/                    the Shinylive entry (app.py, requirements.txt)
demo/                     a public-domain clip, its canned sidecar and subtitles, CREDITS.md
scripts/                  build_pages.py, brand_css.py, shinylive_local.py, rangeserver.py,
                          build_index.py (copied from recordings where noted, §4.4)
tests/  _brand.yml  Makefile  CLAUDE.md  README.md  vid2sub.example.toml

4.2 Dependencies

• Core: pydantic (≥2.10, the Pyodide floor is 2.10.5), pysubs2 (≥1.9), opencc-purepy (≥1.4, behind subs/scripts.py so it can be swapped), charset-normalizer, fastapi, uvicorn, shiny and shinyreact (pinned exactly, matching recordings), chatlas (≥0.23), openai, httpx, numpy, onnxruntime, yt-dlp[default] (lower bound only; a stale yt-dlp breaks on YouTube), platformdirs.
• Extras:
  • [mac]: mlx-qwen3-asr[aligner] (≥0.4.4; macOS 14+ on Apple Silicon).
  • [whisper]: faster-whisper (all platforms, CPU).
  • [nvidia]: faster-whisper plus the CUDA 12 cuBLAS runtime wheels (Windows and Linux).
• Not an extra: qwen-asr. It hard-pins transformers and accelerate, pulls in gradio and flask, is unsatisfiable together with mlx-audio>=0.5, and PyPI’s torch is CPU-only on Windows (council, 2026-10-14). Qwen3-ASR runs on the Mac through mlx-qwen3-asr and on the Spark through a service local-ai builds (§26.1, item 7).
• [tool.uv] conflicts declares any extras that can’t resolve together.
• System tools, checked by vid2sub doctor (§18): ffmpeg and ffprobe (≥6.0), deno (≥2.3; yt-dlp’s only default JavaScript runtime for YouTube).

4.3 Versions and pins

Exact pins for shiny, shinyreact, shinylive and the npm @posit-dev/shinyreact types package, upgraded together by the procedure in recordings’ CLAUDE.md (§24). Lower bounds for engines and yt-dlp. uv.lock committed; CI runs with UV_FROZEN=1.

4.4 Code copied from Dan’s other repos

Copied files carry a header naming the source repo, path and commit, and are relicensed under this repo’s licence where Dan owns both. Planned copies:

• recordings: hosts.py (HostGuard, WriteGuard), the media route, scripts/build_pages.py, brand_css.py, shinylive_local.py, rangeserver.py, _brand.yml, the frontend’s Player.tsx, TranscriptTab.tsx patterns, lib/transcript.ts, lib/theme.ts, ThemeSwitch, sr.ts, vite.config.js, and media.py’s decode (which already carries audio-router’s provenance, so decode has one lineage).
• audio-router: vocabulary_prompt and prompt_sha (stages/transcribe.py), the version-string recipe (worker.py).
• meanwhile: FolderPicker and media/folder.ts, changed to mode: 'readwrite' (§21).
• faster-whisper (MIT, third party): silero_vad_v6.onnx and the torch-free get_speech_timestamps, with its licence (§8.6).

vid2sub does not import recordings or audio-router: recordings core needs Python 3.14 and pins pydantic; audio-router’s engine set is closed. They share a data contract instead (§14.4).

5. Configuration

5.1 vid2sub.toml (Dan, 2026-10-14)

Settings live in one TOML file the user writes. The app never writes it, and it refuses key values: only the names of environment variables that hold keys. Found at --config PATH, else VID2SUB_CONFIG, else the platform config directory (~/.config/vid2sub/vid2sub.toml on macOS and Linux, %APPDATA%\vid2sub\vid2sub.toml on Windows). vid2sub config init prints a starter file to stdout. vid2sub.example.toml in the repo is the template (named .toml, not .env, and never holding a secret).

[server]
port = 8765
open_browser = true

[privacy]
local_hosts = ["127.0.0.1", "localhost", "::1"]   # where a `local = true` model may live

[endpoints.spark]
base_url = "http://127.0.0.1:9100/v1"     # make tunnel (local-ai)
key_env = "SPARK_API_KEY"                 # VID2SUB_SPARK_API_KEY once local-ai Phase 3 issues one

[endpoints.openai]
base_url = "https://api.openai.com/v1"
key_env = "OPENAI_API_KEY"
allow_typed_key = true                    # a key may be typed into the UI for this endpoint (§5.3)

[models.spark-turbo]
catalog = "whisper-large-v3-turbo"        # catalog entry: descriptions, languages (§11)
endpoint = "spark"
model = "whisper-large-v3-turbo"          # the name the server knows
parser = "whisper.cpp"                    # response shape (§9.2)
local = true                              # valid only because spark's host is in local_hosts

[models.spark-gemma]
catalog = "gemma-4-26b-a4b"
endpoint = "spark"
model = "gemma-4-26b-a4b"
task = "translate"
local = true
concurrency = 1
request_extra = { chat_template_kwargs = { enable_thinking = false } }

[models.mac-qwen]
catalog = "qwen3-asr-1.7b"
engine = "mlx-qwen3-asr"                  # in-process; local by definition

[models.openai-whisper]
catalog = "openai-whisper-1"
endpoint = "openai"
model = "whisper-1"
parser = "openai"

[defaults]
targets = ["en"]
formats = ["srt"]
youtube_dir = "~/Movies/vid2sub"
cache_max_gb = 20

5.2 Validation

• Unknown keys anywhere are refused at load, naming the key and the line (an audio-router lesson).
• A key-looking value (sk-…, or any *_key/*_token field holding a value rather than an env var name) is refused, naming only the field.
• local = true is refused unless the model is in-process or its endpoint’s host is in privacy.local_hosts.
• A key is sent over plain http only to a loopback address or a Tailscale address (100.64.0.0/10, *.ts.net); otherwise the endpoint must be https.
• Every config key has a test proving its value reaches the code that uses it (an audio-router lesson).

5.3 Keys

• From the environment, by name, at call time. On Dan’s Mac ~/.secrets exports them through .zshrc; on Windows, setx or a PowerShell profile. A long-running engine keeps the environment it started with, so the 401 hint says to restart vid2sub from a shell that exports the key.
• Typed into the UI (Dan, 2026-10-14): allowed only for an endpoint declared with allow_typed_key = true. The key is held in the engine’s memory for that run, scoped to that endpoint’s origin, never written, logged or sent back to the UI (which shows only “key entered for this session”), and dropped on exit.
• A key goes only to the origin its endpoint declared (§6.4).

5.4 Precedence

CLI flags (per run) > vid2sub.toml > built-in defaults. UI changes are per session and never written; Settings → Copy settings gives the current session’s settings as TOML to paste into the file, never including a key. A model entry’s task defaults to transcribe.

6. Security model

The engine is a local RPC server that lists directories, reads and writes files, runs subprocesses and sends requests carrying API keys, and it runs on other people’s machines. The council’s red team (§27) found three blockers in the first draft; the rules below fix them.

6.1 Binding and the launch token

• The engine binds 127.0.0.1 only.
• Per-launch token, as Jupyter does. On start the engine makes secrets.token_urlsafe(32). It opens the browser through a redirect file (mode 0600, in a private temp directory) that sends the browser to http://127.0.0.1:<port>/auth#t=<token>; the token is in the fragment, so it never reaches the server’s logs or the command line (ps). The /auth page’s script (served as a file, no inline script) posts the token to /auth and receives an HttpOnly, SameSite=Strict cookie whose name includes the port (vid2sub-8765), then replaces the URL. The CLI also prints that URL for opening by hand.
• Every route and the websocket handshake require the cookie, except /auth and its script, and /healthz, which returns only {"ok": true}.

6.2 Origin and write guards

Copied from recordings (hosts.py), and kept because other localhost ports count as same-site:

• HostGuard: HTTP requests with a Host outside the allowlist get 400. A websocket is closed (1008) before accept unless the Host is allowed, the Origin is exactly http://<Host> with the port, and Sec-Fetch-Site is empty, same-origin or none.
• WriteGuard: every response gets Cross-Origin-Resource-Policy: same-origin, Cross-Origin-Opener-Policy: same-origin, X-Frame-Options: DENY, Referrer-Policy: same-origin. Any method other than GET/HEAD/OPTIONS needs a same-origin request with Content-Type: application/json.
• No GET request writes anything or starts a process. Shiny bookmarking stays off.

6.3 Content-Security-Policy and untrusted text

Any script on the app’s origin controls everything, so nothing hostile may run there.

• CSP on every page: default-src 'self'; script-src 'self'; connect-src 'self'; object-src 'none'; base-uri 'none'; frame-ancestors 'none'; img-src 'self' data: blob:; media-src 'self' blob:. Style rules are whatever Shiny and shinyreact need, verified in M2 (§29); never unsafe-eval.
• Untrusted text is rendered as plain text in React: subtitle text (SRT <font> tags, ASS {\…} overrides), MKV tags, YouTube titles, file names, LLM output, error bodies, /v1/models lists. The caption overlay renders <i> only, through a whitelist; never dangerouslySetInnerHTML.
• Fonts are served locally. The app makes no third-party requests.
• /media/ responses send Content-Security-Policy: sandbox, X-Content-Type-Options: nosniff and Cache-Control: no-store.
• Hostile sample files (§22.4) are part of the test suite.

6.4 Endpoints and keys

• Endpoints, key env var names, local flags, the YouTube output folder and the list of models are fixed at launch by vid2sub.toml. The UI only chooses among them. A URL can’t be typed into the UI.
• A key goes only to the origin it was declared with. Every client gets api_key and base_url passed explicitly (never read from OPENAI_API_KEY/OPENAI_BASE_URL by the SDK), with redirects refused (follow_redirects=False), so a 307 can’t re-send audio or a key elsewhere.
• One client factory (engines/clients.py) creates every openai, httpx and chatlas client. A test fails if any other module constructs one (§22.4).

6.5 Paths

• Reads: only files the user added this launch (typed, browsed to, or in a folder they added) and files vid2sub derives from them (sidecar, glossary, subtitles next to the video).
• Writes: only (a) the sidecar, subtitle projections and the glossary next to an added video, (b) the YouTube output folder from vid2sub.toml, (c) the cache (§8.9), (d) per-job scratch.
• The in-app file browser lists home and mounted volumes (/Volumes, drive letters, /media, /mnt), shows only folders, video and subtitle files, and hides dotfiles and credential folders (.ssh, .aws, .gnupg, .config/gh, …).
• Accepted paths are fully qualified. UNC paths only when the user typed or browsed to them, never when read from a file’s contents (a UNC path leaks the NTLM hash).
• safe_component() builds every file name vid2sub writes: Unicode NFC, control and bidi characters stripped, : and other reserved characters replaced, Windows reserved names (CON, NUL, COM1… even with an extension) refused, trailing dots and spaces stripped, ≤255 bytes. YouTube titles go through it.
• Media tokens: secrets.token_urlsafe(32), bound to the session, recording the file’s (dev, inode) so a swapped file isn’t served.

6.6 Writes

safe_write() is the only way vid2sub writes a file:

• write a temporary file in the target directory with O_EXCL | O_NOFOLLOW (on Windows, the equivalent create-new flags), fsync, then os.replace;
• refuse a target that exists and isn’t a regular file (symlinks included);
• never overwrite a file vid2sub didn’t write (one with no projection record in the sidecar, §15.3); the UI asks per file, the CLI needs --overwrite-foreign;
• each Save carries the sidecar hash it started from (compare-and-swap, §14.6), so two tabs or a tab and a job can’t overwrite each other;
• on Windows, a replace that fails with a sharing violation is retried with backoff, then fails with “Plex or antivirus has the file open”.

6.7 Subprocesses

media/tools.py runs every child process:

• absolute executable paths, resolved by doctor’s rules (shutil.which, then known install locations such as /opt/homebrew/bin), never the current directory;
• argument lists, no shell; cwd = the job’s scratch directory, created by mkdtemp (mode 0700);
• a minimal environment from an allowlist (PATH, HOME/USERPROFILE, TMP/TEMP, LANG, SYSTEMROOT, …), so children never inherit keys;
• timeouts proportional to duration.

ffmpeg/ffprobe: -nostdin -protocol_whitelist file -format_whitelist <the demuxers vid2sub accepts>, inputs as file: + absolute path. A playlist disguised as .mp4 can then neither fetch URLs nor read other files.

yt-dlp: run from the engine’s own environment with `–ignore-config –no-plugin-dirs –use-extractors youtube –no-playlist -f bestaudio/best -P -o "%(id)s.%(ext)s" --

`. The URL's host must be `youtube.com`, `www.youtube.com`, `m.youtube.com`, `music.youtube.com` or `youtu.be`, and the video id is validated (ids can start with `-`, hence the `--`). Never `--exec`, `--cookies-from-browser`, `-U` or unsafe-extension options. The downloaded file is renamed to `safe_component("

This site is open source. Improve this page.