feat: add dark source inspection interface
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
# Architecture
|
||||
|
||||
## Current state
|
||||
|
||||
```text
|
||||
Browser
|
||||
-> FastAPI / Jinja interface
|
||||
-> POST /api/inspect
|
||||
-> URL validation
|
||||
-> yt-dlp metadata extraction (download disabled)
|
||||
-> normalized metadata and output previews
|
||||
```
|
||||
|
||||
FastAPI runs inspection in a thread pool. The browser receives a deliberately
|
||||
small response and never receives yt-dlp's signed media URLs. Displayed output
|
||||
options are previews; backend download selectors are not implemented yet.
|
||||
|
||||
## Next application boundary
|
||||
|
||||
Downloads will use durable jobs rather than long-running HTTP requests:
|
||||
|
||||
```text
|
||||
Web service -> SQLite job record <- Worker service
|
||||
|
|
||||
+-> yt-dlp
|
||||
+-> FFmpeg / ffprobe
|
||||
+-> per-job directory
|
||||
```
|
||||
|
||||
The worker will claim one queued job at a time, update progress, validate the
|
||||
finished media, and record the artifact. This preserves state across web
|
||||
restarts and keeps media processing outside request handlers.
|
||||
|
||||
## Planned output presets
|
||||
|
||||
1. **Best available** — highest useful source quality, merging separate video
|
||||
and audio streams without transcoding.
|
||||
2. **Compatible MP4** — prefer the highest H.264 video and AAC audio streams
|
||||
that can be merged into MP4 without transcoding.
|
||||
|
||||
If no compatible combination exists, the first version should report that
|
||||
instead of silently starting an expensive transcode. Transcoding can later be
|
||||
an explicit third outcome with CPU and time limits.
|
||||
|
||||
## Production boundary
|
||||
|
||||
Production will route Cloudflare Access through Cloudflare Tunnel to the web
|
||||
service. The origin will not publish an application port directly.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Security
|
||||
|
||||
## Trust model
|
||||
|
||||
The submitted URL and all remote metadata are untrusted. yt-dlp and FFmpeg
|
||||
process attacker-controlled network content and must not have host privileges
|
||||
or access to unrelated files and services. The current development build is
|
||||
not ready for public exposure.
|
||||
|
||||
## Controls currently implemented
|
||||
|
||||
- port bound only to `127.0.0.1`
|
||||
- non-root, read-only container with a small `/tmp` tmpfs
|
||||
- all Linux capabilities dropped and `no-new-privileges` enabled
|
||||
- yt-dlp plugins disabled and configuration files ignored
|
||||
- inspection only; media download disabled
|
||||
- playlists and multi-video results rejected
|
||||
- only HTTP/HTTPS URLs on ports 80/443 accepted
|
||||
- URL credentials rejected
|
||||
- initially resolved IPv4 and IPv6 addresses must all be globally routable
|
||||
- raw signed media URLs omitted from API responses
|
||||
- remote metadata inserted into the interface as text, not HTML
|
||||
|
||||
## Required before download jobs
|
||||
|
||||
- maximum source duration and estimated output size
|
||||
- per-job timeout and cancellation
|
||||
- one concurrent worker job initially
|
||||
- bounded CPU, memory, process, and temporary-storage use
|
||||
- randomized job directories and server-generated filenames
|
||||
- no user-controlled output templates or command fragments
|
||||
- FFmpeg/ffprobe validation of completed artifacts
|
||||
- safe handling of partial files and failed merges
|
||||
- automatic expiry and deletion
|
||||
- durable job recovery after process or container restarts
|
||||
- logs that exclude signed URLs, cookies, and credentials
|
||||
|
||||
## Required before Internet deployment
|
||||
|
||||
- Cloudflare Access deny-by-default policy
|
||||
- Cloudflare Tunnel with Access-token validation at the origin
|
||||
- no directly reachable origin application port
|
||||
- request-rate and body-size limits
|
||||
- explicit production configuration validation
|
||||
- dependency and container-image update procedure
|
||||
|
||||
## SSRF limitation
|
||||
|
||||
Initial-host validation is necessary but incomplete. Extractors may follow
|
||||
redirects and retrieve manifests, APIs, or media from additional hosts. DNS
|
||||
answers can also change between validation and connection.
|
||||
|
||||
Before Internet deployment, outbound traffic from the processing service must
|
||||
be restricted at the network layer so private, loopback, link-local, multicast,
|
||||
and infrastructure-metadata destinations cannot be reached. URL validation is
|
||||
an additional layer, not the sole SSRF defense.
|
||||
|
||||
## Deferred features
|
||||
|
||||
Browser cookies and authenticated source accounts are excluded. A cookie file
|
||||
is effectively an account credential. Supporting one requires encrypted secret
|
||||
handling, isolated injection, redacted logging, rotation, and a deliberate
|
||||
account-compromise risk review.
|
||||
Reference in New Issue
Block a user