Media Ingest

Media Ingest is a private, self-hosted tool for inspecting and downloading publicly available video for a broadcast-news workflow. It uses yt-dlp for source extraction and FFmpeg for media inspection, stream merging, and conversion when required.

The project is being developed locally with Docker first. Once the local workflow is reliable and hardened, it will be deployed to an Ubuntu VPS behind Cloudflare Access and Cloudflare Tunnel.

Initial scope

The first useful version will support:

one public, non-live video URL per job
source and format inspection before downloading
an original-quality download mode
a compatible H.264/AAC MP4 mode when the source provides suitable streams
background jobs with progress reporting
source-attribution metadata stored alongside the downloaded media
automatic cleanup of expired files

The initial version will not support browser cookies, authenticated source accounts, playlists, live recording, or multiple simultaneous downloads.

Planned architecture

FastAPI web application
server-rendered HTML with a small amount of JavaScript
yt-dlp embedded through its Python API
FFmpeg and ffprobe
SQLite for durable job state
a separate worker process for downloads
Docker Compose for local and production deployments

Development stages

Create a minimal containerized web application with a health check.
Inspect a URL and display sanitized source metadata and formats.
Add a durable job queue and separate worker.
Download media and report progress.
Add original-quality and compatible-MP4 output modes.
Add cleanup, limits, failure handling, and security tests.
Validate the complete local Docker deployment.
Deploy privately to the VPS through Cloudflare Access and Tunnel.

Each stage should produce a small, working, reviewable commit.

Local development

The current milestone is a minimal FastAPI application running in a hardened local container. Docker publishes it only on the PC's loopback interface, so it is not exposed to other devices on the LAN.

Build and start it:

docker compose up --build -d

Check its status and logs:

docker compose ps
docker compose logs --tail=50 web

Open http://127.0.0.1:8000 or request the health endpoint:

curl --fail http://127.0.0.1:8000/health

Inspect a public, non-live video without downloading it:

curl --fail --request POST http://127.0.0.1:8000/api/inspect \
  --header 'Content-Type: application/json' \
  --data '{"url":"https://example.com/video"}'

Inspection accepts only HTTP and HTTPS URLs on standard ports. The initial hostname must resolve entirely to public IP addresses. Playlists and multi-video sources are rejected, and raw signed media URLs are not returned to the browser.

Run the automated tests in an ephemeral container:

docker compose run --rm --build test

Stop the application:

docker compose down

Runtime data, downloaded media, local environment files, and secrets must not be committed to Git.

Documentation

The original project discussion is retained in docs/general-idea.pdf as planning context. Decisions that affect the implementation will be documented in this repository rather than relying on that discussion alone.

Responsible use

This tool is intended for material the operator is authorized to retrieve and use. The operator remains responsible for source terms, copyright, licensing, privacy, and appropriate attribution.

3.5 KiB Raw Blame History