ytdlp/README.md

# Media Ingest

Media Ingest is a private, self-hosted tool for inspecting and downloading
publicly available video for a broadcast-news workflow. It uses yt-dlp for
source extraction and FFmpeg for media inspection, stream merging, and
conversion when required.

The project is being developed locally with Docker first. Once the local
workflow is reliable and hardened, it will be deployed to an Ubuntu VPS behind
Cloudflare Access and Cloudflare Tunnel.

## Initial scope

The first useful version will support:

- one public, non-live video URL per job
- source and format inspection before downloading
- an original-quality download mode
- a compatible H.264/AAC MP4 mode when the source provides suitable streams
- background jobs with progress reporting
- source-attribution metadata stored alongside the downloaded media
- automatic cleanup of expired files

The initial version will not support browser cookies, authenticated source
accounts, playlists, live recording, or multiple simultaneous downloads.

## Planned architecture

- FastAPI web application
- server-rendered HTML with a small amount of JavaScript
- yt-dlp embedded through its Python API
- FFmpeg and ffprobe
- SQLite for durable job state
- a separate worker process for downloads
- Docker Compose for local and production deployments

## Development stages

- [x] Create a minimal containerized web application with a health check.
- [x] Inspect a URL and display sanitized source metadata and formats.
- [x] Add a dark, responsive inspection interface.
- [ ] Define download presets and their exact yt-dlp selectors.
- [ ] Add a durable job queue and separate worker.
- [ ] Download media and report progress.
- [ ] Add cleanup, limits, failure handling, and security tests.
- [ ] Validate the complete local Docker deployment.
- [ ] Deploy privately to the VPS through Cloudflare Access and Tunnel.

Each stage should produce a small, working, reviewable commit.

## Local development

The current milestone provides a browser interface for inspecting public video
sources. It shows normalized source metadata and previews best-available and
compatible-MP4 outcomes. Download jobs are not implemented yet.

The application runs as a non-root user in a read-only container with Linux
capabilities dropped. Docker publishes it only on the PC's loopback interface,
so it is not exposed to other devices on the LAN.

Build and start it:

```bash
docker compose up --build -d
```

Check its status and logs:

```bash
docker compose ps
docker compose logs --tail=50 web
```

Open <http://127.0.0.1:8000> or request the health endpoint:

```bash
curl --fail http://127.0.0.1:8000/health
```

Inspect a public, non-live video without downloading it:

```bash
curl --fail --request POST http://127.0.0.1:8000/api/inspect \
  --header 'Content-Type: application/json' \
  --data '{"url":"https://example.com/video"}'
```

Inspection accepts only HTTP and HTTPS URLs on standard ports. The initial
hostname must resolve entirely to public IP addresses. Playlists and
multi-video sources are rejected, and raw signed media URLs are not returned
to the browser.

The initial URL validation is not yet a complete SSRF defense. Do not expose
this development build to the Internet. See [`docs/SECURITY.md`](docs/SECURITY.md)
for the implemented boundary and work required before deployment.

Run the automated tests in an ephemeral container:

```bash
docker compose run --rm --build test
```

Stop the application:

```bash
docker compose down
```

Runtime data, downloaded media, local environment files, and secrets must not
be committed to Git.

## Documentation

The original project discussion is retained in
[`docs/general-idea.pdf`](docs/general-idea.pdf) as planning context. Decisions
that affect the implementation will be documented in this repository rather
than relying on that discussion alone.

- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) describes the current and
  planned application structure.
- [`docs/SECURITY.md`](docs/SECURITY.md) tracks the trust boundary and
  pre-deployment requirements.

## Responsible use

This tool is intended for material the operator is authorized to retrieve and
use. The operator remains responsible for source terms, copyright, licensing,
privacy, and appropriate attribution.