bot
102 lines
3.0 KiB
Markdown
102 lines
3.0 KiB
Markdown
# Media Ingest
|
|
|
|
Media Ingest is a private, self-hosted tool for inspecting and downloading
|
|
publicly available video for a broadcast-news workflow. It uses yt-dlp for
|
|
source extraction and FFmpeg for media inspection, stream merging, and
|
|
conversion when required.
|
|
|
|
The project is being developed locally with Docker first. Once the local
|
|
workflow is reliable and hardened, it will be deployed to an Ubuntu VPS behind
|
|
Cloudflare Access and Cloudflare Tunnel.
|
|
|
|
## Initial scope
|
|
|
|
The first useful version will support:
|
|
|
|
- one public, non-live video URL per job
|
|
- source and format inspection before downloading
|
|
- an original-quality download mode
|
|
- a compatible H.264/AAC MP4 mode when the source provides suitable streams
|
|
- background jobs with progress reporting
|
|
- source-attribution metadata stored alongside the downloaded media
|
|
- automatic cleanup of expired files
|
|
|
|
The initial version will not support browser cookies, authenticated source
|
|
accounts, playlists, live recording, or multiple simultaneous downloads.
|
|
|
|
## Planned architecture
|
|
|
|
- FastAPI web application
|
|
- server-rendered HTML with a small amount of JavaScript
|
|
- yt-dlp embedded through its Python API
|
|
- FFmpeg and ffprobe
|
|
- SQLite for durable job state
|
|
- a separate worker process for downloads
|
|
- Docker Compose for local and production deployments
|
|
|
|
## Development stages
|
|
|
|
1. Create a minimal containerized web application with a health check.
|
|
2. Inspect a URL and display sanitized source metadata and formats.
|
|
3. Add a durable job queue and separate worker.
|
|
4. Download media and report progress.
|
|
5. Add original-quality and compatible-MP4 output modes.
|
|
6. Add cleanup, limits, failure handling, and security tests.
|
|
7. Validate the complete local Docker deployment.
|
|
8. Deploy privately to the VPS through Cloudflare Access and Tunnel.
|
|
|
|
Each stage should produce a small, working, reviewable commit.
|
|
|
|
## Local development
|
|
|
|
The current milestone is a minimal FastAPI application running in a hardened
|
|
local container. Docker publishes it only on the PC's loopback interface, so it
|
|
is not exposed to other devices on the LAN.
|
|
|
|
Build and start it:
|
|
|
|
```bash
|
|
docker compose up --build -d
|
|
```
|
|
|
|
Check its status and logs:
|
|
|
|
```bash
|
|
docker compose ps
|
|
docker compose logs --tail=50 web
|
|
```
|
|
|
|
Open <http://127.0.0.1:8000> or request the health endpoint:
|
|
|
|
```bash
|
|
curl --fail http://127.0.0.1:8000/health
|
|
```
|
|
|
|
Run the automated tests in an ephemeral container:
|
|
|
|
```bash
|
|
docker compose run --rm --build test
|
|
```
|
|
|
|
Stop the application:
|
|
|
|
```bash
|
|
docker compose down
|
|
```
|
|
|
|
Runtime data, downloaded media, local environment files, and secrets must not
|
|
be committed to Git.
|
|
|
|
## Documentation
|
|
|
|
The original project discussion is retained in
|
|
[`docs/general-idea.pdf`](docs/general-idea.pdf) as planning context. Decisions
|
|
that affect the implementation will be documented in this repository rather
|
|
than relying on that discussion alone.
|
|
|
|
## Responsible use
|
|
|
|
This tool is intended for material the operator is authorized to retrieve and
|
|
use. The operator remains responsible for source terms, copyright, licensing,
|
|
privacy, and appropriate attribution.
|