bot
131 lines
4.3 KiB
Markdown
131 lines
4.3 KiB
Markdown
# Media Ingest
|
|
|
|
Media Ingest is a private, self-hosted tool for inspecting and downloading
|
|
publicly available video for a broadcast-news workflow. It uses yt-dlp for
|
|
source extraction and FFmpeg for media inspection, stream merging, and
|
|
conversion when required.
|
|
|
|
The project is being developed locally with Docker first. Once the local
|
|
workflow is reliable and hardened, it will be deployed to an Ubuntu VPS behind
|
|
Cloudflare Access and Cloudflare Tunnel.
|
|
|
|
## Initial scope
|
|
|
|
The first useful version will support:
|
|
|
|
- one public, non-live video URL per job
|
|
- source and format inspection before downloading
|
|
- an original-quality download mode
|
|
- a compatible H.264/AAC MP4 mode when the source provides suitable streams
|
|
- background jobs with progress reporting
|
|
- source-attribution metadata stored alongside the downloaded media
|
|
- automatic cleanup of expired files
|
|
|
|
The initial version will not support browser cookies, authenticated source
|
|
accounts, playlists, live recording, or multiple simultaneous downloads.
|
|
|
|
## Planned architecture
|
|
|
|
- FastAPI web application
|
|
- server-rendered HTML with a small amount of JavaScript
|
|
- yt-dlp embedded through its Python API
|
|
- FFmpeg and ffprobe
|
|
- SQLite for durable job state
|
|
- a separate worker process for downloads
|
|
- Docker Compose for local and production deployments
|
|
|
|
## Development stages
|
|
|
|
- [x] Create a minimal containerized web application with a health check.
|
|
- [x] Inspect a URL and display sanitized source metadata and formats.
|
|
- [x] Add a dark, responsive inspection interface.
|
|
- [ ] Define download presets and their exact yt-dlp selectors.
|
|
- [ ] Add a durable job queue and separate worker.
|
|
- [ ] Download media and report progress.
|
|
- [ ] Add cleanup, limits, failure handling, and security tests.
|
|
- [ ] Validate the complete local Docker deployment.
|
|
- [ ] Deploy privately to the VPS through Cloudflare Access and Tunnel.
|
|
|
|
Each stage should produce a small, working, reviewable commit.
|
|
|
|
## Local development
|
|
|
|
The current milestone provides a browser interface for inspecting public video
|
|
sources. It shows normalized source metadata and previews best-available and
|
|
compatible-MP4 outcomes. Download jobs are not implemented yet.
|
|
|
|
The application runs as a non-root user in a read-only container with Linux
|
|
capabilities dropped. Docker publishes it only on the PC's loopback interface,
|
|
so it is not exposed to other devices on the LAN.
|
|
|
|
Build and start it:
|
|
|
|
```bash
|
|
docker compose up --build -d
|
|
```
|
|
|
|
Check its status and logs:
|
|
|
|
```bash
|
|
docker compose ps
|
|
docker compose logs --tail=50 web
|
|
```
|
|
|
|
Open <http://127.0.0.1:8000> or request the health endpoint:
|
|
|
|
```bash
|
|
curl --fail http://127.0.0.1:8000/health
|
|
```
|
|
|
|
Inspect a public, non-live video without downloading it:
|
|
|
|
```bash
|
|
curl --fail --request POST http://127.0.0.1:8000/api/inspect \
|
|
--header 'Content-Type: application/json' \
|
|
--data '{"url":"https://example.com/video"}'
|
|
```
|
|
|
|
Inspection accepts only HTTP and HTTPS URLs on standard ports. The initial
|
|
hostname must resolve entirely to public IP addresses. Playlists and
|
|
multi-video sources are rejected, and raw signed media URLs are not returned
|
|
to the browser.
|
|
|
|
The initial URL validation is not yet a complete SSRF defense. Do not expose
|
|
this development build to the Internet. See [`docs/SECURITY.md`](docs/SECURITY.md)
|
|
for the implemented boundary and work required before deployment.
|
|
|
|
Run the automated tests in an ephemeral container:
|
|
|
|
```bash
|
|
docker compose run --rm --build test
|
|
```
|
|
|
|
Stop the application:
|
|
|
|
```bash
|
|
docker compose down
|
|
```
|
|
|
|
Runtime data, downloaded media, local environment files, and secrets must not
|
|
be committed to Git.
|
|
|
|
## Documentation
|
|
|
|
The original project discussion is retained in
|
|
[`docs/general-idea.pdf`](docs/general-idea.pdf) as planning context. Decisions
|
|
that affect the implementation will be documented in this repository rather
|
|
than relying on that discussion alone.
|
|
|
|
- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) describes the current and
|
|
planned application structure.
|
|
- [`docs/CURRENT_STATUS.md`](docs/CURRENT_STATUS.md) provides a concise handoff
|
|
and resume checklist.
|
|
- [`docs/SECURITY.md`](docs/SECURITY.md) tracks the trust boundary and
|
|
pre-deployment requirements.
|
|
|
|
## Responsible use
|
|
|
|
This tool is intended for material the operator is authorized to retrieve and
|
|
use. The operator remains responsible for source terms, copyright, licensing,
|
|
privacy, and appropriate attribution.
|