diff --git a/docs/README.md b/docs/README.md index 459191b..8642b34 100644 --- a/docs/README.md +++ b/docs/README.md @@ -40,6 +40,7 @@ This directory contains the working documentation for the WeatherTool modernizat - [Continuous integration](CONTINUOUS_INTEGRATION.md) — Gitea Actions topology, isolation boundary, and the current CI workflow. - [Third-party notices](THIRD_PARTY_NOTICES.md) — licenses and attribution for adapted interface components. - [Temporary VPS staging plan](VPS_STAGING_PLAN.md) — isolation, authentication, prepared deployment bundle, release, backup, verification, and rollback model for external user testing. +- [VPS release runbook](VPS_RELEASE_RUNBOOK.md) — the literal, copy-pasteable command sequence for shipping and rolling back a release, step by step. ## Documentation rules diff --git a/docs/VPS_RELEASE_RUNBOOK.md b/docs/VPS_RELEASE_RUNBOOK.md new file mode 100644 index 0000000..e362e13 --- /dev/null +++ b/docs/VPS_RELEASE_RUNBOOK.md @@ -0,0 +1,183 @@ +# VPS release runbook + +This is the literal, runnable command sequence for shipping a new WeatherTool +release to the VPS. `VPS_STAGING_PLAN.md` explains *why* the process looks +this way (topology, isolation boundary, acceptance checklist) and gives the +same steps at a policy level; this document is the *how* — copy-pasteable +commands, run manually, in order, every time. There is no automation here +deliberately: continuous deployment is out of scope while the VPS carries +real testers, and every release is a deliberate human checkpoint (see +`CONTINUOUS_INTEGRATION.md` for why CI does not perform this step either). + +Run every command from the Rocky checkout at +`/home/sandbox/Documents/projects/WeatherTool` unless noted otherwise. + +## 0. Preconditions + +```bash +git status --short # must be empty — never build from an uncommitted tree +git log --oneline -1 +``` + +If CI exists for this commit, check it's green before proceeding — nothing +else currently stops a red-CI commit from being deployed. + +## 1. Build the release image + +Always build from the committed tree via `git archive`, never from the +working directory — a dirty or stale build has bitten this project before. + +```bash +SHA=$(git rev-parse HEAD) +echo "Building weathertool:$SHA" +git archive HEAD | docker build -f deploy/vps/Dockerfile -t weathertool:$SHA - +``` + +## 2. Smoke-test the exact image + +Test the literal artifact that will ship, not just "the code" — against a +throwaway Postgres and placeholder credentials, never real ones. + +```bash +docker network create weathertool-smoketest-net +docker run -d --name weathertool-smoketest-pg --network weathertool-smoketest-net \ + -e POSTGRES_DB=smoketest -e POSTGRES_USER=smoketest -e POSTGRES_PASSWORD=smoketestpass \ + postgres:16.1 +until docker exec weathertool-smoketest-pg pg_isready -U smoketest -d smoketest >/dev/null 2>&1; do sleep 1; done + +docker run -d --name weathertool-smoketest --network weathertool-smoketest-net \ + -p 18080:8080 \ + -e POSTGRES_DB=smoketest -e POSTGRES_USER=smoketest -e POSTGRES_PASSWORD=smoketestpass -e POSTGRES_HOST=weathertool-smoketest-pg \ + -e LVGMC_URL=placeholder -e LVGMC_USER=placeholder -e LVGMC_PASSWORD=placeholder \ + -e HARMONIE_EDR_URL=placeholder -e HARMONIE_EDR_API_KEY=placeholder -e HARMONIE_STAC_URL=placeholder -e HARMONIE_STAC_API_KEY=placeholder \ + -e ENABLE_SCHEDULED_JOBS=false -e ENABLE_LVGMC_FTP_JOBS=false -e ENABLE_HARMONIE_JOBS=false \ + weathertool:$SHA + +sleep 6 +docker logs weathertool-smoketest --tail 20 # expect clean startup, no "Fatal error" +curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/ # expect 200 +``` + +Add release-specific checks here (route behavior, a known bug that's +supposedly fixed, a real data path). Then tear down: + +```bash +docker stop weathertool-smoketest weathertool-smoketest-pg +docker rm weathertool-smoketest weathertool-smoketest-pg +docker network rm weathertool-smoketest-net +``` + +If anything looks wrong, stop here. Fix it, commit, and restart from step 1 +with the new commit. + +## 3. Save, checksum, and retain the artifact + +```bash +mkdir -p /tmp/weathertool-release +OUT=/tmp/weathertool-release/weathertool-$SHA.tar.gz +docker save weathertool:$SHA | gzip > "$OUT" +sha256sum "$OUT" | tee "$OUT.sha256" +``` + +## 4. Transfer to the VPS and verify + +```bash +scp "$OUT" "$OUT.sha256" vps:/tmp/ + +ssh vps "cd /tmp && sha256sum -c weathertool-$SHA.tar.gz.sha256" +# must print: weathertool-.tar.gz: OK +``` + +Do not proceed past a checksum mismatch. Re-transfer or re-build. + +## 5. Retain the release on the VPS + +This is the step tonight's releases skipped — restore it. Keeping the +tarball means a future rollback never needs Rocky at all. + +```bash +ssh vps "sudo mkdir -p /srv/weathertool/releases/$SHA && sudo mv /tmp/weathertool-$SHA.tar.gz /tmp/weathertool-$SHA.tar.gz.sha256 /srv/weathertool/releases/$SHA/" +``` + +## 6. Load the image + +```bash +ssh vps "sudo docker load -i /srv/weathertool/releases/$SHA/weathertool-$SHA.tar.gz" +``` + +## 7. Back up `.env.staging` before editing it + +Also restore this — it's how earlier releases (through `ef64895`) protected +against a bad edit to a file that lives outside Git and holds real secrets. + +```bash +ssh vps "sudo mkdir -p /srv/weathertool/backups && sudo cp /srv/weathertool/.env.staging /srv/weathertool/backups/env.staging.before-$SHA && sudo chmod 600 /srv/weathertool/backups/env.staging.before-$SHA" +``` + +## 8. Point at the new image and roll out + +```bash +ssh vps "cd /srv/weathertool && sudo sed -i 's|^WEATHERTOOL_IMAGE=.*|WEATHERTOOL_IMAGE=weathertool:$SHA|' .env.staging && grep WEATHERTOOL_IMAGE .env.staging" +ssh vps "cd /srv/weathertool && sudo docker compose --env-file .env.staging up -d --no-deps app" +``` + +`--no-deps app` is deliberate — this recreates only the WeatherTool +container. PostgreSQL and Authelia are never touched by an application +release. + +If this release also needs a database change (schema, backfill, wipe), +back up PostgreSQL first — see `VPS_STAGING_PLAN.md`'s acceptance +checklist. That is a separate, explicit decision from the steps here. + +## 9. Verify + +```bash +ssh vps "sudo docker ps --filter name=weathertool-uat-app-1 --format '{{.Status}}'" # expect Up ... (healthy) +ssh vps "sudo docker logs weathertool-uat-app-1 --tail 20" # expect clean startup, no errors +ssh vps "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8002/" # expect 200 + +# public boundary — confirm Authelia/Cloudflare gate is unchanged +curl -s -o /dev/null -w "%{http_code}\n" https://laikapstak.li/ # expect 302 (unauthenticated redirect) +curl -s -o /dev/null -w "%{http_code}\n" https://laikapstak.li/api/warnings # expect 401 (unauthenticated API) +``` + +Add release-specific loopback checks for whatever this release actually +changed — a bugfix should be checked against the live VPS data, not assumed +from the Rocky smoke test alone. + +## 10. Clean up and record + +```bash +rm -f /tmp/weathertool-release/weathertool-$SHA.tar.gz* +``` + +The transfer copy in `/tmp` on both ends is temporary and safe to delete — +the real, retained copy is `/srv/weathertool/releases/$SHA/` from step 5. + +Then update: + +- `docs/README.md` — the "Release `X` is deployed..." status line, with the + new image tag and rollback target (the *previous* release's SHA). +- `docs/UPDATE_ROADMAP.md`'s change log table — one row: date, SHA, what + shipped, and exactly how it was verified (not just "yes," the specific + checks run). + +## Rollback + +Only needed if verification in step 9 fails, or a problem surfaces after +release. + +```bash +# PREVIOUS_SHA = the rollback target recorded in the last README release line +ssh vps "sudo docker load -i /srv/weathertool/releases/$PREVIOUS_SHA/weathertool-$PREVIOUS_SHA.tar.gz" # only if that image was pruned locally; usually still loaded +ssh vps "cd /srv/weathertool && sudo sed -i 's|^WEATHERTOOL_IMAGE=.*|WEATHERTOOL_IMAGE=weathertool:$PREVIOUS_SHA|' .env.staging" +ssh vps "cd /srv/weathertool && sudo docker compose --env-file .env.staging up -d --no-deps app" +``` + +Then repeat step 9's verification against the rolled-back release, and +record the rollback in the roadmap change log the same way as a forward +release. + +A database change is not undone by an application rollback — if the release +being rolled back touched the database, that needs its own explicit +decision and the relevant PostgreSQL backup from step 8.