WeatherTool/docs/VPS_RELEASE_RUNBOOK.md

VPS release runbook

This is the literal, runnable command sequence for shipping a new WeatherTool release to the VPS. VPS_STAGING_PLAN.md explains why the process looks this way (topology, isolation boundary, acceptance checklist) and gives the same steps at a policy level; this document is the how — copy-pasteable commands, run manually, in order, every time. There is no automation here deliberately: continuous deployment is out of scope while the VPS carries real testers, and every release is a deliberate human checkpoint (see CONTINUOUS_INTEGRATION.md for why CI does not perform this step either).

Run every command from the Rocky checkout at /home/sandbox/Documents/projects/WeatherTool unless noted otherwise.

0. Preconditions

git status --short   # must be empty — never build from an uncommitted tree
git log --oneline -1

If CI exists for this commit, check it's green before proceeding — nothing else currently stops a red-CI commit from being deployed.

1. Build the release image

Always build from the committed tree via git archive, never from the working directory — a dirty or stale build has bitten this project before.

SHA=$(git rev-parse HEAD)
echo "Building weathertool:$SHA"
git archive HEAD | docker build -f deploy/vps/Dockerfile -t weathertool:$SHA -

2. Smoke-test the exact image

Test the literal artifact that will ship, not just "the code" — against a throwaway Postgres and placeholder credentials, never real ones.

docker network create weathertool-smoketest-net
docker run -d --name weathertool-smoketest-pg --network weathertool-smoketest-net \
  -e POSTGRES_DB=smoketest -e POSTGRES_USER=smoketest -e POSTGRES_PASSWORD=smoketestpass \
  postgres:16.1
until docker exec weathertool-smoketest-pg pg_isready -U smoketest -d smoketest >/dev/null 2>&1; do sleep 1; done

docker run -d --name weathertool-smoketest --network weathertool-smoketest-net \
  -p 18080:8080 \
  -e POSTGRES_DB=smoketest -e POSTGRES_USER=smoketest -e POSTGRES_PASSWORD=smoketestpass -e POSTGRES_HOST=weathertool-smoketest-pg \
  -e LVGMC_URL=placeholder -e LVGMC_USER=placeholder -e LVGMC_PASSWORD=placeholder \
  -e HARMONIE_EDR_URL=placeholder -e HARMONIE_EDR_API_KEY=placeholder -e HARMONIE_STAC_URL=placeholder -e HARMONIE_STAC_API_KEY=placeholder \
  -e ENABLE_SCHEDULED_JOBS=false -e ENABLE_LVGMC_FTP_JOBS=false -e ENABLE_HARMONIE_JOBS=false \
  weathertool:$SHA

sleep 6
docker logs weathertool-smoketest --tail 20   # expect clean startup, no "Fatal error"
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/   # expect 200

Add release-specific checks here (route behavior, a known bug that's supposedly fixed, a real data path). Then tear down:

docker stop weathertool-smoketest weathertool-smoketest-pg
docker rm weathertool-smoketest weathertool-smoketest-pg
docker network rm weathertool-smoketest-net

If anything looks wrong, stop here. Fix it, commit, and restart from step 1 with the new commit.

3. Save, checksum, and retain the artifact

mkdir -p /tmp/weathertool-release
cd /tmp/weathertool-release
docker save weathertool:$SHA | gzip > weathertool-$SHA.tar.gz
sha256sum weathertool-$SHA.tar.gz | tee weathertool-$SHA.tar.gz.sha256

Generate the checksum from a bare filename (after cd-ing into the directory), not a full path — sha256sum -c on the VPS checks the exact path string recorded in the .sha256 file, and a local absolute path won't exist there. (Found by running this step for real, not a hypothetical.)

4. Transfer to the VPS and verify

scp weathertool-$SHA.tar.gz weathertool-$SHA.tar.gz.sha256 vps:/tmp/

ssh vps "cd /tmp && sha256sum -c weathertool-$SHA.tar.gz.sha256"
# must print: weathertool-<SHA>.tar.gz: OK

Do not proceed past a checksum mismatch. Re-transfer or re-build.

5. Retain the release on the VPS

This is the step tonight's releases skipped — restore it. Keeping the tarball means a future rollback never needs Rocky at all.

ssh vps "sudo mkdir -p /srv/weathertool/releases/$SHA && sudo mv /tmp/weathertool-$SHA.tar.gz /tmp/weathertool-$SHA.tar.gz.sha256 /srv/weathertool/releases/$SHA/"

6. Load the image

ssh vps "sudo docker load -i /srv/weathertool/releases/$SHA/weathertool-$SHA.tar.gz"

7. Back up .env.staging before editing it

Also restore this — it's how earlier releases (through ef64895) protected against a bad edit to a file that lives outside Git and holds real secrets.

ssh vps "sudo mkdir -p /srv/weathertool/backups && sudo cp /srv/weathertool/.env.staging /srv/weathertool/backups/env.staging.before-$SHA && sudo chmod 600 /srv/weathertool/backups/env.staging.before-$SHA"

8. Point at the new image and roll out

ssh vps "cd /srv/weathertool && sudo sed -i 's|^WEATHERTOOL_IMAGE=.*|WEATHERTOOL_IMAGE=weathertool:$SHA|' .env.staging && grep WEATHERTOOL_IMAGE .env.staging"
ssh vps "cd /srv/weathertool && sudo docker compose --env-file .env.staging up -d --no-deps app"

--no-deps app is deliberate — this recreates only the WeatherTool container. PostgreSQL and Authelia are never touched by an application release.

If this release also needs a database change (schema, backfill, wipe), back up PostgreSQL first — see VPS_STAGING_PLAN.md's acceptance checklist. That is a separate, explicit decision from the steps here.

9. Verify

ssh vps "sudo docker ps --filter name=weathertool-uat-app-1 --format '{{.Status}}'"   # expect Up ... (healthy)
ssh vps "sudo docker logs weathertool-uat-app-1 --tail 20"                            # expect clean startup, no errors
ssh vps "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8002/"             # expect 200

# public boundary — confirm Authelia/Cloudflare gate is unchanged
curl -s -o /dev/null -w "%{http_code}\n" https://laikapstak.li/          # expect 302 (unauthenticated redirect)
curl -s -o /dev/null -w "%{http_code}\n" https://laikapstak.li/api/warnings   # expect 401 (unauthenticated API)

Add release-specific loopback checks for whatever this release actually changed — a bugfix should be checked against the live VPS data, not assumed from the Rocky smoke test alone.

10. Clean up and record

rm -f /tmp/weathertool-release/weathertool-$SHA.tar.gz*

The transfer copy in /tmp on both ends is temporary and safe to delete — the real, retained copy is /srv/weathertool/releases/$SHA/ from step 5.

Then update:

• docs/README.md — the "Release X is deployed..." status line, with the new image tag and rollback target (the previous release's SHA).
• docs/UPDATE_ROADMAP.md's change log table — one row: date, SHA, what shipped, and exactly how it was verified (not just "yes," the specific checks run).

Rollback

Only needed if verification in step 9 fails, or a problem surfaces after release.

# PREVIOUS_SHA = the rollback target recorded in the last README release line
ssh vps "sudo docker load -i /srv/weathertool/releases/$PREVIOUS_SHA/weathertool-$PREVIOUS_SHA.tar.gz"   # only if that image was pruned locally; usually still loaded
ssh vps "cd /srv/weathertool && sudo sed -i 's|^WEATHERTOOL_IMAGE=.*|WEATHERTOOL_IMAGE=weathertool:$PREVIOUS_SHA|' .env.staging"
ssh vps "cd /srv/weathertool && sudo docker compose --env-file .env.staging up -d --no-deps app"

Then repeat step 9's verification against the rolled-back release, and record the rollback in the roadmap change log the same way as a forward release.

A database change is not undone by an application rollback — if the release being rolled back touched the database, that needs its own explicit decision and the relevant PostgreSQL backup from step 8.