WeatherTool/docs/VPS_RELEASE_RUNBOOK.md

# VPS release runbook

This is the literal, runnable command sequence for shipping a new WeatherTool
release to the VPS. `VPS_STAGING_PLAN.md` explains *why* the process looks
this way (topology, isolation boundary, acceptance checklist) and gives the
same steps at a policy level; this document is the *how* — copy-pasteable
commands, run manually, in order, every time. There is no automation here
deliberately: continuous deployment is out of scope while the VPS carries
real testers, and every release is a deliberate human checkpoint (see
`CONTINUOUS_INTEGRATION.md` for why CI does not perform this step either).

Run every command from the Rocky checkout at
`/home/sandbox/Documents/projects/WeatherTool` unless noted otherwise.

## 0. Preconditions

```bash
git status --short   # must be empty — never build from an uncommitted tree
git log --oneline -1
```

If CI exists for this commit, check it's green before proceeding — nothing
else currently stops a red-CI commit from being deployed.

## 1. Build the release image

Always build from the committed tree via `git archive`, never from the
working directory — a dirty or stale build has bitten this project before.

```bash
SHA=$(git rev-parse HEAD)
echo "Building weathertool:$SHA"
git archive HEAD | docker build -f deploy/vps/Dockerfile -t weathertool:$SHA -
```

## 2. Smoke-test the exact image

Test the literal artifact that will ship, not just "the code" — against a
throwaway Postgres and placeholder credentials, never real ones.

```bash
docker network create weathertool-smoketest-net
docker run -d --name weathertool-smoketest-pg --network weathertool-smoketest-net \
  -e POSTGRES_DB=smoketest -e POSTGRES_USER=smoketest -e POSTGRES_PASSWORD=smoketestpass \
  postgres:16.1
until docker exec weathertool-smoketest-pg pg_isready -U smoketest -d smoketest >/dev/null 2>&1; do sleep 1; done

docker run -d --name weathertool-smoketest --network weathertool-smoketest-net \
  -p 18080:8080 \
  -e POSTGRES_DB=smoketest -e POSTGRES_USER=smoketest -e POSTGRES_PASSWORD=smoketestpass -e POSTGRES_HOST=weathertool-smoketest-pg \
  -e LVGMC_URL=placeholder -e LVGMC_USER=placeholder -e LVGMC_PASSWORD=placeholder \
  -e HARMONIE_EDR_URL=placeholder -e HARMONIE_EDR_API_KEY=placeholder -e HARMONIE_STAC_URL=placeholder -e HARMONIE_STAC_API_KEY=placeholder \
  -e ENABLE_SCHEDULED_JOBS=false -e ENABLE_LVGMC_FTP_JOBS=false -e ENABLE_HARMONIE_JOBS=false \
  weathertool:$SHA

sleep 6
docker logs weathertool-smoketest --tail 20   # expect clean startup, no "Fatal error"
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/   # expect 200
```

Add release-specific checks here (route behavior, a known bug that's
supposedly fixed, a real data path). Then tear down:

```bash
docker stop weathertool-smoketest weathertool-smoketest-pg
docker rm weathertool-smoketest weathertool-smoketest-pg
docker network rm weathertool-smoketest-net
```

If anything looks wrong, stop here. Fix it, commit, and restart from step 1
with the new commit.

## 3. Save, checksum, and retain the artifact

```bash
mkdir -p /tmp/weathertool-release
cd /tmp/weathertool-release
docker save weathertool:$SHA | gzip > weathertool-$SHA.tar.gz
sha256sum weathertool-$SHA.tar.gz | tee weathertool-$SHA.tar.gz.sha256
```

Generate the checksum from a bare filename (after `cd`-ing into the
directory), not a full path — `sha256sum -c` on the VPS checks the exact
path string recorded in the `.sha256` file, and a local absolute path won't
exist there. (Found by running this step for real, not a hypothetical.)

## 4. Transfer to the VPS and verify

```bash
scp weathertool-$SHA.tar.gz weathertool-$SHA.tar.gz.sha256 vps:/tmp/

ssh vps "cd /tmp && sha256sum -c weathertool-$SHA.tar.gz.sha256"
# must print: weathertool-<SHA>.tar.gz: OK
```

Do not proceed past a checksum mismatch. Re-transfer or re-build.

## 5. Retain the release on the VPS

This is the step tonight's releases skipped — restore it. Keeping the
tarball means a future rollback never needs Rocky at all.

```bash
ssh vps "sudo mkdir -p /srv/weathertool/releases/$SHA && sudo mv /tmp/weathertool-$SHA.tar.gz /tmp/weathertool-$SHA.tar.gz.sha256 /srv/weathertool/releases/$SHA/"
```

## 6. Load the image

```bash
ssh vps "sudo docker load -i /srv/weathertool/releases/$SHA/weathertool-$SHA.tar.gz"
```

## 7. Back up `.env.staging` before editing it

Also restore this — it's how earlier releases (through `ef64895`) protected
against a bad edit to a file that lives outside Git and holds real secrets.

```bash
ssh vps "sudo mkdir -p /srv/weathertool/backups && sudo cp /srv/weathertool/.env.staging /srv/weathertool/backups/env.staging.before-$SHA && sudo chmod 600 /srv/weathertool/backups/env.staging.before-$SHA"
```

## 8. Point at the new image and roll out

```bash
ssh vps "cd /srv/weathertool && sudo sed -i 's|^WEATHERTOOL_IMAGE=.*|WEATHERTOOL_IMAGE=weathertool:$SHA|' .env.staging && grep WEATHERTOOL_IMAGE .env.staging"
ssh vps "cd /srv/weathertool && sudo docker compose --env-file .env.staging up -d --no-deps app"
```

`--no-deps app` is deliberate — this recreates only the WeatherTool
container. PostgreSQL and Authelia are never touched by an application
release.

If this release also needs a database change (schema, backfill, wipe),
back up PostgreSQL first — see `VPS_STAGING_PLAN.md`'s acceptance
checklist. That is a separate, explicit decision from the steps here.

## 9. Verify

```bash
ssh vps "sudo docker ps --filter name=weathertool-uat-app-1 --format '{{.Status}}'"   # expect Up ... (healthy)
ssh vps "sudo docker logs weathertool-uat-app-1 --tail 20"                            # expect clean startup, no errors
ssh vps "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8002/"             # expect 200

# public boundary — confirm Authelia/Cloudflare gate is unchanged
curl -s -o /dev/null -w "%{http_code}\n" https://laikapstak.li/          # expect 302 (unauthenticated redirect)
curl -s -o /dev/null -w "%{http_code}\n" https://laikapstak.li/api/warnings   # expect 401 (unauthenticated API)
```

Add release-specific loopback checks for whatever this release actually
changed — a bugfix should be checked against the live VPS data, not assumed
from the Rocky smoke test alone.

## 10. Clean up and record

```bash
rm -f /tmp/weathertool-release/weathertool-$SHA.tar.gz*
```

The transfer copy in `/tmp` on both ends is temporary and safe to delete —
the real, retained copy is `/srv/weathertool/releases/$SHA/` from step 5.

Then update:

- `docs/README.md` — the "Release `X` is deployed..." status line, with the
  new image tag and rollback target (the *previous* release's SHA).
- `docs/UPDATE_ROADMAP.md`'s change log table — one row: date, SHA, what
  shipped, and exactly how it was verified (not just "yes," the specific
  checks run).

## Rollback

Only needed if verification in step 9 fails, or a problem surfaces after
release.

```bash
# PREVIOUS_SHA = the rollback target recorded in the last README release line
ssh vps "sudo docker load -i /srv/weathertool/releases/$PREVIOUS_SHA/weathertool-$PREVIOUS_SHA.tar.gz"   # only if that image was pruned locally; usually still loaded
ssh vps "cd /srv/weathertool && sudo sed -i 's|^WEATHERTOOL_IMAGE=.*|WEATHERTOOL_IMAGE=weathertool:$PREVIOUS_SHA|' .env.staging"
ssh vps "cd /srv/weathertool && sudo docker compose --env-file .env.staging up -d --no-deps app"
```

Then repeat step 9's verification against the rolled-back release, and
record the rollback in the roadmap change log the same way as a forward
release.

A database change is not undone by an application rollback — if the release
being rolled back touched the database, that needs its own explicit
decision and the relevant PostgreSQL backup from step 8.