diff --git a/docs/CONTINUOUS_INTEGRATION.md b/docs/CONTINUOUS_INTEGRATION.md new file mode 100644 index 0000000..5321ff1 --- /dev/null +++ b/docs/CONTINUOUS_INTEGRATION.md @@ -0,0 +1,132 @@ +# Continuous integration + +Last verified: 2026-08-24 + +## Current topology + +WeatherTool uses Gitea Actions for continuous integration: + +```text +bot/WeatherTool repository on gitea.packet.garden + → repository-scoped weathertool-ci-rocky-01 runner on the coding PC + → dedicated Docker-in-Docker daemon + → disposable per-job containers (backend: hseeberger/scala-sbt, frontend: node:20-bookworm) +``` + +Gitea does not execute workflow commands itself. The runner polls Gitea for +jobs and executes them. The runner is registered only to `bot/WeatherTool`; +it is not a user-, organization-, or instance-wide runner. + +The runner host is the Rocky Linux coding PC — the same machine already used +for all compile/build/test/staging work. This is a second, independent +runner instance on that host: HOP already runs its own Forgejo runner there +(`hop-forgejo-runner-*`, registered to a different Forgejo instance's +`bot/hop` repo). The two don't conflict — each is a separate long-polling +client registered to a different server with its own token, own Docker +network, and own Docker-in-Docker daemon; they only share host CPU/RAM, +which is a non-issue for occasional CI runs. + +## Isolation boundary + +Same pattern as HOP's Forgejo runner: the runner uses a dedicated +Docker-in-Docker daemon rather than exposing the coding PC's normal Docker +socket to workflow containers. The nested daemon: + +- runs in its own privileged container because a nested Docker daemon + requires that capability; +- exposes no port on the host or LAN; +- is reachable only through the runner's private Compose network; and +- stores its images and state in the `weathertool-gitea-runner_dind_data` + volume. + +This prevents ordinary WeatherTool workflow jobs from enumerating or +mutating the coding PC's normal development containers (including the +Rocky staging stack itself). + +## Host-local runner files + +The active runner is operated from: + +```text +/home/sandbox/.local/share/weathertool-gitea-runner/ + compose.yml + runner-token +``` + +The directory is mode `0700`. `runner-token` is mode `0600`. It is +deliberately outside the WeatherTool Git repository and was written directly +to disk from an interactive prompt — it was never printed, pasted into +chat, or copied into project documentation. + +Normal operator commands are run from the directory above: + +```bash +docker compose up -d +docker compose ps +docker compose logs --tail=100 runner +docker compose pull +``` + +The expected steady state is: + +- `docker-in-docker`: `Up (healthy)`; +- `runner`: `Up`; and +- `weathertool-ci-rocky-01`: `Idle` in Gitea when no job is queued. + +`Idle` is healthy. It means the runner is authenticated and polling for +work. The runner is available only while the coding PC, Docker, and these +containers are running. + +One known, non-fatal warning in the runner logs: `cannot init cache server, +it will be disabled: mkdir /.cache: permission denied` — the runner +container runs as uid 1000 with no writable `$HOME`, so the `actions/cache` +action type is unavailable. Nothing in the current workflow uses it. Fixable +later by setting `HOME=/data` in the runner's environment if caching +becomes worth adding. + +## WeatherTool workflow + +The version-controlled workflow is: + +```text +.gitea/workflows/ci.yml +``` + +It runs for: + +- pushes to `codex/staging-baseline`; +- pull requests targeting `codex/staging-baseline`; and +- manual `workflow_dispatch` requests. + +Two independent jobs, each with its own container image rather than one +shared runner-label image with ad-hoc installs — this reuses images already +proven for this project rather than bootstrapping a second language runtime +into a single shared container: + +- **`backend`** (`hseeberger/scala-sbt:17.0.2_1.6.2_2.13.8`, the same image + used all session for local Rocky builds): checkout, `sbt -batch test`. +- **`frontend`** (`node:20-bookworm`): checkout, `npm ci`, `npm run + typecheck`, `npm run build`, `npm audit`, `npm audit --omit=dev` — the + same five checks already documented as the manual Rocky verification + routine in `DEVELOPMENT_AND_STAGING.md`. + +Verified before writing the workflow that `sbt test` compiles and passes +with no `.env` file present at all (via a `git archive HEAD` dry run into a +clean scratch directory) — matching exactly what a checkout-only CI job +actually has, since real credentials must never reach CI. The sbt-dotenv +plugin logs a graceful warning and continues; nothing in the current test +suite (`FileNameServiceSpec`; `ParserSpec` has no live assertions) touches +`sys.env` or a live database. + +The first complete green run was verified on 2026-08-24 for commit +`5b88e69` (`ci.yml #1`, 33s). + +## Current boundary + +This workflow is CI only. A green result proves the committed source +compiles, passes its (currently minimal) test suite, typechecks, builds, +and has no known frontend dependency vulnerabilities. It does not deploy to +Rocky staging or the VPS, publish an image, or access any staging/production +secret — the runner's job containers never see `.env`, `.env.staging`, or +any real provider credential. Continuous delivery, and expanding actual test +coverage (see the roadmap's Phase 2), remain separate, not yet started work. diff --git a/docs/README.md b/docs/README.md index 2b7650e..695d34a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,7 +16,8 @@ This directory contains the working documentation for the WeatherTool modernizat - **Brīdinājumi** renders current LVĢMC warning polygons over a production border overlay with feathered severity fills, plus draggable/resizable per-warning weather-symbol placement. Its lon/lat-to-pixel projection is an affine fit calibrated against the same validated city pixel positions Kartes/Faktiskā already use, replacing an earlier bounding-box calibration that drifted up to ~200px on the 3840 canvas. - Confirmed local Monda Regular/Bold files provide interface and generated-graphic typography; weather symbols use normalized transparent image assets. - Release `b0b58d2` is deployed as immutable image `weathertool:b0b58d2f1e0ed47ca13795b64595386ec2f0e0c7`; release `3eddf95` remains the immediate application rollback. On top of the earlier frontend design pass and the timezone/upsert/scheduler-split fixes, this release replaces the hardcoded per-route static-file list with a general SPA fallback — a real, user-reported bug where refreshing `/faktiska` or `/udens-temperatura` 404ed instead of loading the app (those two routes were never added to the old list). A missing `/assets` file still 404s properly rather than silently serving HTML. -- An independent fresh-eyes security review (requested against the local Rocky version only) found a live-verified CRITICAL path-traversal vulnerability plus several HIGH/MEDIUM findings — see the roadmap changelog for `6b9c7cf`/`8c45d8d` for the full list. All are fixed, compiled, live-verified on Rocky, and deployed to the VPS as release `0be325f` (image `weathertool:0be325fbbbf92b03bc8d574dc6f7b9449ef310ea`); release `b0b58d2` remains the immediate application rollback. Rocky's `POSTGRES_PASSWORD` was rotated afterward since the old value had been exposed into the review's own output via `/proc/self/environ`. `LVGMC_PASSWORD` rotation is not yet done — that needs the user's own action via the LVGMC contact. +- An independent fresh-eyes security review (requested against the local Rocky version only) found a live-verified CRITICAL path-traversal vulnerability plus several HIGH/MEDIUM findings — see the roadmap changelog for `6b9c7cf`/`8c45d8d` for the full list. All are fixed, compiled, live-verified on Rocky, and deployed to the VPS as release `0be325f` (image `weathertool:0be325fbbbf92b03bc8d574dc6f7b9449ef310ea`); release `b0b58d2` remains the immediate application rollback. Rocky's `POSTGRES_PASSWORD` was rotated afterward since the old value had been exposed into the review's own output via `/proc/self/environ`. `LVGMC_PASSWORD` rotation is not yet done — that needs the user's own action via the LVGMC contact. A follow-up independent review then caught that the FTP auth-bypass fix was incomplete (a sibling route, `/api/show/lvgmc-forecast`, had the same live-FTP-trigger issue but had only gotten the traversal fix); fixed and deployed as `9bab93d`. That same review flagged several open architectural/testability findings (the `parMapN` scheduler crash-loop pattern, per-request `System.gc()`, an unenforced field-routing invariant, no JDBC connection pooling, and several files under `src/main/scala` that look like tests but aren't) — not yet acted on, tracked for a future session. +- Gitea Actions CI is live: a repository-scoped runner on Rocky (isolated behind its own Docker-in-Docker daemon, same pattern as HOP's Forgejo runner) runs `sbt test` plus the frontend typecheck/build/audit routine on every push/PR to `codex/staging-baseline`. See [Continuous integration](CONTINUOUS_INTEGRATION.md). CI-only for now — it does not deploy anywhere or touch any real credential. - FTP (`ENABLE_LVGMC_FTP_JOBS`) is currently `false` on both Rocky and the VPS — re-testing the morning after enabling it turned up a second, unexplained failure (Rocky alone, VPS confirmed off, ~4 minutes after a successful manual test) that doesn't fit the original two-machine-collision theory. Decided to stop self-testing via trial and error and wait for the user to ask the LVGMC contact directly about the `ltv` account's connection/rate policy, rather than risk repeatedly tripping an unknown limit. See the roadmap Phase 7 for the full diagnosis timeline. - The isolated VPS UAT stack is running and healthy: WeatherTool is bound to `127.0.0.1:8002`, Authelia to `127.0.0.1:9091`, and PostgreSQL has no host port. Public access is routed through Cloudflare, Nginx, and Authelia. - Cloudflare delegation is active, strict origin TLS covers only `laikapstak.li` and `auth.laikapstak.li`, and the public Nginx/Authelia login flow is operational without changing the existing HOP site. @@ -34,6 +35,7 @@ This directory contains the working documentation for the WeatherTool modernizat - [Development and staging](DEVELOPMENT_AND_STAGING.md) — Windows source/Git workflow, Rocky development and verification, VPS deployment, synthetic data, and rollback. - [Product workflows](PRODUCT_WORKFLOWS.md) — the intended purpose and current status of each visible workspace. - [Update roadmap](UPDATE_ROADMAP.md) — phased technical, security, dependency, testing, and UI work. +- [Continuous integration](CONTINUOUS_INTEGRATION.md) — Gitea Actions topology, isolation boundary, and the current CI workflow. - [Third-party notices](THIRD_PARTY_NOTICES.md) — licenses and attribution for adapted interface components. - [Temporary VPS staging plan](VPS_STAGING_PLAN.md) — isolation, authentication, prepared deployment bundle, release, backup, verification, and rollback model for external user testing. diff --git a/docs/UPDATE_ROADMAP.md b/docs/UPDATE_ROADMAP.md index 043846e..10ceb43 100644 --- a/docs/UPDATE_ROADMAP.md +++ b/docs/UPDATE_ROADMAP.md @@ -57,9 +57,10 @@ Status: in progress - [ ] Add database integration tests for aggregation and export behavior. - [ ] Restore and expand CSV parser tests. - [ ] Add GRIB parser boundary and malformed-file tests. -- [ ] Add security tests for invalid fields, filenames, offsets, lengths, and date ranges. -- [ ] Add frontend type checking and critical workflow smoke tests. -- [ ] Run tests automatically before staging deployment. +- [ ] Add security tests for invalid fields, filenames, offsets, lengths, and date ranges. Highest-priority target per an independent review: `ValidateFileName`/`ValidateField`/`ValidateInt` are pure functions with zero I/O and are currently the entire path-traversal/SQL-injection security boundary, verified only by manual curl. +- [ ] `PostgresService.query`'s "list" branch (`byField` match, ~13 hardcoded literal cases, no `case _ =>`) is only safe today because `ValidateField`'s allowlist and `WeatherData`'s case class fields happen to stay in sync with it — nothing enforces that. Add a field to `WeatherData` without updating this match and any request for it with `granularity=hour` throws an uncaught `MatchError` (a real risk given the DMI/open-data provider work in progress). A test iterating every `WeatherData.getKeys` value through this path would catch it before it ships; needs either a test-container Postgres or refactoring SQL-fragment-building apart from execution so it's testable without a DB. +- [x] Add frontend type checking (`npm run typecheck`, part of the dependency-maintenance work in Phase 3) — now also runs automatically in CI (see below). Critical workflow smoke tests remain manual/headless-browser only, not automated. +- [ ] Run tests automatically before staging deployment. Partial: Gitea Actions CI (see `CONTINUOUS_INTEGRATION.md`) now runs `sbt test` and the frontend typecheck/build/audit routine automatically on every push/PR to `codex/staging-baseline` — this is CI, not CD; it isn't yet wired as a required gate before Rocky/VPS deployment, which remains a manual decision independent of CI status. ## Phase 3 — Dependency modernization @@ -103,8 +104,9 @@ Status: pending Status: pending - [ ] Manage custom executors as resources and close them cleanly. -- [ ] Remove explicit `System.gc()` calls. -- [ ] Supervise scheduled jobs independently instead of recursively restarting the application. +- [ ] Remove explicit `System.gc()` calls. Confirmed still present in `DataService.getBinaryChunk` (runs on every GRIB binary-chunk request — likely a latency source on what's probably a hot path; a commented-out `logMemory` block next to it suggests leftover debugging code, not an intentional design choice), per a 2026-08-24 independent review. +- [ ] Supervise scheduled jobs independently instead of recursively restarting the application. Confirmed still fully live by a 2026-08-24 independent review: `Scheduler.scheduleTask` has zero `.attempt`/error handling on any task, none of the task bodies wired into it (FTP fetch, open-data fetch, cleanup, GRIB fetch) handle their own errors either, and `(serverTask, scheduledTasks).parMapN(...)` means any one failure cancels the HTTP server itself — this is the same defect that caused the actual VPS/Rocky crash-loop incidents earlier this session, patched around with feature flags rather than closed. `OpenDataStationService` also has no client timeout configured, unlike the DMI fetch service. `WarningService`/`WaterTemperatureService` already show the correct pattern (`.attempt` + stale-cache fallback) the scheduler tasks should follow. +- [ ] Add a pooled JDBC transactor. `DBConnection.scala` uses `Transactor.fromDriverManager`, which opens a brand-new physical connection for every query — doobie's documented behavior for scripts/tests, not production services. Not urgent at current traffic, but an easy, well-known swap (`HikariTransactor`) worth doing before real load. - [ ] Add application health and readiness endpoints. - [ ] Add structured logging without leaking secrets. - [ ] Define PostgreSQL and GRIB-data backup/restore procedures. @@ -446,4 +448,6 @@ Record completed work here by date and commit after the Git workflow is establis | 2026-08-24 | *(operational, no commit)* | Rotated Rocky's `POSTGRES_PASSWORD` following the security review, since the old value had been exposed into this session's context multiple times (two of my own sloppy shell commands, plus the `/proc/self/environ` read the review used as its traversal proof). Changed the role's actual password via `ALTER ROLE` against the running container (editing `.env` alone has no effect on an already-initialized PostgreSQL data directory), then updated `.env` and recreated the `scala` service. The Postgres container itself was also recreated as a side effect (its own `POSTGRES_PASSWORD` env interpolation changed too), which is harmless — that variable only takes effect on a fresh, empty data directory, not an existing one — but is worth knowing about | Yes — clean scala container startup with no auth errors, and a real query (`/api/query/latest-temperatures/Rīga`) returning live data over the new password | | 2026-08-24 | `0be325f` | Deploy the security-review fixes (`6b9c7cf`, `8c45d8d`) to the VPS as image `weathertool:0be325fbbbf92b03bc8d574dc6f7b9449ef310ea`; release `b0b58d2` remains the immediate application rollback. Application-only release — PostgreSQL and Authelia were not restarted | Yes — exact release image smoke-tested on Rocky against a throwaway local Postgres before transfer (traversal/injection payloads 404, gated FTP route 503, legitimate queries 200); checksum verified on both ends; `app` service recreated cleanly (healthy, no auth/DB errors in logs); loopback re-verification of the same traversal/injection/FTP-gate checks plus real `/api/warnings` and `/api/water-temperatures` responses; public HTTPS confirmed unchanged (302 unauthenticated page, 401 unauthenticated API) | | 2026-08-24 | `9bab93d` | A follow-up independent fresh-eyes review (requested specifically as a second pass, not a rerun of the same pentest) caught that the `6b9c7cf` FTP auth-bypass fix was incomplete: only `/api/fetch/lvgmc/stations` (the one route the original review named) was gated behind `ENABLE_LVGMC_FTP_JOBS`. A sibling route, `/api/show/lvgmc-forecast/{fileName}`, calls the same `fetch.fetchFile` — a real, unauthenticated LVGMC FTP login — and had only gotten the traversal fix (`ValidateFileName`), not the auth-bypass fix, because the two routes were fixed along different mental categories (traversal batch vs. the one named auth-bypass fix) instead of by tracing every caller of the dangerous method. Gated this route the same way; traced every HTTP-reachable caller of `fetch.fetchFile`/`fetchWeatherStations` this time to confirm no others remain. Deployed to both Rocky and the VPS (image `weathertool:9bab93daa6b3a0fb99165a8fac9accfa3a3ccbf1`) immediately given this was live and exploitable in production | Yes — Scala tests, Rocky rebuild and curl verification (both FTP routes 503 while the flag is off), VPS image smoke-tested against a throwaway local Postgres before transfer, checksum verified, `app` service recreated cleanly, loopback re-verification of both gated routes, public HTTPS unchanged (302/401) | +| 2026-08-24 | *(review, no commit)* | A follow-up independent fresh-eyes review (explicitly scoped as a second, different pass — not a rerun of the original pentest) sanity-checked the `6b9c7cf`/`8c45d8d` fixes and did a broader skeptical pass. Caught the incomplete FTP fix documented under `9bab93d` above. Confirmed `ValidateFileName`'s minor `"."` edge case is harmless (resolves to a directory, not exploitable) and found no other unvalidated `Fragment.const` usages. Surfaced several open architectural/testability findings now tracked in Phase 2/Phase 5 above: the `parMapN` scheduler crash-loop pattern (still fully live), per-request `System.gc()` in `getBinaryChunk`, `PostgresService.query`'s unenforced field-routing invariant, no JDBC connection pooling, and several files under `src/main/scala` (`DataServiceTest`, `FetchServiceTest` ×2, `OpenDataStationServiceTest`, `GribParserTest`) that look like tests but are actually `main`-method scripts never run by `sbt test` — compiled straight into the production jar, at least one capable of writing to the real database or hitting live endpoints if run by mistake | +| 2026-08-24 | `5b88e69` | Add Gitea Actions CI: a repository-scoped `weathertool-ci-rocky-01` runner on Rocky (isolated behind its own Docker-in-Docker daemon, mirroring the isolation pattern already proven by HOP's Forgejo runner on the same host — the two coexist without conflict, each an independent client registered to a different server) runs `sbt test` and the frontend typecheck/build/audit routine on every push/PR to `codex/staging-baseline` plus manual dispatch. Verified `sbt test` compiles and passes with no `.env` present at all (via a `git archive HEAD` dry run) before writing the workflow, since CI must never see real credentials. Repo mirrored to Gitea (`bot/WeatherTool`) as an additional remote alongside the existing `rocky`/`origin` ones — deliberately not replacing either, pending an open question about the GitHub `origin` (`guntisdev/WeatherTool`) that won't be resolved until 2026-08-26. See `CONTINUOUS_INTEGRATION.md` | Yes — first run (`ci.yml #1`, commit `5b88e69`) verified green in the Gitea Actions UI (33s), and confirmed via the runner's nested Docker daemon that both job images (`hseeberger/scala-sbt`, `node:20-bookworm`) were actually pulled and used, not just orchestration logs | | 2026-08-24 | `2c44888`–`b0b58d2` | Replace the hardcoded per-route static-file list (`/station`, `/cities`, `/latvia`, `/database`, `/harmonie`, `/lvgmc-forecast`, `/bridinajumi`) with a general SPA fallback: real files serve as-is, anything else falls back to `index.html` so the SolidJS router handles it client-side — fixes a real, user-reported bug where a direct hit (e.g. a browser refresh) on `/faktiska` or `/udens-temperatura` 404ed instead of loading the app, since those two routes were never added to the old list. A missing file under `/assets` specifically still 404s properly rather than silently serving HTML, so a stale tab after a future deploy gets a clean error instead of a confusing JS parse failure | Yes — first attempt silently no-op'd because the fix was built into a `git archive HEAD` image before being committed (classic mistake, caught immediately by re-testing and finding identical old behavior); after committing, verified via curl on all previously-working routes, both previously-broken routes, a missing asset (404), a real asset (200), and `/api` (200), then a full headless-browser render check (zero console errors, real data, correct nav state) on a genuine direct hit — not just HTTP status codes; deployed to VPS in release `b0b58d2`, verified the same way through both the loopback port and the public domain (Authelia gate still correctly redirects unauthenticated requests) |