diff --git a/docs/README.md b/docs/README.md index 67f8cb8..f04ee64 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,16 +11,16 @@ This directory contains the working documentation for the WeatherTool modernizat - PostgreSQL is private to the project Compose network; only the Scala application publishes a host port. - The operator-facing workspaces now use the Latvian workflow names **Stacijas**, **Kartes**, **Faktiskā**, **Ūdens**, **Brīdinājumi**, **Apskats**, **Arhīvs**, **Harmonie**, and **LVĢMC**. Kartes retains custom analytical map outputs, while Faktiskā is a fixed 13-position, latest-temperature newsroom workflow with a locked 3840×1440 export. - Faktiskā symbol placement is automatic after manual image selection and is anchored to each rendered temperature badge. -- **Ūdens** uses separate authoritative 1920×1080 and 3840×1440 production templates. Its six value rectangles are measured per template and the text is optically centered from rendered glyph bounds; both exports have been visually validated. -- **Brīdinājumi** renders current LVĢMC warning polygons over a production border overlay with feathered severity fills. Its lon/lat-to-pixel projection is an affine fit calibrated against the same validated city pixel positions Kartes/Faktiskā already use, replacing an earlier bounding-box calibration that drifted up to ~200px on the 3840 canvas. +- **Ūdens** auto-populates its six ranges on load with real per-zone water-temperature min/max (65 LVĢMC stations classified into the 6 named zones), with manual override and reset still available. Uses separate authoritative 1920×1080 and 3840×1440 production templates; both exports have been visually validated. +- **Brīdinājumi** renders current LVĢMC warning polygons over a production border overlay with feathered severity fills, plus draggable/resizable per-warning weather-symbol placement. Its lon/lat-to-pixel projection is an affine fit calibrated against the same validated city pixel positions Kartes/Faktiskā already use, replacing an earlier bounding-box calibration that drifted up to ~200px on the 3840 canvas. - Confirmed local Monda Regular/Bold files provide interface and generated-graphic typography; weather symbols use normalized transparent image assets. -- Release `6185dbf` is deployed as immutable image `weathertool:6185dbfdb0752db7d715f1061d74baab3c43122e`; release `b5150ab` remains the immediate application rollback. This release hardens LVĢMC warning fetching (bounded retry during upstream outages, per-warning fetch isolation, a CKAN filter verified against the live API, and visible logging for a previously-silent vertex-order fallback) on top of the full Brīdinājumi workspace, JSON API, Latvian workflow labels, and calibrated water-temperature work. +- Release `138f57c8` is deployed as immutable image `weathertool:138f57c808631dc17396b22ac0305b670fbcf776`; release `6185dbf` remains the immediate application rollback. This is the full go-live release: real open-data station and water-temperature ingestion, the scheduler safety split (`ENABLE_SCHEDULED_JOBS` vs. `ENABLE_LEGACY_PROVIDER_JOBS`), synthetic-data-generation removal, the dead `METEO_*` path deleted, draggable Brīdinājumi symbol placement, and real per-zone water-temperature ranges — on top of the hardened LVĢMC warning fetching, full Brīdinājumi workspace, JSON API, and Latvian workflow labels already live in `6185dbf`. - The isolated VPS UAT stack is running and healthy: WeatherTool is bound to `127.0.0.1:8002`, Authelia to `127.0.0.1:9091`, and PostgreSQL has no host port. Public access is routed through Cloudflare, Nginx, and Authelia. - Cloudflare delegation is active, strict origin TLS covers only `laikapstak.li` and `auth.laikapstak.li`, and the public Nginx/Authelia login flow is operational without changing the existing HOP site. -- The VPS database contains a deliberately limited 14-day, 34-city synthetic dataset for newsroom visual testing. It must be cleared before authorized real-provider ingestion begins. +- The VPS `weather` table now holds only real LVĢMC open-data station observations; its original 14-day synthetic dataset was backed up to `/srv/weathertool/backups/pre-real-data-release/` and wiped on 2026-08-23 as part of the `138f57c8` release. The scheduled open-data job runs on the VPS (minutes 15/45); the legacy FTP/HARMONIE jobs stay off pending real credentials. - Approved 1920×1080 and 3840×1440 PNG production bases are now the rendering source for Faktiskā and Ūdens temperatūra; code draws only the changing values, selected weather symbols, and wind data over those fixed newsroom graphics. - Browser branding assets and Latvian Open Graph/Twitter metadata are included for favicon, Apple home-screen icon, and link-preview support. Public crawler access still depends on the Nginx/Authelia policy used for the metadata and preview image. -- Browser verification is complete for the deployed `6185dbf` release, including Brīdinājumi against real live LVĢMC data (a genuinely severe live wind warning tested the hardened fetch path end to end). Newsroom testers will be given the domain first, without walkthrough, to see how intuitive the workflow is before any guidance is provided. +- Browser verification is complete for the deployed `138f57c8` release: exact release image smoke-tested on Rocky before transfer, VPS PostgreSQL backed up before the synthetic-data wipe, checksum verified on both ends, container health/loopback/public HTTPS confirmed, and the scheduled open-data job observed firing and landing 1,577 real rows within ~2 minutes of the deploy. Newsroom testers will be given the domain first, without walkthrough, to see how intuitive the workflow is before any guidance is provided. - Frontend dependency maintenance is complete: Solid runtime and Vite tooling were updated, obsolete packages were removed, TypeScript checking was added, and a clean Rocky `npm ci`, typecheck, production build, full audit, and production-only audit all pass with zero known vulnerabilities. - This is not yet approved or hardened for workplace production. diff --git a/docs/UPDATE_ROADMAP.md b/docs/UPDATE_ROADMAP.md index 5792bba..b5257a7 100644 --- a/docs/UPDATE_ROADMAP.md +++ b/docs/UPDATE_ROADMAP.md @@ -7,7 +7,7 @@ This document tracks proposed WeatherTool improvements. Work should be delivered - **Windows source workspace:** source editing, review, and Git operations only; do not install dependencies, compile, build, run, or test here. - **Rocky development and staging:** the sole compile, build, development-runtime, and test environment, with production-like Docker staging at `http://192.168.1.101:9190`. - **Git over SSH:** Windows pushes reviewed commits to a private bare repository on Rocky; the Rocky staging checkout pulls those commits and rebuilds. -- **Ubuntu VPS deployment:** release `6185dbf` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia with a limited 14-day synthetic dataset; the VPS does not compile or build the project. +- **Ubuntu VPS deployment:** release `138f57c8` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia, ingesting real LVĢMC open-data station/water-temperature observations on a schedule (synthetic data wiped 2026-08-23); the VPS does not compile or build the project. - **Workplace production:** remains separate until changes are reviewed, tested, and explicitly approved for workplace use. Do not synchronize `.env`, database directories, generated dependencies, build output, or provider credentials between machines. @@ -178,10 +178,16 @@ running in parallel until each real source is proven, not cut over in one step. real/fake data — both shown identically as "older observation" — is a real hazard, not just messiness: removed `dev/seed_weather.sql`, the `seed` Compose service, and the README section describing it. -- [ ] Wipe the synthetic rows already sitting in the `weather` table (a - separate step from stopping generation — the seed already ran once - before this decision) so history going forward is real data only, not - a mix. Cities with no matching open-data station (Valmiera, Cēsis) +- [x] Wipe the synthetic rows already sitting in the VPS `weather` table + (backed up to `/srv/weathertool/backups/pre-real-data-release/` first) + as part of the 2026-08-23 VPS release, so VPS history going forward is + real data only, not a mix — confirmed Valmiera (no matching open-data + station) now shows no data rather than a stale synthetic fallback. + **Rocky's own `weather` table still holds its original synthetic rows + mixed with real ones (449k rows as of 2026-08-23) — not wiped yet, + since Rocky is a development/staging environment rather than the + newsroom-facing UAT domain; revisit if it starts interfering with + testing.** Cities with no matching open-data station (Valmiera, Cēsis) should show no data rather than falling back to old synthetic rows — deferred by design, revisit once a real source is found or accepted as permanently unavailable. @@ -231,20 +237,20 @@ Status: in progress - [x] Provision a dedicated `/srv/weathertool` tree, Compose project, private network, database storage, and loopback-only application and Authelia ports. - [x] Configure Authelia with one temporary shared account, Argon2id password storage, rate limiting, and temporary IP bans. - [x] Add an exact-host Cloudflare origin certificate and Nginx authorization routing without disrupting the existing HOP site. -- [x] Build and verify commit-addressed images on Rocky, transfer and checksum-verify them on the VPS, and update only the application service. Current full-SHA image: `weathertool:6185dbfdb0752db7d715f1061d74baab3c43122e`; PostgreSQL and Authelia were not restarted. +- [x] Build and verify commit-addressed images on Rocky, transfer and checksum-verify them on the VPS, and update only the application service. Current full-SHA image: `weathertool:138f57c808631dc17396b22ac0305b670fbcf776`; PostgreSQL and Authelia were not restarted. - [ ] Complete UAT verification: public authentication and application health pass; direct-origin blocking, logout, throttling, PNG downloads, backups, logs, and rollback remain. - [ ] Share the domain with newsroom testers without a walkthrough first, to observe unassisted intuitiveness, before providing any guidance. - [ ] Run the month-long user test with manual releases and record feedback before any workplace-production decision. ## Known current limitations -- The fixed production PNG bases, browser/social metadata, reviewed frontend dependency updates, stabilized header navigation, authenticated API routing, the full Brīdinājumi warning-map workspace, and hardened LVĢMC warning fetching are deployed in UAT release `6185dbf`; newsroom workflow validation is in progress. +- The fixed production PNG bases, browser/social metadata, reviewed frontend dependency updates, stabilized header navigation, authenticated API routing, the full Brīdinājumi warning-map workspace (including draggable per-warning symbol placement), hardened LVĢMC warning fetching, real open-data station/water-temperature ingestion, and the scheduler safety split are deployed in UAT release `138f57c8`; newsroom workflow validation is in progress. - Link-preview crawlers cannot authenticate through Authelia; the final Nginx policy must deliberately expose only the minimum preview metadata/assets if WhatsApp previews are required. -- Staging uses synthetic PostgreSQL station data. +- The VPS `weather` table now holds only real open-data station observations (synthetic rows wiped 2026-08-23); Rocky's own `weather` table still has its original synthetic rows mixed in with real ones, not yet wiped. - LVGMC forecast CSV fixtures are not yet available. - HARMONIE GRIB fixtures are not yet available. -- Scheduled provider downloads are disabled in development and staging. +- The legacy private LVGMC FTP feed and DMI HARMONIE forecast feed remain gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials, in both development and staging (Rocky and VPS). - Existing automated test coverage is minimal. - Direct refreshes on newer frontend routes can return 404 until the backend gains a general SPA fallback. - Full Docker build context scanning on Rocky can fail on the container-owned `postgres/` bind directory; do not loosen its permissions. @@ -284,11 +290,13 @@ Record completed work here by date and commit after the Git workflow is establis | 2026-08-22 | `6fbf672` | Split Brīdinājumi warning selection (checkbox) from viewing detail (chip click) and widen the detail popup | Yes — Rocky typecheck, build, and headless-browser verification that checkbox clicks change the map selection without opening the popup and chip clicks open the correct warning's detail without changing the selection; deployed to VPS in release `b5150ab` on 2026-08-22 | | 2026-08-22 | `56b8716` | Replace single-warning Brīdinājumi selection with multi-select checkboxes and severity-ordered polygon compositing, and restyle chips as narrower, fully severity-colored, white-icon/white-text buttons with a ported Uiverse.io checkbox component | Yes — Rocky typecheck, build, and headless-browser screenshot verification of multiple simultaneous checked warnings compositing correctly on the map; deployed to VPS in release `b5150ab` on 2026-08-22 | | 2026-08-22 | `6185dbf` | Harden `WarningService` against partial upstream failures: bounded outage retry, per-warning fetch isolation, a live-API-verified CKAN polygon filter, and visible logging for a previously-silent vertex-order default | Yes — Scala tests, Rocky full-stack rebuild verified against real live LVĢMC data (including a genuinely severe 52,345-point wind-warning polygon rendering correctly); deployed to VPS in release `6185dbf` on 2026-08-22, verified against live data on the VPS itself | -| 2026-08-23 | `c97875d` | Add draggable, resizable per-warning weather-symbol placement to Brīdinājumi, reusing Faktiskā's symbol set and image cache | Yes — Rocky typecheck, build, and headless-browser mouse-drag/resize verification against fixture warning data (the live LVĢMC feed had zero active warnings at test time); confirmed the exported PNG excludes the preview-only drag/resize handles; VPS deployment pending | -| 2026-08-23 | `893a09a` | Add `fetch.lvgmc.OpenDataStationService`: real, free, keyless current station observations from LVĢMC's open-data portal, scheduled alongside (not replacing) the private FTP feed | Yes — Scala tests, manual `sbt runMain` fetch+write against the real feed on Rocky staging (1,577 rows, 33 cities), confirmed in `weather` via direct SQL, Faktiskā's 13 fixed stations rendering real temperatures with correct stale-fallback for the one uncovered city (Valmiera), and Kartes' aggregate query API returning sane blended values; VPS deployment pending | -| 2026-08-23 | `cb25316` | Stop generating synthetic weather data (remove `dev/seed_weather.sql`, the `seed` Compose service, and the README section describing it) now that real data is verified and this tool may plausibly be used against live broadcasts | Yes — `docker compose config` valid, running Rocky stack unaffected (profile-gated service, no running containers touched); VPS deployment pending | -| 2026-08-23 | `6ddfe73` | Split scheduled jobs (`ENABLE_SCHEDULED_JOBS` vs. new `ENABLE_LEGACY_PROVIDER_JOBS`) so the working open-data station fetch runs without the FTP/Harmonie jobs' placeholder credentials crash-looping the whole app via `parMapN` | Yes — Scala tests, Rocky rebuild deployed with the flag on: logs confirm only the safe jobs scheduled, FTP/Harmonie never attempted, app stayed up; VPS deployment pending | -| 2026-08-23 | `39fcb3e` | Remove the dead `METEO_*` fetch path (`fetch/csv/FetchService.scala`, its empty test, and its env vars everywhere) — confirmed via git history it's the same underlying LVĢMC data as the FTP/open-data paths, just an earlier, superseded delivery mechanism that was never wired into anything running | Yes — Scala tests (2 suites now, down from 3, the deleted one was empty); VPS deployment pending | -| 2026-08-23 | `4036d24`–`4b92272` | Add `fetch.lvgmc.WaterTemperatureService`: real, free, keyless water temperatures for all 6 Ūdens zones (2 coastal `SEDUT` stations for Jūra/Līcis, 4 inland `WTEMD` stations for Kurzeme/Zemgale/Vidzeme/Latgale), fetch-on-demand with a 15-minute in-memory cache mirroring `WarningService` rather than the persisted station-observation path, since Ūdens has never stored history | Yes — Scala tests, Rocky rebuild, direct API curl confirming real values for all 6 zones; caught and fixed one station-selection error (Daugavpils only reports water level, not temperature — swapped for Ludza) via direct verification before committing; VPS deployment pending | -| 2026-08-23 | `5a92465` | Auto-populate Ūdens's min/max fields from `/api/water-temperatures` on load, mirroring Faktiskā's fetch/override/reset pattern, with per-zone manual-override tracking and an "Atiestatīt" reset button | Yes — Rocky typecheck, build, and headless-browser verification (Playwright in Docker, since `chromium-cli` wasn't available in this environment) confirming auto-populated values match the API, manual edit correctly flips a zone to "Manuāli" and enables its reset button, and reset correctly restores the fetched value; noted along the way that `/udens-temperatura` 404s on a direct hit (no SPA fallback yet, consistent with the known limitation already listed) but works via client-side nav; VPS deployment pending | -| 2026-08-23 | `9eca9eb` | Fix the one-station-per-zone design catching a real problem after deploy: every zone showed the same value duplicated as both min and max ("19...19"). Classified all 65 LVĢMC stations that report water temperature (56 inland WTEMD, 9 coastal SEDUT) into the 6 zones by geography and report the real min/max across each zone's currently-reporting stations, dropping readings over 12h stale so one stuck sensor can't skew a range | Yes — Scala tests, Rocky rebuild, direct API curl confirming a genuine spread per zone (e.g. Vidzeme 11.4–18.7°C, traced the low point to a real fresh reading from a colder headwater station near Cēsis, not a stale-sensor artifact), and headless-browser screenshot confirming the UI renders the real spread (e.g. "11...19") with no console errors; VPS deployment pending | +| 2026-08-23 | `c97875d` | Add draggable, resizable per-warning weather-symbol placement to Brīdinājumi, reusing Faktiskā's symbol set and image cache | Yes — Rocky typecheck, build, and headless-browser mouse-drag/resize verification against fixture warning data (the live LVĢMC feed had zero active warnings at test time); confirmed the exported PNG excludes the preview-only drag/resize handles; deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `893a09a` | Add `fetch.lvgmc.OpenDataStationService`: real, free, keyless current station observations from LVĢMC's open-data portal, scheduled alongside (not replacing) the private FTP feed | Yes — Scala tests, manual `sbt runMain` fetch+write against the real feed on Rocky staging (1,577 rows, 33 cities), confirmed in `weather` via direct SQL, Faktiskā's 13 fixed stations rendering real temperatures with correct stale-fallback for the one uncovered city (Valmiera), and Kartes' aggregate query API returning sane blended values; deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `cb25316` | Stop generating synthetic weather data (remove `dev/seed_weather.sql`, the `seed` Compose service, and the README section describing it) now that real data is verified and this tool may plausibly be used against live broadcasts | Yes — `docker compose config` valid, running Rocky stack unaffected (profile-gated service, no running containers touched); deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `6ddfe73` | Split scheduled jobs (`ENABLE_SCHEDULED_JOBS` vs. new `ENABLE_LEGACY_PROVIDER_JOBS`) so the working open-data station fetch runs without the FTP/Harmonie jobs' placeholder credentials crash-looping the whole app via `parMapN` | Yes — Scala tests, Rocky rebuild deployed with the flag on: logs confirm only the safe jobs scheduled, FTP/Harmonie never attempted, app stayed up; deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `39fcb3e` | Remove the dead `METEO_*` fetch path (`fetch/csv/FetchService.scala`, its empty test, and its env vars everywhere) — confirmed via git history it's the same underlying LVĢMC data as the FTP/open-data paths, just an earlier, superseded delivery mechanism that was never wired into anything running | Yes — Scala tests (2 suites now, down from 3, the deleted one was empty); deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `4036d24`–`4b92272` | Add `fetch.lvgmc.WaterTemperatureService`: real, free, keyless water temperatures for all 6 Ūdens zones (2 coastal `SEDUT` stations for Jūra/Līcis, 4 inland `WTEMD` stations for Kurzeme/Zemgale/Vidzeme/Latgale), fetch-on-demand with a 15-minute in-memory cache mirroring `WarningService` rather than the persisted station-observation path, since Ūdens has never stored history | Yes — Scala tests, Rocky rebuild, direct API curl confirming real values for all 6 zones; caught and fixed one station-selection error (Daugavpils only reports water level, not temperature — swapped for Ludza) via direct verification before committing; deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `5a92465` | Auto-populate Ūdens's min/max fields from `/api/water-temperatures` on load, mirroring Faktiskā's fetch/override/reset pattern, with per-zone manual-override tracking and an "Atiestatīt" reset button | Yes — Rocky typecheck, build, and headless-browser verification (Playwright in Docker, since `chromium-cli` wasn't available in this environment) confirming auto-populated values match the API, manual edit correctly flips a zone to "Manuāli" and enables its reset button, and reset correctly restores the fetched value; noted along the way that `/udens-temperatura` 404s on a direct hit (no SPA fallback yet, consistent with the known limitation already listed) but works via client-side nav; deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `9eca9eb` | Fix the one-station-per-zone design catching a real problem after deploy: every zone showed the same value duplicated as both min and max ("19...19"). Classified all 65 LVĢMC stations that report water temperature (56 inland WTEMD, 9 coastal SEDUT) into the 6 zones by geography and report the real min/max across each zone's currently-reporting stations, dropping readings over 12h stale so one stuck sensor can't skew a range | Yes — Scala tests, Rocky rebuild, direct API curl confirming a genuine spread per zone (e.g. Vidzeme 11.4–18.7°C, traced the low point to a real fresh reading from a colder headwater station near Cēsis, not a stale-sensor artifact), and headless-browser screenshot confirming the UI renders the real spread (e.g. "11...19") with no console errors; deployed to VPS in release `138f57c8` on 2026-08-23 | +| 2026-08-23 | `69a5260` | Bring architecture/workflow docs current with real-data ingestion (removed synthetic-seed staging descriptions, documented Ūdens auto-populate, the scheduler split) | Yes — doc-only change, no rebuild needed | +| 2026-08-23 | `138f57c8` | **Full go-live VPS release**: deployed image `weathertool:138f57c808631dc17396b22ac0305b670fbcf776` (everything from `c97875d` through `69a5260` — Brīdinājumi draggable symbol placement, real open-data station/water-temperature ingestion, synthetic-data removal, scheduler split, `METEO_*` deletion, real per-zone water-temperature ranges), flipped `ENABLE_SCHEDULED_JOBS=true` on the VPS now that the open-data path is proven, and wiped the VPS `weather` table's original 12,784 synthetic rows down to 0 (Rocky's own table was left untouched — separate decision, not requested for this release) | Yes — exact release image smoke-tested on Rocky (healthy, real warnings/water-temperature data) before transfer; VPS PostgreSQL backed up to `/srv/weathertool/backups/pre-real-data-release/` before the wipe; checksum verified on both ends; `app` service recreated without touching PostgreSQL/Authelia; confirmed healthy container, working loopback and public HTTPS (302 unauthenticated page, 401 unauthenticated API — matching established behavior), real warnings/water-temperature API responses, the scheduled open-data job firing on schedule and landing 1,577 real rows within ~2 minutes, and a real (non-Valmiera) city query returning genuine current temperatures with Valmiera correctly absent rather than falling back to wiped synthetic data |