# WeatherTool update roadmap

This document tracks proposed WeatherTool improvements. Work should be delivered in small, reviewable phases rather than as one large rewrite. Each phase should leave the application runnable and independently testable.

## Environments and workflow

- **Windows source workspace:** source editing, review, and Git operations only; do not install dependencies, compile, build, run, or test here.
- **Rocky development and staging:** the sole compile, build, development-runtime, and test environment, with production-like Docker staging at `http://192.168.1.101:9190`.
- **Git over SSH:** Windows pushes reviewed commits to a private bare repository on Rocky; the Rocky staging checkout pulls those commits and rebuilds.
- **Ubuntu VPS deployment:** release `6185dbf` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia with a limited 14-day synthetic dataset; the VPS does not compile or build the project.
- **Workplace production:** remains separate until changes are reviewed, tested, and explicitly approved for workplace use.

Do not synchronize `.env`, database directories, generated dependencies, build output, or provider credentials between machines.

## Working principles

1. Make one coherent change at a time.
2. Record the existing behavior before intentionally changing it.
3. Keep dependency updates separate from UI redesign and functional changes.
4. Review and commit on Windows, then deploy and test the same commit on Rocky staging.
5. Use synthetic or sanitized data outside the workplace environment.
6. Never enable external provider schedules with placeholder credentials.
7. Do not connect staging to workplace services without explicit authorization.

## Phase 0 — Reproducible development baseline

Status: in progress

- [x] Review backend, frontend, deployment, and security structure.
- [x] Run the project locally through Docker Desktop.
- [x] Add configurable host port and scheduled-job switch.
- [x] Add deterministic synthetic station data for the repository station set and the Faktiskā-required Valmiera position.
- [x] Create an isolated Rocky Linux staging deployment.
- [x] Keep staging PostgreSQL private to its Compose network.
- [x] Commit the baseline changes and establish the Git-over-SSH workflow.
- [x] Document normal build, seed, frontend deploy, source rollback, and current operational limitations. (Database backup/restore remains pending.)
- [ ] Capture representative screenshots and expected API responses.

## Phase 1 — Behavior discovery and bug inventory

Status: in progress

- [ ] Walk through every page with synthetic data.
- [x] Document the current understood purpose of each workflow; validate it with workplace users over time.
- [x] Separate analytical dashboard features from broadcast-graphic authoring tools on the city-results map.
- [ ] Record unclear controls, missing units, broken states, and layout problems.
- [ ] Fix the `atmPressire` frontend field typo.
- [ ] Fix shifted Database export columns caused by duplicated `tempMax`.
- [ ] Correct malformed integer route handling.
- [ ] Add consistent loading, empty, and error states. (City selection empty state completed.)

## Phase 2 — Test safety net

Status: in progress

- [ ] Add backend route tests for representative station and country queries.
- [ ] Add database integration tests for aggregation and export behavior.
- [ ] Restore and expand CSV parser tests.
- [ ] Add GRIB parser boundary and malformed-file tests.
- [ ] Add security tests for invalid fields, filenames, offsets, lengths, and date ranges.
- [ ] Add frontend type checking and critical workflow smoke tests.
- [ ] Run tests automatically before staging deployment.

## Phase 3 — Dependency modernization

Status: in progress — frontend maintenance complete; backend maintenance pending

The first observed frontend install reported 15 vulnerabilities: 1 critical, 10 high, 3 moderate, and 1 low. The frontend dependency tree was reviewed in controlled groups, obsolete packages were removed, and both the complete and production-only npm audits now report zero known vulnerabilities at commit `de6f279`.

- [x] Capture and review the full npm audit report.
- [x] Establish that the production dependency audit is primarily blocked by `solid-js@1.9.4` resolving vulnerable `seroval@1.2.0`; confirm that the full audit also contains development-tool advisories.
- [x] Update direct frontend dependencies in controlled groups.
- [x] Replace or remove obsolete frontend packages where appropriate.
- [x] Add a committed TypeScript typecheck command and validate the updated frontend with a clean `npm ci`, typecheck, and production build.
- [x] Resolve the remaining transitive build-tool advisories after reviewing the proposed `npm audit fix` changes; verify full and production-only audits at zero.
- [x] Build, smoke-test, checksum, transfer, and deploy a commit-addressed VPS release containing `de6f279` or later without restarting PostgreSQL or Authelia. (`ef64895` deployed 2026-08-22.)
- [ ] Visually compare every page and representative exported PNG in the deployed dependency-maintenance release.
- [ ] Update Scala within the supported 2.13 line before considering larger migration.
- [ ] Update http4s, Doobie, Circe, Cats Effect, Logback, and test libraries in compatible groups.
- [ ] Replace release-candidate dependencies with stable releases where possible.
- [ ] Update Docker base images deliberately and pin reproducible versions.
- [ ] Verify database compatibility and generated artifacts after every group.

Dependency changes must not be combined with a visual redesign unless a package migration strictly requires it.

## Phase 4 — Security hardening

Status: pending

- [ ] Rotate and remove the API key exposed in a source comment (`src/main/scala/fetch/dmi/FetchService.scala:56`, a DMI HARMONIE key introduced 2025-02-01; confirmed the repo's GitHub `origin` is public, so this has been externally exposed for roughly 18 months — decision on rotation vs. a git-history rewrite is still open).
- [ ] Remove credentials from connection-error messages.
- [ ] Protect or remove debug and administrative endpoints.
- [ ] Convert state-changing `GET` routes to appropriate methods.
- [ ] Introduce closed, validated weather-field and aggregation types.
- [ ] Eliminate raw user-controlled SQL identifiers.
- [ ] Validate and constrain filenames, resolved paths, offsets, and byte lengths.
- [ ] Add query-range, response-size, request-rate, and timeout limits.
- [ ] Restrict CORS to intended origins.
- [ ] Define authentication and authorization requirements for workplace deployment.

## Phase 5 — Runtime reliability and operations

Status: pending

- [ ] Manage custom executors as resources and close them cleanly.
- [ ] Remove explicit `System.gc()` calls.
- [ ] Supervise scheduled jobs independently instead of recursively restarting the application.
- [ ] Add application health and readiness endpoints.
- [ ] Add structured logging without leaking secrets.
- [ ] Define PostgreSQL and GRIB-data backup/restore procedures.
- [ ] Add container resource limits and deployment health checks.
- [ ] Document monitoring, update, rollback, and incident procedures.
- [x] Harden `WarningService`: bound LVĢMC retry attempts to once per cache window during an upstream outage, isolate one warning's polygon-fetch failure from the rest instead of failing the whole response, verify the CKAN polygon filter against real column-type ambiguity rather than guessing, and log (rather than silently swallow) a defaulted vertex-order value.

## Phase 6 — Information architecture and UI redesign

Status: pending

- [ ] Identify primary user roles and their most frequent tasks.
- [ ] Separate historical analysis, live station monitoring, database inspection, HARMONIE visualization, and broadcast graphics.
- [ ] Replace technical/internal labels with task-oriented language.
- [x] Replace the copy/paste weather-character workflow with a direct visual font-glyph picker, bulk assignment, and city exceptions.
- [x] Explain and visually separate manual wind and weather-icon inputs from queried data.
- [ ] Add units, legends, contextual help, and clear date semantics.
- [x] Establish the first responsive layout, typography, spacing, and component-system foundation.
- [x] Add an explicit PNG download workflow for broadcast map assets.
- [x] Implement the fixed Faktiskā station set, latest-temperature loading, manual overrides, and locked 3840×1440 export.
- [x] Make Faktiskā weather-symbol selection manual while keeping placement automatic and badge-relative.
- [x] Replace Daira font glyphs with normalized 256×256 transparent image assets across the editor and canvas renderers.
- [x] Add a separate Latvian-named Ūdens workspace with fixed manual fields.
- [x] Calibrate and visually validate independent Ūdens exports at 1920×1080 and 3840×1440 against the supplied production templates.
- [x] Bundle the confirmed Monda Regular/Bold files and replace temporary Rubik rendering.
- [ ] Perform pixel-level Monda comparison against the authoritative production masters.
- [x] Add a separate Latvian-named Brīdinājumi workspace that renders current LVĢMC warning polygons without manual tracing.
- [x] Add production border overlay artwork and feathered warning-polygon fills to Brīdinājumi.
- [x] Replace Brīdinājumi's 4-corner bounding-box projection with an affine fit calibrated against the same validated Kartes/Faktiskā city pixel positions.
- [x] Limit the Brīdinājumi canvas to one operator-selected warning (superseded below by multi-warning selection), remove the on-canvas legend, collapse card detail to phenomenon/severity by default with full detail on click, and pre-fill the title from the selected warning's phenomenon.
- [x] Redesign Brīdinājumi warning cards as compact phenomenon-icon chips and move full warning text into a dialog popup over a dimmed, blurred backdrop.
- [x] Give Brīdinājumi chips a hover state where the severity-colored icon capsule expands to fill the chip, adapted from a reviewed external Uiverse.io reference component.
- [x] Split Brīdinājumi's warning selection (checkbox, drives the map) from viewing a warning's detail (clicking the chip, opens the popup) so choosing a map warning no longer forces the popup open; widen the detail popup for readability.
- [x] Replace single-warning selection with multi-select checkboxes so combined-severity newsroom maps (e.g. yellow + orange wind together) can be produced; re-add severity-ordered compositing of every checked warning's polygons. Restyle chips as narrower, fully severity-colored, white-icon/white-text buttons, and port a reviewed Uiverse.io checkbox component for the selection control.
- [x] Add per-warning editorial weather-symbol placement to Brīdinājumi, reusing Faktiskā's existing symbol set: drag to move, drag a corner handle to resize, stored as canvas-fraction coordinates so placement holds across both export resolutions; the drag/resize affordance is preview-only and never reaches the exported PNG. This is the app's first direct-manipulation canvas control (everywhere else is plain form inputs).
- [ ] Decide whether Brīdinājumi should actively prevent checking warnings of different phenomena together, or continue leaving that as an operator responsibility.
- [ ] Tune Brīdinājumi feather-blur bleed at sharp coastline curves (e.g. the Gulf of Rīga indentation).
- [ ] Test target resolutions and real workplace display conditions.
- [ ] Check keyboard navigation, contrast, focus states, and screen-reader labeling.

## Phase 7 — Real data and production readiness

Status: in progress

Phased plan: real station/temperature data first, then Harmonie once DMI
credentials are registered, then remaining sources. Synthetic data is kept
running in parallel until each real source is proven, not cut over in one step.

- [ ] Confirm the actual workplace deployment topology and current deployed commit.
- [x] Find a credential-free real alternative for current station observations:
      LVĢMC also publishes `hidrometeorologiskie-noverojumi` on `data.gov.lv`
      (the same free, keyless CKAN API already used for Brīdinājumi warnings),
      including a 48h rolling "operational" resource with the same fields the
      private FTP feed provides (minus `visibilityMin`/`dewPoint`/`sunDuration`,
      which aren't published there).
- [x] Add `fetch.lvgmc.OpenDataStationService`, pivoting that feed's tall
      `{STATION_ID, ABBREVIATION, DATETIME, VALUE}` rows into the existing
      `WeatherData` shape via a verified `STATION_ID → city` map (32 of the
      app's 37 tracked cities have a direct station), and schedule it in
      `Main.scala` (minutes 15/45) alongside — not replacing — the existing
      FTP fetch, writing into the same `weather` table via the existing
      `PostgresService.insertInWeatherTable` upsert.
- [x] Verify end to end on Rocky: real rows confirmed in `weather` via direct
      SQL, Faktiskā's 13 fixed stations rendering real current temperatures
      (Valmiera, the one city with no matching open-data station, correctly
      falls back to older synthetic data via the existing stale-observation
      UI rather than breaking), and Kartes' aggregate query API returning
      sane blended real+synthetic values.
- [x] Stop synthetic data generation now that real data is verified and this
      tool may plausibly be used against live broadcasts, where mixed
      real/fake data — both shown identically as "older observation" — is a
      real hazard, not just messiness: removed `dev/seed_weather.sql`, the
      `seed` Compose service, and the README section describing it.
- [ ] Wipe the synthetic rows already sitting in the `weather` table (a
      separate step from stopping generation — the seed already ran once
      before this decision) so history going forward is real data only, not
      a mix. Cities with no matching open-data station (Valmiera, Cēsis)
      should show no data rather than falling back to old synthetic rows —
      deferred by design, revisit once a real source is found or accepted as
      permanently unavailable.
- [ ] Once the open-data fetch has run for a trial period, remove the
      private FTP path (`fetch/lvgmc/FetchService.scala`'s
      `fetchWeatherStations`, its `LVGMC_*` env vars) if it remains unused.
- [ ] Register free DMI Open Data credentials and validate DMI STAC discovery
      and EDR GRIB downloads (code already exists and is already scheduled;
      this is credential registration and verification, not new code —
      confirm first whether DMI's endpoint migration away from
      `dmigw.govcloud.dk` changes what's needed).
- [ ] Investigate open real sources for water temperature (Ūdens) and the
      LVĢMC forecast CSV workspace, following the same open-data-portal
      approach used for station observations.
- [ ] Test scheduled ingestion failure and recovery behavior.
- [ ] Rehearse deployment and rollback using sanitized data.
- [ ] Obtain technical and operational review before workplace rollout.

## Phase 8 — Temporary VPS user acceptance

Status: in progress

- [x] Document the proposed isolated VPS topology and trusted Rocky-to-VPS release flow.
- [x] Choose `laikapstak.li` and `auth.laikapstak.li`, create their proxied Cloudflare DNS records, and activate Cloudflare delegation.
- [x] Record a read-only VPS inventory before provisioning and confirm ports, networks, storage, and capacity do not collide with existing services.
- [x] Provision a dedicated `/srv/weathertool` tree, Compose project, private network, database storage, and loopback-only application and Authelia ports.
- [x] Configure Authelia with one temporary shared account, Argon2id password storage, rate limiting, and temporary IP bans.
- [x] Add an exact-host Cloudflare origin certificate and Nginx authorization routing without disrupting the existing HOP site.
- [x] Build and verify commit-addressed images on Rocky, transfer and checksum-verify them on the VPS, and update only the application service. Current full-SHA image: `weathertool:6185dbfdb0752db7d715f1061d74baab3c43122e`; PostgreSQL and Authelia were not restarted.
- [ ] Complete UAT verification: public authentication and application health pass; direct-origin blocking, logout, throttling, PNG downloads, backups, logs, and rollback remain.
- [ ] Share the domain with newsroom testers without a walkthrough first, to observe unassisted intuitiveness, before providing any guidance.
- [ ] Run the month-long user test with manual releases and record feedback before any workplace-production decision.

## Known current limitations

- The fixed production PNG bases, browser/social metadata, reviewed frontend dependency updates, stabilized header navigation, authenticated API routing, the full Brīdinājumi warning-map workspace, and hardened LVĢMC warning fetching are deployed in UAT release `6185dbf`; newsroom workflow validation is in progress.
- Link-preview crawlers cannot authenticate through Authelia; the final Nginx policy must deliberately expose only the minimum preview metadata/assets if WhatsApp previews are required.

- Staging uses synthetic PostgreSQL station data.
- LVGMC forecast CSV fixtures are not yet available.
- HARMONIE GRIB fixtures are not yet available.
- Scheduled provider downloads are disabled in development and staging.
- Existing automated test coverage is minimal.
- Direct refreshes on newer frontend routes can return 404 until the backend gains a general SPA fallback.
- Full Docker build context scanning on Rocky can fail on the container-owned `postgres/` bind directory; do not loosen its permissions.

## Change log

Record completed work here by date and commit after the Git workflow is established.

| Date | Commit | Summary | Verified on Rocky |
|---|---|---|---|
| 2026-08-18 | `b1fff67` | Docker development baseline, isolated staging, scheduler switch, and synthetic station data | Yes |
| 2026-08-18 | `bb1d9fc` | Home/UI foundation, map preview layout, friendly empty state, and PNG export | Yes |
| 2026-08-18 | `67decb3` | Adaptive temperature badges and title/source overlays | Yes |
| 2026-08-18 | `82199b8` | Separate City Analysis and Faktiskā workflows with direct symbol assignment | Yes |
| 2026-08-18 | `0c78d3c` | Optical centering for bundled weather-font glyphs | Pending Rocky verification |
| 2026-08-19 | `e726291` | Fixed Faktiskā workflow with latest observations, manual overrides, and production-size export | Yes |
| 2026-08-19 | `5913217` | Add Valmiera to the deterministic Faktiskā development data | Yes |
| 2026-08-19 | `632f377` | Arrange manual Faktiskā wind controls horizontally | Yes |
| 2026-08-19 | `340c370`–`3aba9b4` | Anchor and visually calibrate manually selected Faktiskā symbols | Yes |
| 2026-08-19 | `1868041`–`6c9290b` | Add Ūdens temperatūra, bundle and load Monda, fix overlay scaling, and center its locked nameplate | Yes |
| 2026-08-19 | `0d641bd` | Replace runtime Daira glyph rendering with normalized transparent image assets | Yes |
| 2026-08-20 | `754415a` | Match editable title/source overlays to production-safe right margins | Yes |
| 2026-08-20 | `16a8c68` | Deploy approved production map bases, normalized branding assets, favicon, Apple icon, and social metadata to public UAT | Yes — Rocky build and VPS/browser smoke checks |
| 2026-08-21 | `5007517`–`de6f279` | Update Solid runtime and frontend build tooling, add type checking, remove obsolete packages, prune the lockfile, and resolve all npm advisories | Yes — clean `npm ci`, typecheck, production build, full audit, and production-only audit |
| 2026-08-22 | `ef64895` | Deploy the reviewed zero-advisory frontend dependency state as an immutable full-SHA VPS release while leaving PostgreSQL, Authelia, and HOP uninterrupted | Yes — isolated Rocky smoke test, dual-host checksum, matching image ID, container health, loopback, authentication-gate, and HOP checks |
| 2026-08-22 | `36d094e` | Stabilize the header, keep the primary workspaces visible, and move secondary workspaces into an icon-labelled menu | Yes — Rocky browser and Jam navigation checks |
| 2026-08-22 | `df911f1` | Preserve API 401 responses through Authelia/Nginx so frontend queries do not follow cross-origin login redirects | Yes — Rocky and VPS health checks; authenticated browser query verified after refreshing synthetic data |
| 2026-08-22 | `f3197bf` | Return query API payloads with an explicit JSON content type | Yes — Rocky API and browser verification; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `0608865` | Localize the operator-facing workspace names and concise workflow copy in Latvian | Yes — Rocky browser verification; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `7f09181` | Independently calibrate both Ūdens templates and optically center Monda values using visible glyph bounds | Yes — both native-resolution PNG exports visually validated on Rocky; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `bc5bee1` | Add the Brīdinājumi LVĢMC warning-map workflow with metadata/polygon ingestion, day/phenomenon filtering, and manual title entry | Yes — Rocky browser verification; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `13115f0` | Add production border overlay artwork and feathered warning-polygon fills to Brīdinājumi | Yes — Rocky browser verification; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `4395935` | Replace Brīdinājumi's 4-corner bounding-box projection, which drifted up to ~200px on the 3840 canvas, with an affine fit calibrated against the validated Kartes/Faktiskā city pixel positions | Yes — Rocky typecheck, build, and headless-browser screenshot comparison at both native export resolutions; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `924e410` | Limit Brīdinājumi to one selected warning per export, remove the on-canvas legend, collapse card detail until clicked, and pre-fill the title from the selected warning's phenomenon | Yes — Rocky typecheck, build, and headless-browser screenshot verification of selection switching and card detail; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `cafc75a` | Redesign Brīdinājumi warning cards as compact phenomenon-icon chips and move full warning text into a dialog popup over a dimmed, blurred backdrop | Yes — Rocky typecheck, build, and headless-browser screenshot verification of the popup open/close flow; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `70fd24c` | Give Brīdinājumi chips a hover-expanding severity capsule, adapted from a reviewed Uiverse.io reference component | Yes — Rocky typecheck, build, and headless-browser screenshot verification of rest/hover/active chip states; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `6fbf672` | Split Brīdinājumi warning selection (checkbox) from viewing detail (chip click) and widen the detail popup | Yes — Rocky typecheck, build, and headless-browser verification that checkbox clicks change the map selection without opening the popup and chip clicks open the correct warning's detail without changing the selection; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `56b8716` | Replace single-warning Brīdinājumi selection with multi-select checkboxes and severity-ordered polygon compositing, and restyle chips as narrower, fully severity-colored, white-icon/white-text buttons with a ported Uiverse.io checkbox component | Yes — Rocky typecheck, build, and headless-browser screenshot verification of multiple simultaneous checked warnings compositing correctly on the map; deployed to VPS in release `b5150ab` on 2026-08-22 |
| 2026-08-22 | `6185dbf` | Harden `WarningService` against partial upstream failures: bounded outage retry, per-warning fetch isolation, a live-API-verified CKAN polygon filter, and visible logging for a previously-silent vertex-order default | Yes — Scala tests, Rocky full-stack rebuild verified against real live LVĢMC data (including a genuinely severe 52,345-point wind-warning polygon rendering correctly); deployed to VPS in release `6185dbf` on 2026-08-22, verified against live data on the VPS itself |
| 2026-08-23 | `c97875d` | Add draggable, resizable per-warning weather-symbol placement to Brīdinājumi, reusing Faktiskā's symbol set and image cache | Yes — Rocky typecheck, build, and headless-browser mouse-drag/resize verification against fixture warning data (the live LVĢMC feed had zero active warnings at test time); confirmed the exported PNG excludes the preview-only drag/resize handles; VPS deployment pending |
| 2026-08-23 | `893a09a` | Add `fetch.lvgmc.OpenDataStationService`: real, free, keyless current station observations from LVĢMC's open-data portal, scheduled alongside (not replacing) the private FTP feed | Yes — Scala tests, manual `sbt runMain` fetch+write against the real feed on Rocky staging (1,577 rows, 33 cities), confirmed in `weather` via direct SQL, Faktiskā's 13 fixed stations rendering real temperatures with correct stale-fallback for the one uncovered city (Valmiera), and Kartes' aggregate query API returning sane blended values; VPS deployment pending |
| 2026-08-23 | `cb25316` | Stop generating synthetic weather data (remove `dev/seed_weather.sql`, the `seed` Compose service, and the README section describing it) now that real data is verified and this tool may plausibly be used against live broadcasts | Yes — `docker compose config` valid, running Rocky stack unaffected (profile-gated service, no running containers touched); VPS deployment pending |