Catch up documentation on tonight's real-data/timezone work

Changelog rows for the COALESCE upsert fix, both timezone fixes, the
ENABLE_LVGMC_FTP_JOBS/ENABLE_HARMONIE_JOBS split, and the VPS compose
LVGMC templating fix — none had one yet. Updated the stale "currently
deployed" references (were still e446f1ff) to the actual final
release (3eddf95). Recorded the DMI Harmonie research (no API key
needed, new domain, verified live) and the 5-step verification plan
for actually wiring it up, deferred to a dedicated session. One line
about the VPS's first FTP fetch is marked pending — a background
watch is still confirming it as of this commit; will follow up once
it resolves.

docs/UPDATE_ROADMAP.md

@@ -7,7 +7,7 @@ This document tracks proposed WeatherTool improvements. Work should be delivered
 - **Windows source workspace:** source editing, review, and Git operations only; do not install dependencies, compile, build, run, or test here.
 - **Rocky development and staging:** the sole compile, build, development-runtime, and test environment, with production-like Docker staging at `http://192.168.1.101:9190`.
 - **Git over SSH:** Windows pushes reviewed commits to a private bare repository on Rocky; the Rocky staging checkout pulls those commits and rebuilds.
-- **Ubuntu VPS deployment:** release `e446f1ff` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia, ingesting real LVĢMC open-data station/water-temperature observations on a schedule (synthetic data wiped 2026-08-23); the VPS does not compile or build the project.
+- **Ubuntu VPS deployment:** release `3eddf95` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia, ingesting real LVĢMC open-data station/water-temperature observations on a schedule, plus real LVGMC FTP station data as of 2026-08-23 (`ENABLE_LVGMC_FTP_JOBS=true`, real credentials); the `weather` table was wiped and re-populated fresh once the UTC/local timezone mismatch between the two sources was fixed. The VPS does not compile or build the project.
 - **Workplace production:** remains separate until changes are reviewed, tested, and explicitly approved for workplace use.
 
 Do not synchronize `.env`, database directories, generated dependencies, build output, or provider credentials between machines.
@@ -213,10 +213,7 @@ running in parallel until each real source is proven, not cut over in one step.
       different timelines and a single flag would have enabled Harmonie
       (still placeholder) the moment FTP's were ready — caught this before
       it could crash-loop the app (the Grib job was ~15 min from its first
-      scheduled run when this was noticed). FTP enabled on Rocky; Harmonie
-      stays off pending real DMI credentials, still the user's own
-      follow-up. Worth confirming DMI's endpoint migration away from
-      `dmigw.govcloud.dk` before wiring anything.
+      scheduled run when this was noticed). FTP enabled on Rocky.
       Also found and fixed a real UTC-vs-local timezone mismatch while
       verifying FTP alongside open-data: both write into `weather.dateTime`
       with no conversion, but the open-data portal's DATETIME is UTC while
@@ -231,6 +228,79 @@ running in parallel until each real source is proven, not cut over in one step.
       always carry null visibilityMin/dewPoint/sunDuration and an empty
       phenomena array — a blind overwrite would silently erase FTP's real
       values for those fields whenever open-data's write landed later.
+- [x] Went live on both Rocky and the VPS with all of the above: backed up
+      and wiped both `weather` tables (Rocky: `/tmp` scratch backup this
+      session; VPS: `/srv/weathertool/backups/pre-timezone-fix-wipe/`) now
+      that the timezone bug is fixed, so history going forward is correctly
+      and consistently timestamped rather than mixing the old UTC-mislabeled
+      open-data rows with correctly-local FTP rows. Fixed a second real bug
+      found only when wiring up the VPS specifically: `deploy/vps/compose.yml`
+      hardcoded `LVGMC_USER`/`PASSWORD`/`URL` to inert placeholder strings
+      directly in the file (unlike `POSTGRES_*`, which already read from
+      `.env.staging`) — real credentials in `.env.staging` alone would have
+      had no effect until this was switched to the same `${VAR}` substitution
+      pattern. VPS now runs release `3eddf95`/`dc04f66` with real FTP
+      credentials (added directly on the VPS by the user, never typed into
+      this session) and `ENABLE_LVGMC_FTP_JOBS=true`; watching the container
+      logs for VPS's first scheduled FTP fetch specifically (not just the
+      open-data job, which fires more often and was initially mistaken for
+      it on one premature check) to confirm real rows land before leaving it
+      unattended overnight — outbound FTP from the VPS network is a real,
+      non-hypothetical risk worth checking, not something safe to assume
+      just because it worked from Rocky. **Result pending as of this
+      writing — update this line once confirmed.**
+- [x] Researched DMI HARMONIE credentials while waiting on the above and
+      confirmed live (not just from search results, which claimed a specific
+      date that wasn't independently verified): DMI's Forecast Data EDR and
+      STAC APIs no longer validate an API key at all — a direct request to
+      both the old `dmigw.govcloud.dk` and the new `opendataapi.dmi.dk` with
+      *no* `api-key` parameter returned real GRIB/STAC data from both. New
+      collection-items STAC path confirmed as
+      `https://opendataapi.dmi.dk/v1/forecastdata/collections/harmonie_dini_sf/items`,
+      with the same `properties.modelRun`/`properties.datetime` schema
+      `fetch.dmi.FetchService` already expects — no response-shape changes
+      needed, just the domain and dropping real key handling (the app's own
+      config loader still needs *some* non-empty string for
+      `HARMONIE_EDR_API_KEY`/`HARMONIE_STAC_API_KEY`, but DMI ignores it).
+      Not wired up yet — see "Harmonie verification (planned)" below.
+- [ ] **Harmonie verification (planned, next session)** — the user confirmed
+      this data is used daily today (exported as a numbered PNG frame
+      sequence per forecast hour, zipped, imported into After Effects for
+      the on-air animated forecast loop — confirmed directly in
+      `SlideShow.tsx`'s `downloadImagesAsZip`), so getting it working
+      correctly matters, but the code has real, untested risk after a long
+      dormant period (no HARMONIE fixtures, no configured provider access
+      in staging, ever, per the existing product-workflow notes). Real data
+      access itself is solved (see above); what's unverified is everything
+      downstream. Plan, in order:
+      1. Fetch one real current GRIB file and confirm `parse.grib.GribParser`
+         decodes it cleanly with sane values — the low-level GRIB2 parsing
+         reads discipline/category/product generically per the spec, so
+         likely still fine, but not yet checked against a real current file.
+      2. Verify each frontend field renderer (`draw/temperature.ts`,
+         `windDirection.ts`, `precipitation.ts`, `snowDepth.ts`) against real
+         data via screenshot — check in particular whether "Crop Latvia"
+         (`interfaces.ts`'s `CROP_BOUNDS = { x: 1906-1-660, y: 840, width:
+         660, height: 620, angle: 26 }`) still aligns correctly. This is a
+         hardcoded pixel/grid offset into HARMONIE's specific grid
+         domain/rotation — there's already one commented-out earlier version
+         of this constant, meaning it's been hand-recalibrated at least once
+         before, so it's a real, not hypothetical, risk if DMI's grid
+         definition has shifted at all since.
+      3. Verify the frame-export ZIP workflow end-to-end with real
+         multi-hour data — frame count, naming, timing — against the user's
+         own older HARMONIE exports and existing After Effects projects,
+         which they have on hand for direct comparison.
+      4. Only then: update `HARMONIE_EDR_URL`/`HARMONIE_STAC_URL` to
+         `opendataapi.dmi.dk` (see above), set
+         `HARMONIE_EDR_API_KEY`/`HARMONIE_STAC_API_KEY` to any non-empty
+         placeholder (DMI no longer validates them), flip
+         `ENABLE_HARMONIE_JOBS=true`.
+      5. Lower priority, later: the Harmonie workspace is still raw/English
+         dev-tool UI ("Crop Latvia", "Contour", "Interpolate" checkboxes,
+         unstyled file list) unlike the polished, Latvian, newly-redesigned
+         Faktiskā/Ūdens/Brīdinājumi workspaces — worth the same treatment
+         once the data/rendering itself is trusted.
 - [x] Add real water temperature data for Ūdens: `fetch.lvgmc.WaterTemperatureService`
       fetches LVĢMC's open hydrological data (`data.gov.lv`, same free CKAN
       API), mapping one representative real station per named zone (coastal
@@ -326,3 +396,8 @@ Record completed work here by date and commit after the Git workflow is establis
 | 2026-08-23 | `f78b0a26` | Deploy the full frontend design pass (background/accent rework, Faktiskā decluttering, temperature-badge centering fix, header nav) to the VPS — no database or scheduler changes, application-only release | Yes — exact release image smoke-tested on Rocky (bundle hash and headless-browser screenshot confirmed against the known-good local build) before transfer; checksum verified on both ends; `app` service recreated without touching PostgreSQL/Authelia; confirmed healthy container, matching bundle hash on the VPS, working loopback and public HTTPS (302 unauthenticated page, 401 unauthenticated API), and real warnings/water-temperature API responses |
 | 2026-08-23 | `d7439a2b` | Fresh-eyes review of the whole design pass turned up two real issues, fixed here: `WindInputs` had no Latvian branching and was showing "Wind direction"/"Wind speed"/"Gusts" in English right under the freshly-translated "Temperatūra un vējš" heading on Faktiskā (threaded the same `productionTemplate` flag `MapView.tsx` already uses elsewhere), and a confirmed-unreferenced `.assignedSymbol` CSS rule was removed. The review also flagged two things left alone: a pre-existing (not introduced this session) `.symbolPalette`/`.selectedPreview`/`.currentSymbol` cross-file class collision in `weatherIcons.css` currently masked by `!important` rather than actually resolved, and a naming nit on "Mākoņi pilsētām" (the section assigns arbitrary weather symbols, not just clouds) — both worth a look later, not blocking | Yes — Rocky typecheck, build, and headless-browser screenshot confirming the wind fields now render in Latvian; deployed to VPS in release `d7439a2b`, exact release image smoke-tested (bundle hash matched) before transfer, checksum verified, container healthy, public HTTPS/API checks passing |
 | 2026-08-23 | `e446f1f` | Consolidate `.symbolPalette`/`.selectedPreview`/`.currentSymbol` (used by both Faktiskā's `IconInputs.tsx` and Brīdinājumi's `Warnings.tsx`) into one real definition in `weatherIcons.css`, the file both actually import, removing the dead/duplicate versions in `mapGraphics.css` and the `!important` size patches that were masking the collision. "Mākoņi pilsētām" naming confirmed intentional — the newsroom users are non-technical and the same people who'll use this long-term, so intuitive-but-imprecise beats literally-correct-but-jargony | Yes — Rocky typecheck, build, headless-browser screenshot on Faktiskā (pixel-identical to before), and computed-style verification on both Faktiskā and Brīdinājumi confirming identical resolved CSS with no `!important`; caught one incidental fix for free (`.currentSymbol.empty`'s Inter font was silently losing to the `!important` rule, normal cascade now applies it correctly); deployed to VPS in release `e446f1ff`, exact release image smoke-tested (bundle hash matched) before transfer, checksum verified, container healthy, public HTTPS/API checks passing |
+| 2026-08-23 | `2b5dff6` | Make the `weather` table upsert non-destructive: `ON CONFLICT DO UPDATE` was a blind full-row overwrite, so open-data's always-null `visibilityMin`/`dewPoint`/`sunDuration` and always-empty `phenomena` would silently erase real FTP values whenever open-data's write landed later for the same row. Switched to `COALESCE(excluded.field, weather.field)`, with `NULLIF` against an empty array specifically for `phenomena` since Scala's `List[String]` never maps to SQL `NULL` | Yes — Scala tests; found while wiring up real FTP credentials, before any real dual-source writes had happened |
+| 2026-08-23 | `cab94d0` | Fix a real UTC-vs-local timezone mismatch: the open-data portal's `DATETIME` is UTC, stored into `weather.dateTime` with no conversion, while FTP's "Laiks" column is already local and also stored as-is — silently present since open-data went live (`893a09a`), invisible until a second, correctly-labeled source existed alongside it. Fixed by converting to `Europe/Riga` at ingestion in `OpenDataStationService` | Yes — Scala tests; verified live: at real local time 20:31 EEST the fix produces `dateTime=20:00` instead of the previous `17:00`, matching what FTP writes for the same real hour |
+| 2026-08-23 | `3315f00` | Fix the same UTC-vs-local mismatch in Ūdens's `observedAt` display (`WaterTemperatureService`) — same open-data portal, same root cause, display-only (internal recency filtering was already self-consistent either way) | Yes — Scala tests; verified live via `/api/water-temperatures`: `observedAt` now matches real local time |
+| 2026-08-23 | `3eddf95` | Split `ENABLE_LEGACY_PROVIDER_JOBS` into independent `ENABLE_LVGMC_FTP_JOBS`/`ENABLE_HARMONIE_JOBS` — real LVGMC FTP credentials arrived today, real DMI HARMONIE credentials haven't, and the combined flag would have enabled both together, crash-looping the app on HARMONIE's still-placeholder values via `parMapN` | Yes — Scala tests; caught before it could happen (the Grib job was ~15 min from its first scheduled run when noticed) and reverted within under a minute; re-verified after the fix that only "Fetch Weather Stations" scheduled, not "Fetch Grib" |
+| 2026-08-23 | `dc04f66` | Fix `deploy/vps/compose.yml` hardcoding `LVGMC_USER`/`PASSWORD`/`URL` to inert placeholder strings directly in the file (unlike `POSTGRES_*`, which already read from `.env.staging`) — real credentials added to `.env.staging` alone would have had no effect until this switched to the same `${VAR}` substitution pattern. Enabled `ENABLE_LVGMC_FTP_JOBS` on the VPS | Yes — `docker compose config` syntax valid; VPS `app` container came up healthy with real credentials loaded (would have crashed immediately on missing-config if not) |