From a22872a328c6ec7d6fd9b5039bc7facbdaae94a2 Mon Sep 17 00:00:00 2001 From: b0txec Date: Sun, 23 Aug 2026 10:09:51 +0300 Subject: [PATCH] Document the real open-data station ingestion path (Phase 1) --- docs/ARCHITECTURE.md | 2 +- docs/UPDATE_ROADMAP.md | 41 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 38 insertions(+), 5 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 29109fe..8afa06a 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -29,7 +29,7 @@ Staging currently exercises the PostgreSQL → API → frontend path with synthe - Serves JSON API routes and the built frontend from `web/dist`. - Queries and aggregates station observations in PostgreSQL. -- Contains ingestion/parsing code for LVGMC station and forecast data. +- Contains ingestion/parsing code for LVGMC station and forecast data, both the private FTP feed (`fetch.lvgmc.FetchService`) and, as of 2026-08-23, a free/keyless open-data alternative (`fetch.lvgmc.OpenDataStationService`) that pivots LVĢMC's public `data.gov.lv` station-observation feed into the same `weather` table — the two run in parallel during evaluation; see `docs/UPDATE_ROADMAP.md` Phase 7. - Fetches LVĢMC open hydrometeorological-warning metadata and polygon coordinates on request, with a 5-minute in-memory cache and stale-data fallback if the upstream call fails. - Contains HARMONIE discovery, download, GRIB parsing, and rendering support. - Starts scheduled ingestion/cleanup tasks unless `ENABLE_SCHEDULED_JOBS=false`. diff --git a/docs/UPDATE_ROADMAP.md b/docs/UPDATE_ROADMAP.md index dca0022..8ac1f20 100644 --- a/docs/UPDATE_ROADMAP.md +++ b/docs/UPDATE_ROADMAP.md @@ -147,12 +147,44 @@ Status: pending ## Phase 7 — Real data and production readiness -Status: pending +Status: in progress + +Phased plan: real station/temperature data first, then Harmonie once DMI +credentials are registered, then remaining sources. Synthetic data is kept +running in parallel until each real source is proven, not cut over in one step. - [ ] Confirm the actual workplace deployment topology and current deployed commit. -- [ ] Obtain authorized development credentials or representative fixtures. -- [ ] Validate LVGMC station and forecast CSV ingestion. -- [ ] Validate DMI STAC discovery and EDR GRIB downloads. +- [x] Find a credential-free real alternative for current station observations: + LVĢMC also publishes `hidrometeorologiskie-noverojumi` on `data.gov.lv` + (the same free, keyless CKAN API already used for Brīdinājumi warnings), + including a 48h rolling "operational" resource with the same fields the + private FTP feed provides (minus `visibilityMin`/`dewPoint`/`sunDuration`, + which aren't published there). +- [x] Add `fetch.lvgmc.OpenDataStationService`, pivoting that feed's tall + `{STATION_ID, ABBREVIATION, DATETIME, VALUE}` rows into the existing + `WeatherData` shape via a verified `STATION_ID → city` map (32 of the + app's 37 tracked cities have a direct station), and schedule it in + `Main.scala` (minutes 15/45) alongside — not replacing — the existing + FTP fetch, writing into the same `weather` table via the existing + `PostgresService.insertInWeatherTable` upsert. +- [x] Verify end to end on Rocky: real rows confirmed in `weather` via direct + SQL, Faktiskā's 13 fixed stations rendering real current temperatures + (Valmiera, the one city with no matching open-data station, correctly + falls back to older synthetic data via the existing stale-observation + UI rather than breaking), and Kartes' aggregate query API returning + sane blended real+synthetic values. +- [ ] Let the open-data fetch run for a trial period, then remove the private + FTP path (`fetch/lvgmc/FetchService.scala`'s `fetchWeatherStations`, + its `LVGMC_*` env vars) and the synthetic seed machinery + (`dev/seed_weather.sql`, the `seed` Compose service) once confident. +- [ ] Register free DMI Open Data credentials and validate DMI STAC discovery + and EDR GRIB downloads (code already exists and is already scheduled; + this is credential registration and verification, not new code — + confirm first whether DMI's endpoint migration away from + `dmigw.govcloud.dk` changes what's needed). +- [ ] Investigate open real sources for water temperature (Ūdens) and the + LVĢMC forecast CSV workspace, following the same open-data-portal + approach used for station observations. - [ ] Test scheduled ingestion failure and recovery behavior. - [ ] Rehearse deployment and rollback using sanitized data. - [ ] Obtain technical and operational review before workplace rollout. @@ -221,3 +253,4 @@ Record completed work here by date and commit after the Git workflow is establis | 2026-08-22 | `56b8716` | Replace single-warning Brīdinājumi selection with multi-select checkboxes and severity-ordered polygon compositing, and restyle chips as narrower, fully severity-colored, white-icon/white-text buttons with a ported Uiverse.io checkbox component | Yes — Rocky typecheck, build, and headless-browser screenshot verification of multiple simultaneous checked warnings compositing correctly on the map; deployed to VPS in release `b5150ab` on 2026-08-22 | | 2026-08-22 | `6185dbf` | Harden `WarningService` against partial upstream failures: bounded outage retry, per-warning fetch isolation, a live-API-verified CKAN polygon filter, and visible logging for a previously-silent vertex-order default | Yes — Scala tests, Rocky full-stack rebuild verified against real live LVĢMC data (including a genuinely severe 52,345-point wind-warning polygon rendering correctly); deployed to VPS in release `6185dbf` on 2026-08-22, verified against live data on the VPS itself | | 2026-08-23 | `c97875d` | Add draggable, resizable per-warning weather-symbol placement to Brīdinājumi, reusing Faktiskā's symbol set and image cache | Yes — Rocky typecheck, build, and headless-browser mouse-drag/resize verification against fixture warning data (the live LVĢMC feed had zero active warnings at test time); confirmed the exported PNG excludes the preview-only drag/resize handles; VPS deployment pending | +| 2026-08-23 | `893a09a` | Add `fetch.lvgmc.OpenDataStationService`: real, free, keyless current station observations from LVĢMC's open-data portal, scheduled alongside (not replacing) the private FTP feed | Yes — Scala tests, manual `sbt runMain` fetch+write against the real feed on Rocky staging (1,577 rows, 33 cities), confirmed in `weather` via direct SQL, Faktiskā's 13 fixed stations rendering real temperatures with correct stale-fallback for the one uncovered city (Valmiera), and Kartes' aggregate query API returning sane blended values; VPS deployment pending |