Document the real open-data station ingestion path (Phase 1)
This commit is contained in:
b0txec
@@ -29,7 +29,7 @@ Staging currently exercises the PostgreSQL → API → frontend path with synthe
|
||||
|
||||
- Serves JSON API routes and the built frontend from `web/dist`.
|
||||
- Queries and aggregates station observations in PostgreSQL.
|
||||
- Contains ingestion/parsing code for LVGMC station and forecast data.
|
||||
- Contains ingestion/parsing code for LVGMC station and forecast data, both the private FTP feed (`fetch.lvgmc.FetchService`) and, as of 2026-08-23, a free/keyless open-data alternative (`fetch.lvgmc.OpenDataStationService`) that pivots LVĢMC's public `data.gov.lv` station-observation feed into the same `weather` table — the two run in parallel during evaluation; see `docs/UPDATE_ROADMAP.md` Phase 7.
|
||||
- Fetches LVĢMC open hydrometeorological-warning metadata and polygon coordinates on request, with a 5-minute in-memory cache and stale-data fallback if the upstream call fails.
|
||||
- Contains HARMONIE discovery, download, GRIB parsing, and rendering support.
|
||||
- Starts scheduled ingestion/cleanup tasks unless `ENABLE_SCHEDULED_JOBS=false`.
|
||||
|
||||
+37
-4
@@ -147,12 +147,44 @@ Status: pending
|
||||
|
||||
## Phase 7 — Real data and production readiness
|
||||
|
||||
Status: pending
|
||||
Status: in progress
|
||||
|
||||
Phased plan: real station/temperature data first, then Harmonie once DMI
|
||||
credentials are registered, then remaining sources. Synthetic data is kept
|
||||
running in parallel until each real source is proven, not cut over in one step.
|
||||
|
||||
- [ ] Confirm the actual workplace deployment topology and current deployed commit.
|
||||
- [ ] Obtain authorized development credentials or representative fixtures.
|
||||
- [ ] Validate LVGMC station and forecast CSV ingestion.
|
||||
- [ ] Validate DMI STAC discovery and EDR GRIB downloads.
|
||||
- [x] Find a credential-free real alternative for current station observations:
|
||||
LVĢMC also publishes `hidrometeorologiskie-noverojumi` on `data.gov.lv`
|
||||
(the same free, keyless CKAN API already used for Brīdinājumi warnings),
|
||||
including a 48h rolling "operational" resource with the same fields the
|
||||
private FTP feed provides (minus `visibilityMin`/`dewPoint`/`sunDuration`,
|
||||
which aren't published there).
|
||||
- [x] Add `fetch.lvgmc.OpenDataStationService`, pivoting that feed's tall
|
||||
`{STATION_ID, ABBREVIATION, DATETIME, VALUE}` rows into the existing
|
||||
`WeatherData` shape via a verified `STATION_ID → city` map (32 of the
|
||||
app's 37 tracked cities have a direct station), and schedule it in
|
||||
`Main.scala` (minutes 15/45) alongside — not replacing — the existing
|
||||
FTP fetch, writing into the same `weather` table via the existing
|
||||
`PostgresService.insertInWeatherTable` upsert.
|
||||
- [x] Verify end to end on Rocky: real rows confirmed in `weather` via direct
|
||||
SQL, Faktiskā's 13 fixed stations rendering real current temperatures
|
||||
(Valmiera, the one city with no matching open-data station, correctly
|
||||
falls back to older synthetic data via the existing stale-observation
|
||||
UI rather than breaking), and Kartes' aggregate query API returning
|
||||
sane blended real+synthetic values.
|
||||
- [ ] Let the open-data fetch run for a trial period, then remove the private
|
||||
FTP path (`fetch/lvgmc/FetchService.scala`'s `fetchWeatherStations`,
|
||||
its `LVGMC_*` env vars) and the synthetic seed machinery
|
||||
(`dev/seed_weather.sql`, the `seed` Compose service) once confident.
|
||||
- [ ] Register free DMI Open Data credentials and validate DMI STAC discovery
|
||||
and EDR GRIB downloads (code already exists and is already scheduled;
|
||||
this is credential registration and verification, not new code —
|
||||
confirm first whether DMI's endpoint migration away from
|
||||
`dmigw.govcloud.dk` changes what's needed).
|
||||
- [ ] Investigate open real sources for water temperature (Ūdens) and the
|
||||
LVĢMC forecast CSV workspace, following the same open-data-portal
|
||||
approach used for station observations.
|
||||
- [ ] Test scheduled ingestion failure and recovery behavior.
|
||||
- [ ] Rehearse deployment and rollback using sanitized data.
|
||||
- [ ] Obtain technical and operational review before workplace rollout.
|
||||
@@ -221,3 +253,4 @@ Record completed work here by date and commit after the Git workflow is establis
|
||||
| 2026-08-22 | `56b8716` | Replace single-warning Brīdinājumi selection with multi-select checkboxes and severity-ordered polygon compositing, and restyle chips as narrower, fully severity-colored, white-icon/white-text buttons with a ported Uiverse.io checkbox component | Yes — Rocky typecheck, build, and headless-browser screenshot verification of multiple simultaneous checked warnings compositing correctly on the map; deployed to VPS in release `b5150ab` on 2026-08-22 |
|
||||
| 2026-08-22 | `6185dbf` | Harden `WarningService` against partial upstream failures: bounded outage retry, per-warning fetch isolation, a live-API-verified CKAN polygon filter, and visible logging for a previously-silent vertex-order default | Yes — Scala tests, Rocky full-stack rebuild verified against real live LVĢMC data (including a genuinely severe 52,345-point wind-warning polygon rendering correctly); deployed to VPS in release `6185dbf` on 2026-08-22, verified against live data on the VPS itself |
|
||||
| 2026-08-23 | `c97875d` | Add draggable, resizable per-warning weather-symbol placement to Brīdinājumi, reusing Faktiskā's symbol set and image cache | Yes — Rocky typecheck, build, and headless-browser mouse-drag/resize verification against fixture warning data (the live LVĢMC feed had zero active warnings at test time); confirmed the exported PNG excludes the preview-only drag/resize handles; VPS deployment pending |
|
||||
| 2026-08-23 | `893a09a` | Add `fetch.lvgmc.OpenDataStationService`: real, free, keyless current station observations from LVĢMC's open-data portal, scheduled alongside (not replacing) the private FTP feed | Yes — Scala tests, manual `sbt runMain` fetch+write against the real feed on Rocky staging (1,577 rows, 33 cities), confirmed in `weather` via direct SQL, Faktiskā's 13 fixed stations rendering real temperatures with correct stale-fallback for the one uncovered city (Valmiera), and Kartes' aggregate query API returning sane blended values; VPS deployment pending |
|
||||
|
||||
Reference in New Issue
Block a user