Split ENABLE_LEGACY_PROVIDER_JOBS into independent FTP/HARMONIE flags

Real LVGMC FTP credentials arrived today; real DMI HARMONIE
credentials haven't. The combined flag would have enabled both the
moment FTP's were ready, crash-looping the app on HARMONIE's still-
placeholder values via parMapN — caught this before it happened
(the Grib job was ~15 min from its first scheduled run). Split into
ENABLE_LVGMC_FTP_JOBS and ENABLE_HARMONIE_JOBS so each provider can
be enabled independently as its own credentials become real.
This commit is contained in:
b0txec
2026-08-23 21:32:11 +03:00
parent 3315f00fb1
commit 3eddf95008
8 changed files with 64 additions and 24 deletions
+30 -8
View File
@@ -202,13 +202,35 @@ running in parallel until each real source is proven, not cut over in one step.
independently gates the FTP station fetch and Harmonie fetch. Turned
the former on for Rocky staging; real station data now accumulates
automatically every 30 minutes with no manual trigger needed.
- [ ] FTP (`fetch/lvgmc/FetchService.scala`) and Harmonie/DMI credentials are
the user's own follow-up (asking the original dev about the FTP
partner feed; registering for free DMI Open Data). Both stay gated
behind `ENABLE_LEGACY_PROVIDER_JOBS`/missing keys — not removed — so
no code changes are needed once real credentials arrive, just setting
them and flipping the flag. Worth confirming DMI's endpoint migration
away from `dmigw.govcloud.dk` before wiring anything.
- [x] FTP (`fetch/lvgmc/FetchService.scala`) credentials obtained 2026-08-23
(`ftp.meteo.lv`, user `ltv`) and verified working against both a
sample file and the real production file (`Latvija_faktiskais_laiks.csv`)
via a manual `sbt runMain` check before any scheduling was touched.
No code changes were needed to use them — confirming the original
design intent. Split the combined `ENABLE_LEGACY_PROVIDER_JOBS` flag
into independent `ENABLE_LVGMC_FTP_JOBS`/`ENABLE_HARMONIE_JOBS` flags
first, since real credentials for the two providers arrive on
different timelines and a single flag would have enabled Harmonie
(still placeholder) the moment FTP's were ready — caught this before
it could crash-loop the app (the Grib job was ~15 min from its first
scheduled run when this was noticed). FTP enabled on Rocky; Harmonie
stays off pending real DMI credentials, still the user's own
follow-up. Worth confirming DMI's endpoint migration away from
`dmigw.govcloud.dk` before wiring anything.
Also found and fixed a real UTC-vs-local timezone mismatch while
verifying FTP alongside open-data: both write into `weather.dateTime`
with no conversion, but the open-data portal's DATETIME is UTC while
FTP's "Laiks" column is already local — silently present since
open-data went live, only becoming an active problem now that a
second, correctly-labeled source exists alongside it. Fixed by
converting open-data's timestamps to Europe/Riga at ingestion
(`OpenDataStationService`) and in the Ūdens water-temperature display
(`WaterTemperatureService`, same root cause, display-only). Also
hardened `insert_weather_table.sql`'s upsert from a blind overwrite to
`COALESCE(excluded.field, weather.field)`, since open-data's rows
always carry null visibilityMin/dewPoint/sunDuration and an empty
phenomena array — a blind overwrite would silently erase FTP's real
values for those fields whenever open-data's write landed later.
- [x] Add real water temperature data for Ūdens: `fetch.lvgmc.WaterTemperatureService`
fetches LVĢMC's open hydrological data (`data.gov.lv`, same free CKAN
API), mapping one representative real station per named zone (coastal
@@ -247,7 +269,7 @@ Status: in progress
- Both the VPS and Rocky `weather` tables now hold only real open-data station observations; their original synthetic rows were backed up and wiped 2026-08-23.
- LVGMC forecast CSV fixtures are not yet available.
- HARMONIE GRIB fixtures are not yet available.
- The legacy private LVGMC FTP feed and DMI HARMONIE forecast feed remain gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials, in both development and staging (Rocky and VPS).
- The private LVGMC FTP feed has real credentials as of 2026-08-23 and is enabled on Rocky (`ENABLE_LVGMC_FTP_JOBS=true`); the DMI HARMONIE forecast feed remains gated (`ENABLE_HARMONIE_JOBS`, default off) pending real credentials.
- Existing automated test coverage is minimal.
- Direct refreshes on newer frontend routes can return 404 until the backend gains a general SPA fallback.
- Full Docker build context scanning on Rocky can fail on the container-owned `postgres/` bind directory; do not loosen its permissions.