Split ENABLE_LEGACY_PROVIDER_JOBS into independent FTP/HARMONIE flags
Real LVGMC FTP credentials arrived today; real DMI HARMONIE credentials haven't. The combined flag would have enabled both the moment FTP's were ready, crash-looping the app on HARMONIE's still- placeholder values via parMapN — caught this before it happened (the Grib job was ~15 min from its first scheduled run). Split into ENABLE_LVGMC_FTP_JOBS and ENABLE_HARMONIE_JOBS so each provider can be enabled independently as its own credentials become real.
This commit is contained in:
+30
-8
@@ -202,13 +202,35 @@ running in parallel until each real source is proven, not cut over in one step.
|
||||
independently gates the FTP station fetch and Harmonie fetch. Turned
|
||||
the former on for Rocky staging; real station data now accumulates
|
||||
automatically every 30 minutes with no manual trigger needed.
|
||||
- [ ] FTP (`fetch/lvgmc/FetchService.scala`) and Harmonie/DMI credentials are
|
||||
the user's own follow-up (asking the original dev about the FTP
|
||||
partner feed; registering for free DMI Open Data). Both stay gated
|
||||
behind `ENABLE_LEGACY_PROVIDER_JOBS`/missing keys — not removed — so
|
||||
no code changes are needed once real credentials arrive, just setting
|
||||
them and flipping the flag. Worth confirming DMI's endpoint migration
|
||||
away from `dmigw.govcloud.dk` before wiring anything.
|
||||
- [x] FTP (`fetch/lvgmc/FetchService.scala`) credentials obtained 2026-08-23
|
||||
(`ftp.meteo.lv`, user `ltv`) and verified working against both a
|
||||
sample file and the real production file (`Latvija_faktiskais_laiks.csv`)
|
||||
via a manual `sbt runMain` check before any scheduling was touched.
|
||||
No code changes were needed to use them — confirming the original
|
||||
design intent. Split the combined `ENABLE_LEGACY_PROVIDER_JOBS` flag
|
||||
into independent `ENABLE_LVGMC_FTP_JOBS`/`ENABLE_HARMONIE_JOBS` flags
|
||||
first, since real credentials for the two providers arrive on
|
||||
different timelines and a single flag would have enabled Harmonie
|
||||
(still placeholder) the moment FTP's were ready — caught this before
|
||||
it could crash-loop the app (the Grib job was ~15 min from its first
|
||||
scheduled run when this was noticed). FTP enabled on Rocky; Harmonie
|
||||
stays off pending real DMI credentials, still the user's own
|
||||
follow-up. Worth confirming DMI's endpoint migration away from
|
||||
`dmigw.govcloud.dk` before wiring anything.
|
||||
Also found and fixed a real UTC-vs-local timezone mismatch while
|
||||
verifying FTP alongside open-data: both write into `weather.dateTime`
|
||||
with no conversion, but the open-data portal's DATETIME is UTC while
|
||||
FTP's "Laiks" column is already local — silently present since
|
||||
open-data went live, only becoming an active problem now that a
|
||||
second, correctly-labeled source exists alongside it. Fixed by
|
||||
converting open-data's timestamps to Europe/Riga at ingestion
|
||||
(`OpenDataStationService`) and in the Ūdens water-temperature display
|
||||
(`WaterTemperatureService`, same root cause, display-only). Also
|
||||
hardened `insert_weather_table.sql`'s upsert from a blind overwrite to
|
||||
`COALESCE(excluded.field, weather.field)`, since open-data's rows
|
||||
always carry null visibilityMin/dewPoint/sunDuration and an empty
|
||||
phenomena array — a blind overwrite would silently erase FTP's real
|
||||
values for those fields whenever open-data's write landed later.
|
||||
- [x] Add real water temperature data for Ūdens: `fetch.lvgmc.WaterTemperatureService`
|
||||
fetches LVĢMC's open hydrological data (`data.gov.lv`, same free CKAN
|
||||
API), mapping one representative real station per named zone (coastal
|
||||
@@ -247,7 +269,7 @@ Status: in progress
|
||||
- Both the VPS and Rocky `weather` tables now hold only real open-data station observations; their original synthetic rows were backed up and wiped 2026-08-23.
|
||||
- LVGMC forecast CSV fixtures are not yet available.
|
||||
- HARMONIE GRIB fixtures are not yet available.
|
||||
- The legacy private LVGMC FTP feed and DMI HARMONIE forecast feed remain gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials, in both development and staging (Rocky and VPS).
|
||||
- The private LVGMC FTP feed has real credentials as of 2026-08-23 and is enabled on Rocky (`ENABLE_LVGMC_FTP_JOBS=true`); the DMI HARMONIE forecast feed remains gated (`ENABLE_HARMONIE_JOBS`, default off) pending real credentials.
|
||||
- Existing automated test coverage is minimal.
|
||||
- Direct refreshes on newer frontend routes can return 404 until the backend gains a general SPA fallback.
|
||||
- Full Docker build context scanning on Rocky can fail on the container-owned `postgres/` bind directory; do not loosen its permissions.
|
||||
|
||||
Reference in New Issue
Block a user