Split ENABLE_LEGACY_PROVIDER_JOBS into independent FTP/HARMONIE flags
Real LVGMC FTP credentials arrived today; real DMI HARMONIE credentials haven't. The combined flag would have enabled both the moment FTP's were ready, crash-looping the app on HARMONIE's still- placeholder values via parMapN — caught this before it happened (the Grib job was ~15 min from its first scheduled run). Split into ENABLE_LVGMC_FTP_JOBS and ENABLE_HARMONIE_JOBS so each provider can be enabled independently as its own credentials become real.
This commit is contained in:
b0txec
@@ -26,7 +26,7 @@ DMI HARMONIE APIs (legacy, gated off) --------------+
|
||||
Canvas preview and PNG export
|
||||
```
|
||||
|
||||
Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open-data observations (station temperatures, water temperatures, warnings) as of 2026-08-23. The private FTP station feed and the DMI HARMONIE forecast feed still exist in code but are gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials — see `docs/UPDATE_ROADMAP.md` Phase 7.
|
||||
Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open-data observations (station temperatures, water temperatures, warnings) as of 2026-08-23. The private FTP station feed is real and enabled on Rocky as of 2026-08-23 (`ENABLE_LVGMC_FTP_JOBS=true`); the DMI HARMONIE forecast feed still exists in code but stays gated (`ENABLE_HARMONIE_JOBS`, default off) pending real credentials — see `docs/UPDATE_ROADMAP.md` Phase 7.
|
||||
|
||||
## Components
|
||||
|
||||
@@ -37,7 +37,7 @@ Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open
|
||||
- Contains ingestion/parsing code for LVGMC station data: a free/keyless open-data path (`fetch.lvgmc.OpenDataStationService`) that pivots LVĢMC's public `data.gov.lv` station-observation feed into the `weather` table, and the original private FTP feed (`fetch.lvgmc.FetchService`), kept but gated off pending real credentials.
|
||||
- Fetches LVĢMC open hydrometeorological-warning metadata and polygon coordinates on request (`fetch.warnings.WarningService`), and water temperatures per named zone (`fetch.lvgmc.WaterTemperatureService`) — both free/keyless `data.gov.lv` reads with a short in-memory cache and stale-data fallback if the upstream call fails; neither persists to PostgreSQL.
|
||||
- Contains HARMONIE discovery, download, GRIB parsing, and rendering support, gated off pending real DMI credentials.
|
||||
- Starts the safe scheduled ingestion/cleanup tasks (open-data stations, GRIB cleanup) unless `ENABLE_SCHEDULED_JOBS=false`. The legacy FTP station fetch and HARMONIE fetch are scheduled independently and only run if `ENABLE_LEGACY_PROVIDER_JOBS=true` — kept separate so placeholder credentials on the legacy path can't crash-loop the whole app (both paths use `cats.effect.IO`'s `parMapN`, which cancels and fails every sibling task the instant one throws).
|
||||
- Starts the safe scheduled ingestion/cleanup tasks (open-data stations, GRIB cleanup) unless `ENABLE_SCHEDULED_JOBS=false`. The legacy FTP station fetch and HARMONIE fetch are each gated by their own flag (`ENABLE_LVGMC_FTP_JOBS`, `ENABLE_HARMONIE_JOBS`) rather than one combined flag — real credentials for the two providers arrive on different timelines, and a single flag would enable both together the moment either one's were ready, crash-looping the app on whichever provider still had placeholders (both paths use `cats.effect.IO`'s `parMapN`, which cancels and fails every sibling task the instant one throws).
|
||||
|
||||
### SolidJS frontend
|
||||
|
||||
@@ -54,7 +54,7 @@ Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open
|
||||
|
||||
## Main observation-data flow
|
||||
|
||||
1. A scheduled ingestion job parses provider station data (the open-data path by default; the legacy FTP path only if `ENABLE_LEGACY_PROVIDER_JOBS=true`).
|
||||
1. A scheduled ingestion job parses provider station data (the open-data path always; the FTP path too if `ENABLE_LVGMC_FTP_JOBS=true`).
|
||||
2. Observations are stored in the `weather` table.
|
||||
3. The frontend requests an API route containing cities, time range, granularity, field, and aggregate key.
|
||||
4. The backend performs the database query and returns JSON.
|
||||
|
||||
@@ -30,7 +30,7 @@ codex/staging-baseline
|
||||
## Safety boundaries
|
||||
|
||||
- Do not commit `.env`, credentials, database data, `web/node_modules`, or `web/dist`.
|
||||
- Keep `ENABLE_LEGACY_PROVIDER_JOBS=false` wherever the private LVGMC FTP or DMI HARMONIE credentials are still placeholders — that path uses `parMapN`, which cancels and fails every scheduled task the instant one throws, so a placeholder credential there crash-loops the whole app. `ENABLE_SCHEDULED_JOBS` gates only the free/keyless open-data station fetch and GRIB cleanup, and is safe to leave on.
|
||||
- Keep `ENABLE_LVGMC_FTP_JOBS=false` and/or `ENABLE_HARMONIE_JOBS=false` wherever that specific provider's credentials are still placeholders — both paths use `parMapN`, which cancels and fails every scheduled task the instant one throws, so a placeholder credential crash-loops the whole app. The two are gated independently since real credentials for each arrive on different timelines. `ENABLE_SCHEDULED_JOBS` gates only the free/keyless open-data station fetch and GRIB cleanup, and is safe to leave on.
|
||||
- Do not publish the staging PostgreSQL port.
|
||||
- Do not run `npm audit fix` without reviewing the proposed dependency changes.
|
||||
- Do not change ownership or permissions of the container-managed `postgres/` directory as a deployment workaround.
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ This directory contains the working documentation for the WeatherTool modernizat
|
||||
|
||||
- Windows is restricted to source editing, review, and Git operations. Rocky is the sole compile, build, development-runtime, and test environment. The Ubuntu VPS is a deployment target only.
|
||||
- A production-like staging copy runs through Docker Compose on Rocky Linux at `http://192.168.1.101:9190`.
|
||||
- Staging ingests real LVĢMC station and water-temperature observations from the free `data.gov.lv` open-data feeds; synthetic data generation was removed once real ingestion was verified. The private LVGMC FTP feed and DMI HARMONIE forecast feed still exist in code but stay gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials.
|
||||
- Staging ingests real LVĢMC station and water-temperature observations from the free `data.gov.lv` open-data feeds; synthetic data generation was removed once real ingestion was verified. The private LVGMC FTP feed and DMI HARMONIE forecast feed are each gated independently (`ENABLE_LVGMC_FTP_JOBS`, `ENABLE_HARMONIE_JOBS`, both default off) since real credentials for the two arrive on different timelines — FTP creds were obtained and verified 2026-08-23; HARMONIE's are still pending.
|
||||
- The safe scheduled jobs (open-data station ingestion, GRIB cleanup) run in staging; the legacy FTP/HARMONIE jobs stay off by default so placeholder credentials can't crash-loop the app.
|
||||
- PostgreSQL is private to the project Compose network; only the Scala application publishes a host port.
|
||||
- The operator-facing workspaces now use the Latvian workflow names **Stacijas**, **Kartes**, **Faktiskā**, **Ūdens**, **Brīdinājumi**, **Apskats**, **Arhīvs**, **Harmonie**, and **LVĢMC**. Kartes retains custom analytical map outputs, while Faktiskā is a fixed 13-position, latest-temperature newsroom workflow with a locked 3840×1440 export.
|
||||
|
||||
+30
-8
@@ -202,13 +202,35 @@ running in parallel until each real source is proven, not cut over in one step.
|
||||
independently gates the FTP station fetch and Harmonie fetch. Turned
|
||||
the former on for Rocky staging; real station data now accumulates
|
||||
automatically every 30 minutes with no manual trigger needed.
|
||||
- [ ] FTP (`fetch/lvgmc/FetchService.scala`) and Harmonie/DMI credentials are
|
||||
the user's own follow-up (asking the original dev about the FTP
|
||||
partner feed; registering for free DMI Open Data). Both stay gated
|
||||
behind `ENABLE_LEGACY_PROVIDER_JOBS`/missing keys — not removed — so
|
||||
no code changes are needed once real credentials arrive, just setting
|
||||
them and flipping the flag. Worth confirming DMI's endpoint migration
|
||||
away from `dmigw.govcloud.dk` before wiring anything.
|
||||
- [x] FTP (`fetch/lvgmc/FetchService.scala`) credentials obtained 2026-08-23
|
||||
(`ftp.meteo.lv`, user `ltv`) and verified working against both a
|
||||
sample file and the real production file (`Latvija_faktiskais_laiks.csv`)
|
||||
via a manual `sbt runMain` check before any scheduling was touched.
|
||||
No code changes were needed to use them — confirming the original
|
||||
design intent. Split the combined `ENABLE_LEGACY_PROVIDER_JOBS` flag
|
||||
into independent `ENABLE_LVGMC_FTP_JOBS`/`ENABLE_HARMONIE_JOBS` flags
|
||||
first, since real credentials for the two providers arrive on
|
||||
different timelines and a single flag would have enabled Harmonie
|
||||
(still placeholder) the moment FTP's were ready — caught this before
|
||||
it could crash-loop the app (the Grib job was ~15 min from its first
|
||||
scheduled run when this was noticed). FTP enabled on Rocky; Harmonie
|
||||
stays off pending real DMI credentials, still the user's own
|
||||
follow-up. Worth confirming DMI's endpoint migration away from
|
||||
`dmigw.govcloud.dk` before wiring anything.
|
||||
Also found and fixed a real UTC-vs-local timezone mismatch while
|
||||
verifying FTP alongside open-data: both write into `weather.dateTime`
|
||||
with no conversion, but the open-data portal's DATETIME is UTC while
|
||||
FTP's "Laiks" column is already local — silently present since
|
||||
open-data went live, only becoming an active problem now that a
|
||||
second, correctly-labeled source exists alongside it. Fixed by
|
||||
converting open-data's timestamps to Europe/Riga at ingestion
|
||||
(`OpenDataStationService`) and in the Ūdens water-temperature display
|
||||
(`WaterTemperatureService`, same root cause, display-only). Also
|
||||
hardened `insert_weather_table.sql`'s upsert from a blind overwrite to
|
||||
`COALESCE(excluded.field, weather.field)`, since open-data's rows
|
||||
always carry null visibilityMin/dewPoint/sunDuration and an empty
|
||||
phenomena array — a blind overwrite would silently erase FTP's real
|
||||
values for those fields whenever open-data's write landed later.
|
||||
- [x] Add real water temperature data for Ūdens: `fetch.lvgmc.WaterTemperatureService`
|
||||
fetches LVĢMC's open hydrological data (`data.gov.lv`, same free CKAN
|
||||
API), mapping one representative real station per named zone (coastal
|
||||
@@ -247,7 +269,7 @@ Status: in progress
|
||||
- Both the VPS and Rocky `weather` tables now hold only real open-data station observations; their original synthetic rows were backed up and wiped 2026-08-23.
|
||||
- LVGMC forecast CSV fixtures are not yet available.
|
||||
- HARMONIE GRIB fixtures are not yet available.
|
||||
- The legacy private LVGMC FTP feed and DMI HARMONIE forecast feed remain gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials, in both development and staging (Rocky and VPS).
|
||||
- The private LVGMC FTP feed has real credentials as of 2026-08-23 and is enabled on Rocky (`ENABLE_LVGMC_FTP_JOBS=true`); the DMI HARMONIE forecast feed remains gated (`ENABLE_HARMONIE_JOBS`, default off) pending real credentials.
|
||||
- Existing automated test coverage is minimal.
|
||||
- Direct refreshes on newer frontend routes can return 404 until the backend gains a general SPA fallback.
|
||||
- Full Docker build context scanning on Rocky can fail on the container-owned `postgres/` bind directory; do not loosen its permissions.
|
||||
|
||||
Reference in New Issue
Block a user