Split ENABLE_LEGACY_PROVIDER_JOBS into independent FTP/HARMONIE flags

Real LVGMC FTP credentials arrived today; real DMI HARMONIE
credentials haven't. The combined flag would have enabled both the
moment FTP's were ready, crash-looping the app on HARMONIE's still-
placeholder values via parMapN — caught this before it happened
(the Grib job was ~15 min from its first scheduled run). Split into
ENABLE_LVGMC_FTP_JOBS and ENABLE_HARMONIE_JOBS so each provider can
be enabled independently as its own credentials become real.

.env-sample

@@ -6,11 +6,13 @@ POSTGRES_HOST=postgres
 APP_BIND_ADDRESS=127.0.0.1
 APP_PORT=9090
 ENABLE_SCHEDULED_JOBS=true
-# Gates the FTP station fetch and DMI Harmonie fetch specifically, separately
-# from ENABLE_SCHEDULED_JOBS, since both require real credentials below.
-# Leave false until those are real; both jobs throw on placeholder values,
+# Gate the FTP station fetch and DMI Harmonie fetch independently, separately
+# from ENABLE_SCHEDULED_JOBS, since each requires its own real credentials
+# below and they tend to arrive on different timelines. Leave false until a
+# given one's credentials are real; each job throws on placeholder values,
 # and would otherwise crash-loop the whole app rather than just fail quietly.
-ENABLE_LEGACY_PROVIDER_JOBS=false
+ENABLE_LVGMC_FTP_JOBS=false
+ENABLE_HARMONIE_JOBS=false
 
 LVGMC_URL=aaa
 LVGMC_USER=aaa

deploy/vps/compose.yml

@@ -32,11 +32,12 @@ services:
       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
       POSTGRES_HOST: postgres
       ENABLE_SCHEDULED_JOBS: "true"
-      ENABLE_LEGACY_PROVIDER_JOBS: "false"
+      ENABLE_LVGMC_FTP_JOBS: "false"
+      ENABLE_HARMONIE_JOBS: "false"
       DEBUG: "false"
       # The legacy FTP/HARMONIE provider services are still constructed at
-      # startup even though ENABLE_LEGACY_PROVIDER_JOBS keeps their scheduled
-      # jobs off, so inert values are required until real credentials exist.
+      # startup even though these flags keep their scheduled jobs off, so
+      # inert values are required until each one's real credentials exist.
       LVGMC_USER: disabled
       LVGMC_PASSWORD: disabled
       LVGMC_URL: https://invalid.local/

docker-compose.yml

@@ -34,7 +34,8 @@ services:
       HARMONIE_STAC_API_KEY: ${HARMONIE_STAC_API_KEY}
       HARMONIE_STAC_URL: ${HARMONIE_STAC_URL}
       ENABLE_SCHEDULED_JOBS: ${ENABLE_SCHEDULED_JOBS:-true}
-      ENABLE_LEGACY_PROVIDER_JOBS: ${ENABLE_LEGACY_PROVIDER_JOBS:-false}
+      ENABLE_LVGMC_FTP_JOBS: ${ENABLE_LVGMC_FTP_JOBS:-false}
+      ENABLE_HARMONIE_JOBS: ${ENABLE_HARMONIE_JOBS:-false}
     build:
       context: .
       dockerfile: Dockerfile.local

docs/ARCHITECTURE.md

@@ -26,7 +26,7 @@ DMI HARMONIE APIs (legacy, gated off) --------------+
                                   Canvas preview and PNG export
 ```
 
-Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open-data observations (station temperatures, water temperatures, warnings) as of 2026-08-23. The private FTP station feed and the DMI HARMONIE forecast feed still exist in code but are gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials — see `docs/UPDATE_ROADMAP.md` Phase 7.
+Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open-data observations (station temperatures, water temperatures, warnings) as of 2026-08-23. The private FTP station feed is real and enabled on Rocky as of 2026-08-23 (`ENABLE_LVGMC_FTP_JOBS=true`); the DMI HARMONIE forecast feed still exists in code but stays gated (`ENABLE_HARMONIE_JOBS`, default off) pending real credentials — see `docs/UPDATE_ROADMAP.md` Phase 7.
 
 ## Components
 
@@ -37,7 +37,7 @@ Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open
 - Contains ingestion/parsing code for LVGMC station data: a free/keyless open-data path (`fetch.lvgmc.OpenDataStationService`) that pivots LVĢMC's public `data.gov.lv` station-observation feed into the `weather` table, and the original private FTP feed (`fetch.lvgmc.FetchService`), kept but gated off pending real credentials.
 - Fetches LVĢMC open hydrometeorological-warning metadata and polygon coordinates on request (`fetch.warnings.WarningService`), and water temperatures per named zone (`fetch.lvgmc.WaterTemperatureService`) — both free/keyless `data.gov.lv` reads with a short in-memory cache and stale-data fallback if the upstream call fails; neither persists to PostgreSQL.
 - Contains HARMONIE discovery, download, GRIB parsing, and rendering support, gated off pending real DMI credentials.
-- Starts the safe scheduled ingestion/cleanup tasks (open-data stations, GRIB cleanup) unless `ENABLE_SCHEDULED_JOBS=false`. The legacy FTP station fetch and HARMONIE fetch are scheduled independently and only run if `ENABLE_LEGACY_PROVIDER_JOBS=true` — kept separate so placeholder credentials on the legacy path can't crash-loop the whole app (both paths use `cats.effect.IO`'s `parMapN`, which cancels and fails every sibling task the instant one throws).
+- Starts the safe scheduled ingestion/cleanup tasks (open-data stations, GRIB cleanup) unless `ENABLE_SCHEDULED_JOBS=false`. The legacy FTP station fetch and HARMONIE fetch are each gated by their own flag (`ENABLE_LVGMC_FTP_JOBS`, `ENABLE_HARMONIE_JOBS`) rather than one combined flag — real credentials for the two providers arrive on different timelines, and a single flag would enable both together the moment either one's were ready, crash-looping the app on whichever provider still had placeholders (both paths use `cats.effect.IO`'s `parMapN`, which cancels and fails every sibling task the instant one throws).
 
 ### SolidJS frontend
 
@@ -54,7 +54,7 @@ Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open
 
 ## Main observation-data flow
 
-1. A scheduled ingestion job parses provider station data (the open-data path by default; the legacy FTP path only if `ENABLE_LEGACY_PROVIDER_JOBS=true`).
+1. A scheduled ingestion job parses provider station data (the open-data path always; the FTP path too if `ENABLE_LVGMC_FTP_JOBS=true`).
 2. Observations are stored in the `weather` table.
 3. The frontend requests an API route containing cities, time range, granularity, field, and aggregate key.
 4. The backend performs the database query and returns JSON.

docs/DEVELOPMENT_AND_STAGING.md

@@ -30,7 +30,7 @@ codex/staging-baseline
 ## Safety boundaries
 
 - Do not commit `.env`, credentials, database data, `web/node_modules`, or `web/dist`.
-- Keep `ENABLE_LEGACY_PROVIDER_JOBS=false` wherever the private LVGMC FTP or DMI HARMONIE credentials are still placeholders — that path uses `parMapN`, which cancels and fails every scheduled task the instant one throws, so a placeholder credential there crash-loops the whole app. `ENABLE_SCHEDULED_JOBS` gates only the free/keyless open-data station fetch and GRIB cleanup, and is safe to leave on.
+- Keep `ENABLE_LVGMC_FTP_JOBS=false` and/or `ENABLE_HARMONIE_JOBS=false` wherever that specific provider's credentials are still placeholders — both paths use `parMapN`, which cancels and fails every scheduled task the instant one throws, so a placeholder credential crash-loops the whole app. The two are gated independently since real credentials for each arrive on different timelines. `ENABLE_SCHEDULED_JOBS` gates only the free/keyless open-data station fetch and GRIB cleanup, and is safe to leave on.
 - Do not publish the staging PostgreSQL port.
 - Do not run `npm audit fix` without reviewing the proposed dependency changes.
 - Do not change ownership or permissions of the container-managed `postgres/` directory as a deployment workaround.

docs/README.md

@@ -6,7 +6,7 @@ This directory contains the working documentation for the WeatherTool modernizat
 
 - Windows is restricted to source editing, review, and Git operations. Rocky is the sole compile, build, development-runtime, and test environment. The Ubuntu VPS is a deployment target only.
 - A production-like staging copy runs through Docker Compose on Rocky Linux at `http://192.168.1.101:9190`.
-- Staging ingests real LVĢMC station and water-temperature observations from the free `data.gov.lv` open-data feeds; synthetic data generation was removed once real ingestion was verified. The private LVGMC FTP feed and DMI HARMONIE forecast feed still exist in code but stay gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials.
+- Staging ingests real LVĢMC station and water-temperature observations from the free `data.gov.lv` open-data feeds; synthetic data generation was removed once real ingestion was verified. The private LVGMC FTP feed and DMI HARMONIE forecast feed are each gated independently (`ENABLE_LVGMC_FTP_JOBS`, `ENABLE_HARMONIE_JOBS`, both default off) since real credentials for the two arrive on different timelines — FTP creds were obtained and verified 2026-08-23; HARMONIE's are still pending.
 - The safe scheduled jobs (open-data station ingestion, GRIB cleanup) run in staging; the legacy FTP/HARMONIE jobs stay off by default so placeholder credentials can't crash-loop the app.
 - PostgreSQL is private to the project Compose network; only the Scala application publishes a host port.
 - The operator-facing workspaces now use the Latvian workflow names **Stacijas**, **Kartes**, **Faktiskā**, **Ūdens**, **Brīdinājumi**, **Apskats**, **Arhīvs**, **Harmonie**, and **LVĢMC**. Kartes retains custom analytical map outputs, while Faktiskā is a fixed 13-position, latest-temperature newsroom workflow with a locked 3840×1440 export.

docs/UPDATE_ROADMAP.md

@@ -202,13 +202,35 @@ running in parallel until each real source is proven, not cut over in one step.
       independently gates the FTP station fetch and Harmonie fetch. Turned
       the former on for Rocky staging; real station data now accumulates
       automatically every 30 minutes with no manual trigger needed.
-- [ ] FTP (`fetch/lvgmc/FetchService.scala`) and Harmonie/DMI credentials are
-      the user's own follow-up (asking the original dev about the FTP
-      partner feed; registering for free DMI Open Data). Both stay gated
-      behind `ENABLE_LEGACY_PROVIDER_JOBS`/missing keys — not removed — so
-      no code changes are needed once real credentials arrive, just setting
-      them and flipping the flag. Worth confirming DMI's endpoint migration
-      away from `dmigw.govcloud.dk` before wiring anything.
+- [x] FTP (`fetch/lvgmc/FetchService.scala`) credentials obtained 2026-08-23
+      (`ftp.meteo.lv`, user `ltv`) and verified working against both a
+      sample file and the real production file (`Latvija_faktiskais_laiks.csv`)
+      via a manual `sbt runMain` check before any scheduling was touched.
+      No code changes were needed to use them — confirming the original
+      design intent. Split the combined `ENABLE_LEGACY_PROVIDER_JOBS` flag
+      into independent `ENABLE_LVGMC_FTP_JOBS`/`ENABLE_HARMONIE_JOBS` flags
+      first, since real credentials for the two providers arrive on
+      different timelines and a single flag would have enabled Harmonie
+      (still placeholder) the moment FTP's were ready — caught this before
+      it could crash-loop the app (the Grib job was ~15 min from its first
+      scheduled run when this was noticed). FTP enabled on Rocky; Harmonie
+      stays off pending real DMI credentials, still the user's own
+      follow-up. Worth confirming DMI's endpoint migration away from
+      `dmigw.govcloud.dk` before wiring anything.
+      Also found and fixed a real UTC-vs-local timezone mismatch while
+      verifying FTP alongside open-data: both write into `weather.dateTime`
+      with no conversion, but the open-data portal's DATETIME is UTC while
+      FTP's "Laiks" column is already local — silently present since
+      open-data went live, only becoming an active problem now that a
+      second, correctly-labeled source exists alongside it. Fixed by
+      converting open-data's timestamps to Europe/Riga at ingestion
+      (`OpenDataStationService`) and in the Ūdens water-temperature display
+      (`WaterTemperatureService`, same root cause, display-only). Also
+      hardened `insert_weather_table.sql`'s upsert from a blind overwrite to
+      `COALESCE(excluded.field, weather.field)`, since open-data's rows
+      always carry null visibilityMin/dewPoint/sunDuration and an empty
+      phenomena array — a blind overwrite would silently erase FTP's real
+      values for those fields whenever open-data's write landed later.
 - [x] Add real water temperature data for Ūdens: `fetch.lvgmc.WaterTemperatureService`
       fetches LVĢMC's open hydrological data (`data.gov.lv`, same free CKAN
       API), mapping one representative real station per named zone (coastal
@@ -247,7 +269,7 @@ Status: in progress
 - Both the VPS and Rocky `weather` tables now hold only real open-data station observations; their original synthetic rows were backed up and wiped 2026-08-23.
 - LVGMC forecast CSV fixtures are not yet available.
 - HARMONIE GRIB fixtures are not yet available.
-- The legacy private LVGMC FTP feed and DMI HARMONIE forecast feed remain gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials, in both development and staging (Rocky and VPS).
+- The private LVGMC FTP feed has real credentials as of 2026-08-23 and is enabled on Rocky (`ENABLE_LVGMC_FTP_JOBS=true`); the DMI HARMONIE forecast feed remains gated (`ENABLE_HARMONIE_JOBS`, default off) pending real credentials.
 - Existing automated test coverage is minimal.
 - Direct refreshes on newer frontend routes can return 404 until the backend gains a general SPA fallback.
 - Full Docker build context scanning on Rocky can fail on the container-owned `postgres/` bind directory; do not loosen its permissions.

src/main/scala/Main.scala

@@ -68,12 +68,26 @@ object Main extends IOApp {
         else
           (fetchOpenDataStationsTask, cleanupTask).parMapN((_, _) => ())
 
-      legacyProviderTasks =
-        if (sys.env.get("ENABLE_LEGACY_PROVIDER_JOBS").exists(_.equalsIgnoreCase("true")))
-          (fetchStationsTask, fetchGribTask).parMapN((_, _) => ())
+      // FTP (LVGMC) and HARMONIE (DMI) are gated independently: each only
+      // throws once its own real credentials arrive, and they arrive on
+      // different timelines (FTP creds obtained 2026-08-23; HARMONIE still
+      // pending). A single combined flag would enable both together the
+      // moment either one's credentials were ready, crash-looping the app
+      // on whichever provider still had placeholders.
+      ftpStationTask =
+        if (sys.env.get("ENABLE_LVGMC_FTP_JOBS").exists(_.equalsIgnoreCase("true")))
+          fetchStationsTask
         else
           IO.never[Unit]
 
+      harmonieTask =
+        if (sys.env.get("ENABLE_HARMONIE_JOBS").exists(_.equalsIgnoreCase("true")))
+          fetchGribTask
+        else
+          IO.never[Unit]
+
+      legacyProviderTasks = (ftpStationTask, harmonieTask).parMapN((_, _) => ())
+
       scheduledTasks = (safeTasks, legacyProviderTasks).parMapN((_, _) => ())
 
       exitCode <- (serverTask, scheduledTasks).parMapN((_, _) => ExitCode.Success)