Bring architecture/workflow docs current with real-data ingestion
ARCHITECTURE.md, PRODUCT_WORKFLOWS.md, DEVELOPMENT_AND_STAGING.md, and README.md still described the removed synthetic-seed staging setup and Ūdens as pure manual entry. Updates this session (open-data station ingestion, the scheduler split, water-temperature auto-populate) weren't reflected outside UPDATE_ROADMAP.md's changelog.
This commit is contained in:
+26
-21
@@ -5,23 +5,28 @@
|
||||
WeatherTool is a Scala backend, SolidJS frontend, and PostgreSQL database used to inspect Latvian weather observations and prepare map-based newsroom graphics.
|
||||
|
||||
```text
|
||||
External providers Local/staging workflow
|
||||
LVGMC station FTP/CSV ----+ Synthetic SQL seed
|
||||
LVGMC forecast CSV -------+ |
|
||||
DMI HARMONIE APIs --------+ v
|
||||
v PostgreSQL
|
||||
Scala ingestion |
|
||||
| |
|
||||
+---------> Scala/http4s API
|
||||
|
|
||||
v
|
||||
SolidJS web UI
|
||||
|
|
||||
v
|
||||
Canvas preview and PNG export
|
||||
External providers
|
||||
LVĢMC open-data station/hydro feeds (data.gov.lv) --+
|
||||
LVGMC station FTP/CSV (legacy, gated off) ----------+
|
||||
LVGMC forecast CSV (legacy, gated off) --------------+
|
||||
DMI HARMONIE APIs (legacy, gated off) --------------+
|
||||
v
|
||||
Scala ingestion
|
||||
|
|
||||
v
|
||||
PostgreSQL
|
||||
|
|
||||
v
|
||||
Scala/http4s API
|
||||
|
|
||||
v
|
||||
SolidJS web UI
|
||||
|
|
||||
v
|
||||
Canvas preview and PNG export
|
||||
```
|
||||
|
||||
Staging currently exercises the PostgreSQL → API → frontend path with synthetic observations. It does not emulate LVGMC forecast files or HARMONIE GRIB files.
|
||||
Staging exercises the PostgreSQL → API → frontend path with real LVĢMC open-data observations (station temperatures, water temperatures, warnings) as of 2026-08-23. The private FTP station feed and the DMI HARMONIE forecast feed still exist in code but are gated behind `ENABLE_LEGACY_PROVIDER_JOBS` (default off) pending real credentials — see `docs/UPDATE_ROADMAP.md` Phase 7.
|
||||
|
||||
## Components
|
||||
|
||||
@@ -29,10 +34,10 @@ Staging currently exercises the PostgreSQL → API → frontend path with synthe
|
||||
|
||||
- Serves JSON API routes and the built frontend from `web/dist`.
|
||||
- Queries and aggregates station observations in PostgreSQL.
|
||||
- Contains ingestion/parsing code for LVGMC station and forecast data, both the private FTP feed (`fetch.lvgmc.FetchService`) and, as of 2026-08-23, a free/keyless open-data alternative (`fetch.lvgmc.OpenDataStationService`) that pivots LVĢMC's public `data.gov.lv` station-observation feed into the same `weather` table — the two run in parallel during evaluation; see `docs/UPDATE_ROADMAP.md` Phase 7.
|
||||
- Fetches LVĢMC open hydrometeorological-warning metadata and polygon coordinates on request, with a 5-minute in-memory cache and stale-data fallback if the upstream call fails.
|
||||
- Contains HARMONIE discovery, download, GRIB parsing, and rendering support.
|
||||
- Starts scheduled ingestion/cleanup tasks unless `ENABLE_SCHEDULED_JOBS=false`.
|
||||
- Contains ingestion/parsing code for LVGMC station data: a free/keyless open-data path (`fetch.lvgmc.OpenDataStationService`) that pivots LVĢMC's public `data.gov.lv` station-observation feed into the `weather` table, and the original private FTP feed (`fetch.lvgmc.FetchService`), kept but gated off pending real credentials.
|
||||
- Fetches LVĢMC open hydrometeorological-warning metadata and polygon coordinates on request (`fetch.warnings.WarningService`), and water temperatures per named zone (`fetch.lvgmc.WaterTemperatureService`) — both free/keyless `data.gov.lv` reads with a short in-memory cache and stale-data fallback if the upstream call fails; neither persists to PostgreSQL.
|
||||
- Contains HARMONIE discovery, download, GRIB parsing, and rendering support, gated off pending real DMI credentials.
|
||||
- Starts the safe scheduled ingestion/cleanup tasks (open-data stations, GRIB cleanup) unless `ENABLE_SCHEDULED_JOBS=false`. The legacy FTP station fetch and HARMONIE fetch are scheduled independently and only run if `ENABLE_LEGACY_PROVIDER_JOBS=true` — kept separate so placeholder credentials on the legacy path can't crash-loop the whole app (both paths use `cats.effect.IO`'s `parMapN`, which cancels and fails every sibling task the instant one throws).
|
||||
|
||||
### SolidJS frontend
|
||||
|
||||
@@ -49,7 +54,7 @@ Staging currently exercises the PostgreSQL → API → frontend path with synthe
|
||||
|
||||
## Main observation-data flow
|
||||
|
||||
1. An ingestion job parses provider station data, or the development seed inserts synthetic rows.
|
||||
1. A scheduled ingestion job parses provider station data (the open-data path by default; the legacy FTP path only if `ENABLE_LEGACY_PROVIDER_JOBS=true`).
|
||||
2. Observations are stored in the `weather` table.
|
||||
3. The frontend requests an API route containing cities, time range, granularity, field, and aggregate key.
|
||||
4. The backend performs the database query and returns JSON.
|
||||
@@ -79,7 +84,7 @@ The renderer attaches each normalized 256×256 weather image to the measured edg
|
||||
|
||||
### Ūdens
|
||||
|
||||
Ūdens is an independent fixed-template renderer. Its six ranges are manual editorial inputs rather than station-query results. The 1920×1080 and 3840×1440 products use separate authoritative templates and six separately measured value rectangles. Canvas `TextMetrics` visible-glyph bounds center the Monda text optically inside each rectangle. Preview scaling never changes native export geometry, and both output sizes have been visually validated against the supplied newsroom templates.
|
||||
Ūdens is an independent fixed-template renderer. As of 2026-08-23 its six ranges auto-populate from `fetch.lvgmc.WaterTemperatureService` on load — real per-zone min/max, not station-query results from PostgreSQL — with manual override and a per-zone "Atiestatīt" reset still fully available, mirroring the auto-load/override/reset pattern Faktiskā already used. Each of the six named zones (Jūra, Līcis, Kurzeme, Zemgale, Vidzeme, Latgale) is a geographic grouping of every currently-reporting LVĢMC hydrological station in that area (56 inland `WTEMD` + 9 coastal `SEDUT` stations nationwide, classified by approximate historical-region boundaries — the mapping isn't derived from an authoritative region field, since the open-data portal doesn't publish one), so the displayed range reflects genuine spread across that zone's stations rather than one station's value shown twice. Readings older than 12h are dropped so one stuck sensor can't skew a zone's range. The 1920×1080 and 3840×1440 products use separate authoritative templates and six separately measured value rectangles. Canvas `TextMetrics` visible-glyph bounds center the Monda text optically inside each rectangle. Preview scaling never changes native export geometry, and both output sizes have been visually validated against the supplied newsroom templates.
|
||||
|
||||
Monda Regular and Bold are bundled locally and used by the interface and generated text. Weather symbols are bundled as independent transparent PNG assets.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user