The frontend job succeeded after the runner HOME fix, but backend failed at
a different point: actions/checkout is a JavaScript action and needs a Node
runtime inside the job container to run it at all (exec: "node": executable
file not found in $PATH), and hseeberger/scala-sbt has no Node installed.
Same lesson HOP's own CI setup already documented ("JavaScript actions need
Node in the job image") -- mirroring their fix: install git+nodejs via apt
before the checkout step.
Both jobs in the first two CI runs failed identically at the checkout step:
"Unable to clone https://github.com/actions/checkout ... mkdir /.cache:
permission denied". The runner container runs as a fixed uid with no
writable $HOME, so act cloning action sources into its cache directory
failed outright -- this affects every remote action reference, not just the
actions/cache action type as I'd assumed earlier. Fixed by setting
HOME=/data (the already-writable bind-mounted data directory) on the
runner container; not a source change, just re-triggering CI to confirm.
Adds CONTINUOUS_INTEGRATION.md (mirrors HOP's CI doc structure), links it
from README, and records the CI changelog entry plus the follow-up review's
open findings (scheduler crash-loop, System.gc(), connection pooling, the
PostgresService field-routing invariant, and the mislabeled main-method
"tests") in the roadmap's Phase 2/Phase 5 checklists.
Runs on push/PR to codex/staging-baseline plus manual dispatch: sbt test for
the backend, and npm ci/typecheck/build/audit (full + production-only) for
the frontend, matching the checks already documented as the manual Rocky
verification routine. Verified sbt test compiles and passes with no .env
file present (matching what a checkout-only CI job will actually have) via
a git-archive dry run before writing this.
Two separate jobs, each with its own container image, rather than one
shared runner-label image with ad-hoc installs — reuses the exact
hseeberger/scala-sbt image already used all session for local backend
builds, and a plain node:20-bookworm for the frontend, so nothing needs
apt-get bootstrapping of a second language runtime into the same container.
Caught by an independent follow-up review: the FTP auth-bypass fix in
6b9c7cf only gated /api/fetch/lvgmc/stations, the one route the original
security review named. This sibling route calls the same fetch.fetchFile,
which opens a real, unauthenticated FTP login to LVGMC regardless of any
caller-supplied filename, and was only getting the traversal fix
(ValidateFileName) applied to it, not the auth-bypass fix. Traced every
caller of fetch.fetchFile/fetchWeatherStations in Server.scala/Main.scala
this time to confirm these are the only two HTTP-reachable call sites and
both are now gated.
Release 0be325f (image weathertool:0be325fbbbf92b03bc8d574dc6f7b9449ef310ea)
ships the path-traversal/auth-bypass/DoS and SQL-injection fixes to
production; b0b58d2 remains the rollback target.
Records the path-traversal/auth-bypass/DoS fixes (6b9c7cf), the SQL
injection fix (8c45d8d), and the subsequent Postgres password rotation on
Rocky in the roadmap changelog and Phase 4 checklist, and notes current
status in README/DEVELOPMENT_AND_STAGING. VPS deployment of these fixes is
still pending.
field on that route reached PostgresService.query unvalidated, which splices
it into SQL via Fragment.const (unescaped) whenever the aggregate key is
min/max/avg/sum/distinct, or whenever granularity is "hour" in the list
branch. Add ValidateField (allowlists WeatherData's known field names, same
pattern AggFieldList already uses for /query/country) and apply it to the
field path segment. AggregateKey values reaching Fragment.const elsewhere are
already safe since they come from a closed ADT, not raw user input.
- Add ValidateFileName (allowlist regex, rejects .. and separators) and apply
it to every route that concatenates a raw path segment into a filesystem or
remote FTP path: /show/lvgmc-forecast, /show/grib, /grib/binary-chunk, and
/debug/file. Previously an unauthenticated caller could read arbitrary
files, including /proc/self/environ (leaks LVGMC_PASSWORD/POSTGRES_PASSWORD).
- Harden ValidateInt to reject negative integers.
- Gate /api/fetch/lvgmc/stations behind ENABLE_LVGMC_FTP_JOBS so it can no
longer trigger a real, unauthenticated FTP login regardless of the flag;
stop leaking error.getMessage in its response.
- Add an explicit /api/* catch-all (NotFound) so an unmatched API route can
never fall through to the SPA fallback and be served index.html as a 200.
- Cap binary-chunk read length at 64MB to prevent an unbounded allocation.
The SPA fallback (2c44888/b0b58d2) is deployed to VPS as b0b58d2,
verified via curl and headless-browser on real routes, a missing
asset, and /api, both locally and through the public domain.
Also updated the FTP saga: re-testing this morning found a Rocky-only
failure with the VPS confirmed off, which the two-machine-collision
theory from last night can't explain. Decided to stop self-testing
and wait for a real answer from LVGMC about the account's connection
policy rather than keep guessing through trial and error.
Direct hits on client-side routes not in the hardcoded list (e.g. a
browser refresh on /faktiska or /udens-temperatura) 404ed instead of
loading the app — a known limitation that was actually hit in
production. Real files now serve as-is; anything else falls back to
index.html so the SolidJS router handles it, matching the existing
"TODO rewrite in more generic way" comment. Future routes need no
backend changes.
The ltv account fails to authenticate from the VPS specifically
(works fine from Rocky with the same credentials, network path to
ftp.meteo.lv:21 confirmed fine from the VPS too) — narrowed to either
a password transcription error in .env.staging or LVGMC IP-allowlisting
the account, neither confirmable without inspecting the password.
Caught via the overnight watch this was set up for rather than
assumed safe; reverted ENABLE_LVGMC_FTP_JOBS to false on the VPS
immediately once the failure was confirmed real, restoring stable
operation. FTP stays enabled on Rocky, where it works.
Changelog rows for the COALESCE upsert fix, both timezone fixes, the
ENABLE_LVGMC_FTP_JOBS/ENABLE_HARMONIE_JOBS split, and the VPS compose
LVGMC templating fix — none had one yet. Updated the stale "currently
deployed" references (were still e446f1ff) to the actual final
release (3eddf95). Recorded the DMI Harmonie research (no API key
needed, new domain, verified live) and the 5-step verification plan
for actually wiring it up, deferred to a dedicated session. One line
about the VPS's first FTP fetch is marked pending — a background
watch is still confirming it as of this commit; will follow up once
it resolves.
deploy/vps/compose.yml hardcoded LVGMC_USER/PASSWORD/URL to inert
placeholder strings directly in the file, unlike POSTGRES_* which
already read from .env.staging — so simply adding real values to
.env.staging would have had no effect. Switched to the same
${VAR} substitution pattern, and enabled ENABLE_LVGMC_FTP_JOBS now
that real credentials exist. HARMONIE stays on placeholders pending
real DMI credentials.
Real LVGMC FTP credentials arrived today; real DMI HARMONIE
credentials haven't. The combined flag would have enabled both the
moment FTP's were ready, crash-looping the app on HARMONIE's still-
placeholder values via parMapN — caught this before it happened
(the Grib job was ~15 min from its first scheduled run). Split into
ENABLE_LVGMC_FTP_JOBS and ENABLE_HARMONIE_JOBS so each provider can
be enabled independently as its own credentials become real.
Same root cause as the station-observation fix (this uses the same
open-data portal, same UTC DATETIME field): the raw UTC string was
passed straight through to the frontend as a display label, which
JS's Date parser then reads as local time for a string with no
timezone suffix — silently showing observation times 2-3h behind
the newsroom's actual clock. Internal recency filtering (isRecent)
was already self-consistent either way; this only affects display.
Found while wiring up real FTP credentials: the open-data portal's
DATETIME field is UTC, but was being stored into weather.dateTime
as-is with no conversion — while the private FTP feed's "Laiks"
column is already Latvia local time and also stored as-is. Both
paths write the same column, so the table has been silently holding
two timelines 2-3h apart since open-data went live, invisible only
because FTP was never actually running with real data until today.
Verified live: at real local time 20:31 EEST, the fix now correctly
produces dateTime=20:00 instead of 17:00, matching what FTP writes
for the same real hour.
ON CONFLICT DO UPDATE was a blind full-row overwrite. Open-data
station rows always carry null visibilityMin/dewPoint/sunDuration
and an empty phenomena array (fields it doesn't publish), so once a
second source (FTP) writes real values for those fields, a later
open-data write for the same (city, dateTime) would silently null
them back out. Switched to COALESCE(excluded.field, weather.field)
so a missing value from one source never erases a real value the
other already wrote; phenomena needs NULLIF against an empty array
specifically, since Scala's List[String] never maps to SQL NULL.
.symbolPalette/.selectedPreview/.currentSymbol are used by both
Faktiskā's IconInputs.tsx and Brīdinājumi's Warnings.tsx, but were
only defined in mapGraphics.css (Faktiskā's own stylesheet, twice,
with a fully dead first version) and then size-patched for both
consumers via !important overrides in weatherIcons.css. Moved the one
real definition into weatherIcons.css, the file both pages actually
import, and removed the dead/duplicate/!important versions.
No visual change intended — verified via headless-browser screenshot
on Faktiskā and computed-style checks on both pages. One incidental
fix: .currentSymbol.empty's Inter font was silently losing to the
!important rule; normal cascade now applies it correctly again.
Found by a fresh-eyes review after today's design pass: WindInputs
had no productionTemplate branching, so it kept showing "Wind
direction"/"Wind speed"/"Gusts" in English right under the freshly-
translated "Temperatūra un vējš" heading. Threaded the same
productionTemplate flag MapView.tsx already uses for its other
labels. Also removed .assignedSymbol, an unreferenced CSS rule the
same review flagged as dead.
Brīdinājumi was tucked in the "Vairāk" overflow menu despite being a
primary newsroom workflow. Moved it into the always-visible nav
alongside Stacijas/Kartes/Faktiskā/Ūdens, and gave every visible nav
item a small icon (reusing the same icons as their Home cards) so the
header matches the icon treatment the overflow menu already had.
canvas textBaseline "middle" centers on font em-box metrics, not
visible ink — digits have no descenders, so they read as sitting too
high in their boxes. Center on the actual glyph bounds instead,
matching the technique already used correctly in Ūdens's drawRanges.
Shared by Faktiskā and the older Kartes comparison map.
Both "Jaunākās temperatūras" and "Kartes noformējums" dumped their
full content on load, forcing a scroll past both just to reach the
map/export on every visit. Both now collapse by default (Faktiskā
only; the older non-production Kartes overlay panel keeps its current
always-open behavior) behind a toggle that still surfaces a manual-
override count so nothing is silently hidden.
Also fixed a real bug found in the process: Faktiskā's resolution
buttons carried both "resolutionChoices" and "faktiskaResolutionChoices"
classes, and the former (defined in waterTemperature.css for a narrow
sidebar layout) was winning the cascade in this wide-content context,
forcing full-width, left-aligned buttons instead of the intended
compact pair.
Restructured "Kartes noformējums" into distinct bordered cards instead
of one continuous flow, translated the still-English "Weather symbols"
/"City assignments" block to Latvian ("Laikapstākļu simboli"/"Mākoņi
pilsētām"), and collapsed the city list to show only active exceptions
by default with a "Rādīt pārējās pilsētas" button to reveal the rest.
Replaced the flat near-white page background with a layered blue-gray
gradient (from a sampled palette) for actual visual depth, while
keeping buttons/icons/links on the original saturated accent blue so
interactive elements still stand out against the calmer backdrop.
Also removed the "Testa dati" badge on Home, a synthetic-data leftover
now that both Rocky and the VPS run on real data only.
Deployed everything since 6185dbf (Brīdinājumi draggable symbols, real
open-data station/water-temperature ingestion, synthetic-data removal,
scheduler split, METEO_* deletion, real per-zone water-temperature
ranges), flipped ENABLE_SCHEDULED_JOBS=true on the VPS, and wiped the
VPS weather table's synthetic rows after a PostgreSQL backup.
Flips ENABLE_SCHEDULED_JOBS on for the VPS release: the open-data
station and water-temperature paths are proven safe on Rocky.
ENABLE_LEGACY_PROVIDER_JOBS stays off pending real FTP/HARMONIE
credentials.
ARCHITECTURE.md, PRODUCT_WORKFLOWS.md, DEVELOPMENT_AND_STAGING.md, and
README.md still described the removed synthetic-seed staging setup and
Ūdens as pure manual entry. Updates this session (open-data station
ingestion, the scheduler split, water-temperature auto-populate)
weren't reflected outside UPDATE_ROADMAP.md's changelog.
Classify all 65 LVĢMC stations reporting water temperature (56 inland
WTEMD + 9 coastal SEDUT) into the six Ūdens zones by geography, and
report the real min/max across each zone's currently-reporting
stations instead of one hand-picked station's single value shown
twice. Drops readings older than 12h so a stuck sensor can't skew a
zone's range.
Confirmed via git history it's not a separate vendor: the very first
commit (2023-04-13) included real sample CSVs from it with a Latvian
header identical to the LVGMC/open-data fields — same underlying LVĢMC
data, just an earlier delivery mechanism superseded by the FTP feed and
never removed. Never wired into anything that runs (Server.scala's
import was already commented out).
b0txec