Caught by an independent follow-up review: the FTP auth-bypass fix in
6b9c7cf only gated /api/fetch/lvgmc/stations, the one route the original
security review named. This sibling route calls the same fetch.fetchFile,
which opens a real, unauthenticated FTP login to LVGMC regardless of any
caller-supplied filename, and was only getting the traversal fix
(ValidateFileName) applied to it, not the auth-bypass fix. Traced every
caller of fetch.fetchFile/fetchWeatherStations in Server.scala/Main.scala
this time to confirm these are the only two HTTP-reachable call sites and
both are now gated.
field on that route reached PostgresService.query unvalidated, which splices
it into SQL via Fragment.const (unescaped) whenever the aggregate key is
min/max/avg/sum/distinct, or whenever granularity is "hour" in the list
branch. Add ValidateField (allowlists WeatherData's known field names, same
pattern AggFieldList already uses for /query/country) and apply it to the
field path segment. AggregateKey values reaching Fragment.const elsewhere are
already safe since they come from a closed ADT, not raw user input.
- Add ValidateFileName (allowlist regex, rejects .. and separators) and apply
it to every route that concatenates a raw path segment into a filesystem or
remote FTP path: /show/lvgmc-forecast, /show/grib, /grib/binary-chunk, and
/debug/file. Previously an unauthenticated caller could read arbitrary
files, including /proc/self/environ (leaks LVGMC_PASSWORD/POSTGRES_PASSWORD).
- Harden ValidateInt to reject negative integers.
- Gate /api/fetch/lvgmc/stations behind ENABLE_LVGMC_FTP_JOBS so it can no
longer trigger a real, unauthenticated FTP login regardless of the flag;
stop leaking error.getMessage in its response.
- Add an explicit /api/* catch-all (NotFound) so an unmatched API route can
never fall through to the SPA fallback and be served index.html as a 200.
- Cap binary-chunk read length at 64MB to prevent an unbounded allocation.
Direct hits on client-side routes not in the hardcoded list (e.g. a
browser refresh on /faktiska or /udens-temperatura) 404ed instead of
loading the app — a known limitation that was actually hit in
production. Real files now serve as-is; anything else falls back to
index.html so the SolidJS router handles it, matching the existing
"TODO rewrite in more generic way" comment. Future routes need no
backend changes.
Real LVGMC FTP credentials arrived today; real DMI HARMONIE
credentials haven't. The combined flag would have enabled both the
moment FTP's were ready, crash-looping the app on HARMONIE's still-
placeholder values via parMapN — caught this before it happened
(the Grib job was ~15 min from its first scheduled run). Split into
ENABLE_LVGMC_FTP_JOBS and ENABLE_HARMONIE_JOBS so each provider can
be enabled independently as its own credentials become real.
Same root cause as the station-observation fix (this uses the same
open-data portal, same UTC DATETIME field): the raw UTC string was
passed straight through to the frontend as a display label, which
JS's Date parser then reads as local time for a string with no
timezone suffix — silently showing observation times 2-3h behind
the newsroom's actual clock. Internal recency filtering (isRecent)
was already self-consistent either way; this only affects display.
Found while wiring up real FTP credentials: the open-data portal's
DATETIME field is UTC, but was being stored into weather.dateTime
as-is with no conversion — while the private FTP feed's "Laiks"
column is already Latvia local time and also stored as-is. Both
paths write the same column, so the table has been silently holding
two timelines 2-3h apart since open-data went live, invisible only
because FTP was never actually running with real data until today.
Verified live: at real local time 20:31 EEST, the fix now correctly
produces dateTime=20:00 instead of 17:00, matching what FTP writes
for the same real hour.
ON CONFLICT DO UPDATE was a blind full-row overwrite. Open-data
station rows always carry null visibilityMin/dewPoint/sunDuration
and an empty phenomena array (fields it doesn't publish), so once a
second source (FTP) writes real values for those fields, a later
open-data write for the same (city, dateTime) would silently null
them back out. Switched to COALESCE(excluded.field, weather.field)
so a missing value from one source never erases a real value the
other already wrote; phenomena needs NULLIF against an empty array
specifically, since Scala's List[String] never maps to SQL NULL.
Classify all 65 LVĢMC stations reporting water temperature (56 inland
WTEMD + 9 coastal SEDUT) into the six Ūdens zones by geography, and
report the real min/max across each zone's currently-reporting
stations instead of one hand-picked station's single value shown
twice. Drops readings older than 12h so a stuck sensor can't skew a
zone's range.
Confirmed via git history it's not a separate vendor: the very first
commit (2023-04-13) included real sample CSVs from it with a Latvian
header identical to the LVGMC/open-data fields — same underlying LVĢMC
data, just an earlier delivery mechanism superseded by the FTP feed and
never removed. Never wired into anything that runs (Server.scala's
import was already commented out).
b0txec