Revert VPS FTP after it crash-looped the app; document the diagnosis
The ltv account fails to authenticate from the VPS specifically (works fine from Rocky with the same credentials, network path to ftp.meteo.lv:21 confirmed fine from the VPS too) — narrowed to either a password transcription error in .env.staging or LVGMC IP-allowlisting the account, neither confirmable without inspecting the password. Caught via the overnight watch this was set up for rather than assumed safe; reverted ENABLE_LVGMC_FTP_JOBS to false on the VPS immediately once the failure was confirmed real, restoring stable operation. FTP stays enabled on Rocky, where it works.
This commit is contained in:
b0txec
@@ -32,10 +32,15 @@ services:
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
POSTGRES_HOST: postgres
|
||||
ENABLE_SCHEDULED_JOBS: "true"
|
||||
ENABLE_LVGMC_FTP_JOBS: "true"
|
||||
# Briefly "true" on 2026-08-23, reverted same night: the ltv account
|
||||
# fails to authenticate from the VPS specifically (works fine from
|
||||
# Rocky), which crash-looped the whole app in-process via parMapN
|
||||
# before this was caught and reverted. Leave false until the VPS-side
|
||||
# auth failure is diagnosed and re-verified — see docs/UPDATE_ROADMAP.md.
|
||||
ENABLE_LVGMC_FTP_JOBS: "false"
|
||||
ENABLE_HARMONIE_JOBS: "false"
|
||||
DEBUG: "false"
|
||||
# LVGMC_* now read real values from .env.staging (real FTP credentials
|
||||
# LVGMC_* read real values from .env.staging (real FTP credentials
|
||||
# obtained 2026-08-23). HARMONIE stays on inert placeholders until real
|
||||
# DMI credentials exist — the provider service is still constructed at
|
||||
# startup even with its scheduled job off, so it needs some value.
|
||||
|
||||
+3
-2
@@ -6,7 +6,7 @@ This directory contains the working documentation for the WeatherTool modernizat
|
||||
|
||||
- Windows is restricted to source editing, review, and Git operations. Rocky is the sole compile, build, development-runtime, and test environment. The Ubuntu VPS is a deployment target only.
|
||||
- A production-like staging copy runs through Docker Compose on Rocky Linux at `http://192.168.1.101:9190`.
|
||||
- Both Rocky and the VPS now ingest real data from two independent LVĢMC sources at once: the free `data.gov.lv` open-data feed and the private FTP feed (`ftp.meteo.lv`, real credentials obtained and verified 2026-08-23). The two are gated independently (`ENABLE_LVGMC_FTP_JOBS`, `ENABLE_HARMONIE_JOBS`) since real credentials for FTP and DMI HARMONIE arrive on different timelines — FTP is on; HARMONIE's credentials turned out not to be needed at all (DMI dropped its API key requirement — confirmed live, see the roadmap changelog) but wiring it up is deferred to a dedicated verification session, since the GRIB-parsing and map-rendering code has real, untested risk (a hardcoded crop/rotation calibration that may not match the current model grid).
|
||||
- Rocky now ingests real data from two independent LVĢMC sources at once: the free `data.gov.lv` open-data feed and the private FTP feed (`ftp.meteo.lv`, real credentials obtained and verified 2026-08-23). The VPS currently runs open-data only — FTP was briefly enabled there too but reverted the same night after the `ltv` account failed to authenticate from the VPS specifically (see below). The two providers are gated independently (`ENABLE_LVGMC_FTP_JOBS`, `ENABLE_HARMONIE_JOBS`) since real credentials for FTP and DMI HARMONIE arrive on different timelines. HARMONIE's credentials turned out not to be needed at all (DMI dropped its API key requirement — confirmed live, see the roadmap changelog) but wiring it up is deferred to a dedicated verification session, since the GRIB-parsing and map-rendering code has real, untested risk (a hardcoded crop/rotation calibration that may not match the current model grid).
|
||||
- A real UTC-vs-local timezone mismatch between the two sources (open-data's timestamps were UTC, FTP's already local, both stored in the same column with no conversion) was found and fixed before enabling both together; both `weather` tables were backed up and wiped for a clean, consistently-timestamped restart.
|
||||
- The safe scheduled jobs (open-data station ingestion, GRIB cleanup) always run; the FTP job now runs too since real credentials exist; HARMONIE's job stays off pending its own implementation work.
|
||||
- PostgreSQL is private to the project Compose network; only the Scala application publishes a host port.
|
||||
@@ -21,7 +21,8 @@ This directory contains the working documentation for the WeatherTool modernizat
|
||||
- The VPS `weather` table was wiped twice on 2026-08-23: first to remove the original 14-day synthetic dataset (backed up to `/srv/weathertool/backups/pre-real-data-release/`), then again after the UTC/local timezone fix (backed up to `/srv/weathertool/backups/pre-timezone-fix-wipe/`) so the two real sources it now holds — open-data (minutes 15/45) and FTP (minutes 11/13/23/30) — are consistently timestamped from a clean start.
|
||||
- Approved 1920×1080 and 3840×1440 PNG production bases are now the rendering source for Faktiskā and Ūdens temperatūra; code draws only the changing values, selected weather symbols, and wind data over those fixed newsroom graphics.
|
||||
- Browser branding assets and Latvian Open Graph/Twitter metadata are included for favicon, Apple home-screen icon, and link-preview support. Public crawler access still depends on the Nginx/Authelia policy used for the metadata and preview image.
|
||||
- Browser and API verification is complete for the deployed `3eddf95` release: exact release image smoke-tested on Rocky before transfer (bundle hash and correct-local-time API responses matched the known-good local build), checksum verified on both ends, container health/loopback/public HTTPS confirmed. VPS's first scheduled FTP fetch specifically (not just open-data, which fires more often) is being watched for overnight to confirm outbound FTP actually works from the VPS network before relying on it unattended. Newsroom testers will be given the domain first, without walkthrough, to see how intuitive the workflow is before any guidance is provided.
|
||||
- Browser and API verification is complete for the deployed `3eddf95` release: exact release image smoke-tested on Rocky before transfer (bundle hash and correct-local-time API responses matched the known-good local build), checksum verified on both ends, container health/loopback/public HTTPS confirmed.
|
||||
- **VPS FTP is currently disabled again** (`ENABLE_LVGMC_FTP_JOBS=false`), despite being enabled earlier tonight. Watching for its first scheduled fetch (rather than assuming it was safe just because Rocky worked) caught a real problem: the `ltv` account fails to authenticate from the VPS specifically, and because a scheduled-task failure cascades through `parMapN` up to the app's top-level error handler, this was actively crash-looping the entire app in-process (not a Docker-level restart, but the whole app — DB, HTTP server, schedulers — tearing down and rebuilding every ~5s, repeating every ~2 minutes as the schedule re-fired). Reverted immediately; confirmed stable afterward. VPS network connectivity to `ftp.meteo.lv:21` itself is fine (verified with a raw connection test); this is specifically the `ltv` account failing from the VPS's IP, not a firewall issue — see the roadmap changelog for the full diagnosis and next steps (check the password transcription, or ask LVGMC whether the account is IP-restricted). FTP stays enabled on Rocky, where it's confirmed working.
|
||||
- Frontend dependency maintenance is complete: Solid runtime and Vite tooling were updated, obsolete packages were removed, TypeScript checking was added, and a clean Rocky `npm ci`, typecheck, production build, full audit, and production-only audit all pass with zero known vulnerabilities.
|
||||
- This is not yet approved or hardened for workplace production.
|
||||
|
||||
|
||||
+39
-9
@@ -7,7 +7,7 @@ This document tracks proposed WeatherTool improvements. Work should be delivered
|
||||
- **Windows source workspace:** source editing, review, and Git operations only; do not install dependencies, compile, build, run, or test here.
|
||||
- **Rocky development and staging:** the sole compile, build, development-runtime, and test environment, with production-like Docker staging at `http://192.168.1.101:9190`.
|
||||
- **Git over SSH:** Windows pushes reviewed commits to a private bare repository on Rocky; the Rocky staging checkout pulls those commits and rebuilds.
|
||||
- **Ubuntu VPS deployment:** release `3eddf95` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia, ingesting real LVĢMC open-data station/water-temperature observations on a schedule, plus real LVGMC FTP station data as of 2026-08-23 (`ENABLE_LVGMC_FTP_JOBS=true`, real credentials); the `weather` table was wiped and re-populated fresh once the UTC/local timezone mismatch between the two sources was fixed. The VPS does not compile or build the project.
|
||||
- **Ubuntu VPS deployment:** release `3eddf95` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia, ingesting real LVĢMC open-data station/water-temperature observations on a schedule; the `weather` table was wiped and re-populated fresh once the UTC/local timezone mismatch between sources was fixed. FTP (`ENABLE_LVGMC_FTP_JOBS`) was briefly enabled 2026-08-23 but reverted to `false` the same night after the `ltv` account failed to authenticate from the VPS specifically (crash-looped the app in-process until caught and reverted) — real credentials work fine from Rocky, not yet from the VPS; see the changelog for full diagnosis. The VPS does not compile or build the project.
|
||||
- **Workplace production:** remains separate until changes are reviewed, tested, and explicitly approved for workplace use.
|
||||
|
||||
Do not synchronize `.env`, database directories, generated dependencies, build output, or provider credentials between machines.
|
||||
@@ -241,14 +241,44 @@ running in parallel until each real source is proven, not cut over in one step.
|
||||
had no effect until this was switched to the same `${VAR}` substitution
|
||||
pattern. VPS now runs release `3eddf95`/`dc04f66` with real FTP
|
||||
credentials (added directly on the VPS by the user, never typed into
|
||||
this session) and `ENABLE_LVGMC_FTP_JOBS=true`; watching the container
|
||||
logs for VPS's first scheduled FTP fetch specifically (not just the
|
||||
open-data job, which fires more often and was initially mistaken for
|
||||
it on one premature check) to confirm real rows land before leaving it
|
||||
unattended overnight — outbound FTP from the VPS network is a real,
|
||||
non-hypothetical risk worth checking, not something safe to assume
|
||||
just because it worked from Rocky. **Result pending as of this
|
||||
writing — update this line once confirmed.**
|
||||
this session) and `ENABLE_LVGMC_FTP_JOBS=true`. Good thing this was
|
||||
watched instead of assumed safe: the VPS's first scheduled FTP fetch
|
||||
(19:13 UTC) failed with `Failed to login to FTP server`, and because
|
||||
`parMapN` cancels every sibling task the instant one throws, this
|
||||
cascaded all the way up through `Main.scala`'s top-level
|
||||
`handleErrorWith` — which tears down and rebuilds the *entire* app
|
||||
(DB connections, HTTP server, schedulers) and retries every ~5s. Not a
|
||||
Docker-level restart (`RestartCount` stayed `0`, since the JVM process
|
||||
itself never exited) but a real in-process crash-loop on the public
|
||||
site, repeating every ~2 minutes as the FTP schedule re-fired. Reverted
|
||||
`ENABLE_LVGMC_FTP_JOBS` to `false` on the VPS immediately (the user was
|
||||
already asleep; this is undoing a flag flipped minutes earlier as part
|
||||
of the same authorized change, not new destructive action, so didn't
|
||||
wait) — confirmed stable afterward, no fatal errors, public site
|
||||
responding normally again.
|
||||
|
||||
Diagnosed before reverting further: from the VPS, `ftp.meteo.lv:21` is
|
||||
reachable and returns a normal vsFTPd banner, and an anonymous login
|
||||
attempt correctly gets rejected (`530 Login incorrect`) — so this is
|
||||
**not** a network/firewall problem, outbound FTP works fine from the
|
||||
VPS. The real `ltv` account specifically fails to authenticate from
|
||||
the VPS despite working from Rocky a couple of hours earlier with the
|
||||
same credentials. Two candidate causes, neither confirmed (didn't
|
||||
inspect the actual password value to check):
|
||||
1. A transcription error when the password was typed into the VPS's
|
||||
`.env.staging` (manual entry, real risk for a long decoded string).
|
||||
2. LVGMC IP-allowlists the `ltv` account to specific source addresses
|
||||
— common for partner/business FTP accounts — and the VPS's outbound
|
||||
IP isn't on that list even though Rocky's apparently is. The VPS's
|
||||
outbound IP is `57.128.250.38`, in case it needs to be given to the
|
||||
LVGMC contact for allowlisting.
|
||||
**Next step (morning, user): double-check the password in
|
||||
`/srv/weathertool/.env.staging` on the VPS character-for-character
|
||||
against the decoded value; if it's correct, ask the LVGMC contact
|
||||
whether the `ltv` account is IP-restricted and whether the VPS's
|
||||
outbound IP needs adding.** FTP stays on for Rocky (working there);
|
||||
VPS runs open-data only until this is resolved and re-verified the
|
||||
same careful way — watch-and-confirm, not assume-and-move-on.
|
||||
- [x] Researched DMI HARMONIE credentials while waiting on the above and
|
||||
confirmed live (not just from search results, which claimed a specific
|
||||
date that wasn't independently verified): DMI's Forecast Data EDR and
|
||||
|
||||
Reference in New Issue
Block a user