b0txec
212 lines
17 KiB
Markdown
212 lines
17 KiB
Markdown
# WeatherTool update roadmap
|
||
|
||
This document tracks proposed WeatherTool improvements. Work should be delivered in small, reviewable phases rather than as one large rewrite. Each phase should leave the application runnable and independently testable.
|
||
|
||
## Environments and workflow
|
||
|
||
- **Windows source workspace:** source editing, review, and Git operations only; do not install dependencies, compile, build, run, or test here.
|
||
- **Rocky development and staging:** the sole compile, build, development-runtime, and test environment, with production-like Docker staging at `http://192.168.1.101:9190`.
|
||
- **Git over SSH:** Windows pushes reviewed commits to a private bare repository on Rocky; the Rocky staging checkout pulls those commits and rebuilds.
|
||
- **Ubuntu VPS deployment:** release `ef64895` is publicly operational behind Cloudflare strict TLS, Nginx, and Authelia with a limited 14-day synthetic dataset; the VPS does not compile or build the project.
|
||
- **Workplace production:** remains separate until changes are reviewed, tested, and explicitly approved for workplace use.
|
||
|
||
Do not synchronize `.env`, database directories, generated dependencies, build output, or provider credentials between machines.
|
||
|
||
## Working principles
|
||
|
||
1. Make one coherent change at a time.
|
||
2. Record the existing behavior before intentionally changing it.
|
||
3. Keep dependency updates separate from UI redesign and functional changes.
|
||
4. Review and commit on Windows, then deploy and test the same commit on Rocky staging.
|
||
5. Use synthetic or sanitized data outside the workplace environment.
|
||
6. Never enable external provider schedules with placeholder credentials.
|
||
7. Do not connect staging to workplace services without explicit authorization.
|
||
|
||
## Phase 0 — Reproducible development baseline
|
||
|
||
Status: in progress
|
||
|
||
- [x] Review backend, frontend, deployment, and security structure.
|
||
- [x] Run the project locally through Docker Desktop.
|
||
- [x] Add configurable host port and scheduled-job switch.
|
||
- [x] Add deterministic synthetic station data for the repository station set and the Faktiskā-required Valmiera position.
|
||
- [x] Create an isolated Rocky Linux staging deployment.
|
||
- [x] Keep staging PostgreSQL private to its Compose network.
|
||
- [x] Commit the baseline changes and establish the Git-over-SSH workflow.
|
||
- [x] Document normal build, seed, frontend deploy, source rollback, and current operational limitations. (Database backup/restore remains pending.)
|
||
- [ ] Capture representative screenshots and expected API responses.
|
||
|
||
## Phase 1 — Behavior discovery and bug inventory
|
||
|
||
Status: in progress
|
||
|
||
- [ ] Walk through every page with synthetic data.
|
||
- [x] Document the current understood purpose of each workflow; validate it with workplace users over time.
|
||
- [x] Separate analytical dashboard features from broadcast-graphic authoring tools on the city-results map.
|
||
- [ ] Record unclear controls, missing units, broken states, and layout problems.
|
||
- [ ] Fix the `atmPressire` frontend field typo.
|
||
- [ ] Fix shifted Database export columns caused by duplicated `tempMax`.
|
||
- [ ] Correct malformed integer route handling.
|
||
- [ ] Add consistent loading, empty, and error states. (City selection empty state completed.)
|
||
|
||
## Phase 2 — Test safety net
|
||
|
||
Status: in progress
|
||
|
||
- [ ] Add backend route tests for representative station and country queries.
|
||
- [ ] Add database integration tests for aggregation and export behavior.
|
||
- [ ] Restore and expand CSV parser tests.
|
||
- [ ] Add GRIB parser boundary and malformed-file tests.
|
||
- [ ] Add security tests for invalid fields, filenames, offsets, lengths, and date ranges.
|
||
- [ ] Add frontend type checking and critical workflow smoke tests.
|
||
- [ ] Run tests automatically before staging deployment.
|
||
|
||
## Phase 3 — Dependency modernization
|
||
|
||
Status: in progress — frontend maintenance complete; backend maintenance pending
|
||
|
||
The first observed frontend install reported 15 vulnerabilities: 1 critical, 10 high, 3 moderate, and 1 low. The frontend dependency tree was reviewed in controlled groups, obsolete packages were removed, and both the complete and production-only npm audits now report zero known vulnerabilities at commit `de6f279`.
|
||
|
||
- [x] Capture and review the full npm audit report.
|
||
- [x] Establish that the production dependency audit is primarily blocked by `solid-js@1.9.4` resolving vulnerable `seroval@1.2.0`; confirm that the full audit also contains development-tool advisories.
|
||
- [x] Update direct frontend dependencies in controlled groups.
|
||
- [x] Replace or remove obsolete frontend packages where appropriate.
|
||
- [x] Add a committed TypeScript typecheck command and validate the updated frontend with a clean `npm ci`, typecheck, and production build.
|
||
- [x] Resolve the remaining transitive build-tool advisories after reviewing the proposed `npm audit fix` changes; verify full and production-only audits at zero.
|
||
- [x] Build, smoke-test, checksum, transfer, and deploy a commit-addressed VPS release containing `de6f279` or later without restarting PostgreSQL or Authelia. (`ef64895` deployed 2026-08-22.)
|
||
- [ ] Visually compare every page and representative exported PNG in the deployed dependency-maintenance release.
|
||
- [ ] Update Scala within the supported 2.13 line before considering larger migration.
|
||
- [ ] Update http4s, Doobie, Circe, Cats Effect, Logback, and test libraries in compatible groups.
|
||
- [ ] Replace release-candidate dependencies with stable releases where possible.
|
||
- [ ] Update Docker base images deliberately and pin reproducible versions.
|
||
- [ ] Verify database compatibility and generated artifacts after every group.
|
||
|
||
Dependency changes must not be combined with a visual redesign unless a package migration strictly requires it.
|
||
|
||
## Phase 4 — Security hardening
|
||
|
||
Status: pending
|
||
|
||
- [ ] Rotate and remove the API key exposed in a source comment.
|
||
- [ ] Remove credentials from connection-error messages.
|
||
- [ ] Protect or remove debug and administrative endpoints.
|
||
- [ ] Convert state-changing `GET` routes to appropriate methods.
|
||
- [ ] Introduce closed, validated weather-field and aggregation types.
|
||
- [ ] Eliminate raw user-controlled SQL identifiers.
|
||
- [ ] Validate and constrain filenames, resolved paths, offsets, and byte lengths.
|
||
- [ ] Add query-range, response-size, request-rate, and timeout limits.
|
||
- [ ] Restrict CORS to intended origins.
|
||
- [ ] Define authentication and authorization requirements for workplace deployment.
|
||
|
||
## Phase 5 — Runtime reliability and operations
|
||
|
||
Status: pending
|
||
|
||
- [ ] Manage custom executors as resources and close them cleanly.
|
||
- [ ] Remove explicit `System.gc()` calls.
|
||
- [ ] Supervise scheduled jobs independently instead of recursively restarting the application.
|
||
- [ ] Add application health and readiness endpoints.
|
||
- [ ] Add structured logging without leaking secrets.
|
||
- [ ] Define PostgreSQL and GRIB-data backup/restore procedures.
|
||
- [ ] Add container resource limits and deployment health checks.
|
||
- [ ] Document monitoring, update, rollback, and incident procedures.
|
||
|
||
## Phase 6 — Information architecture and UI redesign
|
||
|
||
Status: pending
|
||
|
||
- [ ] Identify primary user roles and their most frequent tasks.
|
||
- [ ] Separate historical analysis, live station monitoring, database inspection, HARMONIE visualization, and broadcast graphics.
|
||
- [ ] Replace technical/internal labels with task-oriented language.
|
||
- [x] Replace the copy/paste weather-character workflow with a direct visual font-glyph picker, bulk assignment, and city exceptions.
|
||
- [x] Explain and visually separate manual wind and weather-icon inputs from queried data.
|
||
- [ ] Add units, legends, contextual help, and clear date semantics.
|
||
- [x] Establish the first responsive layout, typography, spacing, and component-system foundation.
|
||
- [x] Add an explicit PNG download workflow for broadcast map assets.
|
||
- [x] Implement the fixed Faktiskā station set, latest-temperature loading, manual overrides, and locked 3840×1440 export.
|
||
- [x] Make Faktiskā weather-symbol selection manual while keeping placement automatic and badge-relative.
|
||
- [x] Replace Daira font glyphs with normalized 256×256 transparent image assets across the editor and canvas renderers.
|
||
- [x] Add a separate Latvian-named Ūdens workspace with fixed manual fields.
|
||
- [x] Calibrate and visually validate independent Ūdens exports at 1920×1080 and 3840×1440 against the supplied production templates.
|
||
- [x] Bundle the confirmed Monda Regular/Bold files and replace temporary Rubik rendering.
|
||
- [ ] Perform pixel-level Monda comparison against the authoritative production masters.
|
||
- [x] Add a separate Latvian-named Brīdinājumi workspace that renders current LVĢMC warning polygons without manual tracing.
|
||
- [x] Add production border overlay artwork and feathered warning-polygon fills to Brīdinājumi.
|
||
- [x] Replace Brīdinājumi's 4-corner bounding-box projection with an affine fit calibrated against the same validated Kartes/Faktiskā city pixel positions.
|
||
- [x] Limit the Brīdinājumi canvas to one operator-selected warning, remove the on-canvas legend, collapse card detail to phenomenon/severity by default with full detail on click, and pre-fill the title from the selected warning's phenomenon.
|
||
- [x] Redesign Brīdinājumi warning cards as compact phenomenon-icon chips and move full warning text into a dialog popup over a dimmed, blurred backdrop.
|
||
- [ ] Tune Brīdinājumi feather-blur bleed at sharp coastline curves (e.g. the Gulf of Rīga indentation).
|
||
- [ ] Test target resolutions and real workplace display conditions.
|
||
- [ ] Check keyboard navigation, contrast, focus states, and screen-reader labeling.
|
||
|
||
## Phase 7 — Real data and production readiness
|
||
|
||
Status: pending
|
||
|
||
- [ ] Confirm the actual workplace deployment topology and current deployed commit.
|
||
- [ ] Obtain authorized development credentials or representative fixtures.
|
||
- [ ] Validate LVGMC station and forecast CSV ingestion.
|
||
- [ ] Validate DMI STAC discovery and EDR GRIB downloads.
|
||
- [ ] Test scheduled ingestion failure and recovery behavior.
|
||
- [ ] Rehearse deployment and rollback using sanitized data.
|
||
- [ ] Obtain technical and operational review before workplace rollout.
|
||
|
||
## Phase 8 — Temporary VPS user acceptance
|
||
|
||
Status: in progress
|
||
|
||
- [x] Document the proposed isolated VPS topology and trusted Rocky-to-VPS release flow.
|
||
- [x] Choose `laikapstak.li` and `auth.laikapstak.li`, create their proxied Cloudflare DNS records, and activate Cloudflare delegation.
|
||
- [x] Record a read-only VPS inventory before provisioning and confirm ports, networks, storage, and capacity do not collide with existing services.
|
||
- [x] Provision a dedicated `/srv/weathertool` tree, Compose project, private network, database storage, and loopback-only application and Authelia ports.
|
||
- [x] Configure Authelia with one temporary shared account, Argon2id password storage, rate limiting, and temporary IP bans.
|
||
- [x] Add an exact-host Cloudflare origin certificate and Nginx authorization routing without disrupting the existing HOP site.
|
||
- [x] Build and verify commit-addressed images on Rocky, transfer and checksum-verify them on the VPS, and update only the application service. Current full-SHA image: `weathertool:df911f14eb8a1041822959677e09bb544ce0cdcd`; PostgreSQL and Authelia were not restarted.
|
||
- [ ] Complete UAT verification: public authentication and application health pass; direct-origin blocking, logout, throttling, PNG downloads, backups, logs, and rollback remain.
|
||
- [ ] Run the month-long user test with manual releases and record feedback before any workplace-production decision.
|
||
|
||
## Known current limitations
|
||
|
||
- The fixed production PNG bases, browser/social metadata, reviewed frontend dependency updates, stabilized header navigation, and authenticated API routing are deployed in UAT release `df911f1`; newsroom workflow validation is in progress.
|
||
- Link-preview crawlers cannot authenticate through Authelia; the final Nginx policy must deliberately expose only the minimum preview metadata/assets if WhatsApp previews are required.
|
||
|
||
- Staging uses synthetic PostgreSQL station data.
|
||
- LVGMC forecast CSV fixtures are not yet available.
|
||
- HARMONIE GRIB fixtures are not yet available.
|
||
- Scheduled provider downloads are disabled in development and staging.
|
||
- Existing automated test coverage is minimal.
|
||
- Direct refreshes on newer frontend routes can return 404 until the backend gains a general SPA fallback.
|
||
- Full Docker build context scanning on Rocky can fail on the container-owned `postgres/` bind directory; do not loosen its permissions.
|
||
|
||
## Change log
|
||
|
||
Record completed work here by date and commit after the Git workflow is established.
|
||
|
||
| Date | Commit | Summary | Verified on Rocky |
|
||
|---|---|---|---|
|
||
| 2026-08-18 | `b1fff67` | Docker development baseline, isolated staging, scheduler switch, and synthetic station data | Yes |
|
||
| 2026-08-18 | `bb1d9fc` | Home/UI foundation, map preview layout, friendly empty state, and PNG export | Yes |
|
||
| 2026-08-18 | `67decb3` | Adaptive temperature badges and title/source overlays | Yes |
|
||
| 2026-08-18 | `82199b8` | Separate City Analysis and Faktiskā workflows with direct symbol assignment | Yes |
|
||
| 2026-08-18 | `0c78d3c` | Optical centering for bundled weather-font glyphs | Pending Rocky verification |
|
||
| 2026-08-19 | `e726291` | Fixed Faktiskā workflow with latest observations, manual overrides, and production-size export | Yes |
|
||
| 2026-08-19 | `5913217` | Add Valmiera to the deterministic Faktiskā development data | Yes |
|
||
| 2026-08-19 | `632f377` | Arrange manual Faktiskā wind controls horizontally | Yes |
|
||
| 2026-08-19 | `340c370`–`3aba9b4` | Anchor and visually calibrate manually selected Faktiskā symbols | Yes |
|
||
| 2026-08-19 | `1868041`–`6c9290b` | Add Ūdens temperatūra, bundle and load Monda, fix overlay scaling, and center its locked nameplate | Yes |
|
||
| 2026-08-19 | `0d641bd` | Replace runtime Daira glyph rendering with normalized transparent image assets | Yes |
|
||
| 2026-08-20 | `754415a` | Match editable title/source overlays to production-safe right margins | Yes |
|
||
| 2026-08-20 | `16a8c68` | Deploy approved production map bases, normalized branding assets, favicon, Apple icon, and social metadata to public UAT | Yes — Rocky build and VPS/browser smoke checks |
|
||
| 2026-08-21 | `5007517`–`de6f279` | Update Solid runtime and frontend build tooling, add type checking, remove obsolete packages, prune the lockfile, and resolve all npm advisories | Yes — clean `npm ci`, typecheck, production build, full audit, and production-only audit |
|
||
| 2026-08-22 | `ef64895` | Deploy the reviewed zero-advisory frontend dependency state as an immutable full-SHA VPS release while leaving PostgreSQL, Authelia, and HOP uninterrupted | Yes — isolated Rocky smoke test, dual-host checksum, matching image ID, container health, loopback, authentication-gate, and HOP checks |
|
||
| 2026-08-22 | `36d094e` | Stabilize the header, keep the primary workspaces visible, and move secondary workspaces into an icon-labelled menu | Yes — Rocky browser and Jam navigation checks |
|
||
| 2026-08-22 | `df911f1` | Preserve API 401 responses through Authelia/Nginx so frontend queries do not follow cross-origin login redirects | Yes — Rocky and VPS health checks; authenticated browser query verified after refreshing synthetic data |
|
||
| 2026-08-22 | `f3197bf` | Return query API payloads with an explicit JSON content type | Yes — Rocky API and browser verification; VPS deployment pending |
|
||
| 2026-08-22 | `0608865` | Localize the operator-facing workspace names and concise workflow copy in Latvian | Yes — Rocky browser verification; VPS deployment pending |
|
||
| 2026-08-22 | `7f09181` | Independently calibrate both Ūdens templates and optically center Monda values using visible glyph bounds | Yes — both native-resolution PNG exports visually validated on Rocky; VPS deployment pending |
|
||
| 2026-08-22 | `bc5bee1` | Add the Brīdinājumi LVĢMC warning-map workflow with metadata/polygon ingestion, day/phenomenon filtering, and manual title entry | Yes — Rocky browser verification; VPS deployment pending |
|
||
| 2026-08-22 | `13115f0` | Add production border overlay artwork and feathered warning-polygon fills to Brīdinājumi | Yes — Rocky browser verification; VPS deployment pending |
|
||
| 2026-08-22 | `4395935` | Replace Brīdinājumi's 4-corner bounding-box projection, which drifted up to ~200px on the 3840 canvas, with an affine fit calibrated against the validated Kartes/Faktiskā city pixel positions | Yes — Rocky typecheck, build, and headless-browser screenshot comparison at both native export resolutions; VPS deployment pending |
|
||
| 2026-08-22 | `924e410` | Limit Brīdinājumi to one selected warning per export, remove the on-canvas legend, collapse card detail until clicked, and pre-fill the title from the selected warning's phenomenon | Yes — Rocky typecheck, build, and headless-browser screenshot verification of selection switching and card detail; VPS deployment pending |
|
||
| 2026-08-22 | `cafc75a` | Redesign Brīdinājumi warning cards as compact phenomenon-icon chips and move full warning text into a dialog popup over a dimmed, blurred backdrop | Yes — Rocky typecheck, build, and headless-browser screenshot verification of the popup open/close flow; VPS deployment pending |
|