From 2ebafc558493d3174f61de5e11abf25c750bd19a Mon Sep 17 00:00:00 2001 From: bot Date: Wed, 8 Apr 2026 08:37:09 +0300 Subject: [PATCH] Document WAL corruption root cause and fix in README Co-Authored-By: Claude Sonnet 4.6 --- README.md | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 75e54aa..feabb2b 100644 --- a/README.md +++ b/README.md @@ -418,10 +418,11 @@ Broadcasting an open unauthenticated hotspot. Next step: connect via Parrot lapt - [x] Dashboard query caps — 300-row limits on Clients, Networks, Cross-node to keep UI responsive - [x] Sessions tab — deauth events grouped into sessions by source MAC + 2-min gap, sortable, expandable rows - [x] Dashboard performance overhaul — fixed tab switch latency and Networks tab failing to load (see below) -- [x] DB pruning / retention policy — background task deletes events older than HOT_DAYS every 6 hours +- [x] DB pruning / retention policy — background task deletes events older than HOT_DAYS (7 days) every 6 hours - [x] Alerts tab — dominant reason code added to Most Impersonated Networks and Most Targeted Devices tables - [x] Sessions tab — dominant reason code + description shown per session row and in expanded detail panel - [x] SSE reconnect bug fix — multiple stale EventSource instances were accumulating on reconnect, flooding /api/nodes on tab re-open; fixed with proper close-before-reconnect and in-flight guard +- [x] WAL corruption fix — recurring DB corruption under sustained write load; fixed with RESTART checkpointing every 30 minutes and reduced retention to 7 days (see below) - [ ] Surface assoc events in dashboard (Search results or dedicated view) - [ ] Scan interval control from dashboard - [ ] Dedicated AP for nodes to isolate reconnect churn from regular network traffic @@ -457,7 +458,33 @@ No pruning meant every query got slower every day as the tables grew. The databa - **Composite indexes added** — added `(bssid, received_at)` on `beacon_events` and `deauth_events`, and `(src_mac, received_at)` on `probe_events`. Queries that filter by time and aggregate by BSSID or MAC now use a single composite index instead of two separate ones. -- **Background pruning task** — at startup, a background coroutine runs every 6 hours and deletes rows older than `HOT_DAYS` (30 days) from all event tables, followed by a WAL checkpoint. The database will no longer grow indefinitely. +- **Background pruning task** — at startup, a background coroutine runs every 6 hours and deletes rows older than `HOT_DAYS` from all event tables. `HOT_DAYS` was later reduced from 30 to 7 — see WAL corruption fix below. + +--- + +## WAL corruption fix (2026-04-08) + +### Problem + +The SQLite database corrupted twice within 3 days of operation, both times producing `database disk image is malformed` errors. The dashboard would start returning partial or no results on some endpoints, and the SSE stream would crash. A full DB rebuild was required each time. + +### Root cause + +SQLite in WAL (Write-Ahead Log) mode works by writing all changes to a separate WAL file first, then periodically folding those changes back into the main database file — a process called checkpointing. The default checkpoint mode is **PASSIVE**, which only checkpoints what it can without blocking any active readers. It gives up immediately if a reader is present. + +With 4 nodes generating hundreds of events per minute and the dashboard constantly reading (SSE polling every 2 seconds, plus user interactions), there is almost never a clean window for a passive checkpoint to complete. The WAL file accumulates uncommitted state indefinitely, grows too large, and eventually corrupts the main database file. The prune task added in the earlier performance overhaul also used PASSIVE mode, so it had the same problem. + +A second contributing factor: `HOT_DAYS` was set to 30 days. At the observed data rate (~500k beacon events/day, ~115k probe events/day), the database would grow to several gigabytes before the first prune cycle deleted anything, making the WAL problem worse over time. + +### Fixes applied + +- **RESTART checkpointing** — replaced PASSIVE with `PRAGMA wal_checkpoint(RESTART)` in a new `checkpoint_wal()` function. RESTART waits for all current readers to finish, then checkpoints all WAL frames into the main DB file and resets the WAL back to the beginning. This guarantees the WAL is fully flushed rather than partially checkpointed. + +- **Dedicated checkpoint task** — a new background coroutine runs `checkpoint_wal()` every 30 minutes, independent of the prune cycle. The prune task also calls it after deleting rows, so a large delete is always followed by a full WAL flush. + +- **`wal_autocheckpoint=500`** — set on every connection so SQLite's own automatic checkpoint triggers at ~500 pages (~2MB) instead of the default 1000 pages (~4MB). This gives more frequent small opportunities to checkpoint between the 30-minute forced runs. + +- **HOT_DAYS reduced from 30 to 7** — at current data rates, 7 days of retention keeps the database under ~200MB permanently. Queries stay fast, the WAL stays manageable, and 7 days of history is sufficient for all dashboard views and active investigations. ---