Add BLE layer: firmware, coordinator support, dashboard tab

2026-04-27 14:21:02 +03:00
parent 2ebafc5584
commit 1ca2dc21f5
8 changed files with 807 additions and 13 deletions
@@ -31,12 +31,14 @@ This is a learning/research project covering distributed systems, event-driven a

 ### Active nodes

-| node_id  | MAC               | Location                     |
-|----------|-------------------|------------------------------|
-| F68D6E30 | 44:1b:f6:8d:6e:30 | room 1 (permanent)           |
-| A1D658D4 | e0:72:a1:d6:58:d4 | room 2 (permanent)           |
-| A1D700C4 | e0:72:a1:d7:00:c4 | room 3 (permanent)           |
-| A1D6F190 | e0:72:a1:d6:f1:90 | dev machine (desk, USB)      |
+| node_id  | MAC               | Firmware | Location                     |
+|----------|-------------------|----------|------------------------------|
+| F68D6E30 | 44:1b:f6:8d:6e:30 | WiFi     | room 1 (permanent)           |
+| A1D658D4 | e0:72:a1:d6:58:d4 | WiFi     | room 2 (permanent)           |
+| A1D700C4 | e0:72:a1:d7:00:c4 | BLE      | room 3 (permanent)           |
+| A1D6F190 | e0:72:a1:d6:f1:90 | BLE      | dev machine (desk, USB)      |
+
+WiFi nodes stay on their existing firmware. BLE nodes are reflashed with `firmware/ble_node/`.

 ---

@@ -53,11 +55,14 @@ This is a learning/research project covering distributed systems, event-driven a
 ## Architecture

 ```
-[ESP32 nodes] --UDP 5005--> [udp_ingest.py] --write--> [events.db (SQLite)]
+[WiFi nodes]  --UDP 5005--> [udp_ingest.py] --write--> [events.db (SQLite)]
+[BLE nodes]   --UDP 5005-+                                     |
                                                               |
 [Browser] <--HTTP/SSE 8080-- [dashboard.py] ------read--------+
 ```

+Both node types share the same UDP ingestor and database. They are distinguished by event `type` field. BLE nodes send `ble_adv` and `heartbeat` events; WiFi nodes send `beacon`, `probe`, `deauth`, `assoc`, and `heartbeat` events.
+
 The two coordinator processes are intentionally separate. The ingestor has one job: receive UDP, write to DB. The dashboard has one job: read DB, serve UI. They communicate only through the database. Each is independently restartable without affecting the other.

 ---
@@ -263,6 +268,95 @@ The same binary works on every node — `node_id` is auto-derived from the MAC,

 ---

+## BLE Firmware
+
+BLE nodes run `firmware/ble_node/`. Same hardware (ESP32-S3), same transport (UDP JSON to coordinator), different job.
+
+### What BLE nodes do
+
+- Connect to WiFi `sandbox` for UDP transport only (no WiFi scanning)
+- Run a passive BLE scan continuously (5-second windows, auto-restart)
+- Parse each advertisement: MAC, address type, device name, manufacturer specific data (company ID + payload)
+- Dedup by MAC within a 30-second window to avoid flooding the coordinator
+- Buffer events in a FreeRTOS queue, flush every 5 seconds
+- Send a heartbeat every 10 seconds (same format as WiFi nodes, adds `ble_drops` field)
+
+The ESP32 radio is shared between WiFi and BLE via the hardware coexistence controller. WiFi only transmits briefly during UDP flushes, so BLE scan coverage is near-continuous.
+
+### BLE event format
+
+```json
+{
+  "node_id":  "A1D700C4",
+  "ts":       12345,
+  "type":     "ble_adv",
+  "mac":      "AA:BB:CC:DD:EE:FF",
+  "addr_type": 1,
+  "name":     "DeviceName",
+  "rssi":     -65,
+  "mfr_id":   76,
+  "mfr_data": "1201..."
+}
+```
+
+- `addr_type`: `0` = public (real, stable MAC), `1` = random (rotates periodically)
+- `mfr_id`: 16-bit little-endian Bluetooth company ID, `-1` if the advertisement has no manufacturer data
+- `mfr_data`: hex string of the manufacturer payload **after** the 2-byte company ID, up to 16 bytes
+
+### Known company IDs
+
+| ID (hex) | Vendor |
+|----------|--------|
+| 0x004C | Apple (AirTags, AirPods, iBeacon, FindMy, HomeKit) |
+| 0x0075 | Samsung (SmartTag, Galaxy devices) |
+| 0x0006 | Microsoft (Swift Pair) |
+| 0x00E0 | Google |
+| 0x0059 | Nordic Semiconductor (common in IoT devices) |
+| 0x0138 | Garmin |
+| 0x0499 | Ruuvi Innovations |
+| 0x0157 | Polar Electro |
+| 0x00BD | Fitbit |
+| 0x0171 | Amazon |
+
+Apple type byte (first byte of `mfr_data` when `mfr_id` = 76):
+
+| Byte | Device |
+|------|--------|
+| `02` | iBeacon |
+| `05` | AirDrop |
+| `07` | HomeKit |
+| `09` | AirPods |
+| `0a` | AirPods Pro |
+| `0f` | AirPods 3rd gen |
+| `12` | FindMy / AirTag |
+| `15` | Proximity Pair |
+
+### Flashing a BLE node
+
+Same requirements as WiFi nodes. Flash `firmware/ble_node/` instead of `firmware/node/`:
+
+```bash
+# Compile
+arduino-cli compile --fqbn esp32:esp32:esp32s3 /home/sandbox/Documents/projects/esp32_cluster/firmware/ble_node/
+
+# Flash
+arduino-cli upload --fqbn esp32:esp32:esp32s3 --port /dev/ttyACM0 \
+  /home/sandbox/Documents/projects/esp32_cluster/firmware/ble_node/
+```
+
+### config.h reference (BLE node)
+
+| Constant | Default | Purpose |
+|----------|---------|---------|
+| `BLE_SCAN_DURATION_SECS` | `5` | Passive scan window length |
+| `BLE_FLUSH_INTERVAL_MS` | `5000` | How often to send queued events to coordinator |
+| `HEARTBEAT_INTERVAL_MS` | `10000` | Heartbeat cadence |
+| `BLE_DEDUP_SECS` | `30` | Suppress same MAC within this window |
+| `BLE_DEDUP_CACHE_SIZE` | `300` | Dedup ring-buffer entries |
+| `BLE_QUEUE_SIZE` | `256` | Max events buffered between flushes |
+
+---
+
 ## Orange Pi — running services

 Services are managed by systemd and start automatically on boot.
@@ -381,7 +475,9 @@ The `oui.txt` file is the IEEE public OUI database (39,171 entries as of downloa
 ## Active investigations

 ### Sustained deauth attack on Tuya device
-A persistent automated deauth flood has been running since **2026-04-03**, targeting `38:2C:E5:7E:77:1D` (Tuya Smart Inc. device). Four spoofed source MACs fire simultaneously roughly every hour, all using reason code 2, impersonating real AP BSSIDs in the building. All 4 nodes confirm it. Consistent with an automated WPA2 handshake capture tool. No action taken yet — being passively monitored.
+~~A persistent automated deauth flood has been running since **2026-04-03**~~
+
+**Status as of 2026-04-25: attack has wound down.** Down from hundreds of frames/day to 1–7 frames/day. The direct target `38:2C:E5:7E:77:1D` is no longer being hit. Two of the four attacker MACs (`82:4E:66:47:09:C1`, `62:87:CB:38:2C:20`) have gone silent. `42:8C:46:6E:12:9C` and `62:D9:AA:E8:B4:09` still appear occasionally at very low volume. Conclusion: whoever was running the tool either captured the WPA2 handshake they needed or moved away. Keep in watch list for re-activation.

 Attacker src MACs: `82:4E:66:47:09:C1`, `42:8C:46:6E:12:9C`, `62:D9:AA:E8:B4:09`, `62:87:CB:38:2C:20`

@@ -392,6 +488,64 @@ Broadcasting an open unauthenticated hotspot. Next step: connect via Parrot lapt

 ---

+## BLE observations and notes
+
+### Heap baseline
+
+BLE nodes floor out at **109–127K free heap** vs 222K+ for WiFi nodes. The ~100K difference is the BLE stack overhead (BLEDevice + BLEScan + coexistence controller). This is expected and stable — no downward drift observed after 21 hours. The WiFi-only queue flush model keeps heap use predictable.
+
+### MAC randomization and device counting
+
+BLE unique MAC counts are misleading without context:
+
+- **Apple FindMy / AirTag** rotates MAC on every advertisement interval (seconds). One physical AirTag generates hundreds of apparent "unique" MACs per day. The `mfr_data` payload is more useful for identity than the MAC.
+- **Sony WH-1000XM5** rotates MAC every ~15 minutes. One device appears as ~28 different MACs over a day. Identify by device name (`LE_WH-1000XM5`) rather than MAC.
+- **Public MACs** (addr_type = public) are stable and reliably trackable (B&W PX5, ELK-BLEDOM, ATH-M50xBT2).
+- **Apple Proximity Pair / Nearby Info** are iPhones advertising presence — high volume, all randomized.
+
+### Apple type byte reference (first byte of mfr_data when mfr_id = 76)
+
+Most common types seen in practice:
+
+| Byte | Type | Notes |
+|------|------|-------|
+| `10` | Proximity Pair | iPhone/Watch advertising nearby — very common |
+| `12` | FindMy / AirTag | MAC rotates every advertisement; best RSSI seen: −38 dBm |
+| `16` | Nearby Info | iPhone battery/status broadcast |
+| `0c` | AirPods case / Watch | Seen when case lid is open or Watch is unlocked |
+| `07` | HomeKit | Smart home accessory (lights, locks, sensors) |
+| `09`/`0a`/`0f` | AirPods (various gen) | Seen when case is open or buds are in use |
+| `02` | iBeacon | Retail/venue tracking beacons |
+
+### Identified permanent BLE neighbours (as of 2026-04-26)
+
+| Device | MAC type | Pattern | Notes |
+|--------|----------|---------|-------|
+| Sony WH-1000XM5 | Random (rotates ~15min) | All day + late night | Likely 1 device; user wears them most of the day, leaves connected overnight |
+| B&W PX5 headphones | Public — stable | Weekday afternoons ~12:00–18:00 | Consistent with WFH office hours |
+| ELK-BLEDOM LED strip | Public — stable | Evenings | BLE-controlled RGB LED strip, cheap Chinese controller |
+| Audio-Technica ATH-M50xBT2 | Public — stable | Afternoon sessions | Professional wireless headphones |
+| `0x5148` device (c2:fe:68:e8:01:a6) | Public — stable | Persistent all day | Unknown company ID, payload ASCII "364656", best RSSI −56 dBm — very close, identity unknown |
+| "net" (80:3e:4f:1b:4f:e3) | Public — stable | 07:00–20:00 | Short device name, likely a smart home hub or IoT device |
+
+### Parking structure BLE
+
+The same parking structure that generates hundreds of 70mai dashcam WiFi beacons also contains AirTags. The FindMy/AirTag type (`0x12`) consistently hits −38 dBm best RSSI — some are close enough to be in vehicles parked right outside. At least one is persistent across multiple scans, suggesting a parked vehicle with an AirTag rather than a passing pedestrian.
+
+### WiFi-derived context: notable open networks
+
+For reference during future BLE correlation work, the known open networks in range:
+
+| SSID | Device | Notes |
+|------|--------|-------|
+| `Living Room speaker.n078` | Unknown speaker | Unauthenticated provisioning hotspot, RSSI −38 dBm, same building. **Pending active investigation.** |
+| `yeelink-light-strip2_miap7AB7` | Xiaomi Yeelight LED strip | Stuck in setup mode for 8+ days, open |
+| `xiaomi-fryer-maf65_mibt962C` | Xiaomi smart air fryer | Open setup AP |
+| `zhimi-airp-mb5_mibt5FF0` | Xiaomi Mi Air Purifier MB5 | Open setup AP |
+| `REOLINK-HHRYzJCESD-2.4G` | Reolink IP camera | Permanent neighbour's camera, 8 days uptime |
+
+---
+
 ## Current status

 - [x] Arduino CLI installed, ESP32 core configured
@@ -423,6 +577,9 @@ Broadcasting an open unauthenticated hotspot. Next step: connect via Parrot lapt
 - [x] Sessions tab — dominant reason code + description shown per session row and in expanded detail panel
 - [x] SSE reconnect bug fix — multiple stale EventSource instances were accumulating on reconnect, flooding /api/nodes on tab re-open; fixed with proper close-before-reconnect and in-flight guard
 - [x] WAL corruption fix — recurring DB corruption under sustained write load; fixed with RESTART checkpointing every 30 minutes and reduced retention to 7 days (see below)
+- [x] BLE firmware (`firmware/ble_node/`) — passive scan, dedup, mfr data parsing, heartbeat
+- [x] Coordinator BLE support — `ble_events` table, ingest handler, `ble_drops` in heartbeat
+- [x] Dashboard BLE tab — devices table, manufacturer breakdown, live feed
 - [ ] Surface assoc events in dashboard (Search results or dedicated view)
 - [ ] Scan interval control from dashboard
 - [ ] Dedicated AP for nodes to isolate reconnect churn from regular network traffic