diff --git a/README.md b/README.md index feabb2b..551498e 100644 --- a/README.md +++ b/README.md @@ -31,12 +31,14 @@ This is a learning/research project covering distributed systems, event-driven a ### Active nodes -| node_id | MAC | Location | -|----------|-------------------|------------------------------| -| F68D6E30 | 44:1b:f6:8d:6e:30 | room 1 (permanent) | -| A1D658D4 | e0:72:a1:d6:58:d4 | room 2 (permanent) | -| A1D700C4 | e0:72:a1:d7:00:c4 | room 3 (permanent) | -| A1D6F190 | e0:72:a1:d6:f1:90 | dev machine (desk, USB) | +| node_id | MAC | Firmware | Location | +|----------|-------------------|----------|------------------------------| +| F68D6E30 | 44:1b:f6:8d:6e:30 | WiFi | room 1 (permanent) | +| A1D658D4 | e0:72:a1:d6:58:d4 | WiFi | room 2 (permanent) | +| A1D700C4 | e0:72:a1:d7:00:c4 | BLE | room 3 (permanent) | +| A1D6F190 | e0:72:a1:d6:f1:90 | BLE | dev machine (desk, USB) | + +WiFi nodes stay on their existing firmware. BLE nodes are reflashed with `firmware/ble_node/`. --- @@ -53,11 +55,14 @@ This is a learning/research project covering distributed systems, event-driven a ## Architecture ``` -[ESP32 nodes] --UDP 5005--> [udp_ingest.py] --write--> [events.db (SQLite)] +[WiFi nodes] --UDP 5005--> [udp_ingest.py] --write--> [events.db (SQLite)] +[BLE nodes] --UDP 5005-+ | | [Browser] <--HTTP/SSE 8080-- [dashboard.py] ------read--------+ ``` +Both node types share the same UDP ingestor and database. They are distinguished by event `type` field. BLE nodes send `ble_adv` and `heartbeat` events; WiFi nodes send `beacon`, `probe`, `deauth`, `assoc`, and `heartbeat` events. + The two coordinator processes are intentionally separate. The ingestor has one job: receive UDP, write to DB. The dashboard has one job: read DB, serve UI. They communicate only through the database. Each is independently restartable without affecting the other. --- @@ -263,6 +268,95 @@ The same binary works on every node — `node_id` is auto-derived from the MAC, --- +## BLE Firmware + +BLE nodes run `firmware/ble_node/`. Same hardware (ESP32-S3), same transport (UDP JSON to coordinator), different job. + +### What BLE nodes do + +- Connect to WiFi `sandbox` for UDP transport only (no WiFi scanning) +- Run a passive BLE scan continuously (5-second windows, auto-restart) +- Parse each advertisement: MAC, address type, device name, manufacturer specific data (company ID + payload) +- Dedup by MAC within a 30-second window to avoid flooding the coordinator +- Buffer events in a FreeRTOS queue, flush every 5 seconds +- Send a heartbeat every 10 seconds (same format as WiFi nodes, adds `ble_drops` field) + +The ESP32 radio is shared between WiFi and BLE via the hardware coexistence controller. WiFi only transmits briefly during UDP flushes, so BLE scan coverage is near-continuous. + +### BLE event format + +```json +{ + "node_id": "A1D700C4", + "ts": 12345, + "type": "ble_adv", + "mac": "AA:BB:CC:DD:EE:FF", + "addr_type": 1, + "name": "DeviceName", + "rssi": -65, + "mfr_id": 76, + "mfr_data": "1201..." +} +``` + +- `addr_type`: `0` = public (real, stable MAC), `1` = random (rotates periodically) +- `mfr_id`: 16-bit little-endian Bluetooth company ID, `-1` if the advertisement has no manufacturer data +- `mfr_data`: hex string of the manufacturer payload **after** the 2-byte company ID, up to 16 bytes + +### Known company IDs + +| ID (hex) | Vendor | +|----------|--------| +| 0x004C | Apple (AirTags, AirPods, iBeacon, FindMy, HomeKit) | +| 0x0075 | Samsung (SmartTag, Galaxy devices) | +| 0x0006 | Microsoft (Swift Pair) | +| 0x00E0 | Google | +| 0x0059 | Nordic Semiconductor (common in IoT devices) | +| 0x0138 | Garmin | +| 0x0499 | Ruuvi Innovations | +| 0x0157 | Polar Electro | +| 0x00BD | Fitbit | +| 0x0171 | Amazon | + +Apple type byte (first byte of `mfr_data` when `mfr_id` = 76): + +| Byte | Device | +|------|--------| +| `02` | iBeacon | +| `05` | AirDrop | +| `07` | HomeKit | +| `09` | AirPods | +| `0a` | AirPods Pro | +| `0f` | AirPods 3rd gen | +| `12` | FindMy / AirTag | +| `15` | Proximity Pair | + +### Flashing a BLE node + +Same requirements as WiFi nodes. Flash `firmware/ble_node/` instead of `firmware/node/`: + +```bash +# Compile +arduino-cli compile --fqbn esp32:esp32:esp32s3 /home/sandbox/Documents/projects/esp32_cluster/firmware/ble_node/ + +# Flash +arduino-cli upload --fqbn esp32:esp32:esp32s3 --port /dev/ttyACM0 \ + /home/sandbox/Documents/projects/esp32_cluster/firmware/ble_node/ +``` + +### config.h reference (BLE node) + +| Constant | Default | Purpose | +|----------|---------|---------| +| `BLE_SCAN_DURATION_SECS` | `5` | Passive scan window length | +| `BLE_FLUSH_INTERVAL_MS` | `5000` | How often to send queued events to coordinator | +| `HEARTBEAT_INTERVAL_MS` | `10000` | Heartbeat cadence | +| `BLE_DEDUP_SECS` | `30` | Suppress same MAC within this window | +| `BLE_DEDUP_CACHE_SIZE` | `300` | Dedup ring-buffer entries | +| `BLE_QUEUE_SIZE` | `256` | Max events buffered between flushes | + +--- + ## Orange Pi — running services Services are managed by systemd and start automatically on boot. @@ -381,7 +475,9 @@ The `oui.txt` file is the IEEE public OUI database (39,171 entries as of downloa ## Active investigations ### Sustained deauth attack on Tuya device -A persistent automated deauth flood has been running since **2026-04-03**, targeting `38:2C:E5:7E:77:1D` (Tuya Smart Inc. device). Four spoofed source MACs fire simultaneously roughly every hour, all using reason code 2, impersonating real AP BSSIDs in the building. All 4 nodes confirm it. Consistent with an automated WPA2 handshake capture tool. No action taken yet — being passively monitored. +~~A persistent automated deauth flood has been running since **2026-04-03**~~ + +**Status as of 2026-04-25: attack has wound down.** Down from hundreds of frames/day to 1–7 frames/day. The direct target `38:2C:E5:7E:77:1D` is no longer being hit. Two of the four attacker MACs (`82:4E:66:47:09:C1`, `62:87:CB:38:2C:20`) have gone silent. `42:8C:46:6E:12:9C` and `62:D9:AA:E8:B4:09` still appear occasionally at very low volume. Conclusion: whoever was running the tool either captured the WPA2 handshake they needed or moved away. Keep in watch list for re-activation. Attacker src MACs: `82:4E:66:47:09:C1`, `42:8C:46:6E:12:9C`, `62:D9:AA:E8:B4:09`, `62:87:CB:38:2C:20` @@ -392,6 +488,64 @@ Broadcasting an open unauthenticated hotspot. Next step: connect via Parrot lapt --- +## BLE observations and notes + +### Heap baseline + +BLE nodes floor out at **109–127K free heap** vs 222K+ for WiFi nodes. The ~100K difference is the BLE stack overhead (BLEDevice + BLEScan + coexistence controller). This is expected and stable — no downward drift observed after 21 hours. The WiFi-only queue flush model keeps heap use predictable. + +### MAC randomization and device counting + +BLE unique MAC counts are misleading without context: + +- **Apple FindMy / AirTag** rotates MAC on every advertisement interval (seconds). One physical AirTag generates hundreds of apparent "unique" MACs per day. The `mfr_data` payload is more useful for identity than the MAC. +- **Sony WH-1000XM5** rotates MAC every ~15 minutes. One device appears as ~28 different MACs over a day. Identify by device name (`LE_WH-1000XM5`) rather than MAC. +- **Public MACs** (addr_type = public) are stable and reliably trackable (B&W PX5, ELK-BLEDOM, ATH-M50xBT2). +- **Apple Proximity Pair / Nearby Info** are iPhones advertising presence — high volume, all randomized. + +### Apple type byte reference (first byte of mfr_data when mfr_id = 76) + +Most common types seen in practice: + +| Byte | Type | Notes | +|------|------|-------| +| `10` | Proximity Pair | iPhone/Watch advertising nearby — very common | +| `12` | FindMy / AirTag | MAC rotates every advertisement; best RSSI seen: −38 dBm | +| `16` | Nearby Info | iPhone battery/status broadcast | +| `0c` | AirPods case / Watch | Seen when case lid is open or Watch is unlocked | +| `07` | HomeKit | Smart home accessory (lights, locks, sensors) | +| `09`/`0a`/`0f` | AirPods (various gen) | Seen when case is open or buds are in use | +| `02` | iBeacon | Retail/venue tracking beacons | + +### Identified permanent BLE neighbours (as of 2026-04-26) + +| Device | MAC type | Pattern | Notes | +|--------|----------|---------|-------| +| Sony WH-1000XM5 | Random (rotates ~15min) | All day + late night | Likely 1 device; user wears them most of the day, leaves connected overnight | +| B&W PX5 headphones | Public — stable | Weekday afternoons ~12:00–18:00 | Consistent with WFH office hours | +| ELK-BLEDOM LED strip | Public — stable | Evenings | BLE-controlled RGB LED strip, cheap Chinese controller | +| Audio-Technica ATH-M50xBT2 | Public — stable | Afternoon sessions | Professional wireless headphones | +| `0x5148` device (c2:fe:68:e8:01:a6) | Public — stable | Persistent all day | Unknown company ID, payload ASCII "364656", best RSSI −56 dBm — very close, identity unknown | +| "net" (80:3e:4f:1b:4f:e3) | Public — stable | 07:00–20:00 | Short device name, likely a smart home hub or IoT device | + +### Parking structure BLE + +The same parking structure that generates hundreds of 70mai dashcam WiFi beacons also contains AirTags. The FindMy/AirTag type (`0x12`) consistently hits −38 dBm best RSSI — some are close enough to be in vehicles parked right outside. At least one is persistent across multiple scans, suggesting a parked vehicle with an AirTag rather than a passing pedestrian. + +### WiFi-derived context: notable open networks + +For reference during future BLE correlation work, the known open networks in range: + +| SSID | Device | Notes | +|------|--------|-------| +| `Living Room speaker.n078` | Unknown speaker | Unauthenticated provisioning hotspot, RSSI −38 dBm, same building. **Pending active investigation.** | +| `yeelink-light-strip2_miap7AB7` | Xiaomi Yeelight LED strip | Stuck in setup mode for 8+ days, open | +| `xiaomi-fryer-maf65_mibt962C` | Xiaomi smart air fryer | Open setup AP | +| `zhimi-airp-mb5_mibt5FF0` | Xiaomi Mi Air Purifier MB5 | Open setup AP | +| `REOLINK-HHRYzJCESD-2.4G` | Reolink IP camera | Permanent neighbour's camera, 8 days uptime | + +--- + ## Current status - [x] Arduino CLI installed, ESP32 core configured @@ -423,6 +577,9 @@ Broadcasting an open unauthenticated hotspot. Next step: connect via Parrot lapt - [x] Sessions tab — dominant reason code + description shown per session row and in expanded detail panel - [x] SSE reconnect bug fix — multiple stale EventSource instances were accumulating on reconnect, flooding /api/nodes on tab re-open; fixed with proper close-before-reconnect and in-flight guard - [x] WAL corruption fix — recurring DB corruption under sustained write load; fixed with RESTART checkpointing every 30 minutes and reduced retention to 7 days (see below) +- [x] BLE firmware (`firmware/ble_node/`) — passive scan, dedup, mfr data parsing, heartbeat +- [x] Coordinator BLE support — `ble_events` table, ingest handler, `ble_drops` in heartbeat +- [x] Dashboard BLE tab — devices table, manufacturer breakdown, live feed - [ ] Surface assoc events in dashboard (Search results or dedicated view) - [ ] Scan interval control from dashboard - [ ] Dedicated AP for nodes to isolate reconnect churn from regular network traffic diff --git a/coordinator/dashboard.py b/coordinator/dashboard.py index e6614b8..1836b95 100644 --- a/coordinator/dashboard.py +++ b/coordinator/dashboard.py @@ -131,6 +131,49 @@ def _node_status(last_seen: str | None, threshold: int = ONLINE_SECS) -> str: # ─── API data builders ──────────────────────────────────────────────────────── +# ─── BLE classification ────────────────────────────────────────────────────── + +BLE_COMPANIES: dict[int, str] = { + 0x004C: "Apple", + 0x0075: "Samsung", + 0x0006: "Microsoft", + 0x00E0: "Google", + 0x0059: "Nordic Semiconductor", + 0x0138: "Garmin", + 0x0499: "Ruuvi Innovations", + 0x0157: "Polar Electro", + 0x00BD: "Fitbit", + 0x0171: "Amazon", + 0x004F: "MediaTek", + 0x001D: "Qualcomm", +} + +# Apple manufacturer data: after the 2-byte company ID, byte 0 of mfr_data is the type. +APPLE_TYPES: dict[str, str] = { + "02": "iBeacon", + "05": "AirDrop", + "07": "HomeKit", + "09": "AirPods", + "0a": "AirPods Pro", + "0b": "AirPods 2nd gen", + "0c": "AirPods case / Watch", + "0f": "AirPods 3rd gen", + "12": "FindMy / AirTag", + "13": "AirPods 4th gen", + "15": "Proximity Pair", + "1e": "AirPods Pro 2nd gen", +} + + +def ble_device_type(mfr_id: int | None, mfr_data: str | None) -> str: + if mfr_id is None: + return "Unknown" + if mfr_id == 0x004C and mfr_data and len(mfr_data) >= 2: + return APPLE_TYPES.get(mfr_data[:2].lower(), "Apple device") + vendor = BLE_COMPANIES.get(mfr_id) + return f"{vendor} device" if vendor else f"Unknown (0x{mfr_id:04X})" + + _nodes_cache: tuple[float, list] | None = None _networks_cache: tuple[float, list] | None = None _clients_cache: tuple[float, list] | None = None @@ -138,6 +181,7 @@ _cross_node_cache: tuple[float, dict] | None = None _presence_cache: tuple[float, dict] | None = None _alerts_cache: tuple[float, dict] | None = None _sessions_cache: tuple[float, list] | None = None +_ble_cache: tuple[float, dict] | None = None _NODES_CACHE_TTL = 10 # seconds _NETWORKS_CACHE_TTL = 30 # seconds _CLIENTS_CACHE_TTL = 30 # seconds @@ -145,6 +189,7 @@ _CROSS_NODE_CACHE_TTL = 60 # seconds _PRESENCE_CACHE_TTL = 60 # seconds _ALERTS_CACHE_TTL = 60 # seconds _SESSIONS_CACHE_TTL = 60 # seconds +_BLE_CACHE_TTL = 30 # seconds def build_nodes() -> list[dict]: global _nodes_cache @@ -1190,6 +1235,12 @@ async def api_node_detail(node_id: str): return detail +@app.get("/api/ble") +async def api_ble(): + data = await asyncio.to_thread(build_ble) + return data + + @app.get("/api/stream") async def api_stream(): """SSE stream — polls DB every 2 s, pushes new events to the browser.""" @@ -1225,6 +1276,65 @@ async def api_stream(): ) +def build_ble() -> dict: + global _ble_cache + now = datetime.datetime.utcnow().timestamp() + if _ble_cache and (now - _ble_cache[0]) < _BLE_CACHE_TTL: + return _ble_cache[1] + + cutoff = _hot_cutoff() + + devices = query(""" + SELECT + mac, addr_type, + MAX(name) AS name, + mfr_id, + MAX(mfr_data) AS mfr_data, + MAX(rssi) AS best_rssi, + COUNT(*) AS times_seen, + MIN(received_at) AS first_seen, + MAX(received_at) AS last_seen, + COUNT(DISTINCT node_id) AS node_count + FROM ble_events + WHERE received_at >= ? + GROUP BY mac + ORDER BY times_seen DESC + LIMIT 300 + """, (cutoff,)) + + for d in devices: + d["type"] = ble_device_type(d["mfr_id"], d["mfr_data"]) + d["vendor"] = BLE_COMPANIES.get(d["mfr_id"], "Unknown") if d["mfr_id"] is not None else "Unknown" + + feed = query(""" + SELECT received_at, node_id, mac, addr_type, name, rssi, mfr_id, mfr_data + FROM ble_events + WHERE received_at >= ? + ORDER BY id DESC + LIMIT 100 + """, (cutoff,)) + + for f in feed: + f["type"] = ble_device_type(f["mfr_id"], f["mfr_data"]) + + # Manufacturer breakdown + breakdown = query(""" + SELECT mfr_id, COUNT(DISTINCT mac) AS unique_devices + FROM ble_events + WHERE received_at >= ? + GROUP BY mfr_id + ORDER BY unique_devices DESC + LIMIT 20 + """, (cutoff,)) + + for b in breakdown: + b["vendor"] = BLE_COMPANIES.get(b["mfr_id"], "Unknown") if b["mfr_id"] is not None else "No mfr data" + + result = {"devices": devices, "feed": feed, "breakdown": breakdown} + _ble_cache = (now, result) + return result + + # ─── Entry point ───────────────────────────────────────────────────────────── if __name__ == "__main__": diff --git a/coordinator/static/dashboard.css b/coordinator/static/dashboard.css index 9173306..223f700 100644 --- a/coordinator/static/dashboard.css +++ b/coordinator/static/dashboard.css @@ -607,6 +607,33 @@ td.disassoc-type { color: var(--orange); } border-bottom: 1px solid var(--border); } +/* ── BLE ── */ +.ble-type-apple { color: #a8c8ff; } +.ble-type-samsung { color: #7eb3ff; } +.ble-type-other { color: var(--text-muted); } +.ble-addr-public { color: var(--green); font-size: 10px; } +.ble-addr-random { color: var(--text-muted); font-size: 10px; } + +.ble-breakdown-row { + display: flex; + flex-wrap: wrap; + gap: 8px; + margin-bottom: 4px; +} +.ble-breakdown-chip { + background: var(--surface); + border: 1px solid var(--border); + border-radius: 4px; + padding: 4px 10px; + font-size: 11px; + color: var(--text-muted); +} +.ble-breakdown-chip strong { + color: var(--text); + font-size: 13px; + margin-right: 4px; +} + /* ── Scrollbar ── */ ::-webkit-scrollbar { width: 5px; height: 5px; } ::-webkit-scrollbar-track { background: var(--bg); } diff --git a/coordinator/static/dashboard.js b/coordinator/static/dashboard.js index f47ae95..40822a4 100644 --- a/coordinator/static/dashboard.js +++ b/coordinator/static/dashboard.js @@ -1190,6 +1190,9 @@ function setView(view) { document.getElementById('tab-sessions').classList.toggle('active', view === 'sessions'); document.getElementById('tab-analysis').classList.toggle('active', view === 'analysis'); document.getElementById('tab-search').classList.toggle('active', view === 'search'); + document.getElementById('view-ble').classList.toggle('active', view === 'ble'); + document.getElementById('tab-ble').classList.toggle('active', view === 'ble'); + if (view !== 'ble' && _blePoll) { clearInterval(_blePoll); _blePoll = null; } } function showFeed() { @@ -1280,6 +1283,7 @@ function startSSE() { else if (currentView === 'presence') await fetchPresence(); else if (currentView === 'alerts') await fetchAlerts(); else if (currentView === 'analysis') await showAnalysis(); + else if (currentView === 'ble') await fetchBle(); if (!_nodesFetching) { _nodesFetching = true; @@ -1306,6 +1310,96 @@ function startSSE() { }; } +// ── BLE ─────────────────────────────────────────────────────────────────────── + +let bleData = { devices: [], feed: [], breakdown: [] }; +let _blePoll = null; + +function showBle() { + setView('ble'); + fetchBle(); + if (_blePoll) clearInterval(_blePoll); + _blePoll = setInterval(fetchBle, 5000); +} + +async function fetchBle() { + try { + bleData = await fetch('/api/ble').then(r => r.json()); + renderBle(); + } catch (e) { /* network hiccup */ } +} + +function bleTypeClass(type) { + if (!type) return 'ble-type-other'; + if (type.startsWith('Apple') || type === 'iBeacon' || type.includes('AirPods') || + type.includes('FindMy') || type.includes('AirTag') || type.includes('HomeKit') || + type.includes('AirDrop') || type.includes('Proximity')) + return 'ble-type-apple'; + if (type.startsWith('Samsung')) return 'ble-type-samsung'; + return 'ble-type-other'; +} + +function renderBle() { + // Breakdown chips + const bd = bleData.breakdown || []; + const bdEl = document.getElementById('ble-breakdown'); + document.getElementById('ble-breakdown-count').textContent = bd.length ? `${bd.length} manufacturers` : ''; + bdEl.innerHTML = bd.map(b => { + const label = b.vendor && b.vendor !== 'Unknown' && b.vendor !== 'No mfr data' + ? b.vendor + : b.mfr_id != null ? `0x${b.mfr_id.toString(16).padStart(4,'0').toUpperCase()}` : 'No mfr data'; + return `
${b.unique_devices}${label}
`; + }).join(''); + + // Devices table + const devs = bleData.devices || []; + const devsEl = document.getElementById('ble-devices-body'); + const emptyEl = document.getElementById('ble-devices-empty'); + const tableEl = document.getElementById('ble-devices-table'); + document.getElementById('ble-devices-count').textContent = devs.length ? `${devs.length} unique` : ''; + if (!devs.length) { + emptyEl.style.display = ''; + tableEl.style.display = 'none'; + } else { + emptyEl.style.display = 'none'; + tableEl.style.display = ''; + devsEl.innerHTML = devs.map(d => { + const typeClass = bleTypeClass(d.type); + const addrClass = d.addr_type === 'public' ? 'ble-addr-public' : 'ble-addr-random'; + const name = d.name || ''; + const vendor = d.vendor && d.vendor !== 'Unknown' ? d.vendor : ''; + return ` + ${fmt(d.type)} + ${fmt(d.mac)} + ${name} + ${vendor} + ${d.addr_type || '—'} + ${fmt(d.best_rssi)} dBm + ${fmt(d.times_seen)} + ${fmt(d.node_count)} + ${shortTime(d.last_seen)} + `; + }).join(''); + } + + // Feed + const feed = bleData.feed || []; + document.getElementById('ble-feed-count').textContent = feed.length ? `last ${feed.length}` : ''; + document.getElementById('ble-feed-body').innerHTML = feed.map(f => { + const typeClass = bleTypeClass(f.type); + const addrClass = f.addr_type === 'public' ? 'ble-addr-public' : 'ble-addr-random'; + return ` + ${shortTime(f.received_at)} + ${fmt(f.node_id)} + ${fmt(f.type)} + ${fmt(f.mac)} + ${f.name || ''} + ${fmt(f.rssi)} dBm + ${f.addr_type || '—'} + `; + }).join(''); +} + // ── Boot ────────────────────────────────────────────────────────────────────── initNetworkSort(); init().then(startSSE); diff --git a/coordinator/templates/dashboard.html b/coordinator/templates/dashboard.html index b30f165..a7c4e16 100644 --- a/coordinator/templates/dashboard.html +++ b/coordinator/templates/dashboard.html @@ -33,6 +33,7 @@ + @@ -42,6 +43,7 @@
nodes
online
events (session)
+
data window30 days
+ +
+ + +
+ Manufacturers + +
+
+ + +
+ Devices + + unique MACs — last 7 days +
+ + + + + + + + + +
TypeMACNameVendorAddrBest RSSISeenNodesLast seen
+ + +
+ Recent Advertisements + +
+ + + + + + + + +
TimeNodeTypeMACNameRSSIAddr
+ +
+ diff --git a/coordinator/udp_ingest.py b/coordinator/udp_ingest.py index 0af2bc6..6ab6af2 100644 --- a/coordinator/udp_ingest.py +++ b/coordinator/udp_ingest.py @@ -89,6 +89,26 @@ def init_db(path: str) -> sqlite3.Connection: rssi INTEGER ) """) + conn.execute(""" + CREATE TABLE IF NOT EXISTS ble_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + received_at TEXT NOT NULL, + node_id TEXT, + node_ts INTEGER, + mac TEXT, + addr_type TEXT, + name TEXT, + rssi INTEGER, + mfr_id INTEGER, + mfr_data TEXT + ) + """) + # Safe migration: add ble_drops to heartbeat table if not present yet + try: + conn.execute("ALTER TABLE heartbeat_events ADD COLUMN ble_drops INTEGER") + conn.commit() + except sqlite3.OperationalError: + pass # Indexes for common query patterns conn.executescript(""" CREATE INDEX IF NOT EXISTS idx_beacon_received_at ON beacon_events (received_at); @@ -107,6 +127,10 @@ def init_db(path: str) -> sqlite3.Connection: CREATE INDEX IF NOT EXISTS idx_assoc_node_id ON assoc_events (node_id); CREATE INDEX IF NOT EXISTS idx_assoc_src ON assoc_events (src); CREATE INDEX IF NOT EXISTS idx_assoc_bssid ON assoc_events (bssid); + CREATE INDEX IF NOT EXISTS idx_ble_received_at ON ble_events (received_at); + CREATE INDEX IF NOT EXISTS idx_ble_node_id ON ble_events (node_id); + CREATE INDEX IF NOT EXISTS idx_ble_mac ON ble_events (mac); + CREATE INDEX IF NOT EXISTS idx_ble_mfr_id ON ble_events (mfr_id); """) conn.commit() return conn @@ -136,8 +160,8 @@ def store_heartbeat(conn: sqlite3.Connection, ev: dict, received_at: str): conn.execute(""" INSERT INTO heartbeat_events (received_at, node_id, node_ts, uptime_ms, free_heap, wifi_rssi, - probe_drops, deauth_drops, assoc_drops) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + probe_drops, deauth_drops, assoc_drops, ble_drops) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( received_at, ev.get("node_id"), @@ -148,6 +172,7 @@ def store_heartbeat(conn: sqlite3.Connection, ev: dict, received_at: str): ev.get("probe_drops"), ev.get("deauth_drops"), ev.get("assoc_drops"), + ev.get("ble_drops"), )) conn.commit() @@ -270,6 +295,37 @@ def validate_assoc(ev: dict) -> str | None: return None +def store_ble(conn: sqlite3.Connection, ev: dict, received_at: str): + mfr_id = ev.get("mfr_id") + conn.execute(""" + INSERT INTO ble_events + (received_at, node_id, node_ts, mac, addr_type, name, rssi, mfr_id, mfr_data) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + """, ( + received_at, + ev.get("node_id"), + ev.get("ts"), + ev.get("mac"), + "public" if ev.get("addr_type") == 0 else "random", + ev.get("name", ""), + ev.get("rssi"), + mfr_id if mfr_id is not None and mfr_id >= 0 else None, + ev.get("mfr_data", ""), + )) + conn.commit() + + +def validate_ble(ev: dict) -> str | None: + for field in ("node_id", "mac", "rssi"): + if ev.get(field) is None: + return f"missing field '{field}'" + if not _check_mac(ev["mac"]): + return f"bad mac format: {ev['mac']}" + if not _check_rssi(ev["rssi"]): + return f"rssi out of range: {ev['rssi']}" + return None + + def validate_probe(ev: dict) -> str | None: """Returns an error string if invalid, else None.""" for field in ("node_id", "src_mac", "rssi"): @@ -283,6 +339,8 @@ def validate_probe(ev: dict) -> str | None: # ─── Packet handling ───────────────────────────────────────────────────────── +MAGENTA = "\033[95m" + IMP_COLOR = { "high": "\033[92m", "normal": "\033[0m", @@ -303,7 +361,21 @@ def handle_packet(data: bytes, addr: tuple, conn: sqlite3.Connection): received_at = datetime.datetime.utcnow().isoformat(timespec="seconds") pkt_type = ev.get("type", "beacon") - if pkt_type in ("assoc", "reassoc"): + if pkt_type == "ble_adv": + err = validate_ble(ev) + if err: + print(f"[DROP] {addr[0]} ble_adv — {err}") + return + store_ble(conn, ev, received_at) + node = ev.get("node_id", "?") + mac = ev.get("mac", "?") + name = ev.get("name") or "" + rssi = ev.get("rssi", 0) + mfr_id = ev.get("mfr_id", -1) + at = "pub" if ev.get("addr_type") == 0 else "rnd" + mfr_str = f"mfr=0x{mfr_id:04X}" if mfr_id is not None and mfr_id >= 0 else "mfr=—" + print(f"{MAGENTA}[{received_at}] {node} BLE {mac} ({at}) \"{name}\" {rssi:>4}dBm {mfr_str}{RESET}") + elif pkt_type in ("assoc", "reassoc"): err = validate_assoc(ev) if err: print(f"[DROP] {addr[0]} {pkt_type} — {err}") diff --git a/firmware/ble_node/ble_node.ino b/firmware/ble_node/ble_node.ino new file mode 100644 index 0000000..1a3b97d --- /dev/null +++ b/firmware/ble_node/ble_node.ino @@ -0,0 +1,264 @@ +#include +#include +#include +#include +#include +#include "esp_efuse.h" +#include "esp_mac.h" +#include "freertos/queue.h" +#include "config.h" + +WiFiUDP udp; +String nodeId; + +unsigned long lastHeartbeat = 0; +unsigned long lastFlush = 0; + +static volatile bool scanDone = false; +static volatile uint32_t bleDrops = 0; + +// ─── BLE event ─────────────────────────────────────────────────────────────── + +struct BleEvent { + char mac[18]; + uint8_t addr_type; // 0 = public, 1 = random + char name[32]; + int8_t rssi; + int32_t mfr_id; // company ID (little-endian), -1 if absent + char mfr_data[33]; // hex string of bytes after company ID, up to 16 bytes +}; + +static QueueHandle_t bleQueue; + +// ─── Dedup ─────────────────────────────────────────────────────────────────── + +struct BleDedup { char mac[18]; uint32_t last_ms; }; +static BleDedup dedupCache[BLE_DEDUP_CACHE_SIZE]; +static int dedupNext = 0; + +static bool isDuplicate(const char* mac) { + uint32_t now = millis(); + for (int i = 0; i < BLE_DEDUP_CACHE_SIZE; i++) { + if (dedupCache[i].last_ms == 0) continue; + if (strcmp(dedupCache[i].mac, mac) == 0) { + if ((now - dedupCache[i].last_ms) < (uint32_t)(BLE_DEDUP_SECS * 1000)) + return true; + dedupCache[i].last_ms = now; + return false; + } + } + strncpy(dedupCache[dedupNext].mac, mac, 17); + dedupCache[dedupNext].mac[17] = '\0'; + dedupCache[dedupNext].last_ms = now; + dedupNext = (dedupNext + 1) % BLE_DEDUP_CACHE_SIZE; + return false; +} + +// ─── Advertisement callback ────────────────────────────────────────────────── +// Runs in the BLE task — must be fast and must not call UDP. + +class AdvCallback : public BLEAdvertisedDeviceCallbacks { + void onResult(BLEAdvertisedDevice dev) override { + char mac[18]; + strncpy(mac, dev.getAddress().toString().c_str(), 17); + mac[17] = '\0'; + + if (isDuplicate(mac)) return; + + BleEvent ev; + memcpy(ev.mac, mac, 18); + + // BLE_ADDR_TYPE_PUBLIC = 0, everything else treated as random + ev.addr_type = (dev.getAddressType() == BLE_ADDR_PUBLIC) ? 0 : 1; + ev.rssi = (int8_t)dev.getRSSI(); + ev.mfr_id = -1; + ev.mfr_data[0] = '\0'; + ev.name[0] = '\0'; + + // Device name — strip non-printable bytes + if (dev.haveName()) { + String raw = dev.getName(); + int j = 0; + for (int i = 0; i < (int)raw.length() && j < 31; i++) { + char c = raw[i]; + ev.name[j++] = (c >= 32 && c <= 126) ? c : '?'; + } + ev.name[j] = '\0'; + } + + // Manufacturer specific data (AD type 0xFF) + // First 2 bytes (little-endian) are the company ID; rest is payload. + if (dev.haveManufacturerData()) { + String md = dev.getManufacturerData(); + if (md.length() >= 2) { + ev.mfr_id = (int32_t)((uint8_t)md[0] | ((uint8_t)md[1] << 8)); + int payload_len = min((int)16, (int)md.length() - 2); + for (int i = 0; i < payload_len; i++) { + snprintf(ev.mfr_data + i * 2, 3, "%02x", (uint8_t)md[2 + i]); + } + ev.mfr_data[payload_len * 2] = '\0'; + } + } + + if (xQueueSend(bleQueue, &ev, 0) != pdTRUE) bleDrops++; + } +}; + +static AdvCallback advCallback; +static BLEScan* pBLEScan = nullptr; + +// Called from BLE task when a scan window completes. +static void onScanComplete(BLEScanResults) { + pBLEScan->clearResults(); + scanDone = true; +} + +// ─── Setup ─────────────────────────────────────────────────────────────────── + +void setup() { + Serial.begin(115200); + delay(500); + + uint8_t mac[6]; + esp_efuse_mac_get_default(mac); + char macBuf[9]; + snprintf(macBuf, sizeof(macBuf), "%02X%02X%02X%02X", mac[2], mac[3], mac[4], mac[5]); + nodeId = String(macBuf); + + Serial.printf("\n[BLE NODE] ID: %s\n", nodeId.c_str()); + + memset(dedupCache, 0, sizeof(dedupCache)); + bleQueue = xQueueCreate(BLE_QUEUE_SIZE, sizeof(BleEvent)); + + connectWiFi(); + + BLEDevice::init(""); + pBLEScan = BLEDevice::getScan(); + pBLEScan->setAdvertisedDeviceCallbacks(&advCallback, true); // true = keep duplicates in scan window + pBLEScan->setActiveScan(false); // passive — do not send scan requests + pBLEScan->setInterval(100); // scan interval (ms × 0.625 = 62.5ms) + pBLEScan->setWindow(99); // scan window — near-continuous coverage + + pBLEScan->start(BLE_SCAN_DURATION_SECS, onScanComplete, false); + Serial.println("[BLE] Passive scan started"); +} + +// ─── Loop ──────────────────────────────────────────────────────────────────── + +void loop() { + if (WiFi.status() != WL_CONNECTED) { + Serial.println("[WIFI] Lost connection, reconnecting..."); + connectWiFi(); + return; + } + + unsigned long now = millis(); + + if (now - lastHeartbeat >= HEARTBEAT_INTERVAL_MS) { + lastHeartbeat = now; + sendHeartbeat(); + } + + if (now - lastFlush >= BLE_FLUSH_INTERVAL_MS) { + lastFlush = now; + flushBleQueue(); + } + + // Restart scan after each window completes + if (scanDone) { + scanDone = false; + pBLEScan->start(BLE_SCAN_DURATION_SECS, onScanComplete, false); + } + + delay(10); +} + +// ─── WiFi ──────────────────────────────────────────────────────────────────── + +void connectWiFi() { + Serial.printf("[WIFI] Connecting to %s\n", WIFI_SSID); + WiFi.mode(WIFI_STA); + WiFi.begin(WIFI_SSID, WIFI_PASSWORD); + + int attempts = 0; + while (WiFi.status() != WL_CONNECTED && attempts < 20) { + delay(500); + Serial.print("."); + attempts++; + } + + if (WiFi.status() == WL_CONNECTED) { + Serial.printf("\n[WIFI] Connected — IP: %s\n", WiFi.localIP().toString().c_str()); + } else { + Serial.println("\n[WIFI] Failed to connect, will retry in loop"); + } +} + +// ─── Send ──────────────────────────────────────────────────────────────────── + +void sendBleEvent(const BleEvent& ev) { + // Escape name for JSON + char nameEsc[66]; + int j = 0; + for (int i = 0; ev.name[i] && j < 64; i++) { + if (ev.name[i] == '"' || ev.name[i] == '\\') nameEsc[j++] = '\\'; + nameEsc[j++] = ev.name[i]; + } + nameEsc[j] = '\0'; + + char buf[320]; + snprintf(buf, sizeof(buf), + "{\"node_id\":\"%s\",\"ts\":%lu,\"type\":\"ble_adv\"," + "\"mac\":\"%s\",\"addr_type\":%u," + "\"name\":\"%s\",\"rssi\":%d," + "\"mfr_id\":%ld,\"mfr_data\":\"%s\"}", + nodeId.c_str(), millis(), + ev.mac, (unsigned)ev.addr_type, + nameEsc, (int)ev.rssi, + (long)ev.mfr_id, ev.mfr_data + ); + + udp.beginPacket(COORDINATOR_IP, COORDINATOR_PORT); + udp.print(buf); + udp.endPacket(); + + const char* at = ev.addr_type == 0 ? "pub" : "rnd"; + Serial.printf(" [BLE] %s (%s) \"%s\" %ddBm mfr=0x%04lX data=%s\n", + ev.mac, at, ev.name[0] ? ev.name : "", + (int)ev.rssi, (long)(ev.mfr_id >= 0 ? ev.mfr_id : 0), ev.mfr_data); +} + +void flushBleQueue() { + if (WiFi.status() != WL_CONNECTED) return; + BleEvent ev; + while (xQueueReceive(bleQueue, &ev, 0) == pdTRUE) { + sendBleEvent(ev); + } +} + +// ─── Heartbeat ─────────────────────────────────────────────────────────────── + +void sendHeartbeat() { + unsigned long now = millis(); + char buf[256]; + snprintf(buf, sizeof(buf), + "{\"node_id\":\"%s\",\"ts\":%lu,\"type\":\"heartbeat\"," + "\"uptime_ms\":%lu,\"free_heap\":%lu,\"wifi_rssi\":%d," + "\"probe_drops\":0,\"deauth_drops\":0,\"assoc_drops\":0," + "\"ble_drops\":%lu}", + nodeId.c_str(), now, + now, + (unsigned long)ESP.getFreeHeap(), + WiFi.RSSI(), + (unsigned long)bleDrops + ); + udp.beginPacket(COORDINATOR_IP, COORDINATOR_PORT); + udp.print(buf); + udp.endPacket(); + Serial.printf("[HB] uptime=%lus heap=%luK ap=%ddBm ble_drops=%lu\n", + now / 1000, + (unsigned long)ESP.getFreeHeap() / 1024, + WiFi.RSSI(), + (unsigned long)bleDrops + ); +} diff --git a/firmware/ble_node/config.h b/firmware/ble_node/config.h new file mode 100644 index 0000000..ea9f32d --- /dev/null +++ b/firmware/ble_node/config.h @@ -0,0 +1,23 @@ +#pragma once + +// ─── WiFi (for UDP transport only) ─────────────────────────────────────────── +#define WIFI_SSID "sandbox" +#define WIFI_PASSWORD "Jaunsgads11!!" + +// ─── Coordinator ───────────────────────────────────────────────────────────── +#define COORDINATOR_IP "192.168.1.133" +#define COORDINATOR_PORT 5005 + +// ─── BLE scan ──────────────────────────────────────────────────────────────── +#define BLE_SCAN_DURATION_SECS 5 // each passive scan window (seconds) +#define BLE_FLUSH_INTERVAL_MS 5000 // how often to flush queue to coordinator (ms) +#define HEARTBEAT_INTERVAL_MS 10000 // heartbeat cadence (ms) — same as WiFi nodes + +// ─── Dedup ─────────────────────────────────────────────────────────────────── +// Suppress the same MAC within this window. Randomised-MAC devices will +// reappear when their MAC rotates (typically every 10–15 min), which is fine. +#define BLE_DEDUP_SECS 30 // seconds before the same MAC is allowed again +#define BLE_DEDUP_CACHE_SIZE 300 // ring-buffer size; BLE sees far more MACs than WiFi + +// ─── Queue ─────────────────────────────────────────────────────────────────── +#define BLE_QUEUE_SIZE 256 // events buffered between flushes