# ESP32 WiFi Recon Cluster

## What this is

A distributed WiFi reconnaissance system using ESP32-S3 nodes and an Orange Pi as coordinator.

Each ESP32 node scans the WiFi environment, builds structured events (not raw data), and ships them over UDP to a coordinator. The coordinator stores events to SQLite. A separate web dashboard reads the database and displays everything in real time via SSE.

This is a learning/research project covering distributed systems, event-driven architecture, sensor fusion, and detection engineering concepts.

---

## Hardware

- **Nodes:** ESP32-S3 N16R8 (2 active, scaling later)
- **Coordinator:** Orange Pi at `192.168.1.133`, runs 24/7 as systemd services
- **Dev machine:** This PC at `192.168.1.101` — coding and flashing only
- **Flashing:** Arduino CLI on this PC

### Active nodes

| node_id  | MAC               | Port         |
|----------|-------------------|--------------|
| F68D6E30 | 44:1b:f6:8d:6e:30 | /dev/ttyACM0 |
| A1D6F190 | e0:72:a1:d6:f1:90 | /dev/ttyACM1 |
| A1D658D4 | e0:72:a1:d6:58:d4 | /dev/ttyACM2 |

---

## Network

- **ESP32 nodes connect to:** WiFi SSID `sandbox`
- **Coordinator IP:** `192.168.1.133` (Orange Pi, production)
- **UDP port:** `5005`
- **Dashboard port:** `8080`

---

## Architecture

```
[ESP32 nodes] --UDP 5005--> [udp_ingest.py] --write--> [events.db (SQLite)]
                                                               |
[Browser] <--HTTP/SSE 8080-- [dashboard.py] ------read--------+
```

The two coordinator processes are intentionally separate. The ingestor has one job: receive UDP, write to DB. The dashboard has one job: read DB, serve UI. They communicate only through the database. Each is independently restartable without affecting the other.

---

## Development workflow

**All coding is done on this PC.** The Orange Pi only runs the deployed files — never edit directly on it.

```
[This PC] → edit code → rsync → [Orange Pi] → restart services
```

### Sync to Orange Pi

```bash
bash /home/sandbox/Documents/projects/esp32_cluster/deploy.sh
```

This rsyncs `coordinator/` to `/opt/esp32_cluster/coordinator/` on the Orange Pi, excluding `events.db` (the Pi keeps its own database).

### Restart services after deploy

```bash
ssh root@192.168.1.133 'systemctl restart udp_ingest dashboard'
```

### View live logs on Orange Pi

```bash
# Ingestor (UDP events coming in)
ssh root@192.168.1.133 'journalctl -u udp_ingest -f'

# Dashboard
ssh root@192.168.1.133 'journalctl -u dashboard -f'
```

---

## Project structure

```
esp32_cluster/
├── firmware/
│   └── node/
│       ├── node.ino              # ESP32 node firmware
│       └── config.h              # WiFi creds, coordinator IP, scan/probe config
├── coordinator/
│   ├── udp_ingest.py             # UDP listener — receives packets, validates, writes to SQLite
│   ├── dashboard.py              # FastAPI app — reads SQLite, serves UI + SSE stream
│   ├── templates/
│   │   └── dashboard.html        # Dashboard skeleton HTML
│   ├── static/
│   │   ├── dashboard.css         # All dashboard styles
│   │   └── dashboard.js          # All dashboard logic (vanilla JS)
│   ├── oui.txt                   # IEEE OUI database for vendor lookup (39k entries)
│   ├── udp_ingest.service        # systemd service file for ingestor
│   ├── dashboard.service         # systemd service file for dashboard
│   └── events.db                 # SQLite database — on Orange Pi only, not synced
├── deploy.sh                     # rsync script: this PC → Orange Pi
├── starting_plan.md              # Original architecture design doc
└── README.md                     # This file
```

---

## Firmware

Each node:
- Connects to WiFi `sandbox`
- Scans for nearby APs every 15 seconds (active scan, includes hidden SSIDs)
- Captures probe request frames passively (promiscuous mode) between scans
- Sends a heartbeat packet every 10 seconds (uptime, free heap, WiFi RSSI to router)
- Sends beacon, probe, and heartbeat events as UDP JSON packets to the coordinator
- `node_id` is derived from bytes 2–5 of the ESP32's base MAC address

### Event formats

**Beacon event** (AP discovered):
```json
{
  "node_id": "F68D6E30",
  "ts":      12345,
  "type":    "beacon",
  "conf":    "high",
  "imp":     "normal",
  "ssid":    "NetworkName",
  "bssid":   "AA:BB:CC:DD:EE:FF",
  "rssi":    -65,
  "ch":      6,
  "enc":     "WPA2"
}
```

**Probe event** (client device searching for a network):
```json
{
  "node_id": "F68D6E30",
  "ts":      12345,
  "type":    "probe",
  "conf":    "high",
  "imp":     "normal",
  "src_mac": "AA:BB:CC:DD:EE:FF",
  "ssid":    "HomeNetwork",
  "rssi":    -55
}
```

**Heartbeat event** (node health, sent every 10 seconds):
```json
{
  "node_id":   "F68D6E30",
  "ts":        12345,
  "type":      "heartbeat",
  "uptime_ms": 123456,
  "free_heap": 245000,
  "wifi_rssi": -62
}
```

- `imp`: `high` if RSSI >= -50, `low` if <= -80, else `normal`
- `ssid` in probe events is the network the device is searching for — empty means wildcard (any network)
- Probe dedup: same MAC+SSID suppressed for 30 seconds to avoid flooding
- `wifi_rssi` in heartbeat is the node's own signal to the `sandbox` AP (not a scanned network)
- A node is considered **online** if a heartbeat arrived within the last 30 seconds; falls back to beacon within 60 seconds if no heartbeat has been received yet

### Flashing a node

Requirements: Arduino CLI v1.4.1, ESP32 core v3.3.7, FQBN `esp32:esp32:esp32s3`, user in `dialout` group.

```bash
# Compile
arduino-cli compile --fqbn esp32:esp32:esp32s3 /home/sandbox/Documents/projects/esp32_cluster/firmware/node/

# Flash — adjust port as needed (check with: ls /dev/ttyACM*)
arduino-cli upload --fqbn esp32:esp32:esp32s3 --port /dev/ttyACM0 \
  /home/sandbox/Documents/projects/esp32_cluster/firmware/node/
```

The same binary works on every node — `node_id` is auto-derived from the MAC, no per-node config needed.

> Check group membership with `groups`. You should see `dialout`.

### config.h reference

| Constant          | Default          | Purpose                                   |
|-------------------|------------------|-------------------------------------------|
| `WIFI_SSID`       | `sandbox`        | WiFi network to connect to                |
| `COORDINATOR_IP`  | `192.168.1.133`  | Orange Pi address                         |
| `COORDINATOR_PORT`| `5005`           | UDP port                                  |
| `SCAN_INTERVAL_MS`| `15000`          | How often to run a beacon scan (ms)       |
| `PROBE_SNIFF`     | `1`              | Enable/disable probe sniffing (1/0)       |
| `PROBE_DEDUP_SECS`| `30`             | Suppress duplicate probe MAC+SSID (secs)  |
| `PROBE_QUEUE_SIZE`| `32`             | Max probe events buffered between scans   |

---

## Orange Pi — running services

Services are managed by systemd and start automatically on boot.

```bash
# Status
ssh root@192.168.1.133 'systemctl status udp_ingest dashboard'

# Restart both
ssh root@192.168.1.133 'systemctl restart udp_ingest dashboard'

# Stop
ssh root@192.168.1.133 'systemctl stop udp_ingest dashboard'
```

Service files are in `coordinator/` and deployed to `/etc/systemd/system/` on the Orange Pi.
The coordinator files live at `/opt/esp32_cluster/coordinator/` on the Orange Pi.

---

## Dashboard

Open `http://192.168.1.133:8080` in your browser.

### Views

- **Feed** — live beacon event stream across all nodes, capped at 100 rows, newest on top
- **Networks** — deduplicated AP table (by BSSID): times seen, best/avg RSSI, channel, encryption, node count. Sortable by any column. Each row has a ▶ button to open the RSSI history chart for that network.
- **RSSI history chart** — opens from any Networks row. Shows signal strength over time per node (1h / 2h / 6h / 24h selectable). Each node gets its own coloured line. Useful for spotting interference patterns, seeing how signal fluctuates by time of day, and comparing which node consistently hears a given AP better. Updates live as new scans arrive.
- **Cross-node** — side-by-side per-node RSSI for every AP. Shows which node is physically closer to each network. Filter to multi-node only to focus on confirmed cross-node observations.
- **Clients** — probe request data: unique MACs, vendor (OUI lookup), what SSIDs they're searching for, signal, which nodes saw them.
- **Presence** — three sections driven entirely by probe data:
  - *Present Now* — real (non-randomized) MACs seen 2+ times in the last hour. Filters out the city-center noise of transient devices.
  - *New Arrivals* — devices and networks first seen in the last 24 hours, split into two side-by-side tables.
  - *Regulars* — devices and networks seen on 2 or more distinct calendar days. Takes at least two days of data to populate.
- **Node detail** — click a node in the sidebar for stats: total events, unique SSIDs/BSSIDs, RSSI range, first/last seen, uptime, free heap, AP signal, last 100 events.

### Sidebar

Always visible: node list with online/offline status dot, last seen time, event count, and a compact heartbeat line (uptime · free heap · AP signal).
A node is considered **online** if a heartbeat was received within the last 30 seconds. Falls back to beacon timestamp (60s threshold) if no heartbeat has been received yet.

### Live updates

SSE stream pushes new beacon events as they arrive. The feed updates immediately. All other views (sidebar, networks, chart, cross-node, presence) refresh on a 2-second debounce so a single scan burst doesn't flood the server with requests.

---

## Observations and notes

### Hidden networks

Networks with a blank SSID but valid BSSID, signal, and channel are **hidden networks** — APs configured not to broadcast their name. They are not hidden from passive scanners. The encryption showing as `OPEN` is a firmware parsing artifact; the network itself may be encrypted. The BSSID OUI still identifies the manufacturer.

### Probe requests and MAC randomization

Modern phones and laptops (iOS since ~2020, Android since ~2021) randomize their MAC address for probe requests. A randomized MAC has bit 1 of the first byte set — the dashboard detects this and labels it "Randomized MAC" instead of doing a meaningless OUI lookup.

What probe data tells you:
- **Directed probes** (with SSID): the device has connected to that network in the past
- **Wildcard probes** (empty SSID): the device is searching for any available network
- Probe data maps **client devices**, not infrastructure — complementary to beacon scan data

### OUI lookup

The `oui.txt` file is the IEEE public OUI database (39,171 entries as of download). It maps the first 3 bytes of a real MAC to a manufacturer name. Used in the Clients tab. Refresh it occasionally by re-running `deploy.sh` after downloading a fresh copy from `https://standards-oui.ieee.org/oui/oui.txt`.

---

## Current status

- [x] Arduino CLI installed, ESP32 core configured
- [x] Node firmware: beacon scan + probe sniffing
- [x] Three ESP32-S3 nodes flashed and running
- [x] UDP ingestor (`udp_ingest.py`) — beacon + probe routing with field validation
- [x] Web dashboard: Feed, Networks, Cross-node, Clients, Node detail views
- [x] OUI vendor lookup in Clients view
- [x] Dashboard split into HTML / CSS / JS (no monolithic file)
- [x] Deployed to Orange Pi as systemd services (runs 24/7)
- [x] All three nodes sending to Orange Pi, confirmed live
- [x] Node heartbeat every 10s — uptime, free heap, AP signal, drives online/offline status
- [x] RSSI history chart per network (canvas, per-node coloured lines, 1h/2h/6h/24h range)
- [x] Presence tab — Present Now, New Arrivals, Regulars
- [ ] Scan interval control from dashboard
- [ ] DB pruning / retention policy (events.db grows indefinitely)

---

## Python dependencies

```
fastapi
uvicorn
```

```bash
pip install fastapi uvicorn
```
