Add Presence tab, RSSI chart, heartbeat; fix SSE flood, chart timezone, N+1 clients query

README.md

@@ -231,6 +231,10 @@ Open `http://192.168.1.133:8080` in your browser.
 - **RSSI history chart** — opens from any Networks row. Shows signal strength over time per node (1h / 2h / 6h / 24h selectable). Each node gets its own coloured line. Useful for spotting interference patterns, seeing how signal fluctuates by time of day, and comparing which node consistently hears a given AP better. Updates live as new scans arrive.
 - **Cross-node** — side-by-side per-node RSSI for every AP. Shows which node is physically closer to each network. Filter to multi-node only to focus on confirmed cross-node observations.
 - **Clients** — probe request data: unique MACs, vendor (OUI lookup), what SSIDs they're searching for, signal, which nodes saw them.
+- **Presence** — three sections driven entirely by probe data:
+  - *Present Now* — real (non-randomized) MACs seen 2+ times in the last hour. Filters out the city-center noise of transient devices.
+  - *New Arrivals* — devices and networks first seen in the last 24 hours, split into two side-by-side tables.
+  - *Regulars* — devices and networks seen on 2 or more distinct calendar days. Takes at least two days of data to populate.
 - **Node detail** — click a node in the sidebar for stats: total events, unique SSIDs/BSSIDs, RSSI range, first/last seen, uptime, free heap, AP signal, last 100 events.
 
 ### Sidebar
@@ -240,7 +244,7 @@ A node is considered **online** if a heartbeat was received within the last 30 s
 
 ### Live updates
 
-SSE stream pushes new beacon events as they arrive. The feed updates immediately. All other views (sidebar, networks, chart, cross-node) refresh on a 2-second debounce so a single scan burst doesn't flood the server with requests.
+SSE stream pushes new beacon events as they arrive. The feed updates immediately. All other views (sidebar, networks, chart, cross-node, presence) refresh on a 2-second debounce so a single scan burst doesn't flood the server with requests.
 
 ---
 
@@ -277,7 +281,10 @@ The `oui.txt` file is the IEEE public OUI database (39,171 entries as of downloa
 - [x] Deployed to Orange Pi as systemd services (runs 24/7)
 - [x] All three nodes sending to Orange Pi, confirmed live
 - [x] Node heartbeat every 10s — uptime, free heap, AP signal, drives online/offline status
+- [x] RSSI history chart per network (canvas, per-node coloured lines, 1h/2h/6h/24h range)
+- [x] Presence tab — Present Now, New Arrivals, Regulars
 - [ ] Scan interval control from dashboard
+- [ ] DB pruning / retention policy (events.db grows indefinitely)
 
 ---
 

coordinator/dashboard.py

@@ -170,14 +170,18 @@ def build_clients() -> list[dict]:
         GROUP BY src_mac
         ORDER BY times_seen DESC
     """)
-    # Attach probed SSIDs and OUI vendor info to each row
+    # Fetch all probed SSIDs in one query, group in Python — avoids N+1
+    ssid_rows = query("""
+        SELECT DISTINCT src_mac, ssid FROM probe_events
+        WHERE src_mac IS NOT NULL AND ssid IS NOT NULL AND ssid != ''
+        ORDER BY src_mac, ssid
+    """)
+    ssids_by_mac: dict = {}
+    for s in ssid_rows:
+        ssids_by_mac.setdefault(s["src_mac"], []).append(s["ssid"])
+
     for r in rows:
-        ssid_rows = query("""
+        r["probed_ssids"] = ssids_by_mac.get(r["src_mac"], [])
-            SELECT DISTINCT ssid FROM probe_events
-            WHERE src_mac = ? AND ssid IS NOT NULL AND ssid != ''
-            ORDER BY ssid
-        """, (r["src_mac"],))
-        r["probed_ssids"] = [s["ssid"] for s in ssid_rows]
         vendor, randomized = oui_lookup(r["src_mac"])
         r["vendor"]     = vendor
         r["randomized"] = randomized
@@ -237,6 +241,119 @@ def build_cross_node() -> dict:
     return {"node_ids": node_ids, "networks": result}
 
 
+def build_presence() -> dict:
+    now        = datetime.datetime.utcnow()
+    cutoff_1h  = (now - datetime.timedelta(hours=1)).isoformat(timespec="seconds")
+    cutoff_24h = (now - datetime.timedelta(hours=24)).isoformat(timespec="seconds")
+
+    # ── Present now: real MACs seen 2+ times in the last hour ────────────────
+    present_rows = query("""
+        SELECT src_mac,
+               COUNT(*)              AS times_seen,
+               MAX(rssi)             AS best_rssi,
+               ROUND(AVG(rssi), 1)   AS avg_rssi,
+               MAX(received_at)      AS last_seen,
+               COUNT(DISTINCT node_id) AS node_count
+        FROM probe_events
+        WHERE src_mac IS NOT NULL AND received_at > ?
+        GROUP BY src_mac
+        HAVING COUNT(*) > 1
+        ORDER BY last_seen DESC
+    """, (cutoff_1h,))
+    present = []
+    for r in present_rows:
+        vendor, randomized = oui_lookup(r["src_mac"])
+        if not randomized:
+            r["vendor"] = vendor
+            present.append(r)
+
+    # ── New arrivals: first seen in last 24h ─────────────────────────────────
+    new_mac_rows = query("""
+        SELECT src_mac,
+               MIN(received_at)      AS first_seen,
+               MAX(rssi)             AS best_rssi,
+               COUNT(*)              AS times_seen,
+               COUNT(DISTINCT node_id) AS node_count
+        FROM probe_events
+        WHERE src_mac IS NOT NULL
+        GROUP BY src_mac
+        HAVING MIN(received_at) > ?
+        ORDER BY first_seen DESC
+        LIMIT 100
+    """, (cutoff_24h,))
+    new_macs = []
+    for r in new_mac_rows:
+        vendor, randomized = oui_lookup(r["src_mac"])
+        r["vendor"]     = vendor
+        r["randomized"] = randomized
+        new_macs.append(r)
+
+    new_networks = query("""
+        SELECT bssid,
+               MAX(ssid)             AS ssid,
+               MIN(received_at)      AS first_seen,
+               MAX(rssi)             AS best_rssi,
+               MAX(channel)          AS channel,
+               MAX(encryption)       AS encryption,
+               COUNT(DISTINCT node_id) AS node_count
+        FROM beacon_events
+        WHERE bssid IS NOT NULL
+        GROUP BY bssid
+        HAVING MIN(received_at) > ?
+        ORDER BY first_seen DESC
+        LIMIT 100
+    """, (cutoff_24h,))
+
+    # ── Regulars: seen on 2+ distinct calendar days ───────────────────────────
+    reg_mac_rows = query("""
+        SELECT src_mac,
+               COUNT(DISTINCT date(received_at)) AS days_seen,
+               MIN(received_at)                  AS first_seen,
+               MAX(received_at)                  AS last_seen,
+               MAX(rssi)                         AS best_rssi,
+               COUNT(*)                          AS times_seen,
+               COUNT(DISTINCT node_id)           AS node_count
+        FROM probe_events
+        WHERE src_mac IS NOT NULL
+        GROUP BY src_mac
+        HAVING COUNT(DISTINCT date(received_at)) >= 2
+        ORDER BY days_seen DESC, times_seen DESC
+        LIMIT 50
+    """)
+    reg_macs = []
+    for r in reg_mac_rows:
+        vendor, randomized = oui_lookup(r["src_mac"])
+        r["vendor"]     = vendor
+        r["randomized"] = randomized
+        reg_macs.append(r)
+
+    reg_networks = query("""
+        SELECT bssid,
+               MAX(ssid)                         AS ssid,
+               COUNT(DISTINCT date(received_at)) AS days_seen,
+               MIN(received_at)                  AS first_seen,
+               MAX(received_at)                  AS last_seen,
+               MAX(rssi)                         AS best_rssi,
+               MAX(channel)                      AS channel,
+               MAX(encryption)                   AS encryption,
+               COUNT(DISTINCT node_id)           AS node_count
+        FROM beacon_events
+        WHERE bssid IS NOT NULL
+        GROUP BY bssid
+        HAVING COUNT(DISTINCT date(received_at)) >= 2
+        ORDER BY days_seen DESC, best_rssi DESC
+        LIMIT 50
+    """)
+
+    return {
+        "present":      present,
+        "new_macs":     new_macs,
+        "new_networks": new_networks,
+        "reg_macs":     reg_macs,
+        "reg_networks": reg_networks,
+    }
+
+
 def build_node_detail(node_id: str) -> dict | None:
     rows = query("""
         SELECT
@@ -349,6 +466,11 @@ async def api_cross_node():
     return build_cross_node()
 
 
+@app.get("/api/presence")
+async def api_presence():
+    return build_presence()
+
+
 @app.get("/api/rssi-history")
 async def api_rssi_history(bssid: str, hours: int = 2):
     hours = min(max(hours, 1), 48)

coordinator/static/dashboard.css

@@ -360,6 +360,25 @@ thead th.sort-active::after { content: ' ' attr(data-arrow); }
   letter-spacing: 0.06em;
 }
 
+/* ── Presence ── */
+.presence-split {
+  display: grid;
+  grid-template-columns: 1fr 1fr;
+  gap: 16px;
+}
+.presence-sub {
+  font-size: 10px;
+  color: var(--text-muted);
+  margin-left: auto;
+}
+.presence-sub-label {
+  font-size: 9px;
+  letter-spacing: 0.1em;
+  text-transform: uppercase;
+  color: var(--text-muted);
+  margin-bottom: 6px;
+}
+
 /* ── Scrollbar ── */
 ::-webkit-scrollbar { width: 5px; height: 5px; }
 ::-webkit-scrollbar-track { background: var(--bg); }

coordinator/static/dashboard.js

@@ -8,6 +8,7 @@ let networksData  = [];
 let netSort       = { col: 'times_seen', dir: 'desc' };
 let crossNodeData = { node_ids: [], networks: [] };
 let clientsData   = [];
+let presenceData  = {};
 let chartBssid    = null;
 let chartHours    = 2;
 
@@ -316,6 +317,126 @@ function renderClientsTable() {
   }).join('');
 }
 
+// ── Presence view ─────────────────────────────────────────────────────────────
+async function fetchPresence() {
+  presenceData = await fetch('/api/presence').then(r => r.json());
+  renderPresenceView();
+}
+
+function renderPresenceView() {
+  const p = presenceData;
+
+  // Present Now
+  const presentTbody = document.getElementById('present-tbody');
+  const presentEmpty = document.getElementById('present-empty');
+  const presentCount = document.getElementById('present-count');
+  const present = p.present || [];
+  presentCount.textContent = `${present.length} devices`;
+  if (!present.length) {
+    presentTbody.innerHTML = '';
+    presentEmpty.style.display = 'block';
+  } else {
+    presentEmpty.style.display = 'none';
+    presentTbody.innerHTML = present.map(r => {
+      const nodeCls = r.node_count > 1 ? 'nodes-multi' : 'muted';
+      return `<tr>
+        <td>${fmt(r.src_mac)}</td>
+        <td class="dim">${r.vendor || '—'}</td>
+        <td class="${rssiClass(r.best_rssi)}">${fmt(r.best_rssi)} dBm</td>
+        <td class="${rssiClass(r.avg_rssi)}">${fmt(r.avg_rssi)} dBm</td>
+        <td>${fmt(r.times_seen)}</td>
+        <td class="${nodeCls}">${fmt(r.node_count)}</td>
+        <td class="dim">${shortTime(r.last_seen)}</td>
+      </tr>`;
+    }).join('');
+  }
+
+  // New Arrivals — devices
+  const arrMacsTbody = document.getElementById('arrivals-macs-tbody');
+  const arrMacsEmpty = document.getElementById('arrivals-macs-empty');
+  const arrCount     = document.getElementById('arrivals-count');
+  const newMacs = p.new_macs || [];
+  const newNets = p.new_networks || [];
+  arrCount.textContent = `${newMacs.length} devices · ${newNets.length} networks`;
+  if (!newMacs.length) {
+    arrMacsTbody.innerHTML = '';
+    arrMacsEmpty.style.display = 'block';
+  } else {
+    arrMacsEmpty.style.display = 'none';
+    arrMacsTbody.innerHTML = newMacs.map(r => `<tr>
+      <td>${fmt(r.src_mac)}</td>
+      <td class="dim">${r.vendor || '—'}</td>
+      <td class="dim">${shortTime(r.first_seen)}</td>
+      <td class="${rssiClass(r.best_rssi)}">${fmt(r.best_rssi)} dBm</td>
+      <td>${fmt(r.times_seen)}</td>
+    </tr>`).join('');
+  }
+
+  // New Arrivals — networks
+  const arrNetsTbody = document.getElementById('arrivals-nets-tbody');
+  const arrNetsEmpty = document.getElementById('arrivals-nets-empty');
+  if (!newNets.length) {
+    arrNetsTbody.innerHTML = '';
+    arrNetsEmpty.style.display = 'block';
+  } else {
+    arrNetsEmpty.style.display = 'none';
+    arrNetsTbody.innerHTML = newNets.map(r => {
+      const encCls = (!r.encryption || r.encryption === 'OPEN' || r.encryption === 'None') ? 'enc-open' : 'muted';
+      return `<tr>
+        <td>${fmt(r.ssid, '<hidden>')}</td>
+        <td class="dim">${fmt(r.bssid)}</td>
+        <td class="dim">${shortTime(r.first_seen)}</td>
+        <td class="${rssiClass(r.best_rssi)}">${fmt(r.best_rssi)} dBm</td>
+        <td class="muted">${fmt(r.channel)}</td>
+        <td class="${encCls}">${r.encryption || 'OPEN'}</td>
+      </tr>`;
+    }).join('');
+  }
+
+  // Regulars — devices
+  const regMacsTbody = document.getElementById('regulars-macs-tbody');
+  const regMacsEmpty = document.getElementById('regulars-macs-empty');
+  const regCount     = document.getElementById('regulars-count');
+  const regMacs = p.regular_macs || [];
+  const regNets = p.regular_networks || [];
+  regCount.textContent = `${regMacs.length} devices · ${regNets.length} networks`;
+  if (!regMacs.length) {
+    regMacsTbody.innerHTML = '';
+    regMacsEmpty.style.display = 'block';
+  } else {
+    regMacsEmpty.style.display = 'none';
+    regMacsTbody.innerHTML = regMacs.map(r => `<tr>
+      <td>${fmt(r.src_mac)}</td>
+      <td class="dim">${r.vendor || '—'}</td>
+      <td>${fmt(r.days_seen)}</td>
+      <td class="dim">${shortTime(r.first_seen)}</td>
+      <td class="dim">${shortTime(r.last_seen)}</td>
+      <td class="${rssiClass(r.best_rssi)}">${fmt(r.best_rssi)} dBm</td>
+    </tr>`).join('');
+  }
+
+  // Regulars — networks
+  const regNetsTbody = document.getElementById('regulars-nets-tbody');
+  const regNetsEmpty = document.getElementById('regulars-nets-empty');
+  if (!regNets.length) {
+    regNetsTbody.innerHTML = '';
+    regNetsEmpty.style.display = 'block';
+  } else {
+    regNetsEmpty.style.display = 'none';
+    regNetsTbody.innerHTML = regNets.map(r => {
+      const encCls = (!r.encryption || r.encryption === 'OPEN' || r.encryption === 'None') ? 'enc-open' : 'muted';
+      return `<tr>
+        <td>${fmt(r.ssid, '<hidden>')}</td>
+        <td class="dim">${fmt(r.bssid)}</td>
+        <td>${fmt(r.days_seen)}</td>
+        <td class="dim">${shortTime(r.first_seen)}</td>
+        <td class="${rssiClass(r.best_rssi)}">${fmt(r.best_rssi)} dBm</td>
+        <td class="muted">${fmt(r.channel)}</td>
+      </tr>`;
+    }).join('');
+  }
+}
+
 // ── RSSI chart ────────────────────────────────────────────────────────────────
 async function showNetworkChart(bssid) {
   chartBssid = bssid;
@@ -344,7 +465,7 @@ function renderRssiChart(series) {
   const canvas = document.getElementById('rssi-chart');
   const empty  = document.getElementById('chart-empty');
   const legend = document.getElementById('chart-legend');
-  const nodeIds = Object.keys(series);
+  const nodeIds = Object.keys(series).sort();
   const allPts  = nodeIds.flatMap(id => series[id]);
 
   if (!allPts.length) {
@@ -446,10 +567,12 @@ function setView(view) {
   document.getElementById('view-crossnode').classList.toggle('active',       view === 'crossnode');
   document.getElementById('view-clients').classList.toggle('active',         view === 'clients');
   document.getElementById('view-detail').classList.toggle('active',          view === 'detail');
+  document.getElementById('view-presence').classList.toggle('active',         view === 'presence');
   document.getElementById('tab-feed').classList.toggle('active',       view === 'feed' || view === 'detail');
   document.getElementById('tab-networks').classList.toggle('active',   view === 'networks' || view === 'network-chart');
   document.getElementById('tab-crossnode').classList.toggle('active',  view === 'crossnode');
   document.getElementById('tab-clients').classList.toggle('active',    view === 'clients');
+  document.getElementById('tab-presence').classList.toggle('active',   view === 'presence');
 }
 
 function showFeed() {
@@ -466,6 +589,8 @@ async function showCrossNode() { setView('crossnode'); await fetchCrossNode(); }
 
 async function showClients() { setView('clients'); await fetchClients(); }
 
+async function showPresence() { setView('presence'); await fetchPresence(); }
+
 async function showDetail(node_id) {
   currentNode = node_id;
   setView('detail');
@@ -517,6 +642,7 @@ function startSSE() {
       if      (currentView === 'networks')      await fetchNetworks();
       else if (currentView === 'network-chart') await loadAndRenderChart();
       else if (currentView === 'crossnode')     await fetchCrossNode();
+      else if (currentView === 'presence')      await fetchPresence();
 
       const nodes = await fetch('/api/nodes').then(r => r.json());
       nodeData = Object.fromEntries(nodes.map(n => [n.node_id, n]));

coordinator/templates/dashboard.html

@@ -27,6 +27,7 @@
     <button class="tab-btn"        id="tab-networks"   onclick="showNetworks()">Networks</button>
     <button class="tab-btn"        id="tab-crossnode"  onclick="showCrossNode()">Cross-node</button>
     <button class="tab-btn"        id="tab-clients"    onclick="showClients()">Clients</button>
+    <button class="tab-btn"        id="tab-presence"   onclick="showPresence()">Presence</button>
   </div>
 
   <!-- Sidebar -->
@@ -211,6 +212,98 @@
       </div>
     </div>
 
+    <!-- Presence view -->
+    <div id="view-presence" class="view">
+
+      <!-- Present now -->
+      <div class="section-header">
+        <span class="section-title">Present Now</span>
+        <span class="section-count" id="present-count"></span>
+        <span class="presence-sub">real MACs · seen 2+ times in last hour</span>
+      </div>
+      <div class="tbl-wrap">
+        <table>
+          <thead><tr>
+            <th>MAC</th><th>Vendor</th><th>Best RSSI</th><th>Avg RSSI</th>
+            <th>Times Seen</th><th>Nodes</th><th>Last Seen</th>
+          </tr></thead>
+          <tbody id="present-tbody"></tbody>
+        </table>
+        <div class="empty" id="present-empty" style="display:none">No devices currently present.</div>
+      </div>
+
+      <!-- New arrivals -->
+      <div class="section-header" style="margin-top:8px">
+        <span class="section-title">New Arrivals</span>
+        <span class="section-count" id="arrivals-count"></span>
+        <span class="presence-sub">first seen in last 24h</span>
+      </div>
+      <div class="presence-split">
+        <div>
+          <div class="presence-sub-label">Devices</div>
+          <div class="tbl-wrap">
+            <table>
+              <thead><tr>
+                <th>MAC</th><th>Vendor</th><th>First Seen</th>
+                <th>Best RSSI</th><th>Times Seen</th>
+              </tr></thead>
+              <tbody id="arrivals-macs-tbody"></tbody>
+            </table>
+            <div class="empty" id="arrivals-macs-empty" style="display:none">No new devices.</div>
+          </div>
+        </div>
+        <div>
+          <div class="presence-sub-label">Networks</div>
+          <div class="tbl-wrap">
+            <table>
+              <thead><tr>
+                <th>SSID</th><th>BSSID</th><th>First Seen</th>
+                <th>Best RSSI</th><th>Ch</th><th>Enc</th>
+              </tr></thead>
+              <tbody id="arrivals-nets-tbody"></tbody>
+            </table>
+            <div class="empty" id="arrivals-nets-empty" style="display:none">No new networks.</div>
+          </div>
+        </div>
+      </div>
+
+      <!-- Regulars -->
+      <div class="section-header" style="margin-top:8px">
+        <span class="section-title">Regulars</span>
+        <span class="section-count" id="regulars-count"></span>
+        <span class="presence-sub">seen on 2+ distinct days</span>
+      </div>
+      <div class="presence-split">
+        <div>
+          <div class="presence-sub-label">Devices</div>
+          <div class="tbl-wrap">
+            <table>
+              <thead><tr>
+                <th>MAC</th><th>Vendor</th><th>Days Seen</th>
+                <th>First Seen</th><th>Last Seen</th><th>Best RSSI</th>
+              </tr></thead>
+              <tbody id="regulars-macs-tbody"></tbody>
+            </table>
+            <div class="empty" id="regulars-macs-empty" style="display:none">Regulars appear after 2+ days of data.</div>
+          </div>
+        </div>
+        <div>
+          <div class="presence-sub-label">Networks</div>
+          <div class="tbl-wrap">
+            <table>
+              <thead><tr>
+                <th>SSID</th><th>BSSID</th><th>Days Seen</th>
+                <th>First Seen</th><th>Best RSSI</th><th>Ch</th>
+              </tr></thead>
+              <tbody id="regulars-nets-tbody"></tbody>
+            </table>
+            <div class="empty" id="regulars-nets-empty" style="display:none">Regulars appear after 2+ days of data.</div>
+          </div>
+        </div>
+      </div>
+
+    </div><!-- /#view-presence -->
+
   </div><!-- /#main -->
 </div><!-- /#app -->