extract_iocs() (IPs, domains, hashes) ran on every classified article
but the result was never shown in Telegram alerts, never saved to
validation/results.jsonl, and never printed by review.py — pure wasted
work every poll cycle. MITRE technique extraction stays; that one is
actually used by the validation tooling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Exact URL/text matching missed the same story from a different outlet
(different URL, different wording). Add fuzzy title matching via
difflib, checked both within a single poll batch (the common case —
two feeds returning the same story in the same 5-minute cycle) and
against a rolling window of recently-sent titles in the seen-DB (the
cross-cycle case, e.g. follow-up coverage a few hours later).
seen_articles.db/seen_victims.db retention bumped from 7 to 14 days
(bot_config.SEEN_RETENTION_DAYS) — the window the new title matching
actually needs, and no reason to keep dedup history longer than that.
validation/run_validation.py: switch to the shared bot_config mapping,
and prune results.jsonl to the same 14-day window each poll instead of
growing forever — it's a testing aid, not an archive.
Also fixes a bug where a future-dated (bad clock/feed) article stayed
"recent" indefinitely instead of aging out after 48 hours, and drops
the legacy MD5 url-hash field that SHA-256 fingerprints replaced.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- New alert topic (/on_ransomware) separate from RSS malware articles,
so a busy leak day doesn't bury other malware coverage.
- Filter victims to Latvia/Estonia/Lithuania before they ever touch the
seen-DB — the global feed is already on ransomware.live's own site.
- Retention bumped to 14 days (bot_config.SEEN_RETENTION_DAYS).
- Add is_first_run(): lets the caller distinguish a genuinely fresh
seen-DB from a restart of an already-running bot.
- Fix a naive-vs-aware datetime comparison that would TypeError-crash
a poll cycle if the API ever returned a timestamp without a UTC offset.
- Drop the unused _victim_id/iocs fields from to_article()'s output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Category labels, feed-file mappings, and emoji were copy-pasted across
threat_intel_bot.py, check_feeds.py, and validation/run_validation.py,
which is how the ransomware category split almost missed one of them.
check_feeds.py now imports from bot_config.py instead of keeping its
own copy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bot
bot