- Document the LV/EE/LT ransomware filter and 14-day retention (was
still saying 7 days and "posts to the malware topic" after the
ransomware split).
- Note cross-source dedup and that a restart now re-checks for missed
articles instead of silently marking them seen.
- Setup: create a venv and install from requirements.txt instead of a
bare pip-install line with no version pins, matching what the
systemd unit actually expects at venv/bin/python3.
- Deploying Updates: the rsync command was missing malware_feeds.json
and osint_feeds.json entirely, and put the feed files at the wrong
destination path since rsync doesn't preserve the feeds/ subdirectory
without --relative. Now includes all 5 feed files, bot_config.py,
check_feeds.py, and requirements.txt.
- systemd block updated to the actual current deployment (ubuntu user,
/home/ubuntu/rss-tele-bot), not the old root/tele-bots path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
requirements.txt (new) pins python-telegram-bot, feedparser, aiohttp,
and beautifulsoup4 to their current stable versions — the repo had no
lockfile or pinned deps at all, just a pip-install line in the README.
validation/review.py still had a --severity filter and printed a
quality_score that content_classifier.py stopped producing a while
back (see "Remove severity and quality scoring, classifier is
extraction-only"). Every record scored UNKNOWN/?, so the flag could
never match anything. Removed; --category/--has/--today/--limit and
the extracted-field printing are unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
monitor_feeds() unconditionally ran the startup scan with
initial_run=True, which marks everything unseen as seen WITHOUT
alerting — not just on the bot's very first run, but on every restart.
Anything published while the service was stopped for a deploy was
silently dropped. RSSFeedManager.first_run and the new
RansomwareFetcher.is_first_run() were already available to tell the
two cases apart; monitor_feeds() now actually checks them, and does a
real fetch-and-alert pass for anything missed on a plain restart.
Also splits the per-article classify+send loop out into
_classify_and_send()/_send_ransomware_victims(), each wrapping a
single article/victim in its own try/except — previously one bad
article could raise past the whole batch and skip ransomware polling
for that cycle too. Category/poll-interval config now comes from
bot_config.py instead of a local copy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extract_iocs() (IPs, domains, hashes) ran on every classified article
but the result was never shown in Telegram alerts, never saved to
validation/results.jsonl, and never printed by review.py — pure wasted
work every poll cycle. MITRE technique extraction stays; that one is
actually used by the validation tooling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Exact URL/text matching missed the same story from a different outlet
(different URL, different wording). Add fuzzy title matching via
difflib, checked both within a single poll batch (the common case —
two feeds returning the same story in the same 5-minute cycle) and
against a rolling window of recently-sent titles in the seen-DB (the
cross-cycle case, e.g. follow-up coverage a few hours later).
seen_articles.db/seen_victims.db retention bumped from 7 to 14 days
(bot_config.SEEN_RETENTION_DAYS) — the window the new title matching
actually needs, and no reason to keep dedup history longer than that.
validation/run_validation.py: switch to the shared bot_config mapping,
and prune results.jsonl to the same 14-day window each poll instead of
growing forever — it's a testing aid, not an archive.
Also fixes a bug where a future-dated (bad clock/feed) article stayed
"recent" indefinitely instead of aging out after 48 hours, and drops
the legacy MD5 url-hash field that SHA-256 fingerprints replaced.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- New alert topic (/on_ransomware) separate from RSS malware articles,
so a busy leak day doesn't bury other malware coverage.
- Filter victims to Latvia/Estonia/Lithuania before they ever touch the
seen-DB — the global feed is already on ransomware.live's own site.
- Retention bumped to 14 days (bot_config.SEEN_RETENTION_DAYS).
- Add is_first_run(): lets the caller distinguish a genuinely fresh
seen-DB from a restart of an already-running bot.
- Fix a naive-vs-aware datetime comparison that would TypeError-crash
a poll cycle if the API ever returned a timestamp without a UTC offset.
- Drop the unused _victim_id/iocs fields from to_article()'s output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Category labels, feed-file mappings, and emoji were copy-pasted across
threat_intel_bot.py, check_feeds.py, and validation/run_validation.py,
which is how the ransomware category split almost missed one of them.
check_feeds.py now imports from bot_config.py instead of keeping its
own copy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bot
bot