Sync README with the ransomware/dedup/retention changes, fix deploy docs
- Document the LV/EE/LT ransomware filter and 14-day retention (was still saying 7 days and "posts to the malware topic" after the ransomware split). - Note cross-source dedup and that a restart now re-checks for missed articles instead of silently marking them seen. - Setup: create a venv and install from requirements.txt instead of a bare pip-install line with no version pins, matching what the systemd unit actually expects at venv/bin/python3. - Deploying Updates: the rsync command was missing malware_feeds.json and osint_feeds.json entirely, and put the feed files at the wrong destination path since rsync doesn't preserve the feeds/ subdirectory without --relative. Now includes all 5 feed files, bot_config.py, check_feeds.py, and requirements.txt. - systemd block updated to the actual current deployment (ubuntu user, /home/ubuntu/rss-tele-bot), not the old root/tele-bots path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,9 +7,10 @@ Telegram bot that monitors cybersecurity RSS feeds and ransomware.live victim da
|
||||
```
|
||||
rss_telegram_bot/
|
||||
├── threat_intel_bot.py # Main bot — commands, subscriptions, alert dispatch
|
||||
├── rss_manager.py # Feed fetching, dedup, message formatting
|
||||
├── content_classifier.py # CVE/actor/malware extraction
|
||||
├── ransomware_fetcher.py # ransomware.live PRO API — victim feed
|
||||
├── rss_manager.py # Feed fetching, dedup (incl. cross-source), message formatting
|
||||
├── content_classifier.py # CVE/actor/malware/MITRE extraction
|
||||
├── ransomware_fetcher.py # ransomware.live PRO API — victim feed, LV/EE/LT only
|
||||
├── bot_config.py # Shared category, feed-file, and tuning config
|
||||
├── check_feeds.py # CLI utility to check feed health
|
||||
├── feeds/
|
||||
│ ├── news_feeds.json
|
||||
@@ -22,14 +23,16 @@ rss_telegram_bot/
|
||||
│ └── review.py # pretty-print and filter results.jsonl
|
||||
├── .env # not committed — see Setup
|
||||
├── subscribers.json # auto-managed — chat/topic subscriptions
|
||||
├── seen_articles.db # SQLite — tracks sent RSS articles (7-day retention)
|
||||
└── seen_victims.db # SQLite — tracks sent ransomware victims (7-day retention)
|
||||
├── seen_articles.db # SQLite — tracks sent RSS articles + titles (14-day retention)
|
||||
└── seen_victims.db # SQLite — tracks sent ransomware victims (14-day retention)
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
pip install python-telegram-bot feedparser aiohttp beautifulsoup4
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Create `.env`:
|
||||
@@ -54,7 +57,7 @@ python3 threat_intel_bot.py
|
||||
| `/stats` | Check feed health and subscriber count |
|
||||
| `/help` | Show commands |
|
||||
|
||||
Categories: `news`, `malware`, `threat_intel`, `osint`, `research`
|
||||
Categories: `news`, `malware`, `threat_intel`, `osint`, `research`, `ransomware`
|
||||
|
||||
**Telegram Topics:** Run `/on_<category>` inside each topic to route categories to separate threads.
|
||||
|
||||
@@ -75,10 +78,10 @@ Description
|
||||
🔗 Read Full Article
|
||||
```
|
||||
|
||||
Ransomware victims (posted to the malware topic):
|
||||
Ransomware victims (posted to the `ransomware` topic, separate from `malware` so a busy leak day doesn't bury other malware sources):
|
||||
|
||||
```
|
||||
🦠 company.com claimed by LockBit
|
||||
💰 company.com claimed by LockBit
|
||||
|
||||
Sector: Finance | Country: US
|
||||
Data description from leak post
|
||||
@@ -107,7 +110,7 @@ Add or remove feeds by editing the relevant `feeds/*.json` file, then restart th
|
||||
|
||||
### ransomware.live PRO API
|
||||
|
||||
When `RANSOMWARE_LIVE_API_KEY` is set, the bot polls the PRO API every 5 minutes for newly discovered ransomware victims. New entries are posted to the `malware` topic. Victims are tracked in `seen_victims.db` with 7-day retention.
|
||||
When `RANSOMWARE_LIVE_API_KEY` is set, the bot polls the PRO API every 5 minutes for newly discovered ransomware victims, filtered to companies in Latvia, Estonia, and Lithuania (`ALLOWED_RANSOMWARE_COUNTRIES` in `bot_config.py`) — the global feed is available directly on ransomware.live. New entries are posted to the `ransomware` topic (subscribe with `/on_ransomware`), kept separate from RSS `malware` articles. Victims are tracked in `seen_victims.db` with 14-day retention.
|
||||
|
||||
## Classification
|
||||
|
||||
@@ -150,10 +153,10 @@ After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/root/tele-bots/rss_splited_bot
|
||||
EnvironmentFile=/root/tele-bots/rss_splited_bot/.env
|
||||
ExecStart=/root/tele-bots/rss_splited_bot/venv/bin/python3 threat_intel_bot.py
|
||||
User=ubuntu
|
||||
WorkingDirectory=/home/ubuntu/rss-tele-bot
|
||||
EnvironmentFile=/home/ubuntu/rss-tele-bot/.env
|
||||
ExecStart=/home/ubuntu/rss-tele-bot/venv/bin/python3 threat_intel_bot.py
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
@@ -167,18 +170,21 @@ sudo systemctl enable --now rss-bot
|
||||
sudo journalctl -u rss-bot -f
|
||||
```
|
||||
|
||||
Currently deployed and running this way — `enabled` (survives reboot) and `Restart=always` (survives crashes), so it no longer depends on an active SSH session.
|
||||
|
||||
## Deploying Updates
|
||||
|
||||
From the local project directory:
|
||||
|
||||
```bash
|
||||
rsync -avz ransomware_fetcher.py threat_intel_bot.py rss_manager.py content_classifier.py feeds/threat_intel_feeds.json feeds/news_feeds.json feeds/research_feeds.json root@deployer:/root/tele-bots/rss_splited_bot/
|
||||
ssh root@deployer "systemctl restart rss-bot"
|
||||
rsync -avz --relative ransomware_fetcher.py threat_intel_bot.py rss_manager.py content_classifier.py bot_config.py check_feeds.py requirements.txt ./feeds/threat_intel_feeds.json ./feeds/news_feeds.json ./feeds/research_feeds.json ./feeds/malware_feeds.json ./feeds/osint_feeds.json ubuntu@<vps-ip>:/home/ubuntu/rss-tele-bot/
|
||||
ssh ubuntu@<vps-ip> "sudo systemctl restart rss-bot"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- First run marks all current articles and victims as seen — no flood on startup
|
||||
- First run (empty seen-DB) marks all current articles and victims as seen — no flood on startup. A restart of an already-initialized bot instead fetches and alerts on anything published while it was stopped, so a redeploy doesn't silently drop real alerts.
|
||||
- Polling interval: 5 minutes
|
||||
- Articles published within the last 48 hours are processed (rolling window, not calendar day)
|
||||
- `seen_articles.db` and `seen_victims.db` auto-purge entries older than 7 days
|
||||
- Cross-source duplicates (the same story from a different outlet) are caught by fuzzy title matching, not just exact URL/text matches
|
||||
- `seen_articles.db` and `seen_victims.db` auto-purge entries older than 14 days
|
||||
|
||||
Reference in New Issue
Block a user