diff --git a/README.md b/README.md index cb2a640..3687eeb 100644 --- a/README.md +++ b/README.md @@ -1,298 +1,104 @@ -# Threat Intelligence RSS Telegram Bot +# telegram-rss-bot -A sophisticated Telegram bot that monitors cybersecurity RSS feeds and delivers real-time alerts to Telegram chats or topics. Automatically classifies content by severity (critical/high/medium/low), quality, and category (News, Malware, Threat Intel, OSINT, Research). - -**Perfect for:** Security teams, SOC analysts, threat hunters, and cybersecurity enthusiasts who want curated threat intelligence delivered directly to Telegram. - ---- - -## Features - -- **Multi-Category Feeds:** News, Malware, Threat Intel, OSINT, Research -- **Telegram Topics Support:** Different categories post to different topics in the same group -- **Intelligent Classification:** Auto-classifies articles by severity and quality -- **Duplicate Prevention:** Deduplicates articles across feeds and sources -- **Quality Filtering:** Filter by minimum quality score and severity levels -- **SQLite Tracking:** Persistent tracking of seen articles -- **Async Architecture:** High-performance async/await design -- **Feed Health Monitoring:** `/stats` command shows feed status - ---- - -## Quick Start - -### 1. Prerequisites - -```bash -# Python 3.9+ -python3 --version - -# Install dependencies -pip install python-telegram-bot feedparser aiohttp beautifulsoup4 -``` - -### 2. Get a Telegram Bot Token - -1. Open Telegram and message [@BotFather](https://t.me/BotFather) -2. Send `/newbot` -3. Follow prompts to create your bot -4. Copy the token (looks like `123456789:ABCdefGHIjklMNOpqrsTUVwxyz`) - -### 3. Configure Your Bot - -**Option 1: Environment Variable** -```bash -export BOT_TOKEN="your_bot_token_here" -``` - -**Option 2: Create `.env` File** -```bash -# Create .env file in project directory -echo 'BOT_TOKEN=your_bot_token_here' > .env -``` - -### 4. Run the Bot - -```bash -python3 threat_intel_bot.py -``` - -You should see: -``` -🚀 Threat Intel Bot is running! -📊 0 subscribers loaded -``` - -### 5. Start Receiving Alerts - -In Telegram: -1. Find your bot (search by the name you gave it) -2. Send `/start` to see available commands -3. Enable categories you want: - - `/on_news` - News feeds - - `/on_threat_intel` - Threat intelligence - - `/on_malware` - Malware analysis - - `/on_osint` - OSINT feeds - - `/on_research` - Security research - ---- +Telegram bot that monitors cybersecurity RSS feeds and delivers real-time alerts. Articles are classified by severity and quality before delivery. ## Project Structure ``` -rss/ -├── threat_intel_bot.py # Main bot (commands, subscriptions, alerts) -├── rss_manager.py # RSS feed parsing and article extraction -├── content_classifier.py # Article classification (severity/quality) -├── check_feeds.py # Utility to check feed health -│ -├── feeds/ # RSS feed configurations -│ ├── news_feeds.json # News sources -│ ├── malware_feeds.json # Malware analysis blogs -│ ├── threat_intel_feeds.json # Threat intelligence feeds -│ ├── osint_feeds.json # OSINT resources -│ └── research_feeds.json # Security research -│ -├── subscribers.json # Subscriber data (auto-created) -└── seen_articles.db # SQLite DB tracking seen articles +rss_telegram_bot/ +├── threat_intel_bot.py # Main bot — commands, subscriptions, alert dispatch +├── rss_manager.py # Feed fetching, dedup, message formatting +├── content_classifier.py # Severity/quality scoring, CVE/actor extraction +├── check_feeds.py # CLI utility to check feed health +├── feeds/ +│ ├── news_feeds.json +│ ├── malware_feeds.json +│ ├── threat_intel_feeds.json +│ ├── osint_feeds.json +│ └── research_feeds.json +├── .env # Not committed — BOT_TOKEN + optional filters +├── subscribers.json # Auto-managed — chat/topic subscriptions +└── seen_articles.db # SQLite — tracks sent articles (7-day retention) ``` ---- +## Setup -## File Reference: What to Edit +```bash +pip install python-telegram-bot feedparser aiohttp beautifulsoup4 +``` -| What You Want to Do | File to Edit | Section | -|---------------------|--------------|---------| -| **Add/remove news feeds** | `feeds/news_feeds.json` | Add URL under category | -| **Add/remove threat intel feeds** | `feeds/threat_intel_feeds.json` | Add URL under category | -| **Add/remove malware feeds** | `feeds/malware_feeds.json` | Add URL under category | -| **Add/remove OSINT feeds** | `feeds/osint_feeds.json` | Add URL under category | -| **Add/remove research feeds** | `feeds/research_feeds.json` | Add URL under category | -| **Change quality threshold** | `.env` or environment | Set `MIN_QUALITY_SCORE=50` | -| **Filter by severity** | `.env` or environment | Set `ALLOWED_SEVERITIES=critical,high` | -| **Change polling interval** | `threat_intel_bot.py` | Line 475 (default: 300 seconds) | -| **Customize classification rules** | `content_classifier.py` | Keyword dictionaries | +Create `.env`: ---- +```env +BOT_TOKEN=your_token_here +MIN_QUALITY_SCORE=0 +ALLOWED_SEVERITIES=critical,high,medium,low +``` + +Run: + +```bash +python3 threat_intel_bot.py +``` ## Bot Commands -### User Commands +| Command | Action | +|---|---| +| `/on_` | Subscribe this chat/topic to a category | +| `/off_` | Unsubscribe | +| `/stats` | Check feed health and subscriber count | +| `/help` | Show commands | -| Command | Description | -|---------|-------------| -| `/start` | Show welcome message and available commands | -| `/help` | Same as `/start` | -| `/on_news` | Subscribe to news feeds | -| `/on_malware` | Subscribe to malware analysis | -| `/on_threat_intel` | Subscribe to threat intelligence | -| `/on_osint` | Subscribe to OSINT feeds | -| `/on_research` | Subscribe to security research | -| `/off_news` | Unsubscribe from news | -| `/off_malware` | Unsubscribe from malware | -| `/off_threat_intel` | Unsubscribe from threat intel | -| `/off_osint` | Unsubscribe from OSINT | -| `/off_research` | Unsubscribe from research | -| `/stats` | Check feed health and status | +Categories: `news`, `malware`, `threat_intel`, `osint`, `research` ---- +**Telegram Topics:** Run `/on_` inside each topic to route categories to separate threads. -## Using with Telegram Topics (Forum Groups) +## Feed Management -**Setup:** -1. Create a Telegram group -2. Enable "Topics" in group settings -3. Create topics: "News", "Threat Intel", "Malware", etc. -4. Add your bot to the group - -**Subscribe topics to categories:** -- Open "News" topic → Send `/on_news` -- Open "Threat Intel" topic → Send `/on_threat_intel` -- Open "Malware" topic → Send `/on_malware` -- Open "OSINT" topic → Send `/on_osint` -- Open "Research" topic → Send `/on_research` - -**Result:** Each category posts to its own topic automatically! - ---- - -## Managing RSS Feeds - -### Feed File Structure - -Each feed file (`feeds/*.json`) has this format: +Each `feeds/*.json` file follows this structure: ```json { "category_name": { - "Feed Name": "https://example.com/rss.xml", - "Another Feed": "https://another.com/feed" + "Feed Display Name": "https://example.com/rss.xml" } } ``` -**Example:** Add a new threat intel feed +Add or remove feeds by editing the JSON, then restart the bot. Run `python3 check_feeds.py` to verify feed health before deploying. -**File:** `feeds/threat_intel_feeds.json` +## Classification -```json -{ - "threat_intel": { - "SANS ISC": "https://isc.sans.edu/rssfeed.xml", - "Your New Feed": "https://newfeed.com/rss.xml" - } -} +**Severity** is keyword-based (title + description): +- `critical` — zero-days, active exploitation, RCE, ransomware +- `high` — privesc, auth bypass, code execution, kernel exploits +- `medium` — XSS, CSRF, DoS, memory corruption +- `low` — everything else + +**Quality score (0–100)** factors in: source reputation, content length, presence of CVEs, PoC indicators, threat actor mentions, MITRE techniques. + +Filter via env vars: +```env +MIN_QUALITY_SCORE=50 +ALLOWED_SEVERITIES=critical,high ``` -**After editing:** Restart the bot +## Systemd Service (VPS) -```bash -# Stop: Ctrl+C -# Start: -python3 threat_intel_bot.py -``` - ---- - -## Content Classification - -### How Articles Are Classified - -**Quality Score (0-100):** -- High-quality sources (Project Zero, Mandiant): +30 points -- Deep dive/technical analysis: +25 points -- Proof-of-concept/exploit: +20 points -- Research papers: +15 points -- Vulnerability advisory: +10 points - -**Severity Levels:** -- **Critical:** Zero-days, active exploitation, RCE, ransomware -- **High:** Privilege escalation, auth bypass, code execution -- **Medium:** XSS, CSRF, information disclosure, DoS -- **Low:** Everything else - -### Filtering Articles - -**By Quality Score:** -```bash -# Only show high-quality articles (score 50+) -export MIN_QUALITY_SCORE=50 -python3 threat_intel_bot.py -``` - -**By Severity:** -```bash -# Only critical and high severity -export ALLOWED_SEVERITIES="critical,high" -python3 threat_intel_bot.py -``` - -**Both:** -```bash -export MIN_QUALITY_SCORE=40 -export ALLOWED_SEVERITIES="critical,high,medium" -python3 threat_intel_bot.py -``` - ---- - -## Advanced Configuration - -### Environment Variables - -Create `.env` file: - -```bash -# Required -BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrsTUVwxyz - -# Optional filters -MIN_QUALITY_SCORE=0 # 0-100 (default: 0 = no filtering) -ALLOWED_SEVERITIES=critical,high,medium,low # Comma-separated -``` - ---- - -## Checking Feed Health - -```bash -# Check if all feeds are working -python3 check_feeds.py - -# Or use the bot command -# In Telegram: /stats -``` - -**Output shows:** -- ✅ Online feeds (with article count) -- ❌ Offline/broken feeds (with error) -- Last update time - ---- - -## Running as a Service (Linux) - -### Systemd Service - -**1. Create service file:** - -```bash -sudo nano /etc/systemd/system/threat-intel-bot.service -``` - -**2. Add this content:** +`/etc/systemd/system/rss-bot.service`: ```ini [Unit] -Description=Threat Intelligence RSS Telegram Bot +Description=Threat Intel RSS Telegram Bot After=network.target [Service] Type=simple -User=your_username -WorkingDirectory=/home/your_username/Documents/telegram-bots/rss -Environment="BOT_TOKEN=your_bot_token_here" -ExecStart=/usr/bin/python3 /home/your_username/Documents/telegram-bots/rss/threat_intel_bot.py +User=your_user +WorkingDirectory=/home/your_user/rss_telegram_bot +EnvironmentFile=/home/your_user/rss_telegram_bot/.env +ExecStart=/usr/bin/python3 threat_intel_bot.py Restart=always RestartSec=10 @@ -300,403 +106,15 @@ RestartSec=10 WantedBy=multi-user.target ``` -**3. Enable and start:** - ```bash sudo systemctl daemon-reload -sudo systemctl enable threat-intel-bot -sudo systemctl start threat-intel-bot +sudo systemctl enable --now rss-bot +sudo journalctl -u rss-bot -f ``` -**4. Check status:** +## Notes -```bash -sudo systemctl status threat-intel-bot -sudo journalctl -u threat-intel-bot -f # View logs -``` - ---- - -## Running with Docker (Optional) - -### Dockerfile - -```dockerfile -FROM python:3.11-slim - -WORKDIR /app - -# Install dependencies -RUN pip install python-telegram-bot feedparser aiohttp beautifulsoup4 - -# Copy bot files -COPY . /app - -# Run bot -CMD ["python3", "threat_intel_bot.py"] -``` - -### Docker Compose - -```yaml -version: '3.8' - -services: - threat-intel-bot: - build: . - container_name: threat-intel-bot - restart: unless-stopped - environment: - - BOT_TOKEN=${BOT_TOKEN} - - MIN_QUALITY_SCORE=${MIN_QUALITY_SCORE:-0} - - ALLOWED_SEVERITIES=${ALLOWED_SEVERITIES:-critical,high,medium,low} - volumes: - - ./subscribers.json:/app/subscribers.json - - ./seen_articles.db:/app/seen_articles.db -``` - -**Run:** - -```bash -# Create .env file first -docker compose up -d -``` - ---- - -## Customization - -### Add Your Own Feed Categories - -**1. Edit `threat_intel_bot.py`:** - -Find `CATEGORY_CONFIG` (line 37): - -```python -CATEGORY_CONFIG = { - "news": {"label": "News", "emoji": "📰", "feeds_file": "feeds/news_feeds.json"}, - "malware": {"label": "Malware", "emoji": "🦠", "feeds_file": "feeds/malware_feeds.json"}, - # Add new category: - "your_category": {"label": "Your Category", "emoji": "🔥", "feeds_file": "feeds/your_feeds.json"}, -} -``` - -**2. Create feed file:** - -```bash -nano feeds/your_feeds.json -``` - -```json -{ - "your_category": { - "Feed Name 1": "https://example.com/rss.xml", - "Feed Name 2": "https://another.com/feed" - } -} -``` - -**3. Restart bot** - -Users can now use `/on_your_category` and `/off_your_category` - ---- - -### Customize Classification Keywords - -**File:** `content_classifier.py` - -**Example:** Add new critical keywords - -```python -CRITICAL_KEYWORDS = { - 'zero-day', '0day', 'zero day', - # Add your keywords: - 'your_critical_term', - 'another_urgent_keyword', -} -``` - -**Example:** Add high-quality sources - -```python -HIGH_QUALITY_SOURCES = { - 'watchTowr Labs', 'Doyensec Blog', - # Add your sources (exact name from feed): - 'Your Favorite Security Blog', -} -``` - ---- - -## Troubleshooting - -### Bot Not Responding - -**Check if running:** -```bash -ps aux | grep threat_intel_bot.py -``` - -**View logs:** -```bash -# If running as service -sudo journalctl -u threat-intel-bot -f - -# If running manually, check terminal output -``` - ---- - -### No Alerts Received - -**1. Check subscription:** -``` -In Telegram: /stats -``` - -Should show you're subscribed. - -**2. Check feeds are working:** -```bash -python3 check_feeds.py -``` - -**3. Check quality/severity filters:** -- If `MIN_QUALITY_SCORE=80`, only very high-quality articles pass -- If `ALLOWED_SEVERITIES=critical`, only critical alerts show - -**Lower thresholds:** -```bash -export MIN_QUALITY_SCORE=0 -export ALLOWED_SEVERITIES="critical,high,medium,low" -``` - ---- - -### Feed Parse Errors - -**Symptom:** Logs show "Error fetching feed" - -**Causes:** -- Feed URL is broken/changed -- Feed server is down -- Network connectivity issue - -**Fix:** -1. Check feed URL in browser -2. Update URL in `feeds/*.json` if changed -3. Remove dead feeds -4. Restart bot - ---- - -### SQLite Database Issues - -**Reset seen articles (get all articles again):** - -```bash -# Stop bot -rm seen_articles.db - -# Restart bot (will recreate DB) -python3 threat_intel_bot.py -``` - -**Note:** First run marks all existing articles as seen (no alerts). New articles after that trigger alerts. - ---- - -## Data Files - -### subscribers.json - -**Format:** -```json -{ - "subscribers": { - "123456789": { - "topic_id": null, - "feed_types": ["news", "threat_intel"] - }, - "987654321_12345": { - "topic_id": 12345, - "feed_types": ["malware"] - } - } -} -``` - -**Explanation:** -- Key format: `chat_id` or `chat_id_topic_id` -- `topic_id`: null for private/group, number for forum topics -- `feed_types`: Array of subscribed categories - -**Manual editing:** You can edit this file, but bot does it automatically. - ---- - -### seen_articles.db - -**SQLite database tracking processed articles.** - -**Schema:** -```sql -CREATE TABLE seen_articles ( - article_key TEXT PRIMARY KEY, - seen_at TEXT NOT NULL -); -``` - -**View contents:** -```bash -sqlite3 seen_articles.db "SELECT * FROM seen_articles LIMIT 10;" -``` - ---- - -## Best Practices - -### 1. Start with Default Filters -Don't set quality/severity filters initially - see what you get first, then filter. - -### 2. Use Topics for Organization -If using in a group, enable Topics and route each category to its topic. - -### 3. Monitor Feed Health -Run `/stats` weekly to check for broken feeds. - -### 4. Curate Your Feeds -Start with defaults, add/remove based on signal-to-noise ratio. - -### 5. Run as a Service -Use systemd so bot survives reboots and crashes. - -### 6. Backup Subscriber Data -```bash -cp subscribers.json subscribers.backup.json -``` - ---- - -## Feed Categories Explained - -### News Feeds (`feeds/news_feeds.json`) -Daily cybersecurity news, breach announcements, general updates. -- **Update frequency:** Multiple times per day -- **Volume:** High -- **Use case:** Stay informed on current events - -### Malware Feeds (`feeds/malware_feeds.json`) -Malware analysis, reverse engineering blogs, threat reports. -- **Update frequency:** Daily to weekly -- **Volume:** Medium -- **Use case:** Malware research, threat hunting - -### Threat Intel Feeds (`feeds/threat_intel_feeds.json`) -Vendor threat intelligence, APT reports, threat actor profiles. -- **Update frequency:** Daily to weekly -- **Volume:** Medium -- **Use case:** Threat intelligence, SOC operations - -### OSINT Feeds (`feeds/osint_feeds.json`) -Open-source intelligence, tools, techniques, investigations. -- **Update frequency:** Weekly -- **Volume:** Low to medium -- **Use case:** OSINT research, investigative work - -### Research Feeds (`feeds/research_feeds.json`) -Deep technical research, vulnerability analysis, exploit development. -- **Update frequency:** Weekly to monthly -- **Volume:** Low (high quality) -- **Use case:** Learning, in-depth technical knowledge - ---- - -## Contributing Feeds - -**Want to add a good feed?** Edit the appropriate JSON file and submit a pull request or update your fork. - -**Criteria for good feeds:** -- Reliable RSS/Atom feed -- Regular updates (at least monthly) -- Quality content (no spam/clickbait) -- Relevant to cybersecurity - ---- - -## Performance - -**Typical resource usage:** -- **CPU:** <5% (idle), 10-20% during feed fetch -- **RAM:** ~50-100MB -- **Network:** Minimal (fetches feeds every 5 minutes) -- **Disk:** ~10MB (grows slowly with seen articles DB) - -**Scaling:** -- Tested with 50+ feeds -- Handles 100+ subscribers -- Processes ~500 articles/day - ---- - -## Security Considerations - -### Bot Token -- **Never commit** `.env` file to Git -- Treat bot token like a password -- Regenerate if leaked (via @BotFather) - -### Network Access -- Bot fetches public RSS feeds (outbound HTTPS) -- Telegram API (outbound HTTPS) -- No inbound connections needed - -### Data Privacy -- Subscriber data stored locally in `subscribers.json` -- No data sent to third parties -- Article URLs/metadata only (no personal data) - ---- - -## FAQ - -**Q: How often does the bot check feeds?** -A: Every 5 minutes (configurable in `threat_intel_bot.py` line 475) - -**Q: Can I run multiple bots from same code?** -A: Yes, just use different bot tokens and run in separate directories - -**Q: Does it support private feeds?** -A: No, only public RSS feeds. For private feeds, you'd need to add authentication - -**Q: Can I export articles to a database?** -A: Articles are in `seen_articles.db` (SQLite). You can query it or extend the code - -**Q: Why no alerts on first run?** -A: First run marks existing articles as seen to avoid spam. Only new articles after that trigger alerts - -**Q: Can I get alerts in multiple languages?** -A: Bot messages are in English. Articles are in whatever language the feed provides - ---- - -## License - -This project is open source. Use it for personal or commercial purposes. - -No warranty provided. Use at your own risk. - ---- - -## Credits - -Built with: -- [python-telegram-bot](https://python-telegram-bot.org/) - Telegram Bot API wrapper -- [feedparser](https://feedparser.readthedocs.io/) - RSS/Atom feed parser -- [aiohttp](https://docs.aiohttp.org/) - Async HTTP client -- [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup/) - HTML parsing - ---- - -**Maintained for personal use - built for the cybersecurity community.** +- First run marks all current articles as seen — no flood on startup +- Polling interval: 5 minutes (`asyncio.sleep(300)` in `threat_intel_bot.py`) +- Only today's UTC articles are processed (strict date gate in `rss_manager.py`) +- `seen_articles.db` auto-purges entries older than 7 days diff --git a/feeds/README.md b/feeds/README.md deleted file mode 100644 index 39c6fdf..0000000 --- a/feeds/README.md +++ /dev/null @@ -1,3 +0,0 @@ -This repository contains JSON feed lists used by my Telegram bot. - -The goal is simple: keep curated RSS sources in one place so the bot can automatically stay up to date with news, OSINT, malware, threat intel, and research updates.