bot/telegram-rss-bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Consolidate to single lean README, remove feeds/README.md

Browse Source
This commit is contained in:
bot
2026-04-23 22:18:11 +03:00
parent f72fddc835
commit e3307391c7
2 changed files with 71 additions and 656 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+71 -653
View File
@@ -1,298 +1,104 @@
# Threat Intelligence RSS Telegram Bot # telegram-rss-bot
A sophisticated Telegram bot that monitors cybersecurity RSS feeds and delivers real-time alerts to Telegram chats or topics. Automatically classifies content by severity (critical/high/medium/low), quality, and category (News, Malware, Threat Intel, OSINT, Research). Telegram bot that monitors cybersecurity RSS feeds and delivers real-time alerts. Articles are classified by severity and quality before delivery.
**Perfect for:** Security teams, SOC analysts, threat hunters, and cybersecurity enthusiasts who want curated threat intelligence delivered directly to Telegram.
---
## Features
- **Multi-Category Feeds:** News, Malware, Threat Intel, OSINT, Research
- **Telegram Topics Support:** Different categories post to different topics in the same group
- **Intelligent Classification:** Auto-classifies articles by severity and quality
- **Duplicate Prevention:** Deduplicates articles across feeds and sources
- **Quality Filtering:** Filter by minimum quality score and severity levels
- **SQLite Tracking:** Persistent tracking of seen articles
- **Async Architecture:** High-performance async/await design
- **Feed Health Monitoring:** `/stats` command shows feed status
---
## Quick Start
### 1. Prerequisites
```bash
# Python 3.9+
python3 --version
# Install dependencies
pip install python-telegram-bot feedparser aiohttp beautifulsoup4
```
### 2. Get a Telegram Bot Token
1. Open Telegram and message [@BotFather](https://t.me/BotFather)
2. Send `/newbot`
3. Follow prompts to create your bot
4. Copy the token (looks like `123456789:ABCdefGHIjklMNOpqrsTUVwxyz`)
### 3. Configure Your Bot
**Option 1: Environment Variable**
```bash
export BOT_TOKEN="your_bot_token_here"
```
**Option 2: Create `.env` File**
```bash
# Create .env file in project directory
echo 'BOT_TOKEN=your_bot_token_here' > .env
```
### 4. Run the Bot
```bash
python3 threat_intel_bot.py
```
You should see:
```
🚀 Threat Intel Bot is running!
📊 0 subscribers loaded
```
### 5. Start Receiving Alerts
In Telegram:
1. Find your bot (search by the name you gave it)
2. Send `/start` to see available commands
3. Enable categories you want:
- `/on_news` - News feeds
- `/on_threat_intel` - Threat intelligence
- `/on_malware` - Malware analysis
- `/on_osint` - OSINT feeds
- `/on_research` - Security research
---
## Project Structure ## Project Structure
``` ```
rss/ rss_telegram_bot/
├── threat_intel_bot.py # Main bot (commands, subscriptions, alerts) ├── threat_intel_bot.py # Main bot — commands, subscriptions, alert dispatch
├── rss_manager.py # RSS feed parsing and article extraction ├── rss_manager.py # Feed fetching, dedup, message formatting
├── content_classifier.py # Article classification (severity/quality) ├── content_classifier.py # Severity/quality scoring, CVE/actor extraction
├── check_feeds.py # Utility to check feed health ├── check_feeds.py # CLI utility to check feed health
│ ├── feeds/
├── feeds/ # RSS feed configurations │ ├── news_feeds.json
│ ├── news_feeds.json # News sources │ ├── malware_feeds.json
│ ├── malware_feeds.json # Malware analysis blogs │ ├── threat_intel_feeds.json
│ ├── threat_intel_feeds.json # Threat intelligence feeds │ ├── osint_feeds.json
│ ├── osint_feeds.json # OSINT resources │ └── research_feeds.json
│ └── research_feeds.json # Security research ├── .env # Not committed — BOT_TOKEN + optional filters
│ ├── subscribers.json # Auto-managed — chat/topic subscriptions
├── subscribers.json # Subscriber data (auto-created) └── seen_articles.db # SQLite — tracks sent articles (7-day retention)
└── seen_articles.db # SQLite DB tracking seen articles
``` ```
--- ## Setup
## File Reference: What to Edit ```bash
pip install python-telegram-bot feedparser aiohttp beautifulsoup4
```
| What You Want to Do | File to Edit | Section | Create `.env`:
|---------------------|--------------|---------|
| **Add/remove news feeds** | `feeds/news_feeds.json` | Add URL under category |
| **Add/remove threat intel feeds** | `feeds/threat_intel_feeds.json` | Add URL under category |
| **Add/remove malware feeds** | `feeds/malware_feeds.json` | Add URL under category |
| **Add/remove OSINT feeds** | `feeds/osint_feeds.json` | Add URL under category |
| **Add/remove research feeds** | `feeds/research_feeds.json` | Add URL under category |
| **Change quality threshold** | `.env` or environment | Set `MIN_QUALITY_SCORE=50` |
| **Filter by severity** | `.env` or environment | Set `ALLOWED_SEVERITIES=critical,high` |
| **Change polling interval** | `threat_intel_bot.py` | Line 475 (default: 300 seconds) |
| **Customize classification rules** | `content_classifier.py` | Keyword dictionaries |
--- ```env
BOT_TOKEN=your_token_here
MIN_QUALITY_SCORE=0
ALLOWED_SEVERITIES=critical,high,medium,low
```
Run:
```bash
python3 threat_intel_bot.py
```
## Bot Commands ## Bot Commands
### User Commands | Command | Action |
|---|---|
| `/on_<category>` | Subscribe this chat/topic to a category |
| `/off_<category>` | Unsubscribe |
| `/stats` | Check feed health and subscriber count |
| `/help` | Show commands |
| Command | Description | Categories: `news`, `malware`, `threat_intel`, `osint`, `research`
|---------|-------------|
| `/start` | Show welcome message and available commands |
| `/help` | Same as `/start` |
| `/on_news` | Subscribe to news feeds |
| `/on_malware` | Subscribe to malware analysis |
| `/on_threat_intel` | Subscribe to threat intelligence |
| `/on_osint` | Subscribe to OSINT feeds |
| `/on_research` | Subscribe to security research |
| `/off_news` | Unsubscribe from news |
| `/off_malware` | Unsubscribe from malware |
| `/off_threat_intel` | Unsubscribe from threat intel |
| `/off_osint` | Unsubscribe from OSINT |
| `/off_research` | Unsubscribe from research |
| `/stats` | Check feed health and status |
--- **Telegram Topics:** Run `/on_<category>` inside each topic to route categories to separate threads.
## Using with Telegram Topics (Forum Groups) ## Feed Management
**Setup:** Each `feeds/*.json` file follows this structure:
1. Create a Telegram group
2. Enable "Topics" in group settings
3. Create topics: "News", "Threat Intel", "Malware", etc.
4. Add your bot to the group
**Subscribe topics to categories:**
- Open "News" topic → Send `/on_news`
- Open "Threat Intel" topic → Send `/on_threat_intel`
- Open "Malware" topic → Send `/on_malware`
- Open "OSINT" topic → Send `/on_osint`
- Open "Research" topic → Send `/on_research`
**Result:** Each category posts to its own topic automatically!
---
## Managing RSS Feeds
### Feed File Structure
Each feed file (`feeds/*.json`) has this format:
```json ```json
{ {
"category_name": { "category_name": {
"Feed Name": "https://example.com/rss.xml", "Feed Display Name": "https://example.com/rss.xml"
"Another Feed": "https://another.com/feed"
} }
} }
``` ```
**Example:** Add a new threat intel feed Add or remove feeds by editing the JSON, then restart the bot. Run `python3 check_feeds.py` to verify feed health before deploying.
**File:** `feeds/threat_intel_feeds.json` ## Classification
```json **Severity** is keyword-based (title + description):
{ - `critical` — zero-days, active exploitation, RCE, ransomware
"threat_intel": { - `high` — privesc, auth bypass, code execution, kernel exploits
"SANS ISC": "https://isc.sans.edu/rssfeed.xml", - `medium` — XSS, CSRF, DoS, memory corruption
"Your New Feed": "https://newfeed.com/rss.xml" - `low` — everything else
}
} **Quality score (0–100)** factors in: source reputation, content length, presence of CVEs, PoC indicators, threat actor mentions, MITRE techniques.
Filter via env vars:
```env
MIN_QUALITY_SCORE=50
ALLOWED_SEVERITIES=critical,high
``` ```
**After editing:** Restart the bot ## Systemd Service (VPS)
```bash `/etc/systemd/system/rss-bot.service`:
# Stop: Ctrl+C
# Start:
python3 threat_intel_bot.py
```
---
## Content Classification
### How Articles Are Classified
**Quality Score (0-100):**
- High-quality sources (Project Zero, Mandiant): +30 points
- Deep dive/technical analysis: +25 points
- Proof-of-concept/exploit: +20 points
- Research papers: +15 points
- Vulnerability advisory: +10 points
**Severity Levels:**
- **Critical:** Zero-days, active exploitation, RCE, ransomware
- **High:** Privilege escalation, auth bypass, code execution
- **Medium:** XSS, CSRF, information disclosure, DoS
- **Low:** Everything else
### Filtering Articles
**By Quality Score:**
```bash
# Only show high-quality articles (score 50+)
export MIN_QUALITY_SCORE=50
python3 threat_intel_bot.py
```
**By Severity:**
```bash
# Only critical and high severity
export ALLOWED_SEVERITIES="critical,high"
python3 threat_intel_bot.py
```
**Both:**
```bash
export MIN_QUALITY_SCORE=40
export ALLOWED_SEVERITIES="critical,high,medium"
python3 threat_intel_bot.py
```
---
## Advanced Configuration
### Environment Variables
Create `.env` file:
```bash
# Required
BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrsTUVwxyz
# Optional filters
MIN_QUALITY_SCORE=0 # 0-100 (default: 0 = no filtering)
ALLOWED_SEVERITIES=critical,high,medium,low # Comma-separated
```
---
## Checking Feed Health
```bash
# Check if all feeds are working
python3 check_feeds.py
# Or use the bot command
# In Telegram: /stats
```
**Output shows:**
- ✅ Online feeds (with article count)
- ❌ Offline/broken feeds (with error)
- Last update time
---
## Running as a Service (Linux)
### Systemd Service
**1. Create service file:**
```bash
sudo nano /etc/systemd/system/threat-intel-bot.service
```
**2. Add this content:**
```ini ```ini
[Unit] [Unit]
Description=Threat Intelligence RSS Telegram Bot Description=Threat Intel RSS Telegram Bot
After=network.target After=network.target
[Service] [Service]
Type=simple Type=simple
User=your_username User=your_user
WorkingDirectory=/home/your_username/Documents/telegram-bots/rss WorkingDirectory=/home/your_user/rss_telegram_bot
Environment="BOT_TOKEN=your_bot_token_here" EnvironmentFile=/home/your_user/rss_telegram_bot/.env
ExecStart=/usr/bin/python3 /home/your_username/Documents/telegram-bots/rss/threat_intel_bot.py ExecStart=/usr/bin/python3 threat_intel_bot.py
Restart=always Restart=always
RestartSec=10 RestartSec=10
@@ -300,403 +106,15 @@ RestartSec=10
WantedBy=multi-user.target WantedBy=multi-user.target
``` ```
**3. Enable and start:**
```bash ```bash
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl enable threat-intel-bot sudo systemctl enable --now rss-bot
sudo systemctl start threat-intel-bot sudo journalctl -u rss-bot -f
``` ```
**4. Check status:** ## Notes
```bash - First run marks all current articles as seen — no flood on startup
sudo systemctl status threat-intel-bot - Polling interval: 5 minutes (`asyncio.sleep(300)` in `threat_intel_bot.py`)
sudo journalctl -u threat-intel-bot -f # View logs - Only today's UTC articles are processed (strict date gate in `rss_manager.py`)
``` - `seen_articles.db` auto-purges entries older than 7 days
---
## Running with Docker (Optional)
### Dockerfile
```dockerfile
FROM python:3.11-slim
WORKDIR /app
# Install dependencies
RUN pip install python-telegram-bot feedparser aiohttp beautifulsoup4
# Copy bot files
COPY . /app
# Run bot
CMD ["python3", "threat_intel_bot.py"]
```
### Docker Compose
```yaml
version: '3.8'
services:
threat-intel-bot:
build: .
container_name: threat-intel-bot
restart: unless-stopped
environment:
- BOT_TOKEN=${BOT_TOKEN}
- MIN_QUALITY_SCORE=${MIN_QUALITY_SCORE:-0}
- ALLOWED_SEVERITIES=${ALLOWED_SEVERITIES:-critical,high,medium,low}
volumes:
- ./subscribers.json:/app/subscribers.json
- ./seen_articles.db:/app/seen_articles.db
```
**Run:**
```bash
# Create .env file first
docker compose up -d
```
---
## Customization
### Add Your Own Feed Categories
**1. Edit `threat_intel_bot.py`:**
Find `CATEGORY_CONFIG` (line 37):
```python
CATEGORY_CONFIG = {
"news": {"label": "News", "emoji": "📰", "feeds_file": "feeds/news_feeds.json"},
"malware": {"label": "Malware", "emoji": "🦠", "feeds_file": "feeds/malware_feeds.json"},
# Add new category:
"your_category": {"label": "Your Category", "emoji": "🔥", "feeds_file": "feeds/your_feeds.json"},
}
```
**2. Create feed file:**
```bash
nano feeds/your_feeds.json
```
```json
{
"your_category": {
"Feed Name 1": "https://example.com/rss.xml",
"Feed Name 2": "https://another.com/feed"
}
}
```
**3. Restart bot**
Users can now use `/on_your_category` and `/off_your_category`
---
### Customize Classification Keywords
**File:** `content_classifier.py`
**Example:** Add new critical keywords
```python
CRITICAL_KEYWORDS = {
'zero-day', '0day', 'zero day',
# Add your keywords:
'your_critical_term',
'another_urgent_keyword',
}
```
**Example:** Add high-quality sources
```python
HIGH_QUALITY_SOURCES = {
'watchTowr Labs', 'Doyensec Blog',
# Add your sources (exact name from feed):
'Your Favorite Security Blog',
}
```
---
## Troubleshooting
### Bot Not Responding
**Check if running:**
```bash
ps aux | grep threat_intel_bot.py
```
**View logs:**
```bash
# If running as service
sudo journalctl -u threat-intel-bot -f
# If running manually, check terminal output
```
---
### No Alerts Received
**1. Check subscription:**
```
In Telegram: /stats
```
Should show you're subscribed.
**2. Check feeds are working:**
```bash
python3 check_feeds.py
```
**3. Check quality/severity filters:**
- If `MIN_QUALITY_SCORE=80`, only very high-quality articles pass
- If `ALLOWED_SEVERITIES=critical`, only critical alerts show
**Lower thresholds:**
```bash
export MIN_QUALITY_SCORE=0
export ALLOWED_SEVERITIES="critical,high,medium,low"
```
---
### Feed Parse Errors
**Symptom:** Logs show "Error fetching feed"
**Causes:**
- Feed URL is broken/changed
- Feed server is down
- Network connectivity issue
**Fix:**
1. Check feed URL in browser
2. Update URL in `feeds/*.json` if changed
3. Remove dead feeds
4. Restart bot
---
### SQLite Database Issues
**Reset seen articles (get all articles again):**
```bash
# Stop bot
rm seen_articles.db
# Restart bot (will recreate DB)
python3 threat_intel_bot.py
```
**Note:** First run marks all existing articles as seen (no alerts). New articles after that trigger alerts.
---
## Data Files
### subscribers.json
**Format:**
```json
{
"subscribers": {
"123456789": {
"topic_id": null,
"feed_types": ["news", "threat_intel"]
},
"987654321_12345": {
"topic_id": 12345,
"feed_types": ["malware"]
}
}
}
```
**Explanation:**
- Key format: `chat_id` or `chat_id_topic_id`
- `topic_id`: null for private/group, number for forum topics
- `feed_types`: Array of subscribed categories
**Manual editing:** You can edit this file, but bot does it automatically.
---
### seen_articles.db
**SQLite database tracking processed articles.**
**Schema:**
```sql
CREATE TABLE seen_articles (
article_key TEXT PRIMARY KEY,
seen_at TEXT NOT NULL
);
```
**View contents:**
```bash
sqlite3 seen_articles.db "SELECT * FROM seen_articles LIMIT 10;"
```
---
## Best Practices
### 1. Start with Default Filters
Don't set quality/severity filters initially - see what you get first, then filter.
### 2. Use Topics for Organization
If using in a group, enable Topics and route each category to its topic.
### 3. Monitor Feed Health
Run `/stats` weekly to check for broken feeds.
### 4. Curate Your Feeds
Start with defaults, add/remove based on signal-to-noise ratio.
### 5. Run as a Service
Use systemd so bot survives reboots and crashes.
### 6. Backup Subscriber Data
```bash
cp subscribers.json subscribers.backup.json
```
---
## Feed Categories Explained
### News Feeds (`feeds/news_feeds.json`)
Daily cybersecurity news, breach announcements, general updates.
- **Update frequency:** Multiple times per day
- **Volume:** High
- **Use case:** Stay informed on current events
### Malware Feeds (`feeds/malware_feeds.json`)
Malware analysis, reverse engineering blogs, threat reports.
- **Update frequency:** Daily to weekly
- **Volume:** Medium
- **Use case:** Malware research, threat hunting
### Threat Intel Feeds (`feeds/threat_intel_feeds.json`)
Vendor threat intelligence, APT reports, threat actor profiles.
- **Update frequency:** Daily to weekly
- **Volume:** Medium
- **Use case:** Threat intelligence, SOC operations
### OSINT Feeds (`feeds/osint_feeds.json`)
Open-source intelligence, tools, techniques, investigations.
- **Update frequency:** Weekly
- **Volume:** Low to medium
- **Use case:** OSINT research, investigative work
### Research Feeds (`feeds/research_feeds.json`)
Deep technical research, vulnerability analysis, exploit development.
- **Update frequency:** Weekly to monthly
- **Volume:** Low (high quality)
- **Use case:** Learning, in-depth technical knowledge
---
## Contributing Feeds
**Want to add a good feed?** Edit the appropriate JSON file and submit a pull request or update your fork.
**Criteria for good feeds:**
- Reliable RSS/Atom feed
- Regular updates (at least monthly)
- Quality content (no spam/clickbait)
- Relevant to cybersecurity
---
## Performance
**Typical resource usage:**
- **CPU:** <5% (idle), 10-20% during feed fetch
- **RAM:** ~50-100MB
- **Network:** Minimal (fetches feeds every 5 minutes)
- **Disk:** ~10MB (grows slowly with seen articles DB)
**Scaling:**
- Tested with 50+ feeds
- Handles 100+ subscribers
- Processes ~500 articles/day
---
## Security Considerations
### Bot Token
- **Never commit** `.env` file to Git
- Treat bot token like a password
- Regenerate if leaked (via @BotFather)
### Network Access
- Bot fetches public RSS feeds (outbound HTTPS)
- Telegram API (outbound HTTPS)
- No inbound connections needed
### Data Privacy
- Subscriber data stored locally in `subscribers.json`
- No data sent to third parties
- Article URLs/metadata only (no personal data)
---
## FAQ
**Q: How often does the bot check feeds?**
A: Every 5 minutes (configurable in `threat_intel_bot.py` line 475)
**Q: Can I run multiple bots from same code?**
A: Yes, just use different bot tokens and run in separate directories
**Q: Does it support private feeds?**
A: No, only public RSS feeds. For private feeds, you'd need to add authentication
**Q: Can I export articles to a database?**
A: Articles are in `seen_articles.db` (SQLite). You can query it or extend the code
**Q: Why no alerts on first run?**
A: First run marks existing articles as seen to avoid spam. Only new articles after that trigger alerts
**Q: Can I get alerts in multiple languages?**
A: Bot messages are in English. Articles are in whatever language the feed provides
---
## License
This project is open source. Use it for personal or commercial purposes.
No warranty provided. Use at your own risk.
---
## Credits
Built with:
- [python-telegram-bot](https://python-telegram-bot.org/) - Telegram Bot API wrapper
- [feedparser](https://feedparser.readthedocs.io/) - RSS/Atom feed parser
- [aiohttp](https://docs.aiohttp.org/) - Async HTTP client
- [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup/) - HTML parsing
---
**Maintained for personal use - built for the cybersecurity community.**
feeds/README.md
-3
View File
@@ -1,3 +0,0 @@
This repository contains JSON feed lists used by my Telegram bot.
The goal is simple: keep curated RSS sources in one place so the bot can automatically stay up to date with news, OSINT, malware, threat intel, and research updates.