telegram-rss-bot/README.md

# Threat Intelligence RSS Telegram Bot

A sophisticated Telegram bot that monitors cybersecurity RSS feeds and delivers real-time alerts to Telegram chats or topics. Automatically classifies content by severity (critical/high/medium/low), quality, and category (News, Malware, Threat Intel, OSINT, Research).

**Perfect for:** Security teams, SOC analysts, threat hunters, and cybersecurity enthusiasts who want curated threat intelligence delivered directly to Telegram.

---

## Features

- **Multi-Category Feeds:** News, Malware, Threat Intel, OSINT, Research
- **Telegram Topics Support:** Different categories post to different topics in the same group
- **Intelligent Classification:** Auto-classifies articles by severity and quality
- **Duplicate Prevention:** Deduplicates articles across feeds and sources
- **Quality Filtering:** Filter by minimum quality score and severity levels
- **SQLite Tracking:** Persistent tracking of seen articles
- **Async Architecture:** High-performance async/await design
- **Feed Health Monitoring:** `/stats` command shows feed status

---

## Quick Start

### 1. Prerequisites

```bash
# Python 3.9+
python3 --version

# Install dependencies
pip install python-telegram-bot feedparser aiohttp beautifulsoup4
```

### 2. Get a Telegram Bot Token

1. Open Telegram and message [@BotFather](https://t.me/BotFather)
2. Send `/newbot`
3. Follow prompts to create your bot
4. Copy the token (looks like `123456789:ABCdefGHIjklMNOpqrsTUVwxyz`)

### 3. Configure Your Bot

**Option 1: Environment Variable**
```bash
export BOT_TOKEN="your_bot_token_here"
```

**Option 2: Create `.env` File**
```bash
# Create .env file in project directory
echo 'BOT_TOKEN=your_bot_token_here' > .env
```

### 4. Run the Bot

```bash
python3 threat_intel_bot.py
```

You should see:
```
🚀 Threat Intel Bot is running!
📊 0 subscribers loaded
```

### 5. Start Receiving Alerts

In Telegram:
1. Find your bot (search by the name you gave it)
2. Send `/start` to see available commands
3. Enable categories you want:
   - `/on_news` - News feeds
   - `/on_threat_intel` - Threat intelligence
   - `/on_malware` - Malware analysis
   - `/on_osint` - OSINT feeds
   - `/on_research` - Security research

---

## Project Structure

```
rss/
├── threat_intel_bot.py           # Main bot (commands, subscriptions, alerts)
├── rss_manager.py                # RSS feed parsing and article extraction
├── content_classifier.py         # Article classification (severity/quality)
├── check_feeds.py                # Utility to check feed health
│
├── feeds/                        # RSS feed configurations
│   ├── news_feeds.json          # News sources
│   ├── malware_feeds.json       # Malware analysis blogs
│   ├── threat_intel_feeds.json  # Threat intelligence feeds
│   ├── osint_feeds.json         # OSINT resources
│   └── research_feeds.json      # Security research
│
├── subscribers.json              # Subscriber data (auto-created)
└── seen_articles.db             # SQLite DB tracking seen articles
```

---

## File Reference: What to Edit

| What You Want to Do | File to Edit | Section |
|---------------------|--------------|---------|
| **Add/remove news feeds** | `feeds/news_feeds.json` | Add URL under category |
| **Add/remove threat intel feeds** | `feeds/threat_intel_feeds.json` | Add URL under category |
| **Add/remove malware feeds** | `feeds/malware_feeds.json` | Add URL under category |
| **Add/remove OSINT feeds** | `feeds/osint_feeds.json` | Add URL under category |
| **Add/remove research feeds** | `feeds/research_feeds.json` | Add URL under category |
| **Change quality threshold** | `.env` or environment | Set `MIN_QUALITY_SCORE=50` |
| **Filter by severity** | `.env` or environment | Set `ALLOWED_SEVERITIES=critical,high` |
| **Change polling interval** | `threat_intel_bot.py` | Line 475 (default: 300 seconds) |
| **Customize classification rules** | `content_classifier.py` | Keyword dictionaries |

---

## Bot Commands

### User Commands

| Command | Description |
|---------|-------------|
| `/start` | Show welcome message and available commands |
| `/help` | Same as `/start` |
| `/on_news` | Subscribe to news feeds |
| `/on_malware` | Subscribe to malware analysis |
| `/on_threat_intel` | Subscribe to threat intelligence |
| `/on_osint` | Subscribe to OSINT feeds |
| `/on_research` | Subscribe to security research |
| `/off_news` | Unsubscribe from news |
| `/off_malware` | Unsubscribe from malware |
| `/off_threat_intel` | Unsubscribe from threat intel |
| `/off_osint` | Unsubscribe from OSINT |
| `/off_research` | Unsubscribe from research |
| `/stats` | Check feed health and status |

---

## Using with Telegram Topics (Forum Groups)

**Setup:**
1. Create a Telegram group
2. Enable "Topics" in group settings
3. Create topics: "News", "Threat Intel", "Malware", etc.
4. Add your bot to the group

**Subscribe topics to categories:**
- Open "News" topic → Send `/on_news`
- Open "Threat Intel" topic → Send `/on_threat_intel`
- Open "Malware" topic → Send `/on_malware`
- Open "OSINT" topic → Send `/on_osint`
- Open "Research" topic → Send `/on_research`

**Result:** Each category posts to its own topic automatically!

---

## Managing RSS Feeds

### Feed File Structure

Each feed file (`feeds/*.json`) has this format:

```json
{
  "category_name": {
    "Feed Name": "https://example.com/rss.xml",
    "Another Feed": "https://another.com/feed"
  }
}
```

**Example:** Add a new threat intel feed

**File:** `feeds/threat_intel_feeds.json`

```json
{
  "threat_intel": {
    "SANS ISC": "https://isc.sans.edu/rssfeed.xml",
    "Your New Feed": "https://newfeed.com/rss.xml"
  }
}
```

**After editing:** Restart the bot

```bash
# Stop: Ctrl+C
# Start:
python3 threat_intel_bot.py
```

---

## Content Classification

### How Articles Are Classified

**Quality Score (0-100):**
- High-quality sources (Project Zero, Mandiant): +30 points
- Deep dive/technical analysis: +25 points
- Proof-of-concept/exploit: +20 points
- Research papers: +15 points
- Vulnerability advisory: +10 points

**Severity Levels:**
- **Critical:** Zero-days, active exploitation, RCE, ransomware
- **High:** Privilege escalation, auth bypass, code execution
- **Medium:** XSS, CSRF, information disclosure, DoS
- **Low:** Everything else

### Filtering Articles

**By Quality Score:**
```bash
# Only show high-quality articles (score 50+)
export MIN_QUALITY_SCORE=50
python3 threat_intel_bot.py
```

**By Severity:**
```bash
# Only critical and high severity
export ALLOWED_SEVERITIES="critical,high"
python3 threat_intel_bot.py
```

**Both:**
```bash
export MIN_QUALITY_SCORE=40
export ALLOWED_SEVERITIES="critical,high,medium"
python3 threat_intel_bot.py
```

---

## Advanced Configuration

### Environment Variables

Create `.env` file:

```bash
# Required
BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrsTUVwxyz

# Optional filters
MIN_QUALITY_SCORE=0              # 0-100 (default: 0 = no filtering)
ALLOWED_SEVERITIES=critical,high,medium,low  # Comma-separated
```

---

## Checking Feed Health

```bash
# Check if all feeds are working
python3 check_feeds.py

# Or use the bot command
# In Telegram: /stats
```

**Output shows:**
- ✅ Online feeds (with article count)
- ❌ Offline/broken feeds (with error)
- Last update time

---

## Running as a Service (Linux)

### Systemd Service

**1. Create service file:**

```bash
sudo nano /etc/systemd/system/threat-intel-bot.service
```

**2. Add this content:**

```ini
[Unit]
Description=Threat Intelligence RSS Telegram Bot
After=network.target

[Service]
Type=simple
User=your_username
WorkingDirectory=/home/your_username/Documents/telegram-bots/rss
Environment="BOT_TOKEN=your_bot_token_here"
ExecStart=/usr/bin/python3 /home/your_username/Documents/telegram-bots/rss/threat_intel_bot.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

**3. Enable and start:**

```bash
sudo systemctl daemon-reload
sudo systemctl enable threat-intel-bot
sudo systemctl start threat-intel-bot
```

**4. Check status:**

```bash
sudo systemctl status threat-intel-bot
sudo journalctl -u threat-intel-bot -f  # View logs
```

---

## Running with Docker (Optional)

### Dockerfile

```dockerfile
FROM python:3.11-slim

WORKDIR /app

# Install dependencies
RUN pip install python-telegram-bot feedparser aiohttp beautifulsoup4

# Copy bot files
COPY . /app

# Run bot
CMD ["python3", "threat_intel_bot.py"]
```

### Docker Compose

```yaml
version: '3.8'

services:
  threat-intel-bot:
    build: .
    container_name: threat-intel-bot
    restart: unless-stopped
    environment:
      - BOT_TOKEN=${BOT_TOKEN}
      - MIN_QUALITY_SCORE=${MIN_QUALITY_SCORE:-0}
      - ALLOWED_SEVERITIES=${ALLOWED_SEVERITIES:-critical,high,medium,low}
    volumes:
      - ./subscribers.json:/app/subscribers.json
      - ./seen_articles.db:/app/seen_articles.db
```

**Run:**

```bash
# Create .env file first
docker compose up -d
```

---

## Customization

### Add Your Own Feed Categories

**1. Edit `threat_intel_bot.py`:**

Find `CATEGORY_CONFIG` (line 37):

```python
CATEGORY_CONFIG = {
    "news": {"label": "News", "emoji": "📰", "feeds_file": "feeds/news_feeds.json"},
    "malware": {"label": "Malware", "emoji": "🦠", "feeds_file": "feeds/malware_feeds.json"},
    # Add new category:
    "your_category": {"label": "Your Category", "emoji": "🔥", "feeds_file": "feeds/your_feeds.json"},
}
```

**2. Create feed file:**

```bash
nano feeds/your_feeds.json
```

```json
{
  "your_category": {
    "Feed Name 1": "https://example.com/rss.xml",
    "Feed Name 2": "https://another.com/feed"
  }
}
```

**3. Restart bot**

Users can now use `/on_your_category` and `/off_your_category`

---

### Customize Classification Keywords

**File:** `content_classifier.py`

**Example:** Add new critical keywords

```python
CRITICAL_KEYWORDS = {
    'zero-day', '0day', 'zero day',
    # Add your keywords:
    'your_critical_term',
    'another_urgent_keyword',
}
```

**Example:** Add high-quality sources

```python
HIGH_QUALITY_SOURCES = {
    'watchTowr Labs', 'Doyensec Blog',
    # Add your sources (exact name from feed):
    'Your Favorite Security Blog',
}
```

---

## Troubleshooting

### Bot Not Responding

**Check if running:**
```bash
ps aux | grep threat_intel_bot.py
```

**View logs:**
```bash
# If running as service
sudo journalctl -u threat-intel-bot -f

# If running manually, check terminal output
```

---

### No Alerts Received

**1. Check subscription:**
```
In Telegram: /stats
```

Should show you're subscribed.

**2. Check feeds are working:**
```bash
python3 check_feeds.py
```

**3. Check quality/severity filters:**
- If `MIN_QUALITY_SCORE=80`, only very high-quality articles pass
- If `ALLOWED_SEVERITIES=critical`, only critical alerts show

**Lower thresholds:**
```bash
export MIN_QUALITY_SCORE=0
export ALLOWED_SEVERITIES="critical,high,medium,low"
```

---

### Feed Parse Errors

**Symptom:** Logs show "Error fetching feed"

**Causes:**
- Feed URL is broken/changed
- Feed server is down
- Network connectivity issue

**Fix:**
1. Check feed URL in browser
2. Update URL in `feeds/*.json` if changed
3. Remove dead feeds
4. Restart bot

---

### SQLite Database Issues

**Reset seen articles (get all articles again):**

```bash
# Stop bot
rm seen_articles.db

# Restart bot (will recreate DB)
python3 threat_intel_bot.py
```

**Note:** First run marks all existing articles as seen (no alerts). New articles after that trigger alerts.

---

## Data Files

### subscribers.json

**Format:**
```json
{
  "subscribers": {
    "123456789": {
      "topic_id": null,
      "feed_types": ["news", "threat_intel"]
    },
    "987654321_12345": {
      "topic_id": 12345,
      "feed_types": ["malware"]
    }
  }
}
```

**Explanation:**
- Key format: `chat_id` or `chat_id_topic_id`
- `topic_id`: null for private/group, number for forum topics
- `feed_types`: Array of subscribed categories

**Manual editing:** You can edit this file, but bot does it automatically.

---

### seen_articles.db

**SQLite database tracking processed articles.**

**Schema:**
```sql
CREATE TABLE seen_articles (
    article_key TEXT PRIMARY KEY,
    seen_at TEXT NOT NULL
);
```

**View contents:**
```bash
sqlite3 seen_articles.db "SELECT * FROM seen_articles LIMIT 10;"
```

---

## Best Practices

### 1. Start with Default Filters
Don't set quality/severity filters initially - see what you get first, then filter.

### 2. Use Topics for Organization
If using in a group, enable Topics and route each category to its topic.

### 3. Monitor Feed Health
Run `/stats` weekly to check for broken feeds.

### 4. Curate Your Feeds
Start with defaults, add/remove based on signal-to-noise ratio.

### 5. Run as a Service
Use systemd so bot survives reboots and crashes.

### 6. Backup Subscriber Data
```bash
cp subscribers.json subscribers.backup.json
```

---

## Feed Categories Explained

### News Feeds (`feeds/news_feeds.json`)
Daily cybersecurity news, breach announcements, general updates.
- **Update frequency:** Multiple times per day
- **Volume:** High
- **Use case:** Stay informed on current events

### Malware Feeds (`feeds/malware_feeds.json`)
Malware analysis, reverse engineering blogs, threat reports.
- **Update frequency:** Daily to weekly
- **Volume:** Medium
- **Use case:** Malware research, threat hunting

### Threat Intel Feeds (`feeds/threat_intel_feeds.json`)
Vendor threat intelligence, APT reports, threat actor profiles.
- **Update frequency:** Daily to weekly
- **Volume:** Medium
- **Use case:** Threat intelligence, SOC operations

### OSINT Feeds (`feeds/osint_feeds.json`)
Open-source intelligence, tools, techniques, investigations.
- **Update frequency:** Weekly
- **Volume:** Low to medium
- **Use case:** OSINT research, investigative work

### Research Feeds (`feeds/research_feeds.json`)
Deep technical research, vulnerability analysis, exploit development.
- **Update frequency:** Weekly to monthly
- **Volume:** Low (high quality)
- **Use case:** Learning, in-depth technical knowledge

---

## Contributing Feeds

**Want to add a good feed?** Edit the appropriate JSON file and submit a pull request or update your fork.

**Criteria for good feeds:**
- Reliable RSS/Atom feed
- Regular updates (at least monthly)
- Quality content (no spam/clickbait)
- Relevant to cybersecurity

---

## Performance

**Typical resource usage:**
- **CPU:** <5% (idle), 10-20% during feed fetch
- **RAM:** ~50-100MB
- **Network:** Minimal (fetches feeds every 5 minutes)
- **Disk:** ~10MB (grows slowly with seen articles DB)

**Scaling:**
- Tested with 50+ feeds
- Handles 100+ subscribers
- Processes ~500 articles/day

---

## Security Considerations

### Bot Token
- **Never commit** `.env` file to Git
- Treat bot token like a password
- Regenerate if leaked (via @BotFather)

### Network Access
- Bot fetches public RSS feeds (outbound HTTPS)
- Telegram API (outbound HTTPS)
- No inbound connections needed

### Data Privacy
- Subscriber data stored locally in `subscribers.json`
- No data sent to third parties
- Article URLs/metadata only (no personal data)

---

## FAQ

**Q: How often does the bot check feeds?**
A: Every 5 minutes (configurable in `threat_intel_bot.py` line 475)

**Q: Can I run multiple bots from same code?**
A: Yes, just use different bot tokens and run in separate directories

**Q: Does it support private feeds?**
A: No, only public RSS feeds. For private feeds, you'd need to add authentication

**Q: Can I export articles to a database?**
A: Articles are in `seen_articles.db` (SQLite). You can query it or extend the code

**Q: Why no alerts on first run?**
A: First run marks existing articles as seen to avoid spam. Only new articles after that trigger alerts

**Q: Can I get alerts in multiple languages?**
A: Bot messages are in English. Articles are in whatever language the feed provides

---

## License

This project is open source. Use it for personal or commercial purposes.

No warranty provided. Use at your own risk.

---

## Credits

Built with:
- [python-telegram-bot](https://python-telegram-bot.org/) - Telegram Bot API wrapper
- [feedparser](https://feedparser.readthedocs.io/) - RSS/Atom feed parser
- [aiohttp](https://docs.aiohttp.org/) - Async HTTP client
- [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup/) - HTML parsing

---

**Maintained for personal use - built for the cybersecurity community.**