requirements.txt (new) pins python-telegram-bot, feedparser, aiohttp,
and beautifulsoup4 to their current stable versions — the repo had no
lockfile or pinned deps at all, just a pip-install line in the README.
validation/review.py still had a --severity filter and printed a
quality_score that content_classifier.py stopped producing a while
back (see "Remove severity and quality scoring, classifier is
extraction-only"). Every record scored UNKNOWN/?, so the flag could
never match anything. Removed; --category/--has/--today/--limit and
the extracted-field printing are unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Exact URL/text matching missed the same story from a different outlet
(different URL, different wording). Add fuzzy title matching via
difflib, checked both within a single poll batch (the common case —
two feeds returning the same story in the same 5-minute cycle) and
against a rolling window of recently-sent titles in the seen-DB (the
cross-cycle case, e.g. follow-up coverage a few hours later).
seen_articles.db/seen_victims.db retention bumped from 7 to 14 days
(bot_config.SEEN_RETENTION_DAYS) — the window the new title matching
actually needs, and no reason to keep dedup history longer than that.
validation/run_validation.py: switch to the shared bot_config mapping,
and prune results.jsonl to the same 14-day window each poll instead of
growing forever — it's a testing aid, not an archive.
Also fixes a bug where a future-dated (bad clock/feed) article stayed
"recent" indefinitely instead of aging out after 48 hours, and drops
the legacy MD5 url-hash field that SHA-256 fingerprints replaced.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bot