Distill News : AI-Powered Community News Aggregation Platform

Distill News is an automated news aggregation and synthesis platform powered by AI multi-agents. Its goal is to crawl, aggregate, filter, and summarize news across the internet—ranging from mainstream news APIs to public community forums—presenting content in a clean, structured, and personalized format.

This fork follows DesolateSea/Distillnews with adapted security fixes. Read SECURITY_CONFIGURATION.md before deployment: sessions now use cookies through /api, service ports are restricted, and JWT/OAuth configuration must be supplied explicitly.

Introduction

The Problem

Traditional news outlets often fail to cover everything due to profit motives, regional biases, and limited workforce resources. Meanwhile, public forums and social platforms like Reddit or X overflow with real-time updates and grassroots stories, but lack a unified, noise-free platform for reader consumption.

The Solution

Distill News uses AI agents to extract core news from web scraping, trusted APIs, and real-time community sources. The platform strips away clickbait, ads, and filler, rewriting posts into concise, professional-grade news articles.

By combining structured AI extraction, community discovery, and smart personalization, Distill News provides direct access to reliable local and global updates—without clickbait, corporate spin, or media gatekeeping.

Screenshots

---

Key Features

• AI-Generated News Feed: Aggregates content from news APIs (GNews, MediaStack, RapidAPI) and community platforms like Reddit.
• Noise & Fluff Filtering: Automatically removes clickbait, advertisements, duplicates, and filler content.
• Multi-Agent Processing: Classifies incoming data, extracts key metadata (title, summary, date, location), and formats news into clean markdown.
• Personalized Recommendations: Learns from user reading time and interests, adapting story selection dynamically using topic-weighted preferences.
• RAG-based AI Assistant: Includes context from articles, supports session memory for follow-up questions, and handles both local and global news queries conversationally.
• Newsletter & Newspaper Views: Generates curated email digests and a digital print-style newspaper layout.

---

Getting Started

Fast Local Development (Hot-Reloading)

For instant sub-second hot-reloading without Docker container builds:

cp .env.example .env # only if you do not already have a .env
# Set JWT_SECRET to a random secret of at least 32 bytes and configure your AI keys.
./dev.sh --install
./dev.sh

Requires Python 3.11+ (3.13 recommended), Node.js 20+, and Docker with Compose. The installer creates ./venv and installs Python/frontend dependencies. This starts mongo & redis in Docker and runs the backend, embedding server, optional MCP server, and frontend locally. Open http://localhost:3000; the backend health check is http://127.0.0.1:8000/health. For local HTTP, use COOKIE_SECURE=false, CORS_ORIGINS=http://localhost:3000, and ARTICLE_STORE_BACKEND=file. The runner uses loopback URLs for MongoDB, Redis, and embeddings; set LOCAL_DB_URL / LOCAL_REDIS_URL only if using different local infrastructure. Google login also needs http://localhost:3000 registered as an authorized JavaScript origin in Google Cloud. Email login/registration needs valid SMTP credentials. Use ./dev.sh --no-frontend to run just the Python services.

Full Docker Deployment (Production VM / VPS)

The platform deploys 3 core application containers (frontend, backend, embedding-server) alongside mongo and redis. Minimum system requirement: 2 vCPU, 4GB RAM, 30GB Disk.

1. Configure Swap Space (Recommended for 4GB RAM VMs):

   sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

2. Build and Start Production Stack:

   docker compose up -d --build

Note: Dockerfile.embedding uses CPU-only PyTorch to keep image size lightweight (~1.5GB).

Pipeline TUI

To use the pipeline, run:

python3.13 -m venv venv
source venv/bin/activate
pip install -r requirements-pipelines.txt
./cli.py

System Documentation Suite

For detailed architecture and subsystem documentation, refer to the documentation suite in doc/:

• 🏗️ System Architecture — High-level architecture, container topology, unified embedding microservice.
• 🎨 Frontend Documentation — Next.js 14 App Router, components (HeadlinesBanner, ChatFab), and LocalStorage feature flags.
• ⚡ FastAPI Web Server — REST API routes, authentication, database handles, and weather proxy.
• 🤖 RAG & Chatbot Subsystem — Conversational QA, vector search, and unified embedding integration.
• ⚙️ Ingestion Pipeline & TUI — Scrapers, LLM extraction pipeline, Textual TUI dashboard, and CLI runner.