A system that discovers, filters, scores, and recommends new AI/LLM projects from GitHub every day.
- Automatically discover trending/new AI/LLM projects on GitHub
- Filter and score projects based on relevance and quality
- Generate daily recommendation reports
- Support local/offline reading (download README, stars history, etc.)
| Feature | Description |
|---|---|
| Discovery | Search GitHub for AI/LLM projects using keywords, topics, and trending lists |
| Filtering | Filter by topics, languages, star count, activity recency |
| Scoring | Multi-dimensional scoring (stars, growth rate, activity, relevance) |
| Reports | Generate daily Markdown reports with summaries |
| Local Storage | Cache project data for offline access |
┌─────────────────────────────────────────────────────────────────────┐
│ AI-GitHub-Daily Pipeline │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Discover │───▶│ Filter │───▶│ Score │ │
│ │ │ │ │ │ │ │
│ │ GitHub API │ │ Keywords │ │ Stars │ │
│ │ Trending │ │ Topics │ │ Growth │ │
│ │ Search │ │ Languages │ │ Activity │ │
│ │ │ │ Min Stars │ │ Relevance │ │
│ └─────────────┘ └─────────────┘ └──────┬──────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Store │◀───│ Enrich │◀───│ Rank │ │
│ │ │ │ │ │ │ │
│ │ SQLite DB │ │ README │ │ Sort by │ │
│ │ JSON Cache │ │ Description │ │ score │ │
│ │ │ │ Topics │ │ │ │
│ └─────────────┘ └─────────────┘ └──────┬──────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Report Generator │ │
│ │ │ │
│ │ • Daily Markdown Report │ │
│ │ • Obsidian Note (optional) │ │
│ │ • JSON Data Export │ │
│ │ • RSS Feed (optional) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
| Endpoint | Purpose | Rate Limit |
|---|---|---|
GET /search/repositories |
Search by keywords | 30 req/min (auth: 30/min) |
GET /repos/{owner}/{repo} |
Get repo details | 5000 req/hour (auth) |
GET /repos/{owner}/{repo}/readme |
Get README content | 5000 req/hour (auth) |
GET /repos/{owner}/{repo}/topics |
Get topics | 5000 req/hour (auth) |
GET /repos/{owner}/{repo}/stats/code_frequency |
Activity stats | 5000 req/hour (auth) |
SEARCH_STRATEGIES = {
# Strategy 1: Topic-based search
"by_topics": {
"topics": [
"llm", "large-language-model", "gpt", "chatgpt",
"transformer", "huggingface", "langchain", "openai",
"generative-ai", "prompt-engineering", "rag",
"vector-database", "embedding", "fine-tuning",
"multimodal", "vision-language-model", "diffusion",
"stable-diffusion", "text-to-image", "speech-synthesis"
],
"sort": "stars",
"order": "desc"
},
# Strategy 2: Keyword search
"by_keywords": {
"queries": [
"llm inference",
"llm fine-tuning",
"rag pipeline",
"chatbot framework",
"prompt engineering",
"vector database",
"embedding model",
"multimodal llm",
"ai agent framework",
"llm evaluation"
],
"sort": "updated",
"order": "desc"
},
# Strategy 3: Trending (by creation date)
"trending": {
"created": "last 7 days",
"topics": ["llm", "ai", "machine-learning"],
"sort": "stars",
"order": "desc"
},
# Strategy 4: High growth (created in last 30 days, sorted by stars)
"high_growth": {
"created": "last 30 days",
"stars": ">100",
"sort": "stars",
"order": "desc"
}
}FILTER_CRITERIA = {
# Language filter (prioritize Python, but include others)
"languages": [
"Python", # Primary
"TypeScript", # Web/App frameworks
"JavaScript", # Web tools
"Rust", # High-performance tools
"Go", # Infrastructure
"Java", # Enterprise
"C++", # Performance-critical
],
# Minimum stars (adjust by repo age)
"min_stars": {
"new_7d": 10, # Created in last 7 days
"new_30d": 50, # Created in last 30 days
"new_90d": 100, # Created in last 90 days
"older": 500, # Older repos
},
# Exclude topics
"exclude_topics": [
"awesome-list", # Usually just collections
"tutorial", # Tutorials, not tools
"course", # Courses
"book", # Books
"interview", # Interview prep
],
# Exclude keywords in name/description
"exclude_keywords": [
"awesome-",
"awesome list",
"roadmap",
"interview questions",
"study guide",
"course notes",
],
# Activity requirements
"activity": {
"updated_within_days": 90, # Must have recent activity
}
}Final Score = Σ (Dimension Score × Weight)
┌────────────────┬────────┬────────────────────────────────────────────┐
│ Dimension │ Weight │ Calculation │
├────────────────┼────────┼────────────────────────────────────────────┤
│ Popularity │ 25% │ Based on star count (logarithmic scale) │
│ Growth Rate │ 25% │ Stars gained in last 7/30 days │
│ Activity │ 20% │ Recent commits, issues, PRs │
│ Relevance │ 15% │ Topic and keyword matching │
│ Quality │ 15% │ README quality, documentation, tests │
└────────────────┴────────┴────────────────────────────────────────────┘
class ScoringSystem:
"""Multi-dimensional scoring for GitHub repositories."""
def calculate_popularity_score(stars: int) -> float:
"""
Popularity score based on star count (logarithmic scale).
- 100+ stars: 1.0
- 500+ stars: 2.0
- 1K+ stars: 3.0
- 5K+ stars: 4.0
- 10K+ stars: 5.0
"""
import math
if stars < 100:
return 0.5
return min(5.0, 1.0 + math.log10(stars / 100))
def calculate_growth_score(
stars_today: int,
stars_7d_ago: int,
stars_30d_ago: int
) -> float:
"""
Growth rate score based on recent star gains.
- 7-day growth: 40% weight
- 30-day growth: 60% weight
"""
if stars_7d_ago == 0 or stars_30d_ago == 0:
return 0.0
growth_7d = (stars_today - stars_7d_ago) / stars_7d_ago
growth_30d = (stars_today - stars_30d_ago) / stars_30d_ago
# Normalize to 0-5 scale
score_7d = min(5.0, growth_7d * 10) # 50% growth = 5.0
score_30d = min(5.0, growth_30d * 5) # 100% growth = 5.0
return score_7d * 0.4 + score_30d * 0.6
def calculate_activity_score(
last_commit_days: int,
open_issues: int,
open_prs: int,
commit_frequency: float # commits per week
) -> float:
"""
Activity score based on recent development activity.
"""
score = 0.0
# Recent commit bonus
if last_commit_days <= 7:
score += 2.0
elif last_commit_days <= 30:
score += 1.5
elif last_commit_days <= 90:
score += 1.0
else:
score += 0.5
# Commit frequency bonus
if commit_frequency >= 10:
score += 1.5
elif commit_frequency >= 5:
score += 1.0
elif commit_frequency >= 1:
score += 0.5
# Community engagement (issues/PRs)
if open_issues > 0 and open_prs > 0:
score += 1.5
elif open_issues > 0 or open_prs > 0:
score += 1.0
return min(5.0, score)
def calculate_relevance_score(
topics: list,
description: str,
readme: str,
user_interests: list
) -> float:
"""
Relevance score based on topic and keyword matching.
"""
score = 0.0
# Topic matching
ai_topics = {
"llm", "large-language-model", "gpt", "chatgpt",
"transformer", "huggingface", "langchain", "openai",
"generative-ai", "prompt-engineering", "rag",
"vector-database", "embedding", "fine-tuning",
"multimodal", "diffusion", "ai-agent"
}
matched_topics = set(topics) & ai_topics
score += min(2.0, len(matched_topics) * 0.5)
# User interest matching
if user_interests:
for interest in user_interests:
if interest.lower() in description.lower():
score += 0.5
if interest.lower() in readme.lower():
score += 0.3
return min(5.0, score)
def calculate_quality_score(
has_readme: bool,
readme_length: int,
has_docs: bool,
has_tests: bool,
has_ci: bool,
has_contributing: bool
) -> float:
"""
Quality score based on project documentation and practices.
"""
score = 0.0
if has_readme:
score += 1.0
if readme_length > 5000:
score += 0.5 # Comprehensive README
if has_docs:
score += 1.0
if has_tests:
score += 1.0
if has_ci:
score += 0.5
if has_contributing:
score += 0.5
return min(5.0, score)
def calculate_final_score(
popularity: float,
growth: float,
activity: float,
relevance: float,
quality: float
) -> float:
"""
Calculate weighted final score.
"""
weights = {
'popularity': 0.25,
'growth': 0.25,
'activity': 0.20,
'relevance': 0.15,
'quality': 0.15
}
return (
popularity * weights['popularity'] +
growth * weights['growth'] +
activity * weights['activity'] +
relevance * weights['relevance'] +
quality * weights['quality']
)from dataclasses import dataclass
from datetime import datetime
from typing import Optional
@dataclass
class Repository:
"""GitHub repository data model."""
# Basic info
id: int
name: str
full_name: str # owner/repo
owner: str
description: Optional[str]
url: str
html_url: str
# Statistics
stars: int
forks: int
watchers: int
open_issues: int
# Dates
created_at: datetime
updated_at: datetime
pushed_at: datetime
# Language and topics
language: Optional[str]
topics: list[str]
# License
license: Optional[str]
# Scores
popularity_score: float = 0.0
growth_score: float = 0.0
activity_score: float = 0.0
relevance_score: float = 0.0
quality_score: float = 0.0
final_score: float = 0.0
# Enriched data
readme_content: Optional[str] = None
stars_history: Optional[dict] = None # {date: count}
# Metadata
discovered_at: datetime = None
last_updated: datetime = None@dataclass
class DailyReport:
"""Daily recommendation report."""
date: str # YYYY-MM-DD
generated_at: datetime
# Summary stats
total_discovered: int
total_filtered: int
total_recommended: int
# Categories
categories: dict # {category: [repos]}
# Top picks
top_picks: list[Repository]
# New projects (< 7 days)
new_projects: list[Repository]
# Rising stars (high growth)
rising_stars: list[Repository]
# By category
by_topic: dict # {topic: [repos]}ai-github-daily/
├── README.md
├── pyproject.toml
├── config.example.yaml
├── requirements.txt
│
├── src/
│ └── ai_github_daily/
│ ├── __init__.py
│ ├── cli.py # CLI entry point
│ │
│ ├── core/
│ │ ├── __init__.py
│ │ ├── discoverer.py # GitHub API interactions
│ │ ├── filter.py # Filtering logic
│ │ ├── scorer.py # Scoring system
│ │ └── tracker.py # Track seen repos, history
│ │
│ ├── storage/
│ │ ├── __init__.py
│ │ ├── database.py # SQLite operations
│ │ └── cache.py # File cache
│ │
│ ├── report/
│ │ ├── __init__.py
│ │ ├── generator.py # Report generation
│ │ └── templates.py # Markdown templates
│ │
│ └── utils/
│ ├── __init__.py
│ ├── github_api.py # GitHub API client
│ └── helpers.py # Utility functions
│
├── skills/
│ └── ai-github-daily/
│ ├── SKILL.md # Claude Code skill definition
│ └── scripts/
│ ├── daily_run.py # Main daily script
│ └── cleanup.py # Cleanup old data
│
├── tests/
│ ├── test_discoverer.py
│ ├── test_filter.py
│ ├── test_scorer.py
│ └── test_report.py
│
└── data/
├── cache/ # Cached repo data
├── reports/ # Generated reports
└── ai_github_daily.db # SQLite database
# src/ai_github_daily/core/discoverer.py
import os
import time
from datetime import datetime, timedelta
from typing import List, Optional
import requests
class GitHubDiscoverer:
"""Discover AI/LLM projects from GitHub."""
API_BASE = "https://api.github.com"
def __init__(self, token: Optional[str] = None):
self.token = token or os.environ.get("GITHUB_TOKEN")
self.session = requests.Session()
if self.token:
self.session.headers["Authorization"] = f"token {self.token}"
self.session.headers["Accept"] = "application/vnd.github.v3+json"
def search_by_topic(
self,
topic: str,
sort: str = "stars",
order: str = "desc",
per_page: int = 100,
max_pages: int = 3
) -> List[dict]:
"""Search repositories by topic."""
repos = []
for page in range(1, max_pages + 1):
url = f"{self.API_BASE}/search/repositories"
params = {
"q": f"topic:{topic}",
"sort": sort,
"order": order,
"per_page": per_page,
"page": page
}
response = self.session.get(url, params=params, timeout=30)
response.raise_for_status()
data = response.json()
repos.extend(data.get("items", []))
if len(data.get("items", [])) < per_page:
break
# Rate limiting
time.sleep(1)
return repos
def search_by_keywords(
self,
keywords: List[str],
created_after: Optional[datetime] = None,
min_stars: int = 0,
language: Optional[str] = None,
per_page: int = 100
) -> List[dict]:
"""Search repositories by keywords with filters."""
query_parts = []
# Add keywords
query_parts.append(" OR ".join(keywords))
# Add filters
if created_after:
query_parts.append(f"created:>{created_after.strftime('%Y-%m-%d')}")
if min_stars > 0:
query_parts.append(f"stars:>={min_stars}")
if language:
query_parts.append(f"language:{language}")
query = " ".join(query_parts)
url = f"{self.API_BASE}/search/repositories"
params = {
"q": query,
"sort": "stars",
"order": "desc",
"per_page": per_page
}
response = self.session.get(url, params=params, timeout=30)
response.raise_for_status()
return response.json().get("items", [])
def search_trending(
self,
topics: List[str],
days: int = 7,
per_page: int = 100
) -> List[dict]:
"""Search trending new repositories."""
since = datetime.now() - timedelta(days=days)
query_parts = [
" OR ".join([f"topic:{t}" for t in topics]),
f"created:>{since.strftime('%Y-%m-%d')}"
]
url = f"{self.API_BASE}/search/repositories"
params = {
"q": " ".join(query_parts),
"sort": "stars",
"order": "desc",
"per_page": per_page
}
response = self.session.get(url, params=params, timeout=30)
response.raise_for_status()
return response.json().get("items", [])
def get_repo_details(self, owner: str, repo: str) -> dict:
"""Get detailed repository information."""
url = f"{self.API_BASE}/repos/{owner}/{repo}"
response = self.session.get(url, timeout=30)
response.raise_for_status()
return response.json()
def get_repo_readme(self, owner: str, repo: str) -> Optional[str]:
"""Get repository README content."""
url = f"{self.API_BASE}/repos/{owner}/{repo}/readme"
try:
response = self.session.get(url, timeout=30)
response.raise_for_status()
import base64
content = response.json().get("content", "")
return base64.b64decode(content).decode("utf-8")
except requests.HTTPError:
return None
def get_repo_topics(self, owner: str, repo: str) -> List[str]:
"""Get repository topics."""
url = f"{self.API_BASE}/repos/{owner}/{repo}/topics"
headers = self.session.headers.copy()
headers["Accept"] = "application/vnd.github.mercy-preview+json"
response = self.session.get(url, headers=headers, timeout=30)
response.raise_for_status()
return response.json().get("names", [])# src/ai_github_daily/core/scorer.py
import math
from datetime import datetime, timedelta
from typing import Dict, Optional
class RepositoryScorer:
"""Score repositories based on multiple dimensions."""
# Weights for final score calculation
WEIGHTS = {
'popularity': 0.25,
'growth': 0.25,
'activity': 0.20,
'relevance': 0.15,
'quality': 0.15
}
# AI/LLM related topics
AI_TOPICS = {
"llm", "large-language-model", "gpt", "chatgpt", "claude",
"transformer", "huggingface", "langchain", "openai", "anthropic",
"generative-ai", "prompt-engineering", "rag", "retrieval-augmented-generation",
"vector-database", "embedding", "embeddings", "fine-tuning", "finetuning",
"multimodal", "diffusion", "ai-agent", "autonomous-agents",
"text-to-image", "text-to-speech", "speech-recognition",
"machine-learning", "deep-learning", "neural-network",
"pytorch", "tensorflow", "jax", "onnx",
}
def score(self, repo: dict, stars_history: Optional[Dict] = None) -> dict:
"""Calculate all scores for a repository."""
popularity = self._score_popularity(repo.get("stargazers_count", 0))
growth = self._score_growth(repo, stars_history)
activity = self._score_activity(repo)
relevance = self._score_relevance(repo)
quality = self._score_quality(repo)
final = (
popularity * self.WEIGHTS['popularity'] +
growth * self.WEIGHTS['growth'] +
activity * self.WEIGHTS['activity'] +
relevance * self.WEIGHTS['relevance'] +
quality * self.WEIGHTS['quality']
)
return {
'popularity_score': round(popularity, 2),
'growth_score': round(growth, 2),
'activity_score': round(activity, 2),
'relevance_score': round(relevance, 2),
'quality_score': round(quality, 2),
'final_score': round(final, 2)
}
def _score_popularity(self, stars: int) -> float:
"""Score based on star count (logarithmic scale)."""
if stars < 50:
return 0.5
return min(5.0, 1.0 + math.log10(stars / 50))
def _score_growth(
self,
repo: dict,
stars_history: Optional[Dict] = None
) -> float:
"""Score based on recent growth."""
if not stars_history:
# Estimate based on repo age and total stars
created = datetime.fromisoformat(
repo.get("created_at", "").replace("Z", "+00:00")
)
age_days = (datetime.now(created.tzinfo) - created).days
if age_days < 7:
return min(5.0, repo.get("stargazers_count", 0) / 20)
elif age_days < 30:
return min(5.0, repo.get("stargazers_count", 0) / 100)
else:
return 1.0 # Default for older repos without history
# Calculate actual growth
today = datetime.now().strftime("%Y-%m-%d")
week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
month_ago = (datetime.now() - timedelta(days=30)).strftime("%Y-%m-%d")
stars_today = stars_history.get(today, 0)
stars_week_ago = stars_history.get(week_ago, stars_today)
stars_month_ago = stars_history.get(month_ago, stars_today)
if stars_week_ago == 0:
return 5.0 if stars_today > 0 else 0.0
growth_7d = (stars_today - stars_week_ago) / stars_week_ago
growth_30d = (stars_today - stars_month_ago) / stars_month_ago
score_7d = min(5.0, growth_7d * 10)
score_30d = min(5.0, growth_30d * 5)
return score_7d * 0.4 + score_30d * 0.6
def _score_activity(self, repo: dict) -> float:
"""Score based on recent activity."""
score = 0.0
# Check last update
updated = datetime.fromisoformat(
repo.get("updated_at", "").replace("Z", "+00:00")
)
days_since_update = (datetime.now(updated.tzinfo) - updated).days
if days_since_update <= 7:
score += 2.0
elif days_since_update <= 30:
score += 1.5
elif days_since_update <= 90:
score += 1.0
else:
score += 0.3
# Check issues and PRs
open_issues = repo.get("open_issues_count", 0)
if open_issues > 0:
score += 1.0
# Check if actively developed (has recent push)
pushed = datetime.fromisoformat(
repo.get("pushed_at", "").replace("Z", "+00:00")
)
days_since_push = (datetime.now(pushed.tzinfo) - pushed).days
if days_since_push <= 7:
score += 1.0
elif days_since_push <= 30:
score += 0.5
return min(5.0, score)
def _score_relevance(self, repo: dict) -> float:
"""Score based on AI/LLM relevance."""
score = 0.0
# Check topics
topics = set(repo.get("topics", []))
matched_topics = topics & self.AI_TOPICS
score += min(2.0, len(matched_topics) * 0.5)
# Check description
description = (repo.get("description") or "").lower()
ai_keywords = [
"llm", "gpt", "chatgpt", "claude", "transformer",
"ai", "machine learning", "deep learning", "neural",
"nlp", "natural language", "embedding", "vector",
"rag", "agent", "multimodal", "diffusion"
]
for keyword in ai_keywords:
if keyword in description:
score += 0.3
return min(5.0, score)
def _score_quality(self, repo: dict) -> float:
"""Score based on project quality indicators."""
score = 0.0
# Has description
if repo.get("description"):
score += 1.0
# Has license
if repo.get("license"):
score += 0.5
# Has documentation (homepage)
if repo.get("homepage"):
score += 0.5
# Has topics
if repo.get("topics"):
score += 0.5
# Fork count indicates community interest
forks = repo.get("forks_count", 0)
if forks > 10:
score += 0.5
if forks > 100:
score += 0.5
return min(5.0, score)# src/ai_github_daily/report/generator.py
from datetime import datetime
from typing import List, Dict
from pathlib import Path
class ReportGenerator:
"""Generate daily recommendation reports."""
def __init__(self, output_dir: str = "./reports"):
self.output_dir = Path(output_dir)
self.output_dir.mkdir(parents=True, exist_ok=True)
def generate_daily_report(
self,
repos: List[dict],
date: str = None,
user_interests: List[str] = None
) -> str:
"""Generate daily Markdown report."""
date = date or datetime.now().strftime("%Y-%m-%d")
# Sort by final score
sorted_repos = sorted(
repos,
key=lambda x: x.get("final_score", 0),
reverse=True
)
# Categorize
new_projects = self._filter_new_projects(sorted_repos, days=7)
rising_stars = self._filter_rising_stars(sorted_repos)
by_topic = self._categorize_by_topic(sorted_repos)
# Generate report
report = self._build_report(
date=date,
total=len(repos),
new_projects=new_projects[:5],
rising_stars=rising_stars[:5],
top_picks=sorted_repos[:10],
by_topic=by_topic,
user_interests=user_interests
)
# Save report
report_path = self.output_dir / f"daily_{date}.md"
report_path.write_text(report, encoding="utf-8")
return str(report_path)
def _build_report(
self,
date: str,
total: int,
new_projects: List[dict],
rising_stars: List[dict],
top_picks: List[dict],
by_topic: Dict[str, List[dict]],
user_interests: List[str] = None
) -> str:
"""Build the Markdown report content."""
lines = [
f"# AI/LLM GitHub Daily - {date}",
"",
f"> 发现 {total} 个 AI/LLM 相关项目",
"",
"---",
"",
]
# Top Picks
if top_picks:
lines.extend([
"## 🌟 今日推荐 (Top Picks)",
"",
])
for i, repo in enumerate(top_picks, 1):
lines.extend(self._format_repo_entry(i, repo))
lines.append("")
# New Projects
if new_projects:
lines.extend([
"## 🆕 新项目 (New This Week)",
"",
])
for i, repo in enumerate(new_projects, 1):
lines.extend(self._format_repo_entry(i, repo))
lines.append("")
# Rising Stars
if rising_stars:
lines.extend([
"## 🚀 快速成长 (Rising Stars)",
"",
])
for i, repo in enumerate(rising_stars, 1):
lines.extend(self._format_repo_entry(i, repo))
lines.append("")
# By Topic
if by_topic:
lines.extend([
"## 📂 按主题分类 (By Topic)",
"",
])
for topic, repos in by_topic.items():
if repos:
lines.append(f"### {topic}")
lines.append("")
for repo in repos[:3]:
lines.extend(self._format_repo_entry(0, repo, compact=True))
lines.append("")
return "\n".join(lines)
def _format_repo_entry(
self,
index: int,
repo: dict,
compact: bool = False
) -> List[str]:
"""Format a repository entry."""
name = repo.get("full_name", "")
url = repo.get("html_url", "")
description = repo.get("description", "No description")[:100]
stars = repo.get("stargazers_count", 0)
language = repo.get("language", "Unknown")
score = repo.get("final_score", 0)
topics = repo.get("topics", [])
lines = []
if index > 0:
lines.append(f"### {index}. [{name}]({url})")
else:
lines.append(f"- **[{name}]({url})**")
if not compact:
lines.extend([
"",
f"**描述**: {description}",
"",
f"**语言**: {language} | **Stars**: {stars:,} | **评分**: {score:.2f}",
"",
])
if topics:
lines.append(f"**标签**: {', '.join(topics[:5])}")
lines.append("")
else:
lines.append(f" {description[:50]}... ⭐ {stars:,}")
return lines
def _filter_new_projects(
self,
repos: List[dict],
days: int = 7
) -> List[dict]:
"""Filter projects created within last N days."""
from datetime import datetime, timedelta
cutoff = datetime.now() - timedelta(days=days)
new_repos = []
for repo in repos:
created = datetime.fromisoformat(
repo.get("created_at", "").replace("Z", "+00:00")
)
if created > cutoff:
new_repos.append(repo)
return new_repos
def _filter_rising_stars(self, repos: List[dict]) -> List[dict]:
"""Filter repos with high growth score."""
return [r for r in repos if r.get("growth_score", 0) >= 3.0]
def _categorize_by_topic(self, repos: List[dict]) -> Dict[str, List[dict]]:
"""Categorize repositories by primary topic."""
topic_map = {
"LLM": ["llm", "large-language-model", "gpt", "chatgpt", "claude"],
"Agents": ["ai-agent", "autonomous-agents", "agent"],
"RAG": ["rag", "retrieval-augmented-generation"],
"Vector DB": ["vector-database", "vector-search", "embeddings"],
"Multimodal": ["multimodal", "vision-language", "text-to-image"],
"Fine-tuning": ["fine-tuning", "finetuning", "lora", "peft"],
"Inference": ["inference", "serving", "deployment"],
"Diffusion": ["diffusion", "stable-diffusion", "text-to-image"],
}
categories = {}
for repo in repos:
topics = set(repo.get("topics", []))
for category, keywords in topic_map.items():
if topics & set(keywords):
if category not in categories:
categories[category] = []
categories[category].append(repo)
break
return categories# config.yaml
# GitHub API Token (optional but recommended for higher rate limits)
github_token: "${GITHUB_TOKEN}"
# Search configuration
search:
topics:
- llm
- large-language-model
- gpt
- chatgpt
- transformer
- huggingface
- langchain
- openai
- generative-ai
- prompt-engineering
- rag
- vector-database
- embedding
- fine-tuning
- multimodal
- diffusion
- ai-agent
keywords:
- "llm inference"
- "llm fine-tuning"
- "rag pipeline"
- "chatbot framework"
- "prompt engineering"
- "vector database"
- "embedding model"
- "multimodal llm"
- "ai agent"
languages:
- Python
- TypeScript
- JavaScript
- Rust
- Go
# Filtering configuration
filter:
min_stars:
new_7d: 10
new_30d: 50
older: 100
exclude_topics:
- awesome-list
- tutorial
- course
- book
exclude_keywords:
- "awesome-"
- "roadmap"
- "interview"
# Scoring weights
scoring:
popularity: 0.25
growth: 0.25
activity: 0.20
relevance: 0.15
quality: 0.15
# User interests (for relevance scoring)
user_interests:
- "large language model"
- "rag"
- "ai agent"
- "multimodal"
- "fine-tuning"
# Output configuration
output:
format: markdown
directory: ./reports
# Obsidian integration (optional)
obsidian:
enabled: false
vault_path: "${OBSIDIAN_VAULT_PATH}"
notes_directory: "10_Daily/AI-GitHub"
# Storage configuration
storage:
database: ./data/ai_github_daily.db
cache_directory: ./data/cache
# Scheduling (for automated runs)
schedule:
enabled: false
time: "08:00"
timezone: "Asia/Shanghai"# Discover and recommend
ai-github-daily run # Full daily run
ai-github-daily run --top-n 20 # Specify number of recommendations
# Search only
ai-github-daily search "llm agent" # Search by keyword
# Filter and score
ai-github-daily score # Re-score cached repos
ai-github-daily filter --min-stars 100 # Filter cached repos
# Report generation
ai-github-daily report # Generate today's report
ai-github-daily report --date 2024-01-15 # Specific date
# Data management
ai-github-daily stats # Show statistics
ai-github-daily cleanup --days 30 # Clean old data
ai-github-daily export --format json # Export data
# Configuration
ai-github-daily config init # Initialize configuration
ai-github-daily config show # Show current config# ai-github-daily
Daily AI/LLM project discovery and recommendation from GitHub.
## Triggers
- "discover new ai projects"
- "find llm tools on github"
- "github ai recommendations"
- "daily ai tools"
- "ai-github-daily"
## Commands
### Run Daily Discovery
```bash
python scripts/daily_run.py --config config.yamlpython scripts/daily_run.py --topics "llm,agent,rag" --top-n 10python scripts/daily_run.py --report-only- Markdown report:
reports/daily_YYYY-MM-DD.md - JSON data:
data/daily_YYYY-MM-DD.json - SQLite database:
data/ai_github_daily.db
Edit config.yaml to customize:
- Search topics and keywords
- Filtering criteria
- User interests
- Output format
- All data stored locally
- GitHub API token optional (increases rate limits)
- No telemetry or external tracking
---
## 9. Future Enhancements
### Phase 2 Features
| Feature | Description |
|---------|-------------|
| **Trend Analysis** | Track project growth over time |
| **Similarity Search** | Find similar projects |
| **User Feedback** | Learn from user preferences |
| **Notifications** | Email/Slack notifications |
| **Web UI** | Dashboard for browsing projects |
### Phase 3 Features
| Feature | Description |
|---------|-------------|
| **ML-based Scoring** | Train model on user preferences |
| **Integration APIs** | REST API for integration |
| **Multi-source** | Include HuggingFace, Papers with Code |
| **Collaborative Filtering** | Community recommendations |
---
## 10. Implementation Roadmap
### Week 1: Core Implementation
- [ ] Setup project structure
- [ ] Implement GitHub API client
- [ ] Implement discoverer module
- [ ] Implement filter module
- [ ] Implement scorer module
### Week 2: Storage & Reports
- [ ] Implement SQLite storage
- [ ] Implement cache system
- [ ] Implement report generator
- [ ] Create report templates
### Week 3: CLI & Integration
- [ ] Implement CLI commands
- [ ] Create Claude Code skill
- [ ] Add configuration system
- [ ] Write tests
### Week 4: Polish & Deploy
- [ ] Add documentation
- [ ] Add error handling
- [ ] Add logging
- [ ] Test end-to-end workflow
---
## Summary
This project provides a complete solution for daily AI/LLM project discovery from GitHub. Key features:
1. **Multi-source Discovery**: Topics, keywords, trending
2. **Smart Filtering**: Language, stars, activity
3. **Multi-dimensional Scoring**: Popularity, growth, activity, relevance, quality
4. **Local Storage**: SQLite + JSON cache
5. **Rich Reports**: Markdown with categories
6. **Claude Code Integration**: Ready-to-use skill
7. **Fully Local**: All data stored locally, no tracking