MemeFinder

An intelligent Telegram bot for storing, organizing, and finding your memes using AI OCR and semantic search

MemeFinder transforms meme chaos into searchable order. Send memes to the bot, and it automatically extracts text using OCR, generates semantic descriptions using AI, and stores everything in a database with advanced full-text search capabilities.

Overview

MemeFinder is a Telegram bot that combines optical character recognition (OCR) and AI-powered image captioning to create a searchable meme library. The bot processes images sent to it, extracts text content, generates descriptive captions, and enables instant retrieval through Telegram's inline query interface using PostgreSQL trigram similarity search.

Key capabilities:

• Automatic text extraction and image descriptions via cloud vision APIs (Gemini, Anthropic, OpenAI)
• Fast semantic search powered by PostgreSQL trigram similarity
• S3-compatible storage for meme images
• Bulk archive processing for importing existing meme collections
• Organization through meme groups and storage chats

Features

• Smart Image Processing: Cloud vision APIs (Gemini, Anthropic, OpenAI) for OCR and AI-generated descriptions
• Lightning-Fast Search: PostgreSQL trigram similarity search (pg_trgm) with GIST indexing for fuzzy text matching
• Inline Search: Search and share memes directly from any chat using Telegram inline queries
• Bulk Import: Process entire meme archives (ZIP files) with the /archive command
• S3 Storage: S3-compatible storage backend for reliable image persistence
• Organized Storage: Group memes with custom categories and separate storage chats
• Multi-Provider Vision: Swap between Gemini, Anthropic (Claude), or OpenAI with a config change
• Type-Safe: Full type hints with mypy enforcement

Prerequisites

Using Docker (Recommended):

• Docker and Docker Compose
• Task (optional, for task automation)
• API key for a vision provider (Gemini, Anthropic, or OpenAI)

Local Development:

• Python 3.11+
• PostgreSQL 17
• S3-compatible storage (MinIO, AWS S3, etc.)
• API key for a vision provider

Installation & Setup

1. Clone the Repository

git clone https://gitlab.com/me1376/memefinder.git
cd memefinder

2. Configuration

Create config/local.yaml:

token: "YOUR_TELEGRAM_BOT_TOKEN"

vision:
  provider: gemini  # Options: gemini, anthropic, openai
  api_key: "YOUR_API_KEY"
  model: null  # Optional: Override default model
  rate_limit: null  # Optional: Requests per minute

updates_strategy: polling

database_dns: "postgresql+asyncpg://admin:admin@postgres:5432/memes"

storage:
  endpoint: "minio:9000"
  access_key: "minioadmin"
  secret_key: "minioadmin"
  bucket: "memes"
  secure: false
  public_url: null

Configuration Options:

Option	Type	Description	Example
token	string	Telegram bot token from @BotFather	"1234567890:ABCdefGHI..."
database_dns	string	PostgreSQL connection string (asyncpg format)	"postgresql+asyncpg://user:pass@host:5432/db"
updates_strategy	string	How to receive Telegram updates	"polling" or "webhook"
vision.provider	string	Vision API provider	"gemini", "anthropic", "openai"
vision.api_key	string	API key for the vision provider	"AIza..."
vision.model	string	Override default model (optional)	"gemini-2.0-flash", "claude-sonnet-4-20250514", "gpt-4o-mini"
vision.rate_limit	int	Requests per minute (optional)	60
storage.endpoint	string	S3-compatible endpoint	"minio:9000" or "s3.amazonaws.com"
storage.access_key	string	S3 access key	"minioadmin"
storage.secret_key	string	S3 secret key	"minioadmin"
storage.bucket	string	S3 bucket name	"memes"
storage.secure	bool	Use HTTPS for S3	true or false
storage.public_url	string	Public URL prefix for images (optional)	"https://cdn.example.com"

3. Build and Run with Docker

# Build the Docker image
docker compose build

# Start services (PostgreSQL + Bot)
docker compose up

OR using Task:

# First time setup
task project_setup

# Run the bot with hot-reload
task bot

4. Database Setup

Run Alembic migrations to set up the database schema:

# Using Docker Compose
docker compose run --rm bot python -m bot alembic upgrade head

# OR using Task
task upgrade_db

The initial migration automatically enables the PostgreSQL pg_trgm extension required for trigram similarity search.

Usage

Basic Workflow

1. Send a meme to the bot

   • Bot uploads image to S3 storage
   • Bot extracts text using OCR via cloud vision API
   • Bot generates image description using AI via cloud vision API
   • Automatically forwards to your configured storage chat
   • Saves metadata (S3 URL, OCR text, description) to PostgreSQL

2. Search your memes

   • Type @your_bot_name search query in any Telegram chat
   • Get instant fuzzy-matched results based on OCR text and AI descriptions
   • Share directly to the conversation

Commands

• /archive <zip_url> - Bulk import memes from a ZIP archive
  • Downloads and processes images in batches
  • Rate-limited to comply with Telegram API restrictions
  • Provides real-time progress updates

Example Search

@memefinder_bot funny cat

Returns all memes with text or AI-generated descriptions matching "funny cat" using trigram similarity scoring.

Architecture & Tech Stack

Core Technologies

• Bot Framework: aiogram 3.13.1 - Modern async Telegram bot framework
• Database: PostgreSQL 17 with pg_trgm extension for trigram similarity
• ORM: SQLAlchemy 2.0.35 (async mode with asyncpg driver)
• Vision APIs: Gemini, Anthropic (Claude), or OpenAI - OCR and image captioning
• Object Storage: S3-compatible storage (MinIO, AWS S3, etc.)
• Migrations: Alembic 1.13.3 - Database schema versioning
• Containerization: Docker

Data Model

The database schema consists of four primary tables:

• meme: Stores meme metadata

  • s3_url - S3 storage URL for the image
  • ocr - Extracted text from OCR
  • image_to_text - AI-generated description
  • Foreign keys to chat (storage location) and message_id

• chat: Represents Telegram chats

  • Stores source and storage chat configurations
  • Many-to-many relationship between source and storage chats via source__storage junction table

• memegroup: Organizes memes into collections

  • Many-to-many with meme via memegroup__meme junction table
  • Many-to-many with chat via memegroup__chat junction table

• users: Telegram user accounts

  • Automatically created on first interaction
  • Each user gets a default meme group on creation

Search Implementation

The search system uses PostgreSQL's pg_trgm (trigram) extension for fuzzy text matching:

1. Index Creation: GIST index on concatenated OCR and AI description fields

   CREATE INDEX ON meme USING gist((ocr || ' ' || image_to_text) gist_trgm_ops(siglen=32));

2. Query Strategy: Trigram distance operator (<->>) for similarity scoring

   query = select(Meme).where(
       (Meme.ocr + Meme.image_to_text).op('<->>')(search_text) < 0.4
   ).order_by(
       (Meme.ocr + Meme.image_to_text).op('<->>')(search_text)
   )

3. Similarity Threshold: Configurable threshold (default: 0.6) controls match sensitivity

   • Lower values = more fuzzy matches
   • Higher values = stricter matching

4. Performance: GIST index enables sub-second searches across thousands of memes

AI Processing Pipeline

1. User sends photo to bot
2. Image uploaded to S3 storage
3. Vision API extracts text (OCR) and generates description
4. Combined metadata stored in PostgreSQL with trigram index
5. Photo forwarded to configured storage chat(s)
6. Immediately searchable via inline queries

Development

Project Structure

memefinder/
├── bot/
│   ├── bl/                      # Business logic layer
│   │   ├── meme.py             # Meme processing logic
│   │   └── user.py             # User management
│   ├── controllers/            # Bot request handlers
│   │   ├── consumption/        # Meme ingestion handlers
│   │   │   ├── handlers.py     # Photo consumption, archive processing
│   │   │   └── state.py        # FSM states
│   │   └── search/             # Search functionality
│   │       └── handlers.py     # Inline query search implementation
│   ├── db/                     # Database layer
│   │   ├── models/            # SQLAlchemy ORM models
│   │   │   ├── meme.py        # Meme model
│   │   │   ├── chat.py        # Chat model
│   │   │   ├── memegroup.py   # Meme group model
│   │   │   └── user.py        # User model
│   │   ├── session.py         # Async session management
│   │   └── types.py           # Custom SQLAlchemy types
│   ├── middlewares/           # Aiogram middlewares
│   │   ├── session.py         # DB session per request
│   │   └── user.py            # User creation/update
│   ├── storage/               # Storage backends
│   │   ├── base.py            # Storage interface
│   │   └── s3.py              # S3-compatible storage
│   ├── vision/                # Vision API providers
│   │   ├── base.py            # Provider interface
│   │   ├── factory.py         # Provider factory
│   │   ├── prompt.py          # Shared prompts for OCR/description
│   │   ├── anthropic.py       # Anthropic (Claude) provider
│   │   ├── gemini.py          # Google Gemini provider
│   │   └── openai.py          # OpenAI provider
│   ├── utils/                 # Utility functions
│   │   ├── setup.py           # Config loading
│   │   └── tg.py              # Telegram helpers
│   ├── config.py              # Pydantic configuration models
│   ├── main.py                # Bot entry point
│   └── cli.py                 # CLI commands
├── migrations/                # Alembic database migrations
│   └── versions/              # Migration scripts
├── config/                    # Configuration files (gitignored)
│   └── local.yaml            # Local configuration
├── requirements/
│   ├── prod.txt              # Production dependencies
│   └── tools.txt             # Development tools
├── Dockerfile                # Docker build
├── docker-compose.yaml       # Development environment
├── Taskfile.yaml            # Task automation
└── alembic.ini              # Alembic configuration

Development Tasks

The project uses Task for automation. Available commands:

# Bot execution
task bot                    # Run bot with hot-reload (watchfiles)

# Interactive development
task ipython                # IPython shell with bot context

# Code quality
task mypy                   # Type checking
task flake8                 # Linting
task isort                  # Import sorting
task pre_commit             # Run all pre-commit checks

# Database operations
task upgrade_db             # Apply migrations
task downgrade_db           # Rollback migrations
task make_migrations -- "description"  # Create new migration
task check_db               # Verify schema matches models

# Utilities
task bot_exec               # Shell access to bot container
task setup_precommit_run_hook  # Install git pre-commit hook

Database Migrations

Create a migration:

task make_migrations -- "add new feature"

This runs Alembic's autogenerate to detect schema changes.

Apply migrations:

task upgrade_db

Applies all pending migrations to bring database to latest version.

Rollback:

task downgrade_db  # Rollback one migration
task downgrade_db -- base  # Rollback all migrations

Code Style Requirements

• PEP 8 Compliance: Enforced by flake8
• Type Hints: Required on all functions (checked by mypy)
• Import Sorting: Managed by isort
• Async Patterns: All I/O operations use async/await
• SQLAlchemy 2.0: Use async session management with context vars

Running Pre-commit Checks

Before committing code, run:

task pre_commit

This executes isort, flake8, and mypy in sequence.

Install as git hook:

task setup_precommit_run_hook

Creates .git/hooks/pre-commit that automatically runs checks before commits.

Docker Environment

The docker-compose.yaml defines four services:

• postgres: PostgreSQL 17 database

  • Default credentials: admin/admin
  • Database name: memes
  • Port 8081 (mapped from 5432)

• minio: S3-compatible object storage

  • Default credentials: minioadmin/minioadmin
  • API port: 9000
  • Console port: 9001

• bot: Main bot service

  • Hot-reload with watchfiles
  • Port 8080 (for webhook mode)

• ipython: Interactive Python shell

  • Same environment as bot
  • Access to all bot modules

Contributing

Contributions are welcome! To contribute:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make changes and test thoroughly
4. Run pre-commit checks: task pre_commit
5. Ensure all type checks pass: task mypy
6. Commit with descriptive message: git commit -m 'Add amazing feature'
7. Push to your fork: git push origin feature/amazing-feature
8. Open a Pull Request

Contribution Guidelines

• Follow existing code style and patterns
• Add type hints to all new code
• Update tests if modifying business logic
• Document significant changes in commit messages
• Keep commits focused and atomic

License

This project is licensed under the MIT License - see below for details:

MIT License

Copyright (c) 2024 MemeFinder Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Acknowledgments

This project is built on top of excellent open-source technologies:

• aiogram - Modern async Telegram bot framework
• PostgreSQL - Advanced open-source database
• PostgreSQL pg_trgm extension - Trigram-based text similarity search
• SQLAlchemy - Python SQL toolkit and ORM
• MinIO - High-performance S3-compatible object storage

And cloud vision APIs:

• Google Gemini - Multimodal AI models
• Anthropic Claude - Advanced AI assistant
• OpenAI - GPT-4 Vision API