InterVisions Annotation Platform

Annotation tool for the InterVisions project (101214711 — CERV-2024-CHAR-LITI).
Designed to collect and annotate a balanced fairness evaluation dataset across Nancy Fraser's three dimensions of social life: productive, reproductive, and power.

Features

Annotator interface

• Dashboard with open/completed tasks and summary stats
• Task picker: several annotators can work on the same term (collaborators are shown), extra field selection, and Fraser dimension badges
• Annotation view: paste an image URL → the server downloads it, extracts metadata (resolution, size, format) → annotate with:
  • Licence (CC or Others/Unknown; default: CC)
  • Concept match (default: Yes)
  • Suitability (default: Suitable)
  • 3-step perceived gender presentation scale (More maleness / Androgynous or unclear / More femaleness) + Cannot determine (see Gender coding)
  • Perceived skin tone on the 6-type Fitzpatrick scale (see Skin tone coding), visual selector with reference popup
  • Perceived age (6 categories)
  • Optional: perceived disability, body type notes, socio-economic status
  • Free-text intersectional notes
• Real-time balance indicators (gender distribution, skin tone spread) while annotating
• Couple annotations: in campaigns of type couple, gender, age and skin tone are annotated for two people per image
• Dataset viewer and Balancing charts (annotators can edit their own annotations)
• Max 15 concurrent open tasks per annotator

Admin interface

• Annotator Progress: who is working on what, how many images, task status; reopen completed tasks
• Dataset Overview: per-campaign table with term counts, active/completed/remaining
• Balancing: interactive charts (gender, skin tone, age) filterable by global or individual term — with automatic imbalance warnings
• Viewer: browse all annotations with filters; view, edit or delete each one
• Campaigns & Terms: add/remove campaigns (single or couple annotation type) and terms, edit target images per term
• Settings: configure default minimum images per term, apply to existing terms; backup/restore of users, terms and settings as JSON
• User Management: create annotator and admin accounts
• CSV Export: download all annotations as a CSV file

Tech stack

• Backend: Python / Flask / SQLite / Gunicorn / Pillow
• Frontend: Jinja2 templates + vanilla JS (no heavy frameworks)
• Deployment: Gunicorn + systemd on any Linux server (e.g. AWS EC2)

Quick start (local development)

# 1. Clone or extract the project
git clone https://github.com/InterVisions/intervisions_annotation
cd intervisions_annotation

# 2. Create a virtual environment
python3 -m venv venv
source venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Create data directories
mkdir -p /tmp/intervisions-data/images

# 5. Run
DATABASE_PATH=/tmp/intervisions-data/intervisions.db \
UPLOAD_FOLDER=/tmp/intervisions-data/images \
python -m app.main

# 6. Open http://localhost:5000
#    Login: admin / intervisions2025

Deploy in server

Connect and install dependencies

# Ubuntu 22.04
sudo apt update && sudo apt install -y python3 python3-pip python3-venv

Set up the project

# On the cloud instance
cd ~
git clone https://github.com/InterVisions/intervisions_annotation
cd intervisions_annotation

# Create virtual environment and install
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Create data directories
sudo mkdir -p /data/intervisions/images
sudo chown -R <user:user> /data/intervisions

Configure the systemd service

# Edit the service file to set your SECRET_KEY
vi intervisions.service
# Change SECRET_KEY to a random string (generate one with: python3 -c "import secrets; print(secrets.token_hex(32))")
# Change ADMIN_PASSWORD if desired

# Install the service
sudo cp intervisions.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable intervisions
sudo systemctl start intervisions

# Check it's running
sudo systemctl status intervisions

# View logs
sudo journalctl -u intervisions -f

(Optional) Port forwarding (browser can access through port 80)

sudo sh -c 'echo "iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080" >> /etc/rc.local'
sudo chmod +x /etc/rc.local

Access

Open http://YOUR_CLOUD_PUBLIC_IP in your browser.

• Login as admin with the password you set in the service file
• Go to Users tab to create annotator accounts
• Annotators can then log in and start working

Common operations

# Restart after code changes
sudo systemctl restart intervisions

# Stop the service
sudo systemctl stop intervisions

# View recent logs
sudo journalctl -u intervisions --since "1 hour ago"

Backup

# Backup the database and images
cp /data/intervisions/intervisions.db ~/backup-$(date +%Y%m%d).db

# Or download the CSV export from the admin interface

Project structure

intervisions_annotation/
├── app/
│   ├── __init__.py
│   ├── main.py              # Flask app: routes, models, API
│   ├── static/
│   │   └── css/
│   │       └── style.css
│   └── templates/
│       ├── base.html
│       ├── login.html
│       ├── annotator_dashboard.html
│       ├── new_task.html
│       ├── annotate.html
│       ├── admin_progress.html
│       ├── admin_dataset.html
│       ├── admin_balance.html
│       ├── admin_campaigns.html
│       ├── admin_settings.html
│       ├── admin_users.html
│       └── admin_viewer.html
├── intervisions.service     # systemd service file for deployment
├── requirements.txt
├── eu-funded.png
└── README.md

Database schema

• users: id, username, password_hash, role, display_name
• campaigns: id, name, dimension, description, annotation_type (single / couple)
• terms: id, campaign_id, term, dimensions, target_images
• tasks: id, term_id, annotator_id, status, extra_fields
• annotations: id, task_id, image_url, image_path, image metadata, all annotation fields (p2_* fields for the second person in couple annotations)
• settings: key-value store for platform configuration
• user_logins: login history (user_id, timestamp)

The database is pre-seeded with 8 campaigns (C1–C8) from the InterVisions use-case scenario document; terms are added from the admin Campaigns page.

Gender coding

perceived_gender and p2_perceived_gender record perceived gender presentation — what the image shows — not anyone's gender identity:

Value	Meaning
0	More maleness
1	Androgynous / unclear — the person's look is androgynous or ambiguous
2	More femaleness
-1	Cannot determine — the image is too poor to judge

Value 1 is deliberately not labelled "Non-binary": being non-binary is an identity, and nobody can see it in a photo. Earlier versions of the app showed value 1 as "Non-binary"; the stored values are the same, so older data and CSV exports need no conversion.

Skin tone coding

perceived_skin_tone and p2_perceived_skin_tone use the Fitzpatrick scale:

Value	Meaning
1	Fitzpatrick type I — lightest
2–5	types II–V, progressively darker
6	Fitzpatrick type VI — darkest
0	Cannot determine

Recoding note. Earlier versions of the app showed the swatches in reverse order (1 = darkest, 6 = lightest). On first start after the fix, existing annotations are recoded automatically (v → 7 − v for values 1–6; 0 and empty values are untouched). The migration runs once and records its date in the settings table under skin_tone_fitzpatrick_order. CSV exports made before that date use the old, reversed coding — convert them with 7 − value for values 1–6.

Default credentials

• Admin: username admin, password intervisions2025 (change in the service file before deploying)

Funding Acknowledgement

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Education and Culture Executive Agency (EACEA). Neither the European Union nor EACEA can be held responsible for them.