🤟 SignSense — AI-Powered Real-Time ASL Recognition

Browser-based American Sign Language fingerspelling recognition using MediaPipe Hands, a custom lightweight neural network, and local browser inference.

Author: Saksham Raj Singh Chauhan
Program: B.Tech Computer Science & Engineering — AI & ML
GitHub: @Saksham1105

What the project does

SignSense captures a user's webcam stream, detects a single hand with MediaPipe Hands, converts the 21 hand landmarks into a 63-value feature vector, and classifies the gesture in the browser. Recognised letters can be assembled into words and sentences, with suggestions, text-to-speech, clipboard copy, and clear/delete controls.

All recognition state is kept locally in the browser. The application has no application backend, database, authentication system, or user account system.

Repository structure

SignSense/
├── website/
│   ├── index.html
│   ├── style.css
│   ├── app.js
│   ├── js/
│   │   ├── commands.js
│   │   ├── detector.js
│   │   ├── inference.js
│   │   ├── preprocess.js
│   │   ├── predictor.js
│   │   ├── sentenceBuilder.js
│   │   ├── speech.js
│   │   └── suggestions.js
│   ├── libs/
│   └── model/
│       ├── weights.json
│       └── labels.json
├── training/
│   ├── collect_data.py
│   ├── train_model.py
│   ├── export_weights.py
│   ├── requirements.txt
│   ├── dataset/raw/
│   └── export/
└── README.md

A Python virtual environment is intentionally not committed. Create one locally when needed.

Run the website locally

Use a local HTTP server rather than opening index.html with file://, because webcam access and module/model loading require a browser-served origin.

cd website
python -m http.server 8000

Then open:

http://localhost:8000

The current browser implementation uses the MediaPipe runtime from the configured CDN location, so an active network connection may be required when MediaPipe assets have not already been cached by the browser.

Training pipeline

1. Collect data

python training/collect_data.py

The current collector uses MediaPipe Hands and stores flattened 63-dimensional landmark vectors as .npy files under training/dataset/raw/.

Current labels are:

A B C D E F G H I K L M N O P Q R S T U V W X Y Z
SPACE DELETE NOTHING

The current collector is configured for 350 samples per class, for a maximum of 9,800 samples across the 28 labels.

2. Train the classifier

python training/train_model.py

The current classifier is a compact fully connected network:

63 inputs
  ↓
Dense 128 + BatchNorm + ReLU + Dropout(0.30)
  ↓
Dense 64 + BatchNorm + ReLU + Dropout(0.15)
  ↓
Dense 28 + Softmax

Training uses:

• Adam optimizer
• Learning rate: 0.001
• Sparse categorical cross-entropy
• Batch size: 32
• Up to 60 epochs
• Early stopping
• ReduceLROnPlateau
• Model checkpointing
• Reproducibility seed: 42

The current code creates separate training, validation, and test splits rather than a single 80/20 train/test split.

3. Export the trained model

python training/export_weights.py

The exporter creates a lightweight custom weights.json representation and copies it, together with labels.json, into website/model/.

This is not a standard TensorFlow.js LayersModel export. The browser uses the project's custom JavaScript inference implementation to load the exported weights.

Browser inference flow

Webcam
  ↓
MediaPipe Hands
  ↓
21 landmarks × XYZ
  ↓
63-value feature vector
  ↓
Landmark preprocessing
  ↓
Custom JavaScript neural network
  ↓
Gesture label + confidence
  ↓
Sentence / suggestions / TTS / clipboard

The frontend is built with HTML5, CSS3, and vanilla JavaScript. The browser APIs used by the application include:

• MediaDevices.getUserMedia() for webcam access
• Web Speech API for speech synthesis
• Clipboard API for copying recognised text

Current recognition scope

The current exported model contains 28 labels:

A B C D E F G H I K L M N O P Q R S T U V W X Y Z
SPACE DELETE NOTHING

Therefore, the current model does not contain a J class. NOTHING is a trained class used to represent the no-gesture state.

The current implementation should be described as an ASL fingerspelling/gesture classifier, not as a complete ASL language translator. Dynamic signs, full ASL grammar, and broad continuous-sign translation are outside the current model scope.

Repository hygiene

Do not commit local environments, caches, editor artifacts, or generated Python package directories. Use a local virtual environment such as .venv/ and keep it ignored by Git.

Limitations

• Static gesture classification only
• Single-hand input in the training pipeline
• J is not currently represented in the trained class set
• Recognition quality depends on camera framing and lighting
• Mobile performance and layout may be lower than desktop
• The current project uses a custom browser inference path rather than an official TF.js LayersModel
• MediaPipe runtime assets are loaded from the configured CDN path

Academic documentation note

The implementation in this repository is the source of truth for the current codebase. Older project-report and presentation descriptions may contain different model architecture, class-set, threshold, dataset-size, or TensorFlow.js export details and should be synchronized with this repository before final submission.