- training dataset: https://drive.google.com/file/d/1Isqc7mHGZR8vmFGpl7AzRizI4IINzica/view
- implementation code: https://github.com/tantansir/BlindNav
A prototype system for navigation for visually impaired individuals, based on YOLO semantic segmentation and object detection.
Currently, two modes are supported:
run.py: Basic navigation version (detects sidewalks, zebra crossings, obstacles, and traffic lights)run_llm.py: Version integrated with LLM for generating scene descriptions via natural language (requires Ollama)
video/Example videos (e.g.,china1.mp4,pitts1.mp4)model/YOLO weights (semantic segmentation, object detection, traffic lights)semantic_segmentation_part/Semantic segmentation related codeobject_detection_part/Object detection related codellm_demo/LLM experiment
-
blind_merged.v2/: Contains data configurations for object detection
test/,train/,valid/: Dataset folders (empty; all data are stored on Google Drive)blind_merge_config.json: Configuration for merging the datasetdata.yaml: YAML file for dataset configurationmerge_details.json: Details for merging datasetssampling.py: Sampling script for data processing
-
eda.py: Exploratory data analysis script
-
merge_yolo_datasets.py: Script for merging YOLO datasets
-
train_yolo.py: Training script for YOLO object detection model
-
run.py: Entry point for running object detection with YOLO
- eda_analysis.py: Exploratory data analysis for segmentation datasets
- evaluate_model.py: Script to evaluate segmentation model performance
- final_config.yaml: Final configuration YAML for semantic segmentation
- hyperparameter_tuning.py: Hyperparameter tuning script for segmentation models
- inference_test.py: Inference testing script for segmentation models
- quick_tuning.py: Quick hyperparameter tuning script
- train_seg_advanced.py: Advanced training script for semantic segmentation models
- verify_dataset.py: Dataset verification script for segmentation models
- Python 3.9 or higher (recommended)
- FFmpeg installed (for video decoding/encoding, required on some systems)
- GPU (optional, recommended for faster processing with Torch + CUDA)
ultralytics(YOLO model)opencv-python(cv2, for video reading/writing and drawing)numpypyttsx3(optional, for text-to-speech)requests(only used byrun_llm.pyfor Ollama integration)
Full dependencies are available in requirements.txt.
# 1. Create and activate a virtual environment (optional but recommended)
python -m venv venv
# Windows
venv\Scripts\activate
# macOS / Linux
source venv/bin/activate
# 2. Install dependencies
pip install -r requirements.txtBy default, the scripts will load weights from the model/ directory:
model/yolo-seg.ptSemantic segmentation for blind path / zebra crossingmodel/yolov8n.ptObject detection for pedestrians, vehicles, and obstaclesmodel/trafficlight_best.ptTraffic light detection for pedestrians
If the weight filenames differ, you can specify them via command-line arguments:
python run.py --seg-weights path/to/seg.pt --det-weights path/to/det.pt --tl-weights path/to/trafficlight.ptpython run.py --source video/china1.mp4Common arguments:
--source: Video file path or camera ID (e.g.,0for webcam)--seg-weights: Path to semantic segmentation weights--det-weights: Path to object detection weights--tl-weights: Path to traffic light detection weights (leave empty for no detection)--save-path: Output path for processed video (default:output.mp4)--no-voice: Disable voice announcements--config: Optional, specify a custom navigation configuration JSON
Example:
# Run with an example video and save the output
python run.py --source video/china1.mp4 --save-path output_china1.mp4
# Use webcam 0
python run.py --source 0run_llm.py builds on the basic navigation version, generating a brief natural language description of the current scene via the Ollama LLM.
python run_llm.py --source video/pitts1.mp4Additional arguments:
--llm-model: Ollama model name, default isllama3.2
Example:
python run_llm.py --source video/pitts1.mp4 --llm-model llama3.2-
Install Ollama
Follow the installation instructions for your system on the Ollama website. -
Start the Ollama service
ollama serve- Pull a model (e.g.,
llama3.2)
ollama pull llama3.2When starting the script, it will automatically detect:
- If LLM is available, it will display:
Press 'X' for scene description - If LLM is not available, it will prompt you on how to start or pull the model
The window will display hotkey instructions, including:
QorEsc: Exit the programSpaceorP: Pause/Resume video playbackC: Toggle "crosswalk mode" (detects zebra crossings and traffic lights)X: Inrun_llm.py, trigger LLM to generate a scene description and read it aloud (if voice is enabled)
You can pass a JSON configuration file via --config to override some parameters in NavigationConfig, such as:
- Sidewalk / zebra crossing area threshold
- Obstacle area threshold
- Voice announcements settings and rate
Example (pseudo code):
{
"blind_path_ratio_threshold": 0.005,
"crossing_ratio_threshold": 0.002,
"voice_enabled": true,
"voice_rate": 150
}Then run:
python run.py --source video/china1.mp4 --config config/navigation.json- Currently tested only on a small number of videos, robustness in real-world scenarios needs improvement
- LLM scene descriptions depend on local Ollama service, ensure network and GPU resources are available
- Text-to-speech is based on
pyttsx3, voice quality may vary across platforms