Skip to content
Purv-KabariaPublic

About

AI-powered offline-first CRM for intelligent field lead capture, audio insights, and automated follow-ups. Features multimodal sentiment analysis, timestamped transcript navigation, and Google Meet scheduling. Built with Next.js, PostgreSQL, Whisper, Gemma 3, and scalable PWA architecture.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

81 Commits

Folders and files

Repository files navigation

FinBridge

Lead capture for field sales teams that works without a network, plus a review desk that turns recorded sales conversations into something a manager can scan in a minute.

Built as a three-person team project at HTT 2026 and maintained by Purv Kabaria. The call-analysis pipeline described in Call analysis is the part that has been reworked and benchmarked since.

Frontend Next.js 15 (App Router), React 19, TypeScript, Tailwind CSS 4, installable PWA
Offline store IndexedDB via Dexie
Backend Next.js API routes, Prisma, PostgreSQL, MinIO (S3 API) for audio
Jobs BullMQ on Redis: contact sync, audio, transcription
AI service Python FastAPI: faster-whisper, RoBERTa sentiment (INT8 ONNX), wav2vec2 voice emotion, Gemma 3 via Ollama

Contents

  1. What it does
  2. Architecture
  3. Call analysis
  4. Offline capture and sync
  5. Follow-ups and meeting scheduling
  6. Roles
  7. Running it
  8. Tests and checks
  9. Known limitations

What it does

  1. Capture. A rep at an expo stall saves a lead (name, phone, company, notes) and optionally records a voice note. With no signal, everything is stored on the device and synced later.
  2. Transcribe and analyse. Uploaded audio is queued, transcribed, and analysed: a summary, the moments worth hearing (amounts, dates, commitments), sentiment over the call, and places where the speaker's tone disagrees with their words.
  3. Review. A moderator listens with a timestamped transcript (click a sentence to jump there), reads the analysis, and marks the deal as profitable, engaging or worth pursuing.
  4. Follow up. Replies to follow-up emails are checked for whether the lead wants a meeting, and the moderator can create a Google Meet invite from the dashboard.

Architecture

flowchart LR
    subgraph Device
        UI[PWA capture] --> IDB[(IndexedDB)]
    end
    IDB -- sync when online --> API[Next.js API]
    API --> PG[(PostgreSQL)]
    API --> S3[(MinIO / S3<br/>audio)]
    API -- enqueue --> Q[[BullMQ / Redis]]
    Q --> W[transcribe worker]
    W -- audio URL --> PY[FastAPI AI service]
    PY --> WH[faster-whisper]
    PY --> SA[RoBERTa sentiment<br/>INT8 ONNX]
    PY --> SER[wav2vec2<br/>voice emotion]
    PY --> LLM[Gemma 3 via Ollama<br/>summary, highlights]
    W -- structuredSnapshot --> PG
Loading

Why a queue: transcription and analysis take tens of seconds per clip. Doing it inside the upload request would make the field app wait on a model. The upload returns as soon as the file is stored; the transcribe worker does the slow part and writes the result back to the interaction.

Call analysis

Code: python_backend/utils/sentiment_engine.py, rules in python_backend/utils/tone.py.

Pipeline

  1. Transcribe. Audio is denoised, split at natural pauses (about 25 s per chunk), and transcribed with faster-whisper into timestamped segments.

  2. Score each segment once. Every segment goes through the text-sentiment model in batches. Those per-segment scores are reused three ways, so the model runs once:

    • Sentiment over time: positive minus negative probability per segment.
    • Whole-call sentiment: per-segment scores averaged, weighted by how much was said. (An earlier version classified only the first 1,500 characters, so the end of a long call never counted.)
    • Candidates for the tone check (next step).
  3. Tone vs words. Voice-emotion recognition is slow, and it only matters where the words are clearly positive or negative. So it runs on at most 8 of the most polar segments (polarity at least 0.6, at least 1 s of audio). A segment is flagged when the voice model is at least 50% sure of an emotion that contradicts the words:

    Words Voice Flag
    positive angry or sad "positive words, angry/sad tone"
    negative happy "negative words, upbeat tone"
    anything neutral, or voice model unsure none

    Flags appear in the review dashboard with timestamps. They are prompts for a human to listen, not verdicts.

  4. In parallel: text emotions (GoEmotions RoBERTa), whole-call voice emotion (start, middle and end windows), and an LLM summary plus highlights.

Why INT8 ONNX for sentiment

The sentiment model is cardiffnlp/twitter-roberta-base-sentiment-latest. It is exported to ONNX and dynamically quantised to INT8 (scripts/export_sentiment_onnx.py). The service uses the INT8 model when the file exists and falls back to PyTorch otherwise; the API reports which one ran (meta.sentimentEngine).

Measured with scripts/bench_sentiment.py on 256 call-style segments, batch 32, 4 CPU threads, Intel 12th-gen laptop CPU (raw result):

Backend ms per segment Labels matching PyTorch
PyTorch FP32 21.5 (reference)
ONNX Runtime INT8 7.8 256 / 256
Speed-up 2.74x

Quantisation does shift probabilities slightly (largest difference 0.19 on one segment), which is why the tone check uses a polarity threshold rather than exact probabilities, and why the label agreement is checked rather than assumed.

Offline capture and sync

Code: lib/sync-service.ts, lib/capture-sync.ts.

  • Every record captured offline is written to IndexedDB with syncStatus: "pending". Audio blobs stay on the device until upload succeeds.
  • When the device is back online, contacts sync before their interactions, because an interaction needs its contact to exist on the server.
  • The server upserts contacts by phone number (unique in the schema), so capturing the same lead twice, offline on two days, produces one contact, not two.
  • If the server assigns a different id, the local record is re-keyed to the server id.
  • If an interaction arrives and the server reports its contact missing, the contact is put back to pending and the sync is retried, instead of dropping the interaction.

This is last-write-wins on contact fields; there is no field-level merge. That is a deliberate simplification for a single rep editing their own leads.

Follow-ups and meeting scheduling

  • Follow-up emails can be drafted by the LLM from the call summary and sent through Gmail.
  • Replies are read through the Gmail API. The LLM answers one question: does this person want to meet, and did they suggest times? (classify_meeting_reply returns wants_meeting and suggested_times.)
  • The moderator can then create a Google Calendar event with a Meet link from the dashboard.

Roles

Role Can
User (field) Capture contacts and audio, sync
Moderator Everything above, plus review audio, set deal verdicts, schedule meetings, export CSV
Admin Everything above, plus user and contact management, analytics

Auth is JWT in an HTTP-only cookie; every moderator and admin route re-checks the role server-side.

Running it

Prerequisites: Node 20+, pnpm, PostgreSQL, Redis, MinIO (or any S3), Python 3.11+, ffmpeg, Ollama with gemma3:4b pulled.

# web app
cp .env.example .env            # DATABASE_URL, JWT_SECRET, REDIS_URL, MINIO_*, PYTHON_BACKEND_URL, GOOGLE_*
pnpm install
pnpm prisma:migrate
pnpm dev
pnpm workers                    # contacts, audio and transcribe workers

# AI service
cd python_backend
pip install -r requirements.txt
python scripts/export_sentiment_onnx.py   # optional: enables the INT8 sentiment model (~240 MB, git-ignored)
python main.py

Tests and checks

cd python_backend && python -m pytest tests -q     # tone and aggregation rules, no models needed (14 tests)
npx tsc --noEmit                                    # TypeScript: 0 errors
python python_backend/scripts/bench_sentiment.py   # ONNX vs PyTorch benchmark

Known limitations

  • The voice-emotion model is coarse. superb/wav2vec2-base-superb-er knows four classes (neutral, happy, angry, sad) and was trained on acted English speech. Noisy stall audio and Indian-English or Hindi/Gujarati speech will be less reliable, which is why it only ever raises flags for a human.
  • The text sentiment model was trained on tweets. It handles conversational English well; heavy code-mixing is a weak spot.
  • The tone thresholds (0.6 polarity, 0.5 voice confidence) were set by inspection, not tuned on labelled calls. There is no labelled call dataset in this project to tune them on. They are environment variables (TONE_POLARITY_THRESHOLD, TONE_MAX_SEGMENTS).
  • Contact sync is last-write-wins, as described above.

About

AI-powered offline-first CRM for intelligent field lead capture, audio insights, and automated follow-ups. Features multimodal sentiment analysis, timestamped transcript navigation, and Google Meet scheduling. Built with Next.js, PostgreSQL, Whisper, Gemma 3, and scalable PWA architecture.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages