ButterCut AI

/Overview

ButterCut is an AI co-pilot for short-form video post-production: subtitles in Indian languages, AI B-roll and dubbing for Reels and Shorts. I joined as a full-stack engineer and now lead the backend.

This page covers the two systems I'm proudest of. The first is the subtitle pipeline that turned Seekho, an edtech platform reaching millions of learners, into the company's first enterprise client. The second is the async B-roll pipeline that powers the editor.

/The problem

Seekho publishes a large volume of short educational video in Hindi and Hinglish. Off-the-shelf captioning was landing at 80–90% accuracy. That sounds acceptable until every video needs a human pass, and corrections were taking 1–2 days per video.

Indian-language speech is a hard target for captioning:

• speakers switch languages mid-sentence
• customers expect a specific script (Roman or Devanagari)
• brand names and teacher catchphrases aren't in any general model's vocabulary

/Two passes, not one big prompt

The obvious design is to send the audio to a multimodal LLM and ask for subtitles. That's expensive, and the timestamps drift: they wander after about a minute, and the token bill grows with every second of audio. The pipeline splits the job in two:

• Pass 1, acoustic. A dedicated speech-to-text model produces word-level timings. Audio is cut into VAD-snapped chunks of roughly 30–45 s and transcribed in parallel.
• Pass 2, linguistic. Gemini Flash only ever sees text, in 150-word windows with 8 words of carried context. It handles transliteration, segmentation and emphasis.

Because the LLM never re-listens to the audio, STT token usage dropped 89%, and word timestamps stay anchored to the acoustic model.

1. Ingestffmpeg

   Upload, URL or HLS playlist; audio extracted and normalised

2. Language probe + VAD chunkingSilero VAD

   Detect the spoken language, cut on silence, never mid-word

3. || pass 1 · word-level STT

   Chunk 1

   Chunk 2

   Chunk n
4. Gap repairretry

   Unexplained gaps re-sent as raw WAV and merged back

5. Pass 2 · Gemini FlashLLM

   150-word windows + 8-word context carry: transliteration, segmentation, emphasis

6. Customer style rulesper client

   Spellings, acronyms, homophones, catchphrases

7. Timestamp validationQA

   Rebased to the global timeline, trimmed against trailing silence

↺ Results go back to the editor, the API response or a partner webhook.

/Details that mattered

• Codec-redundancy retry. We caught the STT provider silently dropping words depending on the audio codec. Any gap the pipeline can't explain is re-sent as raw WAV and merged back in.
• Long audio. Each chunk's timestamps are rebased onto the global timeline and validated, so hallucinated or out-of-range times never reach the editor.
• Captions that end on time. An RMS + VAD gate trims word end-times that bleed into trailing silence.
• Style rules as data. Enterprise customers keep a sheet of spellings, acronyms, homophones and catchphrases. It compiles into three things: a compact prompt block, deterministic rewrites and an STT keyterm boost list. The compiled rules stay warm across Lambda invocations and are versioned from the B2B dashboard.
• A partner-grade API. Prefixed API keys hashed with Argon2, webhooks with a structured retryable/category/action error contract, HLS input, and per-job cost accounting traced in Langfuse.

/B-roll, asynchronously

Subtitles are half of an edit. The B-roll pipeline reads the transcript, decides where overlays belong, and generates them in parallel.

1. Ingest1

   Transcript, word timings and video context

2. || 2 · overlay suggestions in parallel

   Sticker

   Text callout

   Poster

   AI video
3. Supervisor3

   Resolves conflicts, pacing and overlap across suggestions

4. Asset generation4

   Parallel multimodal generation for each approved overlay

5. Render5

   Layered composite with aspect-ratio-aware scaling

↺ Jobs arrive over SQS; every stage checkpoints; results over 256 KB spill to S3.

It runs as an AWS SQS worker with S3 offloading and containerized Lambdas, at 99.9% uptime.

On the product side I also built much of the editor: the subtitle editor and its libass-based player, the multi-track timeline, Remotion caption previews and resumable multipart uploads.

/Results

Seekho subtitle accuracy

98%+

Seekho subtitle accuracy

time to corrected captions

1–2 days → live

time to corrected captions

STT tokens

−89%

STT tokens

paying enterprise client

1st

paying enterprise client

Seekho's subtitle accuracy went from 80–90% to 98%+, and corrections went from 1–2 days per video to real time. The pilot converted into the platform's first paying enterprise contract.