Live-Stream Stability — a Continual-Learning POC

This is the live_stream_stability POC. It lives under a continual_learning/poc/ workspace that will hold sibling POCs over time (e.g. compression/), each its own git repo.

What I'm testing. Whether I can take a strong open-weight vision-language model (VLM) and continually pretrain it on the long-duration, first-person-style experience stream of a single subject — video + audio — so that it becomes measurably more knowledgeable and perceptive about that subject's reality over time.

Hypothesis. A base VLM's understanding/recall/perception on Day-0 (before it sees the stream) is measurably lower than after it has absorbed X hours of that subject's continuous feed by Day-N. The long-term vision is a model that becomes a user's experiential alter ego — it sees what they see and accumulates their context.

As the closest public proxy for "one person's continuous life feed," I use IShowSpeed's 35-day livestreamed IRL US tour (2025): 67 videos, ~752.8 hours (~11 h each, 1080p60 VP9/Opus).

I own this work: Murali Nandan Nagarapu — Head of Engineering, Nucleus AI · nmn@withnucleus.ai

---

Where this stands today

Phase	What I set out to do	Status
1 — Acquire	Turn the playlist into an indexed on-disk corpus: download + metadata + ASR + diarization	✅ 67/67 complete
2 — Chunk	Pick the base model + chunking operating point, then cut the corpus into VLM-sized chunks	✅ Complete — 2,290 chunks
3 — Describe	Generate gold per-chunk descriptions (the training targets)	✅ Complete — all 4 granularities (20/10/5/1-min), 61,067 targets
3.1 — Replay mixture	Assemble a capability-aligned anti-forgetting data mix to train alongside our targets	⏳ Next (own thread)
3.2 — Evals	Build the IShowSpeed Day-0/Day-N eval and a general-capability forgetting eval	🔬 In progress — design settled, harnesses built + validated, split frozen, Day-0 baseline running
4 — Train + eval	Continually pretrain on targets + replay mix; measure Day-0 vs Day-N	⏳

I keep the live working state — what's in flight, gotchas, next steps — in HANDOFF.md and handoff/. This README is the stable overview; HANDOFF is the agent/working canvas.

Decisions locked so far: base model Qwen2.5-VL-32B-Instruct; chunk the stream at 20 min / 1 fps at the model's default token budget; training target = generated descriptions (ASR/diarization are inputs, not targets); targets are video-relative with real-world-time woven in; full bf16 + fp32 Adam, ZeRO-3, no LoRA; audio off for v1. Because the finetune is full-parameter, Phase-4 trains on the targets plus a replay mixture (Phase-3.1) to resist catastrophic forgetting, measured by dedicated eval suites (Phase-3.2).

---

The four phases

PHASE-1 — Acquire the corpus ✅

I turn a YouTube playlist into a clean, indexed, on-disk corpus: one folder per video with the media plus all metadata and time-aligned transcripts, all tracked by a single manifest CSV (ishowspeed_tour.csv) that is the spine of the phase — one row per video, static columns from the playlist, runtime columns filled by the downloader and transcript steps.

The pipeline (in data/live_streams/ishowspeed/tour/): build_manifest.py (playlist → CSV) → download.py (yt-dlp, Archivist format chain → MKV + info + description + thumbnail + subs) → fetch_subs.py (YouTube captions) → asr_subs.py (faster-whisper for caption-less videos) → diarize_subs.py (WhisperX + pyannote speaker labels) → rename_files.py (ASCII-safe artifact names) → status.py (progress report). The production pull ran sharded across the 8-node mesh (scripts/node_shard_download.sh), each node taking a slice of the playlist to its local SSD and backing up to GCS.

Status: all 67/67 videos downloaded, transcribed, and diarized.

PHASE-2 — Chunk the stream ✅

A VLM has finite context; 752 hours can't go in at once. So I had to choose (a) a base model and (b) a chunking operating point (length, fps, resolution), then materialize the chunk dataset.

Strategy (deep record: experiments/chunk_length/HANDOVER.md) — a 6-model empirical sweep over a 195-question bank anchored to Day-01 Miami, auto-graded by LLM judges, produced:

• Base model: Qwen2.5-VL-32B-Instruct (best mean score, best OCR of its cost tier, trains on ≤4 H100 with ZeRO-3). 72B is the backup for long chunks / critical OCR.
• Operating point: 20 min / 1 fps / model-default token budget. Quality holds to ~20 min and degrades past it.
• Hard-won lesson: benchmark on your own data. Qwen3-VL scores higher on public VideoMME but its OCR collapsed (~0.86 → ~0.15) on this footage.

The token-budget math (you'll need it in PHASE-3/4). Qwen2.5-VL uses an adaptive global token cap (~12–14K vision tokens), not a fixed per-frame resolution. As chunk_seconds × fps grows, total tokens stay ~constant and per-frame resolution shrinks: pairs = chunk_s × fps / 2, pixels_per_pair ≈ 25.2M / pairs, frame_side ≈ √pixels_per_pair. At 20 min × 1 fps → 600 pairs → ~205-px frames: great temporal coverage, legible for most reasoning, not for small text. Drop to 0.5 fps or use the 72B if fine OCR ever becomes critical.

Productionization (chunk_videos.py). I stream-copy each video into 20-min chunks with ffmpeg's segment muxer (-c copy — lossless, ~2 min/video, keyframe-exact on this VP9), upload the chunk media to GCS, and write one row per chunk into chunks_manifest.csv. The local SSD is touched only transiently (staged → uploaded → deleted), so the durable chunk store is GCS. The job is mesh-distributed: each node chunks the videos it holds locally and writes its own chunks_manifest.d/<node>.csv; --merge unions the per-node parts into the single source-of-truth table (scripts/launch_mesh_chunk.sh fans the run out over SSH and merges).

Result: all 67 videos → 2,290 chunks → 752.7 h in GCS. Integrity verified — 0 duplicate chunk_ids, 0 non-GCS paths, 0 n_chunks mismatches, per-video coverage to <0.1 s. chunks_manifest.csv is the join table PHASE-3 consumes: one row per chunk, media_path → GCS, all PHASE-1 metadata carried.

PHASE-3 — Gold chunk descriptions ✅ (complete — 61,067 targets)

For every chunk I generate a rich textual description — the TARGET the model learns to predict in PHASE-4. ASR/diarization are inputs to the generator, not targets. Each target is a dense natural-language blob (re-rendered from a stored structured ChunkDesc JSON, so I never re-pay the teacher to reformat): an anchor line + ~280-word prose + a minute-by-minute timeline whose every line is exhaustively detailed. The full record (all structured fields, Q&A, every raw model call) is stored so nothing is wasted. Design notebook: handoff/phase-3-describe.md; code: experiments/phase3_describe/.

Teacher choice (blind, frame-grounded judge bake-off — 4 prompts × 3 models, then a 5-min Pro-vs-Flash study). Gold = gemini-3.1-pro-preview with the v3_disciplined prompt — chosen for the better floor (fewest fabrications) on an unattended run, not the flashiest ceiling. It reads VP9-in-MKV (and sub-windows via video_metadata offsets) straight from GCS, no remux, and uses audio only to name visible things (Tier-B knowledge injection: "a tall man" → "Tom Brady"). gemini-3.5-flash lost on detail/accuracy at every scale except the cheap 1-min tier, where it's the bulk worker with Pro re-doing the OCR-heavy subset. Claude Opus (frames+ASR only, no native video/audio) was the grounded-but- anonymous fallback.

The anchor + real-world-time spine. PHASE-4 shuffles chunks, so position can't be recovered from context — it must live in the target tokens. Every target opens with [Day N of 35 · City, ST · ~local time], and the timeline carries periodic real-world-time stamps. RWT is reconstructed deterministically (rwt.py) from spoken time-cues in the ASR (regex → Gemini-Flash filter → piecewise-linear offset(video_pos) fit) — never model-guessed (VLMs are weak at clock time).

Multi-granularity curriculum (the key structural decision). PHASE-4 trains on varied chunk lengths, so I describe at 20 / 10 / 5 / 1-min. Crucially this is NO media re-cut: the finer scales are just offset windows into the existing 20-min chunk files (gen_offset_manifests.py → 4,549 / 9,060 / 45,170 windows); the teacher reads each sub-window via video_metadata offsets and all timeline timestamps are clip-relative (every clip starts 00:00). A shorter window also gets ~4× the pixels/frame, so finer scales read OCR/signage the 20-min pass misses. Per scale:

• 20-min — 2-pass (full-video description + 4× 5-min HIGH-res timeline windows), Pro, run online (run_20min.py). ✅ 2,288 / 2,288.
• 10-min — 2-pass, Pro, Vertex Batch in two waves (batch Pass-1 → parse → batch Pass-2 → stitch). ✅ 4,549 / 4,549.
• 5-min — single HIGH-res call → prose + fields + 15-s-cadence timeline, Pro, Batch. ✅ 9,060 / 9,060.
• 1-min — single call, Flash bulk + Pro re-verifies the OCR-heavy subset (39,547 windows), Batch. ✅ 45,170 / 45,170.

All four ran via submit_finer.sh's resubmit-loop; every batch shard succeeded and a final 8 truncated-output stragglers were recovered online (backfill.py). Total: 61,067 gold targets, verified local == GCS.

Run on Vertex Batch (~50% cheaper, async ≤24 h) via submit_finer.sh, which loops submit→retrieve until 0 chunks are missing (Gemini intermittently over-blocks even at BLOCK_NONE, and a batch can't retry mid-job). All robustness fixes are baked in: clip-relative timestamps, per-request timeout (no worker deadlocks), BLOCK_NONE + retry-on-empty, thread-local clients. Total cost ≈ $7.8k for all four scales at full coverage. A 3-pane explorer (visualization/explorer.html, served Range-capable by visualization/serve.py) reviews any chunk: video · formatted description + the raw training-label string · full record JSON, with a granularity dropdown.

A per-folder map of the pipeline (the two-strategies × four-scales model, every file's role, the chunk-id naming convention, run/resume cookbook) is in experiments/phase3_describe/README.md.

PHASE-3.1 — Anti-forgetting replay mixture ⏳ (next, own thread)

Continually pretraining a 32B instruct model full-parameter, no LoRA on one subject's narrow feed will catastrophically forget general ability unless we replay data from the original distributions. This phase assembles a capability-aligned replay mixture to train alongside our targets. Design (settled with the team): capability-first, not dataset-first (replay what the Phase-3.2 eval measures); the #1 addition over the initial whiteboard is vision replay (general VQA, OCR/doc, video-QA) since the most fragile, expensive-to-rebuild capability of a VLM is its vision stack — and our own targets lean on OCR. The mixture is multi-objective (pretraining-style text + chat-format instruction + vision instruction + our task), the real levers are the task:replay ratio + LR re-warm/decay, and we'll likely add a KL anchor to the base model as cheap insurance for the aggressive finetune. Direction, candidate datasets, and open questions in handoff/phase-3.1-data-mixture.md.

PHASE-3.2 — Eval suites ⏳ (next, own thread)

Build the harnesses that ground us before training — two tracks: (A) the IShowSpeed Day-0 vs Day-N knowledge/perception eval (the hypothesis test, built from the stored per-chunk QA pairs + held-out chunks, blind frame-grounded judging), and (B) a general-capability forgetting eval (vision VQA/OCR/ video + text knowledge/reasoning/instruction) measured on the base model first. Track B's capability list defines the Phase-3.1 replay buckets and tunes its mixture ratio — eval and data co-design.

Settled + built (2026-06-12). Design locked with the team (experiments/phase3.2_evals/DESIGN.md): Track A measures both closed-book recall (the experiential-memory claim) and open-book perception, on trained (recall) vs held-out (generalization) chunks. The held-out split is frozen — 486 of 2,290 parents (wholeday Day 6 Philly / 16 New Orleans / 28 Portland = unseen-city test for v2; withinday chunk_index%8==3 incl. 5 Day-01 = within-day signal for v1), defined at the 20-min parent level so finer windows inherit it (0 leakage). The Track-A bank is 271,817 tagged items auto-derived from the records (stored QA + OCR + day→city + RWT-spine probes); the judge is a cross-family blind, frame-grounded panel (Claude opus-4-8 + Gemini-3.5-Flash, teacher excluded). Track B = lmms-eval + lm-eval-harness on the base model first. Both harnesses are built and validated end-to-end; the Day-0 baseline is running. Per-folder map + cookbook in experiments/phase3.2_evals/README.md; live state in handoff/phase-3.2-evals.md.

PHASE-4 — Continual pretrain + evaluate ⏳

INPUT  = [<|video_start|>] [vision_tokens(chunk @ 20min/1fps)] [<|video_end|>]
TARGET = chunk description + injected metadata (geo, timestamp, day-N-of-35, cross-chunk cues)
LOSS   = causal LM loss on TARGET tokens only — vision input positions are masked

Why this shape. Modern open-weight VLMs do not convert video to text first. A vision encoder turns pixels into continuous embeddings placed in the LLM's token space, interleaved with text tokens; the LLM does next-token prediction over text positions only while attending to the vision tokens. So I train the model to describe what it sees — grounding it in this subject's visual experience — rather than fine-tuning a language head on synthetic captions.

Full bf16 + fp32 Adam, ZeRO-3 sharded, gradient-checkpointed, no LoRA (continual pretraining needs deep rewiring). ~2 a3-mega nodes (16 H100) for 32B full training. Milestones: v1 Day-01 alone (~11 h) → recall + emergent behavior; v2 all 67 (~752.8 h) → scaling + forgetting; v3 audio; v4 more modalities; v5 online/streaming. Recipe + eval plan in handoff/phase-4-train.md.

---

Repository layout

poc/live_stream_stability/              ← this POC (a git repo)
├── README.md                          ← this file (human-facing overview)
├── HANDOFF.md  handoff/               ← agent working canvas + per-phase notes (live state)
├── scripts/                           ← mesh ops: sync_to_gcs, node_shard_download, launch_mesh_chunk
├── data/live_streams/ishowspeed/tour/ ← PHASE-1 pipeline + PHASE-2 chunker (code + seed manifests)
│   ├── *.py  (build_manifest, download, fetch_subs, asr_subs, diarize_subs,
│   │          calib_diarize, rename_files, status, chunk_videos)
│   ├── playlist_raw.jsonl  ishowspeed_tour.csv  ishowspeed_tour.md   ← PHASE-1 manifest seeds
│   └── chunks_manifest.csv                                           ← PHASE-2 chunk table (the join key)
└── experiments/
    ├── chunk_length/                  ← PHASE-2 experiment record (HANDOVER + plan/ qa_bank/ code/ results/)
    └── phase3_describe/               ← PHASE-3 describe pipeline (teacher gen + RWT + batch + explorer)
        ├── README.md                                                      ← the per-folder map (read this first)
        ├── describe_lib · generators · rwt · batch_lib                     ← shared libs (core / prompt / time / batch plumbing)
        ├── rich_pipeline (2-pass: 20+10min) · single_call (1-call: 5+1min) ← the two describe strategies
        ├── run_20min · batch_10min · batch_single · backfill               ← runners (online 20m / two-wave 10m / single 5+1m / recovery)
        ├── gen_offset_manifests · submit_finer.sh                          ← finer-scale manifests + orchestrator
        └── visualization/ (explorer.html · build_data · serve)            ← review explorer (gitignored: descriptions/, manifests/*.csv, visualization/videos/)

Third-party reference (gitignored, re-clonable): TheFrenchGhostys-Ultimate-YouTube-DL-Scripts-Collection — download.py's yt-dlp format chain mirrors that repo's Archivist preset.

---

Data locations

Asset	Location	In git?	In GCS?
Pipeline code, docs, manifest seeds	this repo (NFS /home)	✅	✅ (via repo)
67 source videos (~1.1 TB) + info/desc/thumb/subs	/mnt/localssd/poc/data/live_streams/ishowspeed/tour/videos/ (per node)	❌	✅
20-min chunks — 2,290 files (~1.1 TB) + chunks_manifest.csv	gs://…/tour/chunks/ (media); table in repo + mirrored to GCS	✅ table	✅ media
PHASE-3 gold descriptions (targets, all granularities)	gs://…/tour/descriptions/<gran>/ ; code in repo	✅ code	✅ targets
HF model cache (Qwen2.5-VL-32B etc.)	/mnt/localssd/.hf-home/	❌	❌ (re-downloadable)

Storage discipline. Git is the source of truth for code/docs; GCS is the source of truth for bulk data. The working tree /mnt/localssd/poc/ is rsync'd to gs://nucleus-continual-learning/poc/ (us-east4, additive/never-delete). ISHOWSPEED_DATA_ROOT overrides the data root (default /mnt/localssd/poc/data/live_streams/ishowspeed/tour). The bucket enforces uniform bucket-level access + public-access-prevention — I share objects via signed URLs, not public ACLs.

Why this discipline exists. I lost the original PHASE-1 working data on /mnt/localssd during a SLURM cluster replacement. Git (code/docs/small seeds) plus the GCS backup are now the durable safety net; nothing irreplaceable lives only on the local SSD.

---

Runbooks

conda activate moe          # torch 2.8/cu128, yt-dlp, faster-whisper, whisperx, pyannote, ffmpeg 7.1
cd data/live_streams/ishowspeed/tour

# PHASE-1 (acquire) — already complete; regenerable + resumable:
python build_manifest.py                         # playlist → manifest (refuses to clobber done rows)
python download.py --all --no-comments --parallel 2   # full pull (production ran sharded via scripts/node_shard_download.sh)
python fetch_subs.py && python asr_subs.py --parallel 8 && python diarize_subs.py --parallel 8
python rename_files.py && python status.py

# PHASE-2 (chunk) — per node, idempotent; or fan out across the mesh:
python chunk_videos.py --all                     # chunk this node's done videos → GCS
python chunk_videos.py --merge                   # union per-node parts → chunks_manifest.csv (+ GCS)
python chunk_videos.py --status                  # cross-node totals
bash ../../../../scripts/launch_mesh_chunk.sh    # fan out over all 8 nodes, then merge

Infra. GCP poetic-avenue-438401-a7, zone us-east4-b. SLURM partition a3mega = 8× a3-mega nodes nucla3m-a3meganodeset-[0-7] (8× H100 80 GB each); /home is shared NFS, /mnt/localssd is per-node local; node-to-node SSH is passwordless. HF_TOKEN is in ~/.bashrc.