Building a Music Brain: My YouTube Data Workflow

Turning a passive YouTube browsing habit into a personalized AI-driven recommendation engine requires more than just a script; it requires a full-stack data pipeline. I've been mapping out a system that treats my listening history as a dataset of one, essentially building a self-learning "Auto-DJ" that understands my mood patterns based on actual behavior rather than just "likes."

This isn't a quick weekend build. It's a multi-month deployment involving a browser extension for data collection, a backend for storage, and ML pipelines for analysis. The core goal is to capture "implicit signals"—the things I do without thinking, like skipping a track after four seconds or looping a specific chorus—and translate those into a behavioral profile.

The Data Pipeline Architecture

To make this work, the system has to track the lifecycle of a single song from the moment I click it to the moment it influences a playlist.

1. Data Acquisition (Browser Extension)
A content script monitors the YouTube video element. Instead of just tracking "views," it acts as a stenographer, recording timeupdate, pause, seeked, and ended events. This allows the system to calculate the actual watch percentage and identify specific segments I find appealing.

2. Ingestion (Event API)
To avoid overloading the server, the extension batches these observations into JSON payloads and POSTs them to a FastAPI backend:

{
  "video_id": "xyz123",
  "title": "Artist - Song Name",
  "channel": "OfficialChannel",
  "watch_segments": [[0, 41], [132, 222]],
  "timestamp": "2023-10-27T21:47:00Z",
  "session_id": "sess_98765"
}

3. Storage (PostgreSQL)
The raw events are written directly to PostgreSQL. I keep the raw signal intact because any future changes to the ML logic will require re-processing the original data.

4. Processing (Offline ML Pipeline)
This is where the "forensic" work happens. A nightly pipeline processes the raw events to:

• Classify: Confirm if the video is actually music.
• Resolve: Parse titles and use external lookups for clean metadata.
• Analyze: Use audio embeddings and mood tagging to determine the "vibe."
• Score: Convert watch percentages and replay counts into implicit ratings.

5. Indexing (Vector Store)
The final step involves pushing audio embeddings into pgvector. By storing these as vectors alongside relational metadata, the system can perform similarity searches to find songs that match the specific "fingerprint" of my late-night synthwave sessions or morning lo-fi habits.

This setup transforms a messy history page into a structured AI workflow, allowing for a level of personalization that generic algorithms usually miss.