Building a taste engine for fanfiction discovery

Most recommendation systems fail because they rely on popularity metrics rather than actual content signal. If you've spent time on AO3 or FanFiction.net, you know the struggle: you search by tags or ship, sort by kudos, and still end up with stories that miss the mark. High kudos counts measure community consensus, not individual preference. Even prompting a model like Claude with a description fails because the LLM lacks a specific model of your personal taste and the domain-specific nuances of prose.

I'm currently developing Siagnos to move past surface-level metadata. Instead of just matching tags, the goal is to analyze pacing, emotional tone, and character dynamics to score stories against a personal preference model.

The architecture is designed as a multi-stage pipeline:
1. Metadata scraping (title, tags, summary, word count) via cloudscraper and BeautifulSoup.
2. Embedding generation using HuggingFace sentence transformers.
3. Feature extraction using a local Mistral 7B instance via Ollama to parse summary nuances (tone, character dynamics).
4. A preference model built with XGBoost, trained on actual reading behavior (what is opened, read depth, and return rate).

The tech stack is strictly Python-based: FastAPI for the backend, PostgreSQL for the relational data, and Docker for containerization. I've intentionally avoided using LangChain for the core pipeline; I want to ensure I understand the underlying mechanics before adding layers of abstraction.

Current progress:
I have completed the scraper and the initial embedding proof of concept. The scraper has already processed about 7,000 My Hero Academia fics into a CSV and collected my own reading behavior data for a test set. The PostgreSQL schema is implemented, handling the complex relationships between fics, sessions, and value tables.

Next step is Stage 4: developing a browser extension or bookmarklet to act as the reading tracker. This is the most critical data source. Without real-time session logging, the XGBoost model won't have the ground truth needed to distinguish between a "click" and a "read."

Siagnos is essentially an attempt to apply professional ML principles to a massive, underserved niche. AO3's traffic is enormous, yet the discovery tools remain primitive. By treating reading behavior as a training set, we can move from "popular" to "personally relevant."

https://github.com/P-Kumar-18