Built a cover-only recommender with CLIP embeddings and a

I've been picking apart the By-Its-Cover architecture for a side project, and the constraint that fascinates me is how far you can push cover embeddings alone before the signal collapses. The author explicitly limited themselves to CLIP vectors — no metadata, no descriptions, no author embeddings — just the visual signal. That's either brave or naive depending on whether you've tried clustering book covers by genre.

The semantic search pipeline is a hybrid: CLIP vector search fused with GLiNER-extracted entities queried against the Hardcover API via Reciprocal Rank Fusion. GLiNER runs ONNX, which is the right call for latency, but the NER step feels like a band-aid for CLIP's known weakness on text-heavy covers. When the cover is just a title in a distinctive font, the visual embedding captures something useful. When it's an abstract pattern, the entity extraction carries the load. RRF merging them is standard practice, but I'd want to see the weight tuning — equal weighting assumes both signals have comparable precision, which they rarely do.

The collaborative filtering side uses a two-tower neural hybrid model with DPP diversification. Two-tower is the right architecture for this scale — the item tower stays fixed during serving, only the user tower updates per session. But training on explicit "Dislike/Like/Love" feedback with only a few thousand items and sparse user interactions? That's a cold-start nightmare. The offline retrain cadence (daily full retrain, 2-hour fine-tune) suggests the author knows the embedding drift problem. Eugene Yan's discovery system design post is the right reference here, though the 2-hour incremental update feels aggressive for a model that sees maybe dozens of new ratings per window.

# DPP diversification snippet from bic-learn
def diversify_recommendations(embeddings, k=10, lambda_param=0.5):
    kernel = embeddings @ embeddings.T
    selected = []
    for _ in range(k):
        scores = np.diag(kernel)
        for idx in selected:
            scores -= lambda_param * kernel[idx] ** 2
        next_idx = np.argmax(scores)
        selected.append(next_idx)
    return selected

The diversification logic above is a greedy MAP approximation — standard, but the lambda_param needs per-genre tuning. Fantasy covers cluster tightly; literary fiction spreads out. A fixed lambda will over-diversify one and under-diversify the other.

AWS deployment details are sparse in the writeup, but the async ingestion loop (search → Hardcover API → vector DB) is the growth engine. Every failed search that triggers an API call expands the catalog. Clever flywheel if the Hardcover rate limits hold.

Biggest question: at what catalog size does cover-only CLIP saturate? My guess is 50k-100k items before genre confusion dominates. Would love to see retrieval@k curves as the corpus grows.