High-dimensional correlation isn't vision, it's just math
The "vision" pipeline is honestly kind of a trip when you peel back the abstraction layers. Instead of a visual cortex, the transformer-based architecture takes an image, decomposes it into discrete patches, and flattens those patches into vectors. From the perspective of the model, there is zero fundamental difference between a visual token and a text token from an LLM. It’s just a sequence of numerical representations being processed through a massive weight matrix. It’s not "looking" at whiskers; it’s calculating the probability of certain vector positions appearing together.
The real heavy lifting occurs during the contrastive learning phase. CLIP is essentially a bridge that forces these distinct modalities—text and pixels—into a shared embedding space. When you provide a prompt, the model isn't "thinking" about the concept; it's just finding the statistical neighborhood where those specific vectors reside. It's all just math, no soul, just vibes (and massive amounts of compute).
We need to stop conflating classification with recognition. A human recognizes a cat through a subjective, lived experience. An AI classifies an input by assigning a label based on statistical weight within its training distribution. There is a massive ontological gap between the two. Even as we push toward better contextual relationship modeling, we're still dealing with an alien intelligence that lacks a physical body or a temporal existence to ground those images in. It's just a very sophisticated way of crunching numbers until they look like something we recognize.
If you want to actually see how these embeddings are structured and stop relying on the hype, you should read the original implementation on GitHub:
https://github.com/openai/CLIPIt's basically just a massive matrix multiplication party at that point. Absolute madness.