CaSA: Computing LLM Inference Directly in RAM

The "memory wall" is the single biggest bottleneck for edge AI; fitting a 27B parameter model into a phone's RAM is useless if the SoC bus can't move that data fast enough to generate tokens without overheating the device or killing the battery. Even with extreme quantization like PrismML's ternary weights, the physical act of transferring gigabytes of data across the bus for every single token is inefficient.

CaSA solves this by implementing Processing-In-Memory (PIM) for ternary LLM inference. Instead of moving data to the compute unit, it performs the inference natively inside COTS DRAM using charge-sharing. This effectively bypasses the memory bus entirely, addressing the thermal throttling and power drain issues that plague current NPU implementations.

For those looking for a deep dive into the architecture, the project is available here:

https://github.com/pcdeni/CaSA

Technical Breakdown

• The Problem: Standard LPDDR bandwidth limitations. Even quantized models require massive data movement, leading to high energy consumption per token.
• The Solution: CaSA (Charge-Sharing Architecture) leverages the physical properties of DRAM to execute operations where the data lives.
• Hardware Impact: By eliminating the need to shuttle weights across the SoC bus, it reduces the power overhead of the "data movement" phase of inference.
• Synergy: This works best when paired with ternary weights (1.58-bit), as the simplified weight values make in-memory computation physically viable.

This is a necessary evolution for on-device AI. If we want a truly functional "Semantic Router" or a sophisticated agent running locally on a smartphone, we have to stop treating memory as a passive storage bin and start treating it as a compute element. This is a practical step toward actual AI-capable silicon that doesn't melt the chassis.