CNN Feature Maps: Why 3D Kernels Are Non-Negotiable

The common misconception about Convolutional Neural Networks is that filters are 2D squares sliding over an image. In reality, if you're working with RGB data—like the deepfake detection datasets I've been debugging in PyTorch—the kernels are strictly 3D.

I hit a wall recently trying to map out my tensor shapes, and it clicked: when you see a batch shape like [500, 3, 256, 256], that second dimension (3) isn't just a label; it's a spatial requirement for the kernel.

The Dimensionality Mismatch

If you try to apply a 2D filter to a 3D image tensor (Channels, Height, Width), the math simply fails. For a convolution to work, the kernel depth must exactly match the input channel depth.

• Input Tensor: [Batch, Channels, Height, Width]
• Kernel Shape: [Out_Channels, In_Channels, Kernel_H, Kernel_W]

In my case, for a standard 3x3 convolution on an RGB image, the kernel is actually 3x3x3. It doesn't scan the Red, Green, and Blue channels sequentially; it scans them simultaneously.

How the Calculation Actually Works

The "sliding window" is a 3D cube. As it moves across the image:
1. It performs a dot product across all three channels at once.
2. It sums these values into a single scalar.
3. This result becomes one pixel in the resulting feature map.

This is why the output of a single filter is always a 2D plane, regardless of how many input channels there were. The depth is collapsed during the summation process.

If you're building a custom architecture from scratch, always double-check your in_channels parameter in nn.Conv2d. If that number doesn't align with your input tensor's channel dimension, PyTorch will throw a runtime error immediately.

import torch
import torch.nn as nn

# Input: 1 image, 3 channels (RGB), 256x256
input_tensor = torch.randn(1, 3, 256, 256)

# The kernel is 3x3, but it MUST have 3 input channels to match the image
conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3)

output = conv(input_tensor)
print(output.shape) # torch.Size([1, 16, 254, 254])

The 16 in the output shape represents 16 different 3D filters, each producing its own 2D feature map. Understanding this depth-collapsing mechanism is the only way to properly debug shape errors in an AI workflow.