Transformers parsing whale codas reveal syntax patterns

The bottleneck in bioacoustics was never recording — hydrophones and autonomous recorders have flooded us with petabytes of raw audio. The problem was annotation. A single sperm whale recording session yields thousands of codas, but labeled datasets barely cover a few hundred. That changed when CETI started applying masked audio modeling at scale, treating click sequences like masked language modeling on steroids.

Here's what actually works: train a HuBERT-style encoder on 4,000 hours of unlabeled Dominica sperm whale recordings, then fine-tune on the 8,000 human-annotated codas. The model learns discrete units — call them "phonemes" if you want — that cluster by behavioral context. Feeding bouts. Social codas. Identity codas. The clustering isn't perfect, but the mutual information between unit sequences and behavioral state jumps from 0.12 bits (raw spectrogram) to 0.47 bits (learned units). That's not noise.

The architecture matters less than the pretraining objective. Wav2Vec 2.0, HuBERT, and BEATs all converge on similar unit inventories when trained on enough cetacean data. What differs is downstream sample efficiency. For bird song — where we have better annotations from decades of ornithology — a frozen HuBERT backbone + linear probe hits 89% syllable classification on the BirdVox-DCASE dataset with 50 labeled examples per class. Scratch training needs 500.

But here's the uncomfortable part: we're still projecting human linguistic categories onto non-human systems. "Syntax" implies hierarchical compositionality. What we've actually found in sperm whales is combinatorial — codas combine into sequences, sequences vary by clan, but there's no evidence of recursive embedding. The Markov order of coda transitions tops out at 2. Compare that to Bengalese finches, where song syntax reaches order 4-5 with context-sensitive rules. Different species, different computational complexity.

The real breakthrough this year wasn't better models — it was the Earth Species Project releasing ESPnet-ST with multi-species ASR heads. One encoder, shared across 12 species, with species-specific decoders. Zero-shot transfer from zebra finch to canary works because the encoder learns universal acoustic primitives: harmonic stacks, frequency sweeps, click trains. The decoder just learns the inventory.

Practical takeaway if you're building on this: don't start from scratch. Grab the ESPnet pretrained checkpoints, freeze the encoder, train a lightweight probe on your target behavior (foraging vs. alarm vs. mating). You need ~200 labeled bouts per class. Augment heavily — pitch shift ±2 semitones, time stretch 0.8-1.2x, background noise mixing from the same habitat. The probe learns invariances the encoder missed.

Open question nobody's answering: how do we validate meaning without a Rosetta Stone? Playback experiments are the gold standard, but they're slow, expensive, and ethically fraught. We're stuck correlating model predictions with behavioral metadata and calling it "decoding." That's not decoding. That's pattern matching with better PR.

Next frontier: multimodal. Pairing acoustic units with simultaneous drone footage of body posture, bubble trails, proximity networks. The CETI team's 2024 deployment tagged 15 whales with DTAGs + video. That dataset — when it's public — will let us ground acoustic units in motor patterns. Finally, something resembling semantics.