ProgramBench reverse-engineering benchmark puts LLMs to the test

I've been digging into ProgramBench's new vetted split and the reverse-engineering track is genuinely different from the usual code-generation benchmarks. Instead of "write a function that does X," you hand the model a stripped ELF or PE binary — no symbols, no debug info — and ask it to recover something semantically equivalent in C or Rust. The vetted subset filters out anything that decompilers like Ghidra or IDA Pro already solve trivially, so you're left with cases where control-flow recovery, indirect-call resolution, and data-type inference actually matter.

What surprised me: the benchmark doesn't just score syntactic similarity. Each submission gets compiled, linked against the original's test harness, and run through a differential fuzzer (libFuzzer + AFL++) for 30 minutes. If the regenerated code diverges on any input, it's a hard fail. That means hallucinated helper functions, wrong calling conventions, or off-by-one loop bounds all surface immediately. The leaderboard currently tops out around 34% pass@1 for GPT-4o, 28% for Claude 3.5 Sonnet, and 19% for DeepSeek-Coder-V2 — and those numbers drop another 8-12 points when you enable the "no standard library" flag.

A few practical takeaways if you want to experiment yourself:

1. Strip aggressively before feeding the binary. strip --strip-all --remove-section=.comment --remove-section=.note.* removes the low-hanging fruit that inflates scores. The benchmark's Docker image does this automatically, but local reproduction needs it.

2. Use a two-stage prompt. First pass: "Produce a Ghidra-like pseudo-C listing with all functions, globals, and struct layouts you can infer." Second pass: "Refine the listing into compilable C99, replacing pseudo-constructs with real code, adding missing headers, and fixing calling conventions." Single-shot prompts consistently miss cross-function type constraints.

3. Anchor the entry point. Add a comment like // ENTRY: 0x401000 in the prompt context. Without it, models frequently invent their own main and ignore the real _start/WinMain, causing immediate harness failures.

4. Iterate with compiler feedback. Pipe gcc -Wall -Wextra -Werror -std=c99 -o /dev/null -xc - output back into the context window. Three rounds usually clears 60-70% of syntactic errors before you even hit the fuzzer.

# Quick local smoke test (matches benchmark harness)
docker run --rm -v $(pwd):/work programbench/vetted:latest \
  --binary /work/target.bin \
  --submission /work/recovered.c \
  --timeout 1800 \
  --fuzzer-libfuzzer --fuzzer-aflpp

The most frustrating failure mode I've seen: models correctly recover algorithmic logic (e.g., a custom hash map) but mangle the memory allocator interface — returning malloced pointers where the original used a bump allocator with a custom free list. The fuzzer catches it because the original binary's free list gets corrupted. Fixing this requires the model to infer allocator semantics from usage patterns, not just function signatures.

If anyone's tried fine-tuning on the training split (about 12k binaries × 3 architectures), I'd love to hear whether LoRA on CodeLlama-34B moves the needle past 40%. The benchmark maintainers hinted at a "compiler-optimization" track coming next quarter — binaries built with -O3 -flto -march=native — which should make control-flow recovery significantly nastier.