Debugging a Memory Leak in Closed-Source ERP

My nightly integration job was crashing with a System.OutOfMemoryException, but the server had tens of gigabytes of free RAM. The vendor's advice was a joke: "just call support." I was dealing with a decades-old, closed-source ERP that our financial settlement process relied on—no source code, no debugger, and zero visibility into the actual logic.

The Symptom

The logs were hitting me every night during a specific 20-minute window:

System.OutOfMemoryException
 at System.Text.Encoding.GetChars(Byte[] bytes, Int32 index, Int32 count)
 at System.Text.Encoding.Convert(...)
 at .SQLSupport.Extensions.ToByteArray(String s)
 at .SQLSupport.Helper.Serializer.ToDataBuffer(IEntity entity)
 at .SQLSupport.ServiceProxy.ActivityServiceProxy.GetNextRecord(...)
 ...

At first glance, it looked like a "poisoned record" issue. The stack trace died during string serialization in ToByteArray. I assumed a corrupted field was passing a massive length value, forcing the system to attempt a multi-gigabyte allocation.

The Investigation

I quickly realized the "bad data" theory was a dead end. The errors weren't localized to one dataset; they were popping up across different internal methods and even during security/session setup calls. If the system is throwing OutOfMemory before it even touches your data, the process is already dying.

Then I had to pivot my mindset regarding .NET errors. An OutOfMemoryException doesn't always mean the physical RAM is full. In a 32-bit process, you are capped at a 2GB address space regardless of how much hardware you throw at it.

I pulled up Task Manager to monitor the culprit process. It was sitting at a measly 140 MB. The server had oceans of free RAM, but the process was claiming it couldn't allocate more. This wasn't a hardware bottleneck; it was an address space exhaustion issue.

The "Stakeout" Strategy

Since I couldn't attach a professional profiler to vendor software in a production environment, I had to build my own telemetry. I wrote a simple scheduled script that logged the process's committed memory, working set, and handle count every hour into a CSV. I keyed the script to the service account rather than a PID, so it could track the process even after daily server restarts.

After 24 hours, the data told the real story:

Time CommitMB Handles
10:00 293 2889
11:00 387 3779
12:00 485 4650
13:00 578 5596
... ... ...
21:00 1361 13429
22:00 1451 14361

The leak was undeniable. A 32-bit COM+ component was leaking memory and handles at a rate of roughly 96 MB/hr under load. It wasn't a single massive allocation causing the crash; it was a slow, steady bleed that eventually hit the 2GB ceiling during peak hours.

By treating the black-box process as an observable system rather than a mystery, I found the leak without ever seeing a single line of the vendor's source code.