How OpenAI Caught an 18-Year-Old Bug in libunwind that Ruined ChatGPT
When you run queries for millions, even a one-in-a-billion glitch becomes a daily nightmare. OpenAI just solved an ancient, 'impossible' cold case that had been lurking in core Linux infrastructure since the days of flip phones.
The trouble began with mysterious, seemingly impossible crashes inside Rockset, a high-performance C++ search engine that OpenAI acquired in 2024 to power ChatGPT's data retrieval. Standard functions were returning to completely wild memory addresses, or the stack pointer would randomly shift by 8 bytes. It looked like the laws of physics were breaking, and every logical explanation the engineering team came up with was quickly debunked by the next crash.
Instead of playing doctor and examining individual memory dumps one by one, the team decided to act like epidemiologists. They had ChatGPT write a script to download and catalog a whole year’s worth of crash dumps from their Azure servers. This mass-data approach immediately revealed that they weren't dealing with one bizarre bug, but two. One was a rare, silent hardware failure where a single Azure server's CPU literally forgot how to do basic math. The other was a race condition hiding inside the libunwind library.
This vulnerable piece of code had been sitting quietly in the GNU libunwind library since 2007, serving as a foundational block for handling C++ exceptions on 64-bit systems. The mechanics of the bug are brutally simple. When an exception occurs, the runtime library unwinds the stack to restore CPU registers. During this process, there is a tiny window lasting exactly one CPU instruction—about 100 picoseconds—where the return address is left in a memory zone that the operating system kernel is technically allowed to overwrite. If a system signal happens to land precisely in this 100-picosecond gap, the kernel overwrites the memory, turning the return address into NULL.
Most software never hits this because throwing exceptions and receiving signals at the exact same picosecond is rarer than winning the lottery twice in a row. However, Rockset's architecture threw an absurdly high volume of exceptions while constantly bombarded by system signals. To stop the bleeding, the engineers migrated their stack-unwinding mechanism to libgcc and sent a patch to the upstream libunwind maintainers.
It turns out that even the most pristine, decade-old open-source foundations can harbor microscopic landmines that only explode when subjected to the crushing weight of modern AI workloads. When code runs at the scale of billions of users, the impossible doesn't just become possible—it happens multiple times before lunch.
Source: OpenAI
Comments
This is where the magic happens: AI reads your discussion and rewrites the article based on the most interesting comments. Each strong comment adds points to the meter below. Once the meter is full, the article updates live — no page reload needed.