Last modified

This is the heap checker we use at Google to detect memory leaks in C++ programs. There are three parts to using it: linking the library into an application, running the code, and analyzing the output.

Linking in the Library

The heap-checker is part of tcmalloc, so to install the heap checker into your executable, add -ltcmalloc to the link-time step for your executable. Also, while we don't necessarily recommend this form of usage, it's possible to add in the profiler at run-time using LD_PRELOAD:

% env LD_PRELOAD="/usr/lib/libtcmalloc.so" 

This does not turn on heap checking; it just inserts the code. For that reason, it's practical to just always link -ltcmalloc into a binary while developing; that's what we do at Google. (However, since any user can turn on the profiler by setting an environment variable, it's not necessarily recommended to install heapchecker-linked binaries into a production, running system.) Note that if you wish to use the heap checker, you must also use the tcmalloc memory-allocation library. There is no way currently to use the heap checker separate from tcmalloc.

Running the Code

Note: For security reasons, heap profiling will not write to a file -- and is thus not usable -- for setuid programs.

Whole-program Heap Leak Checking

The recommended way to use the heap checker is in "whole program" mode. In this case, the heap-checker starts tracking memory allocations before the start of main(), and checks again at program-exit. If it finds any memory leaks -- that is, any memory not pointed to by objects that are still "live" at program-exit -- it aborts the program (via exit(1)) and prints a message describing how to track down the memory leak (using pprof).

The heap-checker records the stack trace for each allocation while it is active. This causes a significant increase in memory usage, in addition to slowing your program down.

Here's how to run a program with whole-program heap checking:

1. Define the environment variable HEAPCHECK to the type of heap-checking to do. For instance, to heap-check /usr/local/bin/my_binary_compiled_with_tcmalloc:

   % env HEAPCHECK=normal /usr/local/bin/my_binary_compiled_with_tcmalloc

No other action is required.

Note that since the heap-checker uses the heap-profiling framework internally, it is not possible to run both the heap-checker and heap profiler at the same time.

Flavors of Heap Checking

These are the legal values when running a whole-program heap check:

1. minimal
2. normal
3. strict
4. draconian

"Minimal" heap-checking starts as late as possible ina initialization, meaning you can leak some memory in your initialization routines (that run before main(), say), and not trigger a leak message. If you frequently (and purposefully) leak data in one-time global initializers, "minimal" mode is useful for you. Otherwise, you should avoid it for stricter modes.

"Normal" heap-checking tracks live objects and reports a leak for any data that is not reachable via a live object when the program exits.

"Strict" heap-checking is much like "normal" but has a few extra checks that memory isn't lost in global destructors. In particular, if you have a global variable that allocates memory during program execution, and then "forgets" about the memory in the global destructor (say, by setting the pointer to it to NULL) without freeing it, that will prompt a leak message in "strict" mode, though not in "normal" mode.

"Draconian" heap-checking is appropriate for those who like to be very precise about their memory management, and want the heap-checker to help them enforce it. In "draconian" mode, the heap-checker does not do "live object" checking at all, so it reports a leak unless all allocated memory is freed before program exit. (However, you can use IgnoreObject() to re-enable liveness-checking on an object-by-object basis.)

"Normal" mode, as the name implies, is the one used most often at Google. It's appropriate for everyday heap-checking use.

In addition, there are two other possible modes:

• as-is
• local

as-is is the most flexible mode; it allows you to specify the various knobs of the heap checker explicitly. local activates the explicit heap-check instrumentation, but does not turn on any whole-program leak checking.

Tweaking whole-program checking

In some cases you want to check the whole program for memory leaks, but waiting for after main() exits to do the first whole-program leak check is waiting too long: e.g. in a long-running server one might wish to simply periodically check for leaks while the server is running. In this case, you can call the static method NoGlobalLeaks(), to verify no global leaks have happened as of that point in the program.

Alternately, doing the check after main() exits might be too late. Perhaps you have some objects that are known not to clean up properly at exit. You'd like to do the "at exit" check before those objects are destroyed (since while they're live, any memory they point to will not be considered a leak). In that case, you can call NoGlobalLeaks() manually, near the end of main(), and then call CancelGlobalCheck() to turn off the automatic post-main() check.

Finally, there's a helper macro for "strict" and "draconian" modes, which require all global memory to be freed before program exit. This freeing can be time-consuming and is often unnecessary, since libc cleans up all memory at program-exit for you. If you want the benefits of "strict"/"draconian" modes without the cost of all that freeing, look at REGISTER_HEAPCHECK_CLEANUP (in heap-checker.h). This macro allows you to mark specific cleanup code as active only when the heap-checker is turned on.

Explicit (Partial-program) Heap Leak Checking

  HeapLeakChecker heap_checker("test_foo");
  {
    code that exercises some foo functionality;
    this code should preserve memory allocation state;
  }
  if (!heap_checker.SameHeap()) assert(NULL == "heap memory leak");

   void LeakTwentyBytes() {
     char* a = malloc(20);
     HeapLeakChecker heap_checker("test_malloc");
     char* b = malloc(20);
     free(a);
     // This will pass: it totes up 20 bytes allocated and 20 bytes freed
     assert(heap_checker.BriefNoLeaks());  // doesn't detect that b is leaked
   }

(This is because BriefSameHeap() does not use pprof, which is slower but is better able to track allocations in tricky situations like the above.)

Note that adding in the HeapLeakChecker object merely instruments the code for leak-checking. To actually turn on this leak-checking on a particular run of the executable, you must still run with the heap-checker turned on:

% env HEAPCHECK=local /usr/local/bin/my_binary_compiled_with_tcmalloc

If you want to do whole-program leak checking in addition to this manual leak checking, you can run in normal or some other mode instead: they'll run the "local" checks in addition to the whole-program check.

Disabling Heap-checking of Known Leaks

Sometimes your code has leaks that you know about and are willing to accept. You would like the heap checker to ignore them when checking your program. You can do this by bracketing the code in question with an appropriate heap-checking construct:

   ...
   {
     HeapLeakChecker::Disabler disabler;
     <leaky code>
   }
   ...

Alternately, you can use IgnoreObject(), which takes a pointer to an object to ignore. That memory, and everything reachable from it (by following pointers), is ignored for the purposes of leak checking. You can call UnIgnoreObject() to undo the effects of IgnoreObject().

Tuning the Heap Checker

The heap leak checker has many options, some that trade off running time and accuracy, and others that increase the sensitivity at the risk of returning false positives. For most uses, the range covered by the heap-check flavors is enough, but in specialized cases more control can be helpful.

These options are specified via environment varaiables.

This first set of options controls sensitivity and accuracy. These options are ignored unless you run the heap checker in as-is mode.

HEAP_CHECK_AFTER_DESTRUCTORS	Default: false	When true, do the final leak check after all other global destructors have run. When false, do it after all REGISTER_HEAPCHECK_CLEANUP, typically much earlier in the global-destructor process.
HEAP_CHECK_IGNORE_THREAD_LIVE	Default: true	If true, ignore objects reachable from thread stacks and registers (that is, do not report them as leaks).
HEAP_CHECK_IGNORE_GLOBAL_LIVE	Default: true	If true, ignore objects reachable from global variables and data (that is, do not report them as leaks).

These options modify the behavior of whole-program leak checking.

These options apply to all types of leak checking.

Tips for Handling Detected Leaks

What do you do when the heap leak checker detects a memory leak? First, you should run the reported pprof command; hopefully, that is enough to track down the location where the leak occurs.

If the leak is a real leak, you should fix it!

If you are sure that the reported leaks are not dangerous and there is no good way to fix them, then you can use HeapLeakChecker::Disabler and/or HeapLeakChecker::IgnoreObject() to disable heap-checking for certain parts of the codebase.

In "strict" or "draconian" mode, leaks may be due to incomplete cleanup in the destructors of global variables. If you don't wish to augment the cleanup routines, but still want to run in "strict" or "draconian" mode, consider using REGISTER_HEAPCHECK_CLEANUP.

Hints for Debugging Detected Leaks

Sometimes it can be useful to not only know the exact code that allocates the leaked objects, but also the addresses of the leaked objects. Combining this e.g. with additional logging in the program one can then track which subset of the allocations made at a certain spot in the code are leaked.
To get the addresses of all leaked objects define the environment variable HEAP_CHECK_IDENTIFY_LEAKS to be 1. The object addresses will be reported in the form of addresses of fake immediate callers of the memory allocation routines. Note that the performance of doing leak-checking in this mode can be noticeably worse than the default mode.

One relatively common class of leaks that don't look real is the case of multiple initialization. In such cases the reported leaks are typically things that are linked from some global objects, which are initialized and say never modified again. The non-obvious cause of the leak is frequently the fact that the initialization code for these objects executes more than once.
E.g. if the code of some .cc file is made to be included twice into the binary, then the constructors for global objects defined in that file will execute twice thus leaking the things allocated on the first run.
Similar problems can occur if object initialization is done more explicitly e.g. on demand by a slightly buggy code that does not always ensure only-once initialization.

A more rare but even more puzzling problem can be use of not properly aligned pointers (maybe inside of not properly aligned objects). Normally such pointers are not followed by the leak checker, hence the objects reachable only via such pointers are reported as leaks. If you suspect this case define the environment variable HEAP_CHECK_TEST_POINTER_ALIGNMENT to be 1 and then look closely at the generated leak report messages.

How It Works

When a HeapLeakChecker object is constructed, it dumps a memory-usage profile named <prefix>.<name>-beg.heap to a temporary directory. When *NoLeaks() or *SameHeap() is called (for whole-program checking, this happens automatically at program-exit), it dumps another profile, named <prefix>.<name>-end.heap. (<prefix> is typically determined automatically, and <name> is typically argv[0].) It then compares the two profiles. If the second profile shows more memory use than the first (or, for *SameHeap() calls, any different pattern of memory use than the first), the *NoLeaks() or *SameHeap() function will return false. For "whole program" profiling, this will cause the executable to abort (via exit(1)). In all cases, it will print a message on how to process the dumped profiles to locate leaks.

Detecting Live Objects

At any point during a program's execution, all memory that is accessible at that time is considered "live." This includes global variables, and also any memory that is reachable by following pointers from a global variable. It also includes all memory reachable from the current stack frame and from current CPU registers (this captures local variables). Finally, it includes the thread equivalents of these: thread-local storage and thread heaps, memory reachable from thread-local storage and thread heaps, and memory reachable from thread CPU registers.

In all modes except "draconian," live memory is not considered to be a leak. We detect this by doing a liveness flood, traversing pointers to heap objects starting from some initial memory regions we know to potentially contain live pointer data. Note that this flood might potentially not find some (global) live data region to start the flood from. If you find such, please file a bug.

The liveness flood attempts to treat any properly aligned byte sequences as pointers to heap objects and thinks that it found a good pointer whenever the current heap memory map contains an object with the address whose byte representation we found. Some pointers into not-at-start of object will also work here.

As a result of this simple approach, it's possible (though unlikely) for the flood to be inexact and occasionally result in leaked objects being erroneously determined to be live. For instance, random bit patterns can happen to look like pointers to leaked heap objects. More likely, stale pointer data not corresponding to any live program variables can be still present in memory regions, especially in thread stacks. For instance, depending on how the local malloc is implemented, it may reuse a heap object address:

    char* p = new char[1];   // new might return 0x80000000, say.
    delete p;
    new char[1];             // new might return 0x80000000 again
    // This last new is a leak, but doesn't seem it: p looks like it points to it

In other words, imprecisions in the liveness flood mean that for any heap leak check we might miss some memory leaks. This means that for local leak checks, we might report a memory leak in the local area, even though the leak actually happened before the HeapLeakChecker object was constructed. Note that for whole-program checks, a leak report does always correspond to a real leak (since there's no "before" to have created a false-live object).

While this liveness flood approach is not very portable and not 100% accurate, it works in most cases and saves us from writing a lot of explicit clean up code and other hassles when dealing with thread data.

More Exact Checking via pprof

The perftools profiling tool, pprof, is primarily intended for users to use interactively in order to explore heap and CPU usage. However, the heap-checker can -- and, by default, does - call pprof internally, in order to improve its leak checking.

In particular, the heap-checker calls pprof to utilize the full call-path for all allocations. pprof uses this data to disambiguate allocations. When the time comes to do a SameHeap or NoLeaks check, the heap-checker asks pprof to do this check on an allocation-by-allocation basis, rather than just by comparing global counts.

Here's an example. Consider the following function:

   void LeakTwentyBytes() {
     char* a = malloc(20);
     HeapLeakChecker heap_checker("test_malloc");
     char* b = malloc(20);
     free(a);
     heap_checker.NoLeaks();
   }

Without using pprof, the only thing we will do is count up the number of allocations and frees inside the leak-checked interval. Twenty bytes allocated, twenty bytes freed, and the code looks ok.

With pprof, however, we can track the call-path for each allocation, and account for them separately. In the example function above, there are two call-paths that end in an allocation, one that ends in "LeakTwentyBytes:line1" and one that ends in "LeakTwentyBytes:line3".

Here's how the heap-checker works when it can use pprof in this way:

1. Line 1: Allocate 20 bytes, mark a as having call-path "LeakTwentyBytes:line1", and update the count-map

   count["LeakTwentyByte:line1"] += 20;

2. Line 2: Dump the current count map to a file.
3. Line 3: Allocate 20 bytes, mark b as having call-path "LeakTwentyBytes:line3", and update the count-map:

   count["LeakTwentyByte:line3"] += 20;

4. Line 4: Look up a to find its call-path (stored in line 1), and use that to update the count-map:

   count["LeakTwentyByte:line1"] -= 20;

5. Line 5: Look at each bucket in the current count-map, minus what was dumped in line 2. Here's the diffs we'll have in each bucket:

   count["LeakTwentyByte:line1"] == -20;
   count["LeakTwentyByte:line3"] == 20;
          

   Since at least one bucket has a positive number, we complain of a leak. (Note if line 5 had been SameHeap instead of NoLeaks, we would have complained if any bucket had had a non-zero number.)

Note that one way to visualize the non-pprof mode is that we do the same thing as above, but always use "unknown" as the call-path. That is, our count-map always only has one entry in it: count["unknown"]. Looking at the example above shows how having only one entry in the map can lead to incorrect results.

Here is when pprof is used by the heap-checker:

• NoLeaks() and SameHeap() both use pprof.
• BriefNoLeaks() and BriefSameHeap() do not use pprof.
• QuickNoLeaks and QuickSameHeap() are a kind of compromise: they do not use pprof for their leak check, but if that check happens to find a leak anyway, then they re-do the leak calculation using pprof. This means they do not always find leaks, but when they do, they will be as accurate as possible in their leak report.

Leak-checking and Threads

At the time of HeapLeakChecker's construction and during *NoLeaks()/*SameHeap() calls, we grab a lock and then pause all other threads so other threads do not interfere with recording or analyzing the state of the heap.

In general, leak checking works correctly in the presence of threads. However, thread stack data liveness determination (via base/thread_lister.h) does not work when the program is running under GDB, because the ptrace functionality needed for finding threads is already hooked to by GDB. Conversely, leak checker's ptrace attempts might also interfere with GDB. As a result, GDB can result in potentially false leak reports. For this reason, the heap-checker turns itself off when running under GDB.

Also, thread_lister only works for Linux pthreads; leak checking is unlikely to handle other thread implementations correctly.

As mentioned in the discussion of liveness flooding, thread-stack liveness determination might mis-classify as reachable objects that very recently became unreachable (leaked). This can happen when the pointers to now-logically-unreachable objects are present in the active thread stack frame. In other words, trivial code like the following might not produce the expected leak checking outcome depending on how the compiled code works with the stack:

  int* foo = new int [20];
  HeapLeakChecker check("a_check");
  foo = NULL;
  CHECK(check.NoLeaks());  // this might succeed