mirror of
https://github.com/gperftools/gperftools
synced 2024-12-23 15:52:10 +00:00
e78238d94d
In most practical terms, this expands "official" heap leak checker support to Linux/arm64 and Linux/riscv (mips-en and legacy arm are likely to work & pass tests too now). The code is now explicitly Linux-only, without trying to pretend otherwise. Main goal of this change is to finally amputate linux_syscall_support.h, which we historically had trouble maintaining well. Biggest challenge was around thread listing facility which uses clone (ptrace explicitly fails between threads) and that causes difficulties around parent and child tasks sharing errno. linux_syscall_support stuff had special feature to "redirect" errno accesses. But it caused us for more trouble. We switched to regular syscalls, and errno stamping avoidance is now simply via careful programming. A number of other cleanups is made (such us thread finding codes in procfs which clearly was built for some ages old and odd kernels). sem_post/sem_wait synchronization was previously potentially prone to deadlock (if parent died at bad time). We now use pipe pair for this synchronization and it is fully robust.
535 lines
21 KiB
HTML
535 lines
21 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
|
|
<HTML>
|
|
|
|
<HEAD>
|
|
<link rel="stylesheet" href="designstyle.css">
|
|
<title>Gperftools Heap Leak Checker</title>
|
|
</HEAD>
|
|
|
|
<BODY>
|
|
|
|
<p align=right>
|
|
<i>Last modified
|
|
<script type=text/javascript>
|
|
var lm = new Date(document.lastModified);
|
|
document.write(lm.toDateString());
|
|
</script></i>
|
|
</p>
|
|
|
|
<p>This is the heap checker we use at Google to detect memory leaks in
|
|
C++ programs. There are three parts to using it: linking the library
|
|
into an application, running the code, and analyzing the output.</p>
|
|
|
|
|
|
<H1>Linking in the Library</H1>
|
|
|
|
<p>The heap-checker is part of tcmalloc, so to install the heap
|
|
checker into your executable, add <code>-ltcmalloc</code> to the
|
|
link-time step for your executable. Also, while we don't necessarily
|
|
recommend this form of usage, it's possible to add in the profiler at
|
|
run-time using <code>LD_PRELOAD</code>:</p>
|
|
<pre>% env LD_PRELOAD="/usr/lib/libtcmalloc.so" <binary></pre>
|
|
|
|
<p>This does <i>not</i> turn on heap checking; it just inserts the
|
|
code. For that reason, it's practical to just always link
|
|
<code>-ltcmalloc</code> into a binary while developing; that's what we
|
|
do at Google. (However, since any user can turn on the profiler by
|
|
setting an environment variable, it's not necessarily recommended to
|
|
install heapchecker-linked binaries into a production, running
|
|
system.) Note that if you wish to use the heap checker, you must
|
|
also use the tcmalloc memory-allocation library. There is no way
|
|
currently to use the heap checker separate from tcmalloc.</p>
|
|
|
|
|
|
<h1>Running the Code</h1>
|
|
|
|
<p>Note: For security reasons, heap profiling will not write to a file
|
|
-- and is thus not usable -- for setuid programs.</p>
|
|
|
|
<h2><a name="whole_program">Whole-program Heap Leak Checking</a></h2>
|
|
|
|
<p>The recommended way to use the heap checker is in "whole program"
|
|
mode. In this case, the heap-checker starts tracking memory
|
|
allocations before the start of <code>main()</code>, and checks again
|
|
at program-exit. If it finds any memory leaks -- that is, any memory
|
|
not pointed to by objects that are still "live" at program-exit -- it
|
|
aborts the program (via <code>exit(1)</code>) and prints a message
|
|
describing how to track down the memory leak (using <A
|
|
HREF="heapprofile.html#pprof">pprof</A>).</p>
|
|
|
|
<p>The heap-checker records the stack trace for each allocation while
|
|
it is active. This causes a significant increase in memory usage, in
|
|
addition to slowing your program down.</p>
|
|
|
|
<p>Here's how to run a program with whole-program heap checking:</p>
|
|
|
|
<ol>
|
|
<li> <p>Define the environment variable HEAPCHECK to the <A
|
|
HREF="#types">type of heap-checking</A> to do. For instance,
|
|
to heap-check
|
|
<code>/usr/local/bin/my_binary_compiled_with_tcmalloc</code>:</p>
|
|
<pre>% env HEAPCHECK=normal /usr/local/bin/my_binary_compiled_with_tcmalloc</pre>
|
|
</ol>
|
|
|
|
<p>No other action is required.</p>
|
|
|
|
<p>Note that since the heap-checker uses the heap-profiling framework
|
|
internally, it is not possible to run both the heap-checker and <A
|
|
HREF="heapprofile.html">heap profiler</A> at the same time.</p>
|
|
|
|
|
|
<h3><a name="types">Flavors of Heap Checking</a></h3>
|
|
|
|
<p>These are the legal values when running a whole-program heap
|
|
check:</p>
|
|
<ol>
|
|
<li> <code>minimal</code>
|
|
<li> <code>normal</code>
|
|
<li> <code>strict</code>
|
|
<li> <code>draconian</code>
|
|
</ol>
|
|
|
|
<p>"Minimal" heap-checking starts as late as possible in a
|
|
initialization, meaning you can leak some memory in your
|
|
initialization routines (that run before <code>main()</code>, say),
|
|
and not trigger a leak message. If you frequently (and purposefully)
|
|
leak data in one-time global initializers, "minimal" mode is useful
|
|
for you. Otherwise, you should avoid it for stricter modes.</p>
|
|
|
|
<p>"Normal" heap-checking tracks <A HREF="#live">live objects</A> and
|
|
reports a leak for any data that is not reachable via a live object
|
|
when the program exits.</p>
|
|
|
|
<p>"Strict" heap-checking is much like "normal" but has a few extra
|
|
checks that memory isn't lost in global destructors. In particular,
|
|
if you have a global variable that allocates memory during program
|
|
execution, and then "forgets" about the memory in the global
|
|
destructor (say, by setting the pointer to it to NULL) without freeing
|
|
it, that will prompt a leak message in "strict" mode, though not in
|
|
"normal" mode.</p>
|
|
|
|
<p>"Draconian" heap-checking is appropriate for those who like to be
|
|
very precise about their memory management, and want the heap-checker
|
|
to help them enforce it. In "draconian" mode, the heap-checker does
|
|
not do "live object" checking at all, so it reports a leak unless
|
|
<i>all</i> allocated memory is freed before program exit. (However,
|
|
you can use <A HREF="#disable">IgnoreObject()</A> to re-enable
|
|
liveness-checking on an object-by-object basis.)</p>
|
|
|
|
<p>"Normal" mode, as the name implies, is the one used most often at
|
|
Google. It's appropriate for everyday heap-checking use.</p>
|
|
|
|
<p>In addition, there are two other possible modes:</p>
|
|
<ul>
|
|
<li> <code>as-is</code>
|
|
<li> <code>local</code>
|
|
</ul>
|
|
<p><code>as-is</code> is the most flexible mode; it allows you to
|
|
specify the various <A HREF="#options">knobs</A> of the heap checker
|
|
explicitly. <code>local</code> activates the <A
|
|
HREF="#explicit">explicit heap-check instrumentation</A>, but does not
|
|
turn on any whole-program leak checking.</p>
|
|
|
|
|
|
<h3><A NAME="tweaking">Tweaking whole-program checking</A></h3>
|
|
|
|
<p>In some cases you want to check the whole program for memory leaks,
|
|
but waiting for after <code>main()</code> exits to do the first
|
|
whole-program leak check is waiting too long: e.g. in a long-running
|
|
server one might wish to simply periodically check for leaks while the
|
|
server is running. In this case, you can call the static method
|
|
<code>HeapLeakChecker::NoGlobalLeaks()</code>, to verify no global leaks have happened
|
|
as of that point in the program.</p>
|
|
|
|
<p>Alternately, doing the check after <code>main()</code> exits might
|
|
be too late. Perhaps you have some objects that are known not to
|
|
clean up properly at exit. You'd like to do the "at exit" check
|
|
before those objects are destroyed (since while they're live, any
|
|
memory they point to will not be considered a leak). In that case,
|
|
you can call <code>HeapLeakChecker::NoGlobalLeaks()</code> manually, near the end of
|
|
<code>main()</code>, and then call <code>HeapLeakChecker::CancelGlobalCheck()</code> to
|
|
turn off the automatic post-<code>main()</code> check.</p>
|
|
|
|
<p>Finally, there's a helper macro for "strict" and "draconian" modes,
|
|
which require all global memory to be freed before program exit. This
|
|
freeing can be time-consuming and is often unnecessary, since libc
|
|
cleans up all memory at program-exit for you. If you want the
|
|
benefits of "strict"/"draconian" modes without the cost of all that
|
|
freeing, look at <code>REGISTER_HEAPCHECK_CLEANUP</code> (in
|
|
<code>heap-checker.h</code>). This macro allows you to mark specific
|
|
cleanup code as active only when the heap-checker is turned on.</p>
|
|
|
|
|
|
<h2><a name="explicit">Explicit (Partial-program) Heap Leak Checking</h2>
|
|
|
|
<p>Instead of whole-program checking, you can check certain parts of your
|
|
code to verify they do not have memory leaks. This check verifies that
|
|
between two parts of a program, no memory is allocated without being freed.</p>
|
|
<p>To use this kind of checking code, bracket the code you want
|
|
checked by creating a <code>HeapLeakChecker</code> object at the
|
|
beginning of the code segment, and call
|
|
<code>NoLeaks()</code> at the end. These functions, and all others
|
|
referred to in this file, are declared in
|
|
<code><gperftools/heap-checker.h></code>.
|
|
</p>
|
|
|
|
<p>Here's an example:</p>
|
|
<pre>
|
|
HeapLeakChecker heap_checker("test_foo");
|
|
{
|
|
code that exercises some foo functionality;
|
|
this code should not leak memory;
|
|
}
|
|
if (!heap_checker.NoLeaks()) assert(NULL == "heap memory leak");
|
|
</pre>
|
|
|
|
<p>Note that adding in the <code>HeapLeakChecker</code> object merely
|
|
instruments the code for leak-checking. To actually turn on this
|
|
leak-checking on a particular run of the executable, you must still
|
|
run with the heap-checker turned on:</p>
|
|
<pre>% env HEAPCHECK=local /usr/local/bin/my_binary_compiled_with_tcmalloc</pre>
|
|
<p>If you want to do whole-program leak checking in addition to this
|
|
manual leak checking, you can run in <code>normal</code> or some other
|
|
mode instead: they'll run the "local" checks in addition to the
|
|
whole-program check.</p>
|
|
|
|
|
|
<h2><a name="disable">Disabling Heap-checking of Known Leaks</a></h2>
|
|
|
|
<p>Sometimes your code has leaks that you know about and are willing
|
|
to accept. You would like the heap checker to ignore them when
|
|
checking your program. You can do this by bracketing the code in
|
|
question with an appropriate heap-checking construct:</p>
|
|
<pre>
|
|
...
|
|
{
|
|
HeapLeakChecker::Disabler disabler;
|
|
<leaky code>
|
|
}
|
|
...
|
|
</pre>
|
|
Any objects allocated by <code>leaky code</code> (including inside any
|
|
routines called by <code>leaky code</code>) and any objects reachable
|
|
from such objects are not reported as leaks.
|
|
|
|
<p>Alternately, you can use <code>IgnoreObject()</code>, which takes a
|
|
pointer to an object to ignore. That memory, and everything reachable
|
|
from it (by following pointers), is ignored for the purposes of leak
|
|
checking. You can call <code>UnIgnoreObject()</code> to undo the
|
|
effects of <code>IgnoreObject()</code>.</p>
|
|
|
|
|
|
<h2><a name="options">Tuning the Heap Checker</h2>
|
|
|
|
<p>The heap leak checker has many options, some that trade off running
|
|
time and accuracy, and others that increase the sensitivity at the
|
|
risk of returning false positives. For most uses, the range covered
|
|
by the <A HREF="#types">heap-check flavors</A> is enough, but in
|
|
specialized cases more control can be helpful.</p>
|
|
|
|
<p>
|
|
These options are specified via environment varaiables.
|
|
</p>
|
|
|
|
<p>This first set of options controls sensitivity and accuracy. These
|
|
options are ignored unless you run the heap checker in <A
|
|
HREF="#types">as-is</A> mode.
|
|
|
|
<table frame=box rules=sides cellpadding=5 width=100%>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_AFTER_DESTRUCTORS</code></td>
|
|
<td>Default: false</td>
|
|
<td>
|
|
When true, do the final leak check after all other global
|
|
destructors have run. When false, do it after all
|
|
<code>REGISTER_HEAPCHECK_CLEANUP</code>, typically much earlier in
|
|
the global-destructor process.
|
|
</td>
|
|
</tr>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_IGNORE_THREAD_LIVE</code></td>
|
|
<td>Default: true</td>
|
|
<td>
|
|
If true, ignore objects reachable from thread stacks and registers
|
|
(that is, do not report them as leaks).
|
|
</td>
|
|
</tr>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_IGNORE_GLOBAL_LIVE</code></td>
|
|
<td>Default: true</td>
|
|
<td>
|
|
If true, ignore objects reachable from global variables and data
|
|
(that is, do not report them as leaks).
|
|
</td>
|
|
</tr>
|
|
|
|
</table>
|
|
|
|
<p>These options modify the behavior of whole-program leak
|
|
checking.</p>
|
|
|
|
<table frame=box rules=sides cellpadding=5 width=100%>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_MAX_LEAKS</code></td>
|
|
<td>Default: 20</td>
|
|
<td>
|
|
The maximum number of leaks to be printed to stderr (all leaks are still
|
|
emitted to file output for pprof to visualize). If negative or zero,
|
|
print all the leaks found.
|
|
</td>
|
|
</tr>
|
|
|
|
|
|
</table>
|
|
|
|
<p>These options apply to all types of leak checking.</p>
|
|
|
|
<table frame=box rules=sides cellpadding=5 width=100%>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_IDENTIFY_LEAKS</code></td>
|
|
<td>Default: false</td>
|
|
<td>
|
|
If true, generate the addresses of the leaked objects in the
|
|
generated memory leak profile files.
|
|
</td>
|
|
</tr>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_TEST_POINTER_ALIGNMENT</code></td>
|
|
<td>Default: false</td>
|
|
<td>
|
|
If true, check all leaks to see if they might be due to the use
|
|
of unaligned pointers.
|
|
</td>
|
|
</tr>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_POINTER_SOURCE_ALIGNMENT</code></td>
|
|
<td>Default: sizeof(void*)</td>
|
|
<td>
|
|
Alignment at which all pointers in memory are supposed to be located.
|
|
Use 1 if any alignment is ok.
|
|
</td>
|
|
</tr>
|
|
|
|
<tr valign=top>
|
|
<td><code>PPROF_PATH</code></td>
|
|
<td>Default: pprof</td>
|
|
<td>
|
|
The location of the <code>pprof</code> executable.
|
|
</td>
|
|
</tr>
|
|
|
|
<tr valign=top>
|
|
<td><code>HEAP_CHECK_DUMP_DIRECTORY</code></td>
|
|
<td>Default: /tmp</td>
|
|
<td>
|
|
Where the heap-profile files are kept while the program is running.
|
|
</td>
|
|
</tr>
|
|
|
|
</table>
|
|
|
|
|
|
<h2>Tips for Handling Detected Leaks</h2>
|
|
|
|
<p>What do you do when the heap leak checker detects a memory leak?
|
|
First, you should run the reported <code>pprof</code> command;
|
|
hopefully, that is enough to track down the location where the leak
|
|
occurs.</p>
|
|
|
|
<p>If the leak is a real leak, you should fix it!</p>
|
|
|
|
<p>If you are sure that the reported leaks are not dangerous and there
|
|
is no good way to fix them, then you can use
|
|
<code>HeapLeakChecker::Disabler</code> and/or
|
|
<code>HeapLeakChecker::IgnoreObject()</code> to disable heap-checking
|
|
for certain parts of the codebase.</p>
|
|
|
|
<p>In "strict" or "draconian" mode, leaks may be due to incomplete
|
|
cleanup in the destructors of global variables. If you don't wish to
|
|
augment the cleanup routines, but still want to run in "strict" or
|
|
"draconian" mode, consider using <A
|
|
HREF="#tweaking"><code>REGISTER_HEAPCHECK_CLEANUP</code></A>.</p>
|
|
|
|
<h2>Hints for Debugging Detected Leaks</h2>
|
|
|
|
<p>Sometimes it can be useful to not only know the exact code that
|
|
allocates the leaked objects, but also the addresses of the leaked objects.
|
|
Combining this e.g. with additional logging in the program
|
|
one can then track which subset of the allocations
|
|
made at a certain spot in the code are leaked.
|
|
<br/>
|
|
To get the addresses of all leaked objects
|
|
define the environment variable <code>HEAP_CHECK_IDENTIFY_LEAKS</code>
|
|
to be <code>1</code>.
|
|
The object addresses will be reported in the form of addresses
|
|
of fake immediate callers of the memory allocation routines.
|
|
Note that the performance of doing leak-checking in this mode
|
|
can be noticeably worse than the default mode.
|
|
</p>
|
|
|
|
<p>One relatively common class of leaks that don't look real
|
|
is the case of multiple initialization.
|
|
In such cases the reported leaks are typically things that are
|
|
linked from some global objects,
|
|
which are initialized and say never modified again.
|
|
The non-obvious cause of the leak is frequently the fact that
|
|
the initialization code for these objects executes more than once.
|
|
<br/>
|
|
E.g. if the code of some <code>.cc</code> file is made to be included twice
|
|
into the binary, then the constructors for global objects defined in that file
|
|
will execute twice thus leaking the things allocated on the first run.
|
|
<br/>
|
|
Similar problems can occur if object initialization is done more explicitly
|
|
e.g. on demand by a slightly buggy code
|
|
that does not always ensure only-once initialization.
|
|
</p>
|
|
|
|
<p>
|
|
A more rare but even more puzzling problem can be use of not properly
|
|
aligned pointers (maybe inside of not properly aligned objects).
|
|
Normally such pointers are not followed by the leak checker,
|
|
hence the objects reachable only via such pointers are reported as leaks.
|
|
If you suspect this case
|
|
define the environment variable <code>HEAP_CHECK_TEST_POINTER_ALIGNMENT</code>
|
|
to be <code>1</code>
|
|
and then look closely at the generated leak report messages.
|
|
</p>
|
|
|
|
<h1>How It Works</h1>
|
|
|
|
<p>When a <code>HeapLeakChecker</code> object is constructed, it dumps
|
|
a memory-usage profile named
|
|
<code><prefix>.<name>-beg.heap</code> to a temporary
|
|
directory. When <code>NoLeaks()</code>
|
|
is called (for whole-program checking, this happens automatically at
|
|
program-exit), it dumps another profile, named
|
|
<code><prefix>.<name>-end.heap</code>.
|
|
(<code><prefix></code> is typically determined automatically,
|
|
and <code><name></code> is typically <code>argv[0]</code>.) It
|
|
then compares the two profiles. If the second profile shows
|
|
more memory use than the first, the
|
|
<code>NoLeaks()</code> function will
|
|
return false. For "whole program" profiling, this will cause the
|
|
executable to abort (via <code>exit(1)</code>). In all cases, it will
|
|
print a message on how to process the dumped profiles to locate
|
|
leaks.</p>
|
|
|
|
<h3><A name=live>Detecting Live Objects</A></h3>
|
|
|
|
<p>At any point during a program's execution, all memory that is
|
|
accessible at that time is considered "live." This includes global
|
|
variables, and also any memory that is reachable by following pointers
|
|
from a global variable. It also includes all memory reachable from
|
|
the current stack frame and from current CPU registers (this captures
|
|
local variables). Finally, it includes the thread equivalents of
|
|
these: thread-local storage and thread heaps, memory reachable from
|
|
thread-local storage and thread heaps, and memory reachable from
|
|
thread CPU registers.</p>
|
|
|
|
<p>In all modes except "draconian," live memory is not
|
|
considered to be a leak. We detect this by doing a liveness flood,
|
|
traversing pointers to heap objects starting from some initial memory
|
|
regions we know to potentially contain live pointer data. Note that
|
|
this flood might potentially not find some (global) live data region
|
|
to start the flood from. If you find such, please file a bug.</p>
|
|
|
|
<p>The liveness flood attempts to treat any properly aligned byte
|
|
sequences as pointers to heap objects and thinks that it found a good
|
|
pointer whenever the current heap memory map contains an object with
|
|
the address whose byte representation we found. Some pointers into
|
|
not-at-start of object will also work here.</p>
|
|
|
|
<p>As a result of this simple approach, it's possible (though
|
|
unlikely) for the flood to be inexact and occasionally result in
|
|
leaked objects being erroneously determined to be live. For instance,
|
|
random bit patterns can happen to look like pointers to leaked heap
|
|
objects. More likely, stale pointer data not corresponding to any
|
|
live program variables can be still present in memory regions,
|
|
especially in thread stacks. For instance, depending on how the local
|
|
<code>malloc</code> is implemented, it may reuse a heap object
|
|
address:</p>
|
|
<pre>
|
|
char* p = new char[1]; // new might return 0x80000000, say.
|
|
delete p;
|
|
new char[1]; // new might return 0x80000000 again
|
|
// This last new is a leak, but doesn't seem it: p looks like it points to it
|
|
</pre>
|
|
|
|
<p>In other words, imprecisions in the liveness flood mean that for
|
|
any heap leak check we might miss some memory leaks. This means that
|
|
for local leak checks, we might report a memory leak in the local
|
|
area, even though the leak actually happened before the
|
|
<code>HeapLeakChecker</code> object was constructed. Note that for
|
|
whole-program checks, a leak report <i>does</i> always correspond to a
|
|
real leak (since there's no "before" to have created a false-live
|
|
object).</p>
|
|
|
|
<p>While this liveness flood approach is not very portable and not
|
|
100% accurate, it works in most cases and saves us from writing a lot
|
|
of explicit clean up code and other hassles when dealing with thread
|
|
data.</p>
|
|
|
|
|
|
<h3>Visualizing Leak with <code>pprof</code></h3>
|
|
|
|
<p>
|
|
The heap checker automatically prints basic leak info with stack traces of
|
|
leaked objects' allocation sites, as well as a pprof command line that can be
|
|
used to visualize the call-graph involved in these allocations.
|
|
The latter can be much more useful for a human
|
|
to see where/why the leaks happened, especially if the leaks are numerous.
|
|
</p>
|
|
|
|
<h3>Leak-checking and Threads</h3>
|
|
|
|
<p>At the time of HeapLeakChecker's construction and during
|
|
<code>NoLeaks()</code> calls, we grab a lock
|
|
and then pause all other threads so other threads do not interfere
|
|
with recording or analyzing the state of the heap.</p>
|
|
|
|
<p>In general, leak checking works correctly in the presence of
|
|
threads. However, thread stack data liveness determination (via
|
|
<code>base/linuxthreads.h</code>) does not work when the program is
|
|
running under GDB, because the ptrace functionality needed for finding
|
|
threads is already hooked to by GDB. Conversely, leak checker's
|
|
ptrace attempts might also interfere with GDB. As a result, GDB can
|
|
result in potentially false leak reports. For this reason, the
|
|
heap-checker turns itself off when running under GDB.</p>
|
|
|
|
<p>Also, <code>thread_lister</code> only works for Linux pthreads;
|
|
leak checking is unlikely to handle other thread implementations
|
|
correctly.</p>
|
|
|
|
<p>As mentioned in the discussion of liveness flooding, thread-stack
|
|
liveness determination might mis-classify as reachable objects that
|
|
very recently became unreachable (leaked). This can happen when the
|
|
pointers to now-logically-unreachable objects are present in the
|
|
active thread stack frame. In other words, trivial code like the
|
|
following might not produce the expected leak checking outcome
|
|
depending on how the compiled code works with the stack:</p>
|
|
<pre>
|
|
int* foo = new int [20];
|
|
HeapLeakChecker check("a_check");
|
|
foo = NULL;
|
|
// May fail to trigger.
|
|
if (!heap_checker.NoLeaks()) assert(NULL == "heap memory leak");
|
|
</pre>
|
|
|
|
|
|
<hr>
|
|
<address>Maxim Lifantsev<br>
|
|
<!-- Created: Tue Dec 19 10:43:14 PST 2000 -->
|
|
<!-- hhmts start -->
|
|
Last modified: Fri Jul 13 13:14:33 PDT 2007
|
|
<!-- hhmts end -->
|
|
</address>
|
|
</body>
|
|
</html>
|