This is what other mallocs do (glibc malloc and jemalloc). The idea is
malloc is usually initialized very eary. So if we register atfork
handler at that time, we're likely to be first. And that makes our
atfork handler a bit safer, since there is much less chance of some
other library installing their "take all locks" handler first and
having fork take malloc lock before library's lock and deadlocking.
This should address issue #904.
Without this patch, any user program that enables LeakSanitizer will
see a leak from tcmalloc. Add a weak hook to __lsan_ignore_object,
so that if LeakSanitizer is enabled, the allocation can be ignored.
This reverts commit b82d89cb7c.
Dynamic sized delete support relies on ifunc handler being able to
look up environment variable. The issue is, when stuff is linked with
-z now linker flags, all relocations are performed early. And sadly
ifunc relocations are not treated specially. So when ifunc handler
runs, it cannot rely on any dynamic relocations at all, otherwise
crash is real possibility. So we cannot afford doing it until (and if)
ifunc is fixed.
This was brought to my attention by Fedora people at
https://bugzilla.redhat.com/show_bug.cgi?id=1452813
There is no need to have pointer indirection for root node. This also
helps the case of early free of garbage pointer because we didn't
check root_ pointer for NULL.
Apparently gcc only supports __attribute__((aligned(N))) on functions
only since version 4.3. So lets test it in configure script and only
use when possible. We now use CACHELINE_ALIGNED_FN macro for aligning
functions.
This was caught by unit tests on centos 5. Apparently some early
thingy is trying to do vprintf which calls free(0). Which used to
crash since before size class cache is initialized it'll report
hit (with size class 0) for NULL pointer, so we'd miss the case of
checking NULL pointer free and crash.
The fix is to check for IsInited in the case when thread cache is
null, and if so then we escalte to free_null_or_invalid.
101 is not very early anyways and arg-ful constructor attribute is
only supported since gcc 4.3 (and e.g. rhel 5's compiler fails to
compile it). So there seems to be very little value trying to ask for
priority of 101.
It was reported that SIZE_MAX isn't getting defined in C++ mode when
C++ standard is less than c++11. Because we still want to support
non-c++11 systems (for now), lets make it simple and not depend on
SIZE_MAX (original google-internal code used
std::numeric_limits<ssize_t>::max, but that failed to compile on
msvc).
Fixes issue #887 and issue #889.
I got report that some build environments for
https://github.com/lyft/envoy are having link-time issue due to
linking libunwind. It was happening despite libunwind.h being present,
which is clear bug as without header we won't really use libunwind.
Some tensorflow benchmarks are seeing large regression with elevated
values. So lets stick to old safe default until we understand how to make
larger values work for all workloads.
With TCMALLOC_TRANSFER_NUM_OBJ environment variable we can change
transfer batch size. And with that comes slightly different number of
size classes depending on value of transfer batch size.
We used to have hardcoded number of size classes, so we couldn't
really support any batch size setting.
This commit adds support for dynamic number of size classes (runtime
value returned by Static::num_size_classes()).
This is significant speedup of fast-path of malloc. Large part comes
from avoiding expensive function prologue/epilogue. Which is achieved
by making sure that tc_{malloc,new,free} etc are small functions that
do only tail-calls. We keep only critical path in those functions and
tail-call to slower "full" versions when we need to deal with less
common case. This helps compiler generate much tidier code.
Fast-path readyness check is now different too. We used to have "min
size for slow path" variable, which was set to non-zero value when we
know that thread cache is present and ready. We now have use
thread-cache pointer not equal to NULL as readyness check.
There is special ThreadCache::threadlocal_data_.fast_path_heap copy of
that pointer that can be temporarily nulled to disable malloc fast
path. This is used to enable emergency malloc.
There is also slight change to tracking thread cache size. Instead of
tracking total size of free list, it now tracks size headroom. This
allows for slightly faster deallocation fast-path check where we're
checking headroom to stay above zero. This check is a bit faster than
comparing with max_size_.
This is mostly dropping FastLog2 which was never necessary for
performance, and making sampler to be called always, even if sampling
is disabled (this benefits more for always-sampling case of Google
fork).
We're also getting TryRecordAllocationFast which is not used yet, but
will be as part of subsequent fast-path speedup commit.
Lower bits of page index are still used as index into hash
table. Those lower bits are zeroed, or-ed with size class and
placed into hash table. So checking is just loading value from hash
table, xoring with higher bits of address and checking if resultant
value is lower than 128. Notably, size class 0 is not considered
"invalid" anymore.
Currently on windows, we're depending on uninitialized tcmalloc
variables to detect freeing foreign malloc's chunks. This works
somewhat by chance due to 0-initialized size classes cache working as
cache with no values. But this is about to change, so lets do explicit
initialization.
48 bits is size of x86-64 and arm64 address spaces. So using 2 levels
map for them is slightly faster. We keep 3 levels for small-but-slow
configuration, since 2 levels consume a bit more memory.
This is partial port of Google-internal commit by Sanjay
Ghemawat (same idea, different implementation).
Apparently throw() on functions actually asks compiler to generate code
to detect unexpected exceptions. Which prevents tail calls optimization.
So in order to re-enable this optimization, we simply don't tell
compiler about throw() at all. C++11 noexcept would be even better, but
it is not universally available yet.
So we change to no exception specifications. Which at least for gcc &
clang on Linux (and likely for all ELF platforms, if not just all)
really eliminates all overhead of exceptions.
Subsequent optimization may cause multiple malloc functions in
google_malloc section to be in call stack. Particularly when fast-path
malloc function calls slow-path and compiler chooses to implement such
call as regular call instead of tail-call.
Because we need stacktrace just until first such function, once we find
innermost such frame, we're simply checking if next outer frame is also
google_malloc and consider it instead.
Particularly, hardcoded skip count was relying on certain behavior of
compiler. Namely, that tail calls inside DebugDeallocate path are not
actually implemented as tail calls.
New implementation is using google_malloc section as a marker of malloc
boundary. But in order for this to work, we have to prevent tail-call in
debugallocation's tc_XXX functions. Which is achieved by doing volatile
read of static variable at the end of such functions.
