Previously, the central free list with index '0' was always unused,
since freelist index 'i' tracked spans of length 'i' and there are no
spans of length 0. This meant that there was no freelist for spans of
length 'kMaxPages'. In the default configuration, this corresponds to
1MB, which is a relatively common allocation size in a lot of
applications.
This changes the free list indexing so that index 'i' tracks spans of
length 'i + 1', meaning that free list index 0 is now used and
freelist[kMaxPages - 1] tracks allocations of kMaxPages size (1MB by
default).
This also fixes the stats output to indicate '>128' for the large spans
stats rather than the incorrect '>255' which must have referred to a
historical value of kMaxPages.
No new tests are added since this code is covered by existing tests.
As reported in issue #954, osx clang compiler is able to optimize our
previous detection away while not really having runtime support for
sized delete. So this time we use AC_LINK_IFELSE and more robust code
to prevent compiler from optimizing away sized delete call. This
should reliably catch "bad" compilers.
Special thanks to Alexey Serbin for reporting the issue, suggesting a
fix and verifying it.
Fixes issue #954.
This is implemented via std::set with custom STL allocator that
delegates to PageHeapAllocator. Free large spans are not linked
together via linked list, but inserted into std::set. Spans also store
iterators to std::set positions pointing to them. So that removing
span from set is fast too.
Patch implemented by Aliaksey Kandratsenka and Todd Lipcon based on
earlier research and experimentation by James Golick.
Addresses issue #535
[alkondratenko@gmail.com: added Todd's fix for building on OSX]
[alkondratenko@gmail.com: removed unnecessary Span constructor]
[alkondratenko@gmail.com: added const for SpanSet comparator]
[alkondratenko@gmail.com: added operator != for STLPageHeapAllocator]
malloc_fast_path now receives oom function instead of full allocation
function and windows/patch_function.cc wasn't updated until now. It
caused assertion failures as reported in issue #944.
We're taking advantage of "natural" alignedness of our size classes
and instead of previous loop over size classes looking for suitably
aligned size, we now directly compute right size. See align_size_up
function. And that gives us ability to use our existing malloc
fast-path to make memalign neat and fast in most common
cases. I.e. memalign/aligned_alloc now only tail calls and thus avoids
expensive prologue/epilogue and is almost as fast as regular malloc.
Because somehow clang still builds "this function will not throw" code
even with noexcept. Which breaks performance of
tc_malloc/tc_new_nothrow. The difference with throw() seems to be just
which function is called when unexpected exception happens.
So we work around this sillyness by simply dropping any exception
specification when compiling tcmalloc.
We now clearly separate PERFTOOLS_NOTHROW (used for tc_XXX functions)
and throw()/noexcept (used for operators we define).
The former is basically "nothrow() for our callers, nothing for
us". It is roughly equivalent of what glibc declares for malloc and
friends. If some exception-full C++ code calls such function it
doesn't have to bother setting up exception handling around such
call. Notably, it is still important for those functions to _not have
throw() declarations when we're building tcmalloc. Because C++ throw()
requires setting up handling of unexpected exceptions thrown from
under such functions which we don't want.
The later is necessary to have operators new/delete definitions have
"correct" exception specifications to calm down compiler
warnings. Particularly older clang versions warn if new/delete aren't
defined with correct exception specifications. Also this commit fixes
annoying gcc 7+ warning (and gnu++14 mode) that complains about
throw() being deprecated.
Previous fast-path malloc implementation failed to arrange proper oom
handling for operator new. I.e. operator new is supposed to call new
handler and throw exception, which was not arranged in fast-path case.
Fixed code now passes pointer for oom function to
ThreadCache::FetchFromCentralCache which will call it in oom
condition. Test is added to verify correct behavior.
I've also updated some fast-path-related comments for more accuracy.
Without aliasing performance is likely to be at least partially
affected. There is still concern that aliasing between functions of
different signatures is not 100% safe. We now explicitly list of
architectures where aliasing is known to be safe.
- Add auto-detection of std::align_val_t presence to configure scripts. This
indicates that the compiler supports C++17 operator new/delete overloads
for overaligned types.
- Add auto-detection of -faligned-new compiler option that appeared in gcc 7.
The option allows the compiler to generate calls to the new operators. It is
needed for tests.
- Added overrides for the new operators. The overrides are enabled if the
support for std::align_val_t has been detected. The implementation is mostly
based on the infrastructure used by memalign, which had to be extended to
support being used by C++ operators in addition to C functions. In particular,
the debug version of the library has to distinguish memory allocated by
memalign from that by operator new. The current implementation of sized
overaligned delete operators do not make use of the supplied size argument
except for the debug allocator because it is difficult to calculate the exact
allocation size that was used to allocate memory with alignment. This can be
done in the future.
- Removed forward declaration of std::nothrow_t. This was not portable as
the standard library is not required to provide nothrow_t directly in
namespace std (it could use e.g. an inline namespace within std). The <new>
header needs to be included for std::align_val_t anyway.
- Fixed operator delete[] implementation in libc_override_redefine.h.
- Moved TC_ALIAS definition to the beginning of the file in tcmalloc.cc so that
the macro is defined before its first use in nallocx.
- Added tests to verify the added operators.
[alkondratenko@gmail.com: fixed couple minor warnings, and some
whitespace change]
[alkondratenko@gmail.com: removed addition of TC_ALIAS in debug allocator]
Signed-off-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
This commit is to fix the data race in ThreadCache::SetMaxSize.
ThreadCache::size_left_ is removed and ThreadCache::size_ is
added. ThreadCache::size_left_ was introduced for optimization.
It is updated in several functions of ThreadCache, including the
ThreadCache::SetMaxSize. But thread A can update size_left_ of
thread B via SetMaxSize without protection or synchronization.
There should not be data race around ThreadCache::size_, for it
isn't accessed by multi threads.
The optimization of tail-call in tc_{malloc, new, free} is kept
and no other logics are affected.
Normally the va_end function does not do anything,
but it should be called because some platforms need it.
[alkondratenko@gmail.com: reworded commit message]
Signed-off-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Few lines of code was taken from
/usr/src/contrib/libexecinfo/backtrace.c
[alkondratenko@gmail.com: updated commit message
Signed-off-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
This is what other mallocs do (glibc malloc and jemalloc). The idea is
malloc is usually initialized very eary. So if we register atfork
handler at that time, we're likely to be first. And that makes our
atfork handler a bit safer, since there is much less chance of some
other library installing their "take all locks" handler first and
having fork take malloc lock before library's lock and deadlocking.
This should address issue #904.
Without this patch, any user program that enables LeakSanitizer will
see a leak from tcmalloc. Add a weak hook to __lsan_ignore_object,
so that if LeakSanitizer is enabled, the allocation can be ignored.
This reverts commit b82d89cb7c8781a6028f6f5959cabdc5a273aec3.
Dynamic sized delete support relies on ifunc handler being able to
look up environment variable. The issue is, when stuff is linked with
-z now linker flags, all relocations are performed early. And sadly
ifunc relocations are not treated specially. So when ifunc handler
runs, it cannot rely on any dynamic relocations at all, otherwise
crash is real possibility. So we cannot afford doing it until (and if)
ifunc is fixed.
This was brought to my attention by Fedora people at
https://bugzilla.redhat.com/show_bug.cgi?id=1452813
