We had 2 nearly identical implementations. Thankfully C++ templates
facility lets us produce 2 different runtime functions (for different
type widths) without duplicating source.
Amend github issue #1414
It actually found real (but arguably minor) issue with memory region
map locking.
As part of that we're replacing PageHeap::DeleteAndUnlock that had
somewhat ambitious 'move' of SpinLockHolder, with more straightforward
PageHeap::PrepareAndDelete. Doesn't look like we can support move
thingy with thread annotations.
As noted on github issue #880 'temporarily' thing saves us not just on
freeing thread cache, but also returning thread's share of thread
cache (max_size_) into common pool. And the later has caused trouble
to mongo folk who originally proposed 'temporarily' thing. They claim
they don't use it anymore.
And thus with no users and no clear benefit, it makes no sense for us
to keep this API. For API and ABI compat sake we keep it, but it is
now identical to regular MarkThreadIdle.
Fixes issue #880
This is nearly impossible in practice, but still. Somehow we missed
this logic that DoSampledAllocation always returns actual object, but
in that condition where stacktrace_allocator failed to get us
StackTrace object we ended up returning span instead of it's object.
While there is still plenty of code that takes pageheap_lock outside
of page_heap module for all kinds of reasons, at least
bread-and-butter logic of allocating/deallocating larger chunks of
memory is now handling page heap locking inside PageHeap itself. This
gives us flexibility.
Update issue #1159
I.e. this covers case of arms that by default compile tcmalloc for 8k
logical pages (assuming 4k system pages), but can actually run on
systems with 64k pages.
Closes#1135
Original CL: https://chromiumcodereview.appspot.com/10391178
1. Enable large object pointer offset check in release build.
Following code will now cause a check error:
char* p = reinterpret_cast<char*>(malloc(kMaxSize + 1));
free(p + 1);
2. Remove a duplicated error reporting function "DieFromBadFreePointer",
can use "InvalidGetAllocatedSize".
Reviewed-on: https://chromium-review.googlesource.com/1184335
[alkondratenko@gmail.com] removed some unrelated formatting changes
Signed-off-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
We previously tested wrong assumption that larger than page size size
classes have addresses aligned on page size. New code is making proper
check of size class.
Also added is unit test coverage for this previously failing
condition. And we now also run "assert-ful" unittests for big tcmalloc
too, not only tcmalloc_minimal configuration.
This fixes github issue #1254
This patch adds visibility into the overhead due to fragmentation for each size
class in the tcmalloc central free list, which is helpful when debugging
fragmentation issues.
Original CL:
- https://codereview.chromium.org/1410353005
Add generic.total_physical_bytes property to MallocExtension
The actual physical memory usage of tcmalloc cannot be obtained by
GetNumericProperty. This accounts for the current_allocated_bytes,
fragmentation and malloc metadata, and excludes the unmapped memory
regions. This helps the user to understand how much memory is actually
being used for the allocations that were made.
Reviewed-on: https://chromium-review.googlesource.com/1130803
We aliased functions with different signatures and gcc now correctly
gives warning for that. Originally gcc 5 same code merging feature
caused us to alias more than necessary, but I am not able to reproduce
this problem anymore. So we're now aliasing only compatible functions.
At first I try to add some functions as what Chrome does at their
https://chromium.googlesource.com/chromium/src/+/master/base/allocator/allocator_shim_override_ucrt_symbols_win.h,
but it still fails. So I decide to remove all heap-related objects
from libucrt.lib to see what happens. At the end I find that a lot of
functions in the CRT directly invoke _malloc_base instead of
malloc (and the others alike), hence we need to override them as well.
This should close issue #716.
[alkondratenko@gmail.com: added reference to ticket]
Signed-off-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Recent commit to fix int overflow for implausibly huge allocation
added call to std::min. Notably, first arg was old size divided by
unsigned long 4. And on GNU/Linux i386 size_t is not long. So such
division was promoting first arg to unsigned long while second arg was
still size_t, so just unsigned. And that caused compilation to fail.
Fix is droping 'ul'.
One of recent commits started passing kMaxPages to printf but not used
it. Thankfully compilers gave us warning. Apparently intention was to
print real value of kMaxPages, so this is what we're doing now.
Previously, the central free list with index '0' was always unused,
since freelist index 'i' tracked spans of length 'i' and there are no
spans of length 0. This meant that there was no freelist for spans of
length 'kMaxPages'. In the default configuration, this corresponds to
1MB, which is a relatively common allocation size in a lot of
applications.
This changes the free list indexing so that index 'i' tracks spans of
length 'i + 1', meaning that free list index 0 is now used and
freelist[kMaxPages - 1] tracks allocations of kMaxPages size (1MB by
default).
This also fixes the stats output to indicate '>128' for the large spans
stats rather than the incorrect '>255' which must have referred to a
historical value of kMaxPages.
No new tests are added since this code is covered by existing tests.
We're taking advantage of "natural" alignedness of our size classes
and instead of previous loop over size classes looking for suitably
aligned size, we now directly compute right size. See align_size_up
function. And that gives us ability to use our existing malloc
fast-path to make memalign neat and fast in most common
cases. I.e. memalign/aligned_alloc now only tail calls and thus avoids
expensive prologue/epilogue and is almost as fast as regular malloc.
Because somehow clang still builds "this function will not throw" code
even with noexcept. Which breaks performance of
tc_malloc/tc_new_nothrow. The difference with throw() seems to be just
which function is called when unexpected exception happens.
So we work around this sillyness by simply dropping any exception
specification when compiling tcmalloc.
Previous fast-path malloc implementation failed to arrange proper oom
handling for operator new. I.e. operator new is supposed to call new
handler and throw exception, which was not arranged in fast-path case.
Fixed code now passes pointer for oom function to
ThreadCache::FetchFromCentralCache which will call it in oom
condition. Test is added to verify correct behavior.
I've also updated some fast-path-related comments for more accuracy.
Without aliasing performance is likely to be at least partially
affected. There is still concern that aliasing between functions of
different signatures is not 100% safe. We now explicitly list of
architectures where aliasing is known to be safe.
- Add auto-detection of std::align_val_t presence to configure scripts. This
indicates that the compiler supports C++17 operator new/delete overloads
for overaligned types.
- Add auto-detection of -faligned-new compiler option that appeared in gcc 7.
The option allows the compiler to generate calls to the new operators. It is
needed for tests.
- Added overrides for the new operators. The overrides are enabled if the
support for std::align_val_t has been detected. The implementation is mostly
based on the infrastructure used by memalign, which had to be extended to
support being used by C++ operators in addition to C functions. In particular,
the debug version of the library has to distinguish memory allocated by
memalign from that by operator new. The current implementation of sized
overaligned delete operators do not make use of the supplied size argument
except for the debug allocator because it is difficult to calculate the exact
allocation size that was used to allocate memory with alignment. This can be
done in the future.
- Removed forward declaration of std::nothrow_t. This was not portable as
the standard library is not required to provide nothrow_t directly in
namespace std (it could use e.g. an inline namespace within std). The <new>
header needs to be included for std::align_val_t anyway.
- Fixed operator delete[] implementation in libc_override_redefine.h.
- Moved TC_ALIAS definition to the beginning of the file in tcmalloc.cc so that
the macro is defined before its first use in nallocx.
- Added tests to verify the added operators.
[alkondratenko@gmail.com: fixed couple minor warnings, and some
whitespace change]
[alkondratenko@gmail.com: removed addition of TC_ALIAS in debug allocator]
Signed-off-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Apparently gcc only supports __attribute__((aligned(N))) on functions
only since version 4.3. So lets test it in configure script and only
use when possible. We now use CACHELINE_ALIGNED_FN macro for aligning
functions.
This was caught by unit tests on centos 5. Apparently some early
thingy is trying to do vprintf which calls free(0). Which used to
crash since before size class cache is initialized it'll report
hit (with size class 0) for NULL pointer, so we'd miss the case of
checking NULL pointer free and crash.
The fix is to check for IsInited in the case when thread cache is
null, and if so then we escalte to free_null_or_invalid.
With TCMALLOC_TRANSFER_NUM_OBJ environment variable we can change
transfer batch size. And with that comes slightly different number of
size classes depending on value of transfer batch size.
We used to have hardcoded number of size classes, so we couldn't
really support any batch size setting.
This commit adds support for dynamic number of size classes (runtime
value returned by Static::num_size_classes()).
This is significant speedup of fast-path of malloc. Large part comes
from avoiding expensive function prologue/epilogue. Which is achieved
by making sure that tc_{malloc,new,free} etc are small functions that
do only tail-calls. We keep only critical path in those functions and
tail-call to slower "full" versions when we need to deal with less
common case. This helps compiler generate much tidier code.
Fast-path readyness check is now different too. We used to have "min
size for slow path" variable, which was set to non-zero value when we
know that thread cache is present and ready. We now have use
thread-cache pointer not equal to NULL as readyness check.
There is special ThreadCache::threadlocal_data_.fast_path_heap copy of
that pointer that can be temporarily nulled to disable malloc fast
path. This is used to enable emergency malloc.
There is also slight change to tracking thread cache size. Instead of
tracking total size of free list, it now tracks size headroom. This
allows for slightly faster deallocation fast-path check where we're
checking headroom to stay above zero. This check is a bit faster than
comparing with max_size_.
Lower bits of page index are still used as index into hash
table. Those lower bits are zeroed, or-ed with size class and
placed into hash table. So checking is just loading value from hash
table, xoring with higher bits of address and checking if resultant
value is lower than 128. Notably, size class 0 is not considered
"invalid" anymore.
Currently on windows, we're depending on uninitialized tcmalloc
variables to detect freeing foreign malloc's chunks. This works
somewhat by chance due to 0-initialized size classes cache working as
cache with no values. But this is about to change, so lets do explicit
initialization.
Apparently throw() on functions actually asks compiler to generate code
to detect unexpected exceptions. Which prevents tail calls optimization.
So in order to re-enable this optimization, we simply don't tell
compiler about throw() at all. C++11 noexcept would be even better, but
it is not universally available yet.
So we change to no exception specifications. Which at least for gcc &
clang on Linux (and likely for all ELF platforms, if not just all)
really eliminates all overhead of exceptions.
