This reverts commit be3da70298.
There are reports of crashes and false-positive OOMs from this
patch. Crashes under aggressive decommit mode are understood, but I
have yet to get confirmations whether false-positive OOMs were seen
under aggressive decommit or not. Thus lets revert for now.
Updates issue #1227 and issue #1204.
"mmap" hooks code for linux only hooks sbrk when __sbrk is
defined. But musl doesn't offer this function (reasonably so), so
sbrk hooking doesn't work there. Not big loss, but lets make sure
tests don't fail.
Lets reference this to issue #1198
Clang mostly ignores those anyways, so our tests needed better way to
disable optimizations (clang is quite aggressive replacing new/delete
pair with stack allocation).
The test relies on stuff "mmaper" function not being inlined, but
compilers are getting smarter and latest clang actually does inline it
now. Adding volatile to pointer declaration fixes it.
After change to release page heap lock around returning memory back to
kernel, page heap test got dependency on page heap lock. Which was not
available on windows since relevant symbols are not exported.
Proposed fix is to simply duplicate all needed .cc files in
page_heap_test project instead of linking to dll. This is not perfect
but gets job done, until we figure out better solution (GNU/Linux will
eventually get hidden visibility and will need it).
This fixes github issue 1189.
Whenever tcmalloc memfs allocator runs out of huge memory, it falls back to the
default system allocator, which will start allocating from normal anonymous
memory pool (small pages). For use cases that exclusively want hugepage backed
memory, such as using tcmalloc in combination with Intel SPDK, this is
problematic.
This change adds an environment variable "TCMALLOC_MEMFS_DISABLE_FALLBACK"
which when set to true will ensure that tcmalloc doesn't fallback to using the
system allocator when the system runs out of hugepages. This is false by
default, such that all other tcmalloc consumers will not see any changes.
For those curious, you can see part of our upstream SPDK work here:
https://review.gerrithub.io/c/spdk/spdk/+/475943
This patch adds visibility into the overhead due to fragmentation for each size
class in the tcmalloc central free list, which is helpful when debugging
fragmentation issues.
This fixes the following warning:
src/thread_cache.cc:281:10: warning: static member accessed through instance [readability-static-accessed-through-instance]
return sampler_.GetSamplePeriod();
^~~~~~~~~~~~~~~~~~~~~~~~
tcmalloc::Sampler::
This avoids a deadlock when a library which is being dlopen'ed creates
as part of its static constructors a thread which quickly need to call
malloc. We are still in the dlopen call (so with some internal glibc
mutex taken) when the thread executes code and later needs to call
malloc which in term calls tls_get_addr_tail, which wait for the dlopen
mutex to be unlocked. If later the dlopen'ing thread also calls malloc
as part of its constructors, we are in a deadlock.
Fix is similar to
7852eeb75b
Stack of the dlopening thread:
#0 0x00007fd5406ca93c in __lll_lock_wait () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
#1 0x00007fd5406c45a5 in pthread_mutex_lock () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
... proprietary code in the stack
#9 0x00007fd5074f0367 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at src/ClientImpl.cpp:15
#10 0x00007fd5074f06d7 in _GLOBAL__sub_I_ClientImpl.cpp(void) () at src/ClientImpl.cpp:85
#11 0x00007fd50757aa46 in __do_global_ctors_aux ()
#12 0x00007fd5073e985f in _init () from ...
#13 0x00007fd53bf9dec8 in ?? () from ...
#14 0x00007fd54d637a5d in call_init.part () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
#15 0x00007fd54d637bab in _dl_init () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
#16 0x00007fd54d63c160 in dl_open_worker () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
#17 0x00007fd54d637944 in _dl_catch_error () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
#18 0x00007fd54d63b7d9 in _dl_open () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
#19 0x00007fd54d61f2b9 in dlopen_doit () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2
#20 0x00007fd54d637944 in _dl_catch_error () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
#21 0x00007fd54d61f889 in _dlerror_run () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2
#22 0x00007fd54d61f351 in dlopen@@GLIBC_2.2.5 () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2
Stack of the newly created thread calling tls_get_addr_tail:
#0 0x00007fd5406ca93c in __lll_lock_wait () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
#1 0x00007fd5406c4622 in pthread_mutex_lock () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
#2 0x00007fd54d63a2ed in tls_get_addr_tail () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
#3 0x00007fd53fee877d in tcmalloc::ThreadCache::CreateCacheIfNecessary () at src/thread_cache.cc:344
#4 0x00007fd53fecb4ab in tcmalloc::ThreadCache::GetCache () at src/thread_cache.h:437
#5 0x00007fd53fefeccb in (anonymous namespace)::do_malloc (size=56) at src/tcmalloc.cc:1354
#6 tcmalloc::do_allocate_full<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1762
#7 tcmalloc::allocate_full_cpp_throw_oom (size=56) at src/tcmalloc.cc:1776
#8 0x00007fd53ff01b80 in tcmalloc::dispatch_allocate_full<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1785
#9 malloc_fast_path<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1845
#10 tc_new (size=56) at src/tcmalloc.cc:1980
... proprietary code in the stack
#26 0x00007fd5406c1ef4 in start_thread () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
#27 0x00007fd5403ba01d in clone () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libc.so.6
In some architectures (e.g. powerpc, powerpc64 and powerpc64le), the
macro that specifies the position of the program counter is defined in
header asm/ptrace.h.
When glibc bug 21457 got fixed in version 2.27, header sys/ucontext.h
stopped including signal.h, which indirectly removed asm/ptrace.h too.
Signed-off-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
mmap64 can already been defined and as a result the following error is
raised:
In file included from src/malloc_hook.cc:686:0:
src/malloc_hook_mmap_linux.h: In function ‘void* mmap(void*, size_t, int, int, int, off_t)’:
src/malloc_hook_mmap_linux.h:173:18: error: redefinition of ‘void* mmap(void*, size_t, int, int, int, off_t)’
extern "C" void* mmap(void *start, size_t length, int prot, int flags,
^~~~
In file included from src/malloc_hook.cc:41:0:
src/malloc_hook_mmap_linux.h:159:18: note: ‘void* mmap(void*, size_t, int, int, int, off_t)’ previously defined here
extern "C" void* mmap64(void *start, size_t length, int prot, int flags,
^
Makefile:4874: recipe for target 'src/libtcmalloc_minimal_internal_la-malloc_hook.lo' failed
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
__sbrk is not defined in musl so add a check for it in configure.ac and
replace the check on ifndef __UCLIBC__ by a check on ifdef HAVE__SBRK
in src/malloc_hook_mmap_linux.h
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
__off64_t is a kernel internal symbol, which happens to be user-visible
with glibc, but not necessarily with other C libraries such as musl.
Switching from __off64_t to off64_t allows the code to build properly
with musl.
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Original CL:
- https://codereview.chromium.org/1410353005
Add generic.total_physical_bytes property to MallocExtension
The actual physical memory usage of tcmalloc cannot be obtained by
GetNumericProperty. This accounts for the current_allocated_bytes,
fragmentation and malloc metadata, and excludes the unmapped memory
regions. This helps the user to understand how much memory is actually
being used for the allocations that were made.
Reviewed-on: https://chromium-review.googlesource.com/1130803
Everyone should be using golang pprof from github.com/google/pprof, but
distros still ship our perl version and not everyone is aware of
better pprof yet.
This is another step in completely dropping perl pprof. We still
default to installing it, but hopefully we'll be able to convince
distros to disable this soon.
We still install pprof under pprof-symbolize name because
stack traces symbolization depends on it, and because golang pprof
won't support this feature.
This is related to issue #1038.
