code introduced in commit 9d44b6460a
wrongly attempted to read past the end of the currently-installed dtv
to determine if a dso provides new, not-already-installed tls. this
logic was probably leftover from an earlier draft of the code that
wrongly installed the new dtv before populating it.
it would work if we instead queried the new, not-yet-installed dtv,
but instead, replace the incorrect check with a simple range check
against old_cnt. this also catches modules that have no tls at all
with a single condition.
code introduced in commit 9d44b6460a
wrongly assumed the dso list tail was the right place to find new dtv
storage. however, this is only true if the last-loaded dependency has
tls. the correct place to get it is the dso corresponding to the tls
module list tail. introduce a container_of macro to get it, and use
it.
ultimately, dynamic tls allocation should be refactored so that this
is not an issue. there is no reason to be allocating new dtv space at
each load_library; instead it could happen after all new libraries
have been loaded but before they are committed. such changes may be
made later, but this commit fixes the present regression.
the motivation for this change is twofold. first, it gets the fallback
logic out of the dynamic linker, improving code readability and
organization. second, it provides application code that wants to use
the membarrier syscall, which depends on preregistration of intent
before the process becomes multithreaded unless unbounded latency is
acceptable, with a symbol that, when linked, ensures that this
registration happens.
commit 9d44b6460a inadvertently
contained leftover logic from a previous approach to the fallback
signaling loop. it had no adverse effect, since j was always nonzero
if the loop body was reachable, but it makes no sense to be there with
the current approach to avoid signaling self.
previously, dynamic loading of new libraries with thread-local storage
allocated the storage needed for all existing threads at load-time,
precluding late failure that can't be handled, but left installation
in existing threads to take place lazily on first access. this imposed
an additional memory access and branch on every dynamic tls access,
and imposed a requirement, which was not actually met, that the
dynamic tlsdesc asm functions preserve all call-clobbered registers
before calling C code to to install new dynamic tls on first access.
the x86[_64] versions of this code wrongly omitted saving and
restoring of fpu/vector registers, assuming the compiler would not
generate anything using them in the called C code. the arm and aarch64
versions saved known existing registers, but failed to be future-proof
against expansion of the register file.
now that we track live threads in a list, it's possible to install the
new dynamic tls for each thread at dlopen time. for the most part,
synchronization is not needed, because if a thread has not
synchronized with completion of the dlopen, there is no way it can
meaningfully request access to a slot past the end of the old dtv,
which remains valid for accessing slots which already existed.
however, it is necessary to ensure that, if a thread sees its new dtv
pointer, it sees correct pointers in each of the slots that existed
prior to the dlopen. my understanding is that, on most real-world
coherency architectures including all the ones we presently support, a
built-in consume order guarantees this; however, don't rely on that.
instead, the SYS_membarrier syscall is used to ensure that all threads
see the stores to the slots of their new dtv prior to the installation
of the new dtv. if it is not supported, the same is implemented in
userspace via signals, using the same mechanism as __synccall.
the __tls_get_addr function, variants, and dynamic tlsdesc asm
functions are all updated to remove the fallback paths for claiming
new dynamic tls, and are now all branch-free.
commit a603a75a72 removed attribute
const from __errno_location and pthread_self, and the same reasoning
forced arch definitions of __pthread_self to use volatile asm,
significantly impacting code generation and imposing manual caching of
pointers where the impact might be noticable.
reorder the thread pointer setup and place it across a strong barrier
(symbolic function lookup) so that there is no assumed ordering
between the initialization and the accesses to the thread pointer in
stage 3.
commit 6ba5517a46 modified
__tls_get_addr to offset the address by +DTP_OFFSET (0x8000 on
powerpc, mips, etc.) and adjusted the result of DTPREL relocations by
-DTP_OFFSET to compensate, but missed changing the argument setup for
calls to __tls_get_addr from dlsym.
as explained in commit 6ba5517a46, some
archs use an offset (typicaly -0x8000) with their DTPOFF relocations,
which __tls_get_addr needs to invert. on affected archs, which lack
direct support for large immediates, this can cost multiple extra
instructions in the hot path. instead, incorporate the DTP_OFFSET into
the DTV entries. this means they are no longer valid pointers, so
store them as an array of uintptr_t rather than void *; this also
makes it easier to access slot 0 as a valid slot count.
commit e75b16cf93 left behind cruft in
two places, __reset_tls and __tls_get_new, from back when it was
possible to have uninitialized gap slots indicated by a null pointer
in the DTV. since the concept of null pointer is no longer meaningful
with an offset applied, remove this cruft.
presently there are no archs with both TLSDESC and nonzero DTP_OFFSET,
but the dynamic TLSDESC relocation code is also updated to apply an
inverted offset to its offset field, so that the offset DTV would not
impose a runtime cost in TLSDESC resolver functions.
unlike other asm where the baseline ISA is used, these functions are
hot paths and use ISA-level specializations.
call-clobbered vfp registers are saved before calling __tls_get_new,
since there is no guarantee it won't use them. while setjmp/longjmp
have to use hwcap to decide whether to the fpu is in use, since
application code could be using vfp registers even if libc was
compiled as pure softfloat, __tls_get_new is part of libc and can be
assumed not to have access to vfp registers if tlsdesc.S does not.
thus it suffices just to check the predefined preprocessor macros. the
check for __ARM_PCS_VFP is redundant; !__SOFTFP__ must always be true
if the target ISA level includes fpu instructions/registers.
this facilitates building software that assumes a large default stack
size without any patching to call pthread_setattr_default_np or
pthread_attr_setstacksize at each thread creation site, using just
LDFLAGS.
normally the PT_GNU_STACK header is used only to reflect whether
executable stack is desired, but with GNU ld at least, passing
-Wl,-z,stack-size=N will set a size on the program header. with this
patch, that size will be incorporated into the default stack size
(subject to increase-only rule and DEFAULT_STACK_MAX limit).
both static and dynamic linking honor the program header. for dynamic
linking, all libraries loaded at program start, including preloaded
ones, are considered. dlopened libraries are not considered, for
several reasons. extra logic would be needed to defer processing until
the load of the new library is commited, synchronization woud be
needed since other threads may be running concurrently, and the
effectiveness woud be limited since the larger size would not apply to
threads that already existed at the time of dlopen. programs that will
dlopen code expecting a large stack need to declare the requirement
themselves, or pthread_setattr_default_np can be used.
commits leading up to this one have moved the vast majority of
libc-internal interface declarations to appropriate internal headers,
allowing them to be type-checked and setting the stage to limit their
visibility. the ones that have not yet been moved are mostly
namespace-protected aliases for standard/public interfaces, which
exist to facilitate implementing plain C functions in terms of POSIX
functionality, or C or POSIX functionality in terms of extensions that
are not standardized. some don't quite fit this description, but are
"internally public" interfacs between subsystems of libc.
rather than create a number of newly-named headers to declare these
functions, and having to add explicit include directives for them to
every source file where they're needed, I have introduced a method of
wrapping the corresponding public headers.
parallel to the public headers in $(srcdir)/include, we now have
wrappers in $(srcdir)/src/include that come earlier in the include
path order. they include the public header they're wrapping, then add
declarations for namespace-protected versions of the same interfaces
and any "internally public" interfaces for the subsystem they
correspond to.
along these lines, the wrapper for features.h is now responsible for
the definition of the hidden, weak, and weak_alias macros. this means
source files will no longer need to include any special headers to
access these features.
over time, it is my expectation that the scope of what is "internally
public" will expand, reducing the number of source files which need to
include *_impl.h and related headers down to those which are actually
implementing the corresponding subsystems, not just using them.
eliminate gratuitous glue function for reporting the version, which
was probably leftover from the old dynamic linker design which lacked
a clear barrier for when/how it could access global data. put the
declaration for the data object that replaces it in libc.h where it
can be type checked.
this cleans up what had become widespread direct inline use of "GNU C"
style attributes directly in the source, and lowers the barrier to
increased use of hidden visibility, which will be useful to recovering
some of the efficiency lost when the protected visibility hack was
dropped in commit dc2f368e56, especially
on archs where the PLT ABI is costly.
previously, this operation succeeded, and the relocation results
worked for access from new threads created after dlopen, but produced
invalid accesses (and possibly clobbered other memory) from threads
that already existed.
the way the check is written, it still permits dlopen of libraries
containing initial-exec references to static TLS (TLS in the main
program or in a dynamic library loaded at startup).
tls_id is one-based, whereas [static_]tls_cnt is a count, so
comparison for checking that a given tls_id is dynamic rather than
static needs to use strict inequality.
since slack space at the beginning and/or end of writable load maps is
donated to malloc, the application could obtain valid pointers in
these ranges which dladdr would erroneously identify as part of the
shared object whose mapping they came from.
instead of checking the queried address against the mapping base and
length, check it against the load segments from the program headers,
and only match the dso if it lies within the bounds of one of them.
as a shortcut, if the address does match the range of the mapping but
not any of the load segments, we know it cannot match any other dso
and can immediately return failure.
the early-exit condition for the symbol match loop on exact matches
caused dladdr to produce the first match for an exact match, but the
last match for an inexact match. in the interest of consistency,
require a strictly-closer match to replace an already-found one.
commit 8b8fb7f037 added logic to prevent
matching a symbol with no recorded size (closest-match) when there is
an intervening symbol whose size was recorded, but it only worked when
the intervening symbol was encountered later in the search.
instead of rejecting symbols where addr falls outside their recorded
size during the closest-match search, accept them to find the true
closest-match, then reject such a result only once the search has
finished.
based on patch by Axel Siebenborn, with fixes discussed on the mailing
list after submission and and rebased around the UB fix in commit
e829695fcc.
avoid spurious symbol matches by dladdr beyond symbol size. for
symbols with a size recorded, only match if the queried address lies
within the address range determined by the symbol address and size.
for symbols with no size recorded, the old closest-match behavior is
kept, as long as there is no intervening symbol with a recorded size.
the case where no symbol is matched, but the address does lie within
the memory range of a shared object, is specified as success. fix the
return value and produce a valid (with null dli_sname and dli_saddr)
Dl_info structure.
writable load segments can have size-in-memory larger than their size
in the ELF file, representing bss or equivalent. the initial partial
page has to be zero-filled, and additional anonymous pages have to be
mapped such that accesses don't failt with SIGBUS.
map_library skips redundant MAP_FIXED mapping of the initial
(lowest-address) segment when processing LOAD segments since it was
already mapped when reserving the virtual address range, but in doing
so, inadvertently also skipped the code to fill/map bss. typical
executable and library files have two or more LOAD segments, and the
first one is text/rodata (non-writable) and thus has no bss, but it is
syntactically valid for an ELF program/library to put its writable
segment first, or to have only one segment (everything writable). the
binutils bfd-based linker has been observed to create such programs in
the presence of unusual sections or linker scripts.
fix by moving only the mmap_fixed operation under the conditional
rather than skipping the remainder of the loop body. add a check to
avoid bss processing in the case where the segment is not writable;
this should not happen, but if it does, the change would be a crashing
regression without this check.
In TLS variant I the TLS is above TP (or above a fixed offset from TP)
but on some targets there is a reserved gap above TP before TLS starts.
This matters for the local-exec tls access model when the offsets of
TLS variables from the TP are hard coded by the linker into the
executable, so the libc must compute these offsets the same way as the
linker. The tls offset of the main module has to be
alignup(GAP_ABOVE_TP, main_tls_align).
If there is no TLS in the main module then the gap can be ignored
since musl does not use it and the tls access models of shared
libraries are not affected.
The previous setup only worked if (tls_align & -GAP_ABOVE_TP) == 0
(i.e. TLS did not require large alignment) because the gap was
treated as a fixed offset from TP. Now the TP points at the end
of the pthread struct (which is aligned) and there is a gap above
it (which may also need alignment).
The fix required changing TP_ADJ and __pthread_self on affected
targets (aarch64, arm and sh) and in the tlsdesc asm the offset to
access the dtv changed too.
commit 618b18c78e removed the previous
detection and hardening since it was incorrect. commit
72141795d4 already handled all that
remained for hardening the static-linked case. in the dynamic-linked
case, have the dynamic linker check whether malloc was replaced and
make that information available.
with these changes, the properties documented in commit
c9f415d7ea are restored: if calloc is
not provided, it will behave as malloc+memset, and any of the
memalign-family functions not provided will fail with ENOMEM.
the existing laddr function for fdpic cannot translate ELF virtual
addresses outside of the LOAD segments to runtime addresses because
the fdpic loadmap only covers the logically-mapped part. however the
whole point of reclaim_gaps is to recover the slack space up to the
page boundaries, so it needs to work with such addresses.
add a new laddr_pg function that accepts any address in the page range
for the LOAD segment by expanding the loadmap records out to page
boundaries. only use the new version for reclaim_gaps, so as not to
impact performance of other address lookups.
also, only use laddr_pg for the start address of a gap; the end
address lies one byte beyond the end, potentially in a different page
where it would get mapped differently. instead of mapping end, apply
the length (end-start) to the mapped value of start.
in theory non-absolute origins can only arise when either the main
program is invoked by running ldso as a command (inherently non-suid)
or when dlopen was called with a relative pathname containing at least
one slash. such usage would be inherently insecure in an suid program
anyway, so the old behavior here does not seem to have been insecure.
harden against it anyway.
the rpath fixup code assumed any module's name field would contain at
least one slash, an invariant which is usually met but not in the case
of a main executable loaded from the current working directory by
running ldd or ldso as a command. it would be possible to make this
invariant always hold, but it has a higher runtime allocation cost and
does not seem useful elsewhere, so just patch things up in fixup_rpath
instead.
the Linux and FreeBSD man pages for dladdr document dli_fbase as the
"base address" of the library/module found. normally (e.g. AT_BASE)
the term "base" is used to denote the base address relative to which
p_vaddr addresses are interpreted; however in the case of dladdr's
Dl_info structure, existing implementations define it as the lowest
address of the mapping, which makes sense in the context of
determining which module's memory range the input address falls
within.
since this is a nonstandard interface provided to mimic one provided
by other implementations, adjust it to match their behavior.
commit c49d3c8ada added logic to detect
attempts to load libc.so via another name and instead redirect to the
existing libc, rather than loading two and producing dangerously
inconsistent state. however, the check for and unmapping of the
duplicate libc happened after reclaim_gaps was already called,
donating the slack space around the writable segment to malloc.
subsequent unmapping of the library then invalidated malloc's free
lists.
fix the issue by moving the call to reclaim_gaps out of map_library
into load_library, after the duplicate libc check but before the first
call to calloc, so that the gaps can still be used to satisfy the
allocation of struct dso. this change also eliminates the need for an
ugly hack (temporarily setting runtime=1) to avoid reclaim_gaps when
loading the main program via map_library, which happens when ldso is
invoked as a command.
only programs/libraries erroneously containing a DT_NEEDED reference
to libc.so via an absolute pathname or symlink were affected by this
issue.
previously, the pathname used to load the program was always used as
argv[0]. the default remains the same, but a new --argv0 option can be
used to provide a different value.
a null pointer for a library's deps list was ambiguous: it could
indicate either no dependencies or that the dependency list had not
yet been populated. inability to distinguish could lead to spurious
work when dlopen is called multiple times on a library with no deps,
and due to related bugs, could actually cause other libraries to
falsely appear as dependencies, translating into false positives for
dlsym.
avoid the problem by always initializing the deps pointer, pointing to
an empty list if there are no deps. rather than wasting memory and
introducing another failure path by allocating an empty list per
library, simply share a global dummy list.
further fixes will be needed for related bugs, and much of this code
may end up being replaced.
commit 4ff234f6cb erroneously changed
the condition for running certain code at dlopen time to check whether
the library was already relocated rather than whether it already had
its deps[] table filled. this was out of concern over whether the code
under the conditional would be idempotent/safe to call on an
already-loaded libraries. however, I missed a consideration in the
opposite direction: if a library was loaded at program startup rather
than dlopen, its deps[] table was not yet allocated/filled, and
load_deps needs to be called at dlopen time in order for dlsym to be
able to perform dependency-order symbol lookups.
in order to avoid wasteful allocation of lazy-binding relocation
tables for libraries which were already loaded and relocated at
startup, the check for !p->relocated is not deleted entirely, but
moved to apply only to allocation of these dables.
this change was suggested based on testing done by Timo Teräs almost
two years ago; the branch (and probably call prep overhead) in the
inner loop was found to contribute noticably to total symbol lookup
time. this change will make lookup slightly slower if libraries were
built with only the traditional "sysv" ELF hash table, but based on
how much slower lookup tends to be without the gnu hash table, it
seems reasonable to assume that (1) users building without gnu hash
don't care about dynamic linking performance, and (2) the extra time
spent computing the gnu hash is likely to be dominated by the slowness
of the sysv hash table lookup anyway.
such loading is unsafe, and can happen when programs use their own
logic to locate a .so file then pass the absolute pathname to dlopen,
or if an absolute pathname ends up in DT_NEEDED headers. multiple
loads with only the base name were already precluded, provided libc
was named appropriately, by special-casing standard library names.
one function symbol (in the reserved namespace, but public, since it's
part of the crt1 entry point ABI) and one data symbol are checked.
this way we avoid likely false positives, particularly from libraries
interposing and wrapping functions. there is no hard requirement to
avoid breaking such usage, since trying to run a hook before libc is
even initialized is not a supported usage case, but it's friendlier
not to break things.
traditional lazy relocation with call-time plt resolver is
intentionally not implemented, as it is a huge bug surface and demands
significant amounts of arch-specific code and requires ongoing
maintenance to ensure compatibility with applications which make use
of new additions to the arch's register file in passing function
arguments.
some applications, however, depend on the ability to dlopen modules
which have unsatisfied symbol references at the time they are loaded,
either avoiding use of the affected interfaces or manually loading
another module to provide the missing definition via their own module
dependency tracking outside the ELF data structures. while such usage
is non-conforming, failure to support it has been a significant
obstacle for users/distributions trying to support affected software,
particularly the X.org server.
instead of resolving lazy relocations at call time, this patch saves
unresolved GOT/PLT relocations for deferral and retries them after
each subsequent dlopen until they are resolved. since dlopen is the
only time at which the effective global symbol table can change, this
behavior is not observably different from traditional lazy binding,
and the required code is minimal.
when loading libraries with dlopen, the caller can request that the
library's symbols become part of the global symbol table, or that they
only be used for resolving relocations in the loaded library and its
dependencies. in the latter case, a subsequent dlopen of the same
library can upgrade it to global status.
previously, if a library was upgraded from local to global mode, its
symbols entered the symbol lookup search order at the point where the
library was originally loaded. this means that a new call to dlopen
could change the value of a symbol that already had a visible
definition, an inconsistency which applications could observe.
POSIX is unclear whether this should happen or whether it's permitted
to happen, but the resolution of Austin Group issue #982 made it
formally unspecified.
with this patch, a library whose mode is upgraded from local to global
enters the symbol lookup order at the point where it was made global,
so that symbol resolution before and after the upgrade are consistent.
in order to implement this change, the per-dso global flag is replaced
with a separate set of linked-list pointers for participation in the
global symbol table. this permits the order of dso objects for symbol
resolution to differ from the order used for iteration of all loaded
libraries. it also improves performance of find_sym, by avoiding a
branch per iteration and skipping, and especially in the case where
many non-global libraries have been loaded, by allowing the loop to
skip over them entirely. logic for temporarily adding non-global
libraries to the symbol table for relocation purposes is also mildly
simplified.
A weak symbol definition is not special during dynamic linking, so
don't let a strong definition in a later module override it.
(glibc dynamic linker allows overriding weak definitions if
LD_DYNAMIC_WEAK is set, musl does not.)
STB_GNU_UNIQUE means that the symbol is global, even if it is in a
module that's loaded with RTLD_LOCAL, and all references resolve to
the same definition. This semantics is only relevant for c++ plugin
systems and even there it's often not what the user wants (so it can
be turned off in g++ by -fno-gnu-unique when the c++ shared lib is
compiled). In musl just treat it like STB_GLOBAL.
commit d56460c939 introduced this
regression as part of splitting the tls module list out of the dso
list. the new code added to dlopen's failure path to undo the changes
adding the partially-loaded libraries reset the tls_tail pointer
correctly, but did not clear its link to the next list entry. thus, at
least until the next successful dlopen, the list was not terminated
but ended with an invalid next pointer, which __copy_tls attempted to
follow when a new thread was created.
patch by Mikael Vidstedt.
On s390x, the kernel provides AT_SYSINFO_EHDR, but sets it to zero, if the
program being run does not have a program interpreter. This causes
problems when running the dynamic linker directly.
alpha and s390x gratuitously use 64-bit entries (wasting 2x space and
cache utilization) despite the values always being 32-bit.
based on patch by Bobby Bingham, with changes suggested by Alexander
Monakov to use the public Elf_Symndx type from link.h (and make it
properly variable by arch) rather than adding new internal
infrastructure for handling the type.
If a DT_NEEDED entry was the prefix of a reserved library name
(up to the first dot) then it was incorrectly treated as a libc
reserved name.
e.g. libp.so dependency was not loaded as it matched libpthread
reserved name.
