commits leading up to this one have moved the vast majority of
libc-internal interface declarations to appropriate internal headers,
allowing them to be type-checked and setting the stage to limit their
visibility. the ones that have not yet been moved are mostly
namespace-protected aliases for standard/public interfaces, which
exist to facilitate implementing plain C functions in terms of POSIX
functionality, or C or POSIX functionality in terms of extensions that
are not standardized. some don't quite fit this description, but are
"internally public" interfacs between subsystems of libc.
rather than create a number of newly-named headers to declare these
functions, and having to add explicit include directives for them to
every source file where they're needed, I have introduced a method of
wrapping the corresponding public headers.
parallel to the public headers in $(srcdir)/include, we now have
wrappers in $(srcdir)/src/include that come earlier in the include
path order. they include the public header they're wrapping, then add
declarations for namespace-protected versions of the same interfaces
and any "internally public" interfaces for the subsystem they
correspond to.
along these lines, the wrapper for features.h is now responsible for
the definition of the hidden, weak, and weak_alias macros. this means
source files will no longer need to include any special headers to
access these features.
over time, it is my expectation that the scope of what is "internally
public" will expand, reducing the number of source files which need to
include *_impl.h and related headers down to those which are actually
implementing the corresponding subsystems, not just using them.
previously, MEMOPS_SRCS failed to include arch-specific replacement
files for memcpy, etc., omitting CFLAGS_MEMOPS and thereby potentially
causing build failure if an arch provided C (rather than asm)
replacements for these files.
instead of trying to explicitly include all the files that might have
arch replacements, which is prone to human error, extract final names
to be used out of $(LIBC_OBJS), where the rules for arch replacements
have already been applied. do the same for NOSSP_OBJS, using CRT_OBJS
and LDSO_OBJS rather than repeating ourselves with $(wildcard...) and
explicit pathnames again.
notes by maintainer:
commit 2f853dd6b9 added these rules
because the new system for handling arch-provided replacement files
introduced for out-of-tree builds did not apply to the crt tree.
commit 63bcda4d8f later adapted the
makefile logic so that the crt and ldso trees go through the same
replacement logic as everything else, but failed to remove the
explicit rules that assumed the arch would always provide asm
replacements.
in addition to cleaning things up, removing these spurious rules
allows crti/crtn asm to be omitted by an arch (thereby using the empty
C files instead) if they are not needed.
the old limit was one byte too short to support locale names of the
form xx_XX.UTF-8@modifier where modifier is more than 3 bytes, a form
which various real-world locale names take. the problem could be
avoided by omitting the useless ".UTF-8" part, but users may need to
have it present when operating on mixed-libc systems or when it will
be carried over (e.g. across ssh) to other systems.
the new limit is chosen sufficient for existing/reasonable locale
names while still keeping the size of setlocale's static buffer small.
also add locale_impl.h to the Makefile's list of headers which force
rebuild of source files, to prevent dangerously inconsistent object
files from getting used after this change.
this follows the principle of having the source tree layout define
build semantics. it also makes it possible for crt/$(ARCH) to define
additional installable files, which may be needed for midipix and
other future targets.
commit 2f853dd6b9 moved the error
handling for $(ARCH) not being set such that it applied to all
targets, including clean and distclean. previously these targets
worked even in an unconfigured tree. to restore the old behavior, make
most of the makefile body conditional on $(ARCH) being set/non-empty
and produce the error via a fake "all" target in the conditional
branch for the case where $(ARCH) is empty.
prior to commit 2f853dd6b9 which
overhauled the makefile for out-of-tree builds, crt/*.c files were
replaceable by crt/$(ARCH)/*.s, and top-level ldso/ did not exist (its
files were under src/ldso). since then, crti.o and crtn.o have been
hard-coded as arch-specific, but none of the other files in crt/ or
ldso/ were replaceable at all.
in preparation for easy integration with midipix, which has a port of
musl to windows, it needs to be possible to override the ELF-specific
code in these files. making the same arch-replacements system work
throughout the whole source tree also improves consistency and removes
the need for some file-specific rules (crti.o and crtn.o) in the
makefile.
previous work overhauling the dynamic linker made it so that linking
libc with -Bsymbolic-functions was no longer mandatory, but the
configure logic that forced --disable-shared when ld failed to accept
the option was left in place.
this commit removes the hard-coded -Bsymbolic-functions from the
Makefile and changes the configure test to one that simply adds it to
the auto-detected LDFLAGS on success.
this sets the stage for the first phase of the bits deduplication.
bits headers which are identical for "most" archs will be moved to
arch/generic/bits.
these were not covered by the parent-level rules with the new build
system. in the old build system, the equivalent files were often in
arch/$(ARCH)/src and likewise lacked the suppression. this could lead
to early crashing (before thread pointer init) when libc itself was
built with stack protector enabled.
now that .lo and .o files differ only by whether -fPIC is passed (and
no longer at the source level based on the SHARED macro), it's
possible to use the same object files for both static and shared libc
when the compiler would produce PIC for the static files anyway. this
happens if the user has included -fPIC in their CFLAGS or if the
compiler has been configured to produce PIE by default.
we use the .lo files for both, and still append -fPIC to the CFLAGS,
rather than using the .o files so that libc.so does not break
catastrophically if the user later removes -fPIC from CFLAGS in
config.mak or on the make command line. this also ensures that we get
full -fPIC in case -fpic, -fPIE, or some other lesser-PIC option was
passed in CFLAGS.
this eliminates the last need for the SHARED macro to control how
files in the src tree are compiled. the same code is used for both
libc.a and libc.so, with additional code for the dynamic linker (from
the new ldso tree) being added to libc.so but not libc.a. separate .o
and .lo object files still exist for the src tree, but the only
difference is that the .lo files are built as PIC.
in the future, if/when we add dlopen support for static-linked
programs, much of the code in dynlink.c may be moved back into the src
tree, but properly factored into separate source files. in that case,
the code in the ldso tree will be reduced to just the dynamic linker
entry point, self-relocation, and loading of libraries needed by the
main application.
the lib dir is automatically created if needed by the out-of-tree
build logic, and now that all generated files are in obj and lib,
deleting them is much simpler. using "rm -rf" is also more thorough,
as it picks up object files that were left around from source files
that no longer exist or which are no longer to be used because an
arch-specific replacement file was added or removed.
as of commit af21a82ccc, .sub files are
no longer in use. removing the makefile machinery to handle them not
only cleans up and simplifies the makefile, but also significantly
reduces make's startup time.
commit 2f853dd6b9 failed to replicate
the old makefile logic that caused arch/arm/src/arm/atomics.s to be
built. since this was the only .s file under arch/*/src, rather than
trying to reproduce the old logic, I'm just moving it up a level and
adjusting the glob pattern in the makefile to catch it. eventually
arch/*/src will probably be removed in favor of moving all these files
to appropriate src/*/$(ARCH) locations.
this allows the rules for .o and .lo files to be identical, with -fPIC
and -DSHARED added for .lo files via target-specific variable append.
this is arguably cleaner now and will allow more cleanup and removal
of redundant rule bodies after other prerequisite changes are made.
previously, replacement files provided in $(ARCH) dirs under src/ had
to be .s files. in order to replace a file with C source, an empty .s
file was needed there to suppress the original file, and a separate .c
file was needed in arch/$(ARCH)/src/.
support for .S is new and is aimed at short-term use eliminating .sub
files. asm source files are still expected not to make any heavy
preprocessor use, just simple conditionals on subarch. eventually most
affected files may be replaced with C source files with minimal inline
asm instead of asm source files.
this change adds support for building musl outside of the source
tree. the implementation is similar to autotools where running
configure in a different directory creates config.mak in the current
working directory and symlinks the makefile, which contains the
logic for creating all necessary directories and resolving paths
relative to the source directory.
to support both in-tree and out-of-tree builds with implicit make
rules, all object files are now placed into a separate directory.
this way, overriding these variables on the make command line (or just
re-passing the originally-passed values when invoking make) won't
suppress use of the flags added by configure.
these per-target CFLAGS adjustments are mandatory additions to the
command line for building the affected targets, not part of the
user-provided CFLAGS for tuning. my intent was always that the
variable append operations would take place after user settings, but
when a variable is set on the command line, it overrides all
definitions in the makefile, including target-specific ones.
based on patch by Szabolcs Nagy.
There is a lot which could be common between i386 and x86_64, but none
of it will be useful for any other arch. These should be useful for
all archs, however.
this functionality is affected by GNU make bug #30653, "intermediate
files incorrectly pruned in parallel builds". on affected versions of
make, parallel builds attempt to compile source files before
alltypes.h is generated.
as noted with commit a91ebdcfac, which
added the use of .SECONDARY, suppression of removal of "intermediate"
files does not seem to be needed at present. if it is needed in the
future, it should be achievable by explicitly mentioning their names
as targets or prerequisites.
at one point, GNU make was removing crt/*.o after producing the copies
in lib/ due to an arcane misfeature for handling "intermediate" files.
the circumstances that caused this are no longer present in our
makefile, but the previous workaround using .PRECIOUS was wrong and
could result in corrupt/partial files being left behind during an
interrupted build. using .SECONDARY is the correct, documented fix
that will prevent deletion of "intermediate" files from ever
resurfacing.
Some functions implemented in asm need to use EBP for purposes other
than acting as a frame pointer. (Notably, it is used for the 6th
argument to syscalls with 6 arguments.) Without frame pointers, GDB
can only show backtraces if it gets CFI information from a
.debug_frame or .eh_frame ELF section.
Rather than littering our asm with ugly .cfi directives, use an awk
script to insert them in the right places during the build process, so
GDB can keep track of where the current stack frame is relative to the
stack pointer. This means GDB can produce beautiful stack traces at
any given point when single-stepping through asm functions.
Additionally, when registers are saved on the stack and later
overwritten, emit ..cfi directives so GDB will know where they were
saved relative to the stack pointer. This way, when you look back up
the stack from within an asm function, you can still reliably print
the values of local variables in the caller.
If this awk script were to understand every possible wild and crazy
contortion that an asm programmer can do with the stack and registers,
and always emit the exact ..cfi directives needed for GDB to know what
the register values were in the preceding stack frame, it would
necessarily be as complex as a full x86 emulator. That way lies
madness.
Hence, we assume that the stack pointer will _only_ ever be adjusted
using push/pop or else add/sub with a constant. We do not attempt to
detect every possible way that a register value could be saved for
later use, just the simple and common ways.
Thanks to Szabolcs Nagy for suggesting numerous improvements to this
code.
musl-clang allows the user to compile musl-powered programs using their
already existent clang install, without the need of a special cross compiler.
it achieves this by wrapping around both the system clang install and the
linker and passing them special flags to re-target musl at runtime.
it does only affect invocations done through the special musl-clang wrapper
script, so that the user setup remains fully intact otherwise.
the clang wrapper consists of the compiler frontend wrapper script,
musl-clang, and the linker wrapper script, ld.musl-clang.
musl-clang makes sure clang invokes ld.musl-clang to link objects; neither
script needs to be in PATH for the wrapper to work.
this overhauls part of the build system in order to support multiple
toolchain wrapper scripts, as opposed to solely the musl-gcc wrapper as
before. it thereby replaces --enable-gcc-wrapper with --enable-wrapper=...,
which has the options 'auto' (the default, detect whether to use wrappers),
'all' (build and install all wrappers), 'no' (don't build any) and finally
the options named after the individual compiler scripts (currently only
'gcc' is available) to build and install only that wrapper.
the old --enable-gcc-wrapper is removed from --help, but still available.
it also modifies the wrappers to use the C compiler specified to the build
system as 'inner' compiler, when applicable. as wrapper detection works by
probing this compiler, it may not work with any other.
static-linked PIE files need startup code to relocate themselves, much
like the dynamic linker does. rcrt1.c reuses the code in dlstart.c,
stage 1 of the dynamic linker, which in turn reuses crt_arch.h, to
achieve static PIE with no new code. only relative relocations are
supported.
existing toolchains that don't yet support static PIE directly can be
repurposed by passing "-shared -Wl,-Bstatic -Wl,-Bsymbolic" instead of
"-static -pie" and substituting rcrt1.o in place of crt1.o.
all libraries being linked must be built as PIC/PIE; TEXTRELs are not
supported at this time.
commit de2b67f8d4 attempted to avoid
having vis.h affect crt files, but the Makefile variable used,
CRT_LIBS, refers to the final output copies in the lib directory, not
the copies in the crt build directory, and thus the -DCRT was not
applied.
while unlikely to be noticed, this regression probably broke
production of PIE executables whose main functions are not in the
executable but rather a shared library.
this is implemented via the build system and does not affect source
files. the idea is to use protected or hidden visibility to prevent
the compiler from pessimizing function calls within a shared (or
position-independent static) libc in the form of overhead setting up
for a call through the PLT. the ld-time symbol binding via the
-Bsymbolic-functions option already optimized out the PLT itself, but
not the code in the caller needed to support a call through the PLT.
on some archs this overhead can be substantial; on others it's
trivial.
this was already essentially possible as a result of the previous
commits changing the dynamic linker/thread pointer bootstrap process.
this commit mainly adds build system infrastructure:
configure no longer attempts to disable stack protector. instead it
simply determines how so the makefile can disable stack protector for
a few translation units used during early startup.
stack protector is also disabled for memcpy and memset since compilers
(incorrectly) generate calls to them on some archs to implement
struct initialization and assignment, and such calls may creep into
early initialization.
no explicit attempt to enable stack protector is made by configure at
this time; any stack protector option supported by the compiler can be
passed to configure in CFLAGS, and if the compiler uses stack
protector by default, this default is respected.
this overhaul further reduces the amount of arch-specific code needed
by the dynamic linker and removes a number of assumptions, including:
- that symbolic function references inside libc are bound at link time
via the linker option -Bsymbolic-functions.
- that libc functions used by the dynamic linker do not require
access to data symbols.
- that static/internal function calls and data accesses can be made
without performing any relocations, or that arch-specific startup
code handled any such relocations needed.
removing these assumptions paves the way for allowing libc.so itself
to be built with stack protector (among other things), and is achieved
by a three-stage bootstrap process:
1. relative relocations are processed with a flat function.
2. symbolic relocations are processed with no external calls/data.
3. main program and dependency libs are processed with a
fully-functional libc/ldso.
reduction in arch-specific code is achived through the following:
- crt_arch.h, used for generating crt1.o, now provides the entry point
for the dynamic linker too.
- asm is no longer responsible for skipping the beginning of argv[]
when ldso is invoked as a command.
- the functionality previously provided by __reloc_self for heavily
GOT-dependent RISC archs is now the arch-agnostic stage-1.
- arch-specific relocation type codes are mapped directly as macros
rather than via an inline translation function/switch statement.
my old, out-of-tree release script that performed a clone rather than
using git archive checked the VERSION file to make sure that it
matched before doing a release. I believe there should be a way to do
the same with git commands (without resorting to checking out the
desired tag) but I have not yet found a way, so care should be taken
when using these targets that the correctness of the VERSION file is
not overlooked.
the main motivation for this change is to aid in debugging. since the
main program's entry point is also named _start, it was difficult to
set breakpoints or quickly identify which _start execution stopped in.
the wildcard function in GNU make includes dangling symlinks; if any
exist under the .git directory, they would get added as dependencies,
causing make to exit with an error due to lacking a rule to build the
missing file.
as far as I can tell, git operations which should force version.h to
be rebuilt must all touch the mtime of the top-level .git directory.
this is still experimental and subject to change. for git checkouts,
an attempt is made to record the exact revision to aid in bug reports
and debugging. no version information is recorded in the static libc.a
or binaries it's linked into.
the historical (non-standardized) install command is really
inappropriate for installing binaries/libraries on a system that
utilizes memory-mapped executable files. rather than replacing an
existing file atomically, it overwrites the existing file. this can
cause running programs to see a partially-modified version of the
file, resulting in unpredictable behavior, or SIGBUS. a MAP_COPY mode
for mmap would get around this problem, but Linux lacks MAP_COPY.
the shell script added with this commit works around the problem by
writing temporary files and moving them into place. unlike the
historical install utility, it also support a -l option for installing
a symbolic link atomically, via the same method.
ln -sf is non-atomic; it unlinks the destination first. instead, make
a temporary link and rename it into place.
this commit also fixes some of the dependency tracking behavior for
the link. depending on the directory it's to be installed in is not
reasonable; it causes a new link to be attempted if the library
directory has been modified, but does not attempt to make a new link
just because libc has been updated. instead, depend on the target to
be linked to. this will ensure that, if prefix has changed but
syslibdir has not, the link will be updated to point to the new
prefix.
instead of subarchs getting their own .s files which are used directly
by the makefile to replace the .c file, they now must provide a .sub
file whose contents are a pathname, relative to the location of the
.sub file, which will substitute for the .c file. essentially these
files are acting as symbolic links, but implemented as text files.
the default subarch is the one whose full name is just the base arch
name, with no suffixes. normally, either the asm in the default
subarch is suitable for all subarch variants, or separate asm is
mandatory for each variant. however, in the case of asm which is
purely for optimization purposes, it's possible to have asm that only
works (or only performs well) on the default subarch, and not any othe
the other variants. thus, I have added a mechanism to give a name to
the default variant, for example "armel" for the default,
little-endian arm. further such default-subarch names can be added in
the future as needed.
