Now when publishing a release, if make-releases-json is usable at the same
place, it will be called to regenerate the json index of available versions.
This will be used to rebuild a releases.json file in each download
directory. It only relies on existing files and sorts them by version,
appends known signatures (md5/sha256) and marks the most recent one as
the latest release.
This aims at addressing github issue #1537.
The list of URLs now also adds pending bugs, reviewed bugs, and code
reports. The redirect is performed on haproxy.org since github URLs
are far too large here.
For each test, the version number is evaluated using a call to awk,
which can be slow to start depending on the versions and OS. This is
only needed for a printf() call to keep only leading digits of each
component, multiply them by 1000 and pad them to 3 digits, something
that's clearly doable in plain shell in a portable way. This is what
this patch does, and it saves yet another 400 ms here on the full
test sequence.
grep is used in the arguments loops to check for features such as OPENSSL
or LUA or services like prometheus-exporter. Let's just look for the words
inside the list, which requires to prepend a delimitor at the beginning of
the list and add one at the end.
run-tegtests is starting to take a lot of time to spot which tests are
eligible, because for each test file a lot of "sed" sub-processes are
launched. This commit eliminates calls to sed by using the shell's
internal processing and parsing the VTC file only once. Instead of
extracting each option one by one from the file, all entries that look
like a valid option are passed to a single case/esac statement and their
value is extracted. Splitting into lists is simply done by adjusting the
IFS depending on the list's delimiter, which, contrary to the // pattern
modifier, is supported on every shell.
This was tested on both bash and dash, and the tests' execution time
dropped by 31% from 8.7 seconds to 6.0 seconds.
The early version of the script used to support passing non-branch
arguments but as it evolved we lost that option. Let's use "--" as a
delimiter after the branch(es) to pass optional file names to filter
on. This is convenient to list missing patches on a specific set of
files.
I forgot about OpenSSL 1.0.2, which neither supports the build_sw target
to build only the software, nor reliably supports parallel builds. Given
that we're building 1.0.2 and 3.0.0, let's stay on the safe side and
keep 1.x sequential.
1/4 of the OpenSSL build time is spent building the docs, let's just
build the software and not the doc, by replacing the "all" target
with "build_sw". With this my build time drops from 1'28 to 1'09.
Nothing was done for the other libs, as it's unknown whether they
provide specific build targets.
Running the "make all" phase on my machine with -j$(nproc) shrinks the
build time from 4'52 to 1'28. It will not be that big of a change in
the CI since it looks like two CPUs are exposed, but it should still
remain a net win. Let's enable it. The install phase obviously remains
sequential however.
In patch 3bad3d5 ("BUILD: Makefile: exclude broken tests by default"),
the default setting of the REGTESTST_TYPE variable was set in the
Makefile instead of the run-regtests.sh script.
Doing it in the Makefile was breaking the use of this environment
varible with make ( REGTESTS_TYPES=slow,default make reg-tests )
This patch move the default setting from the Makefile to
run-regtests.sh. It also change the documentation in `make
reg-tests-help` about the default value.
This patch should be backported where 3bad3d5 is backported.
I badly tested my previous patch forgetting to remove the "+" testing
present in options, and not in services; the list of services do not
have any "+" at the beginning of each service
this patch is fixing commit aabde71332 ("MINOR:
reg-tests: add a way to add service dependency")
Signed-off-by: William Dauchy <wdauchy@gmail.com>
I was looking at writing a simple first test for prometheus but I
realised there is no proper way to exclude it if haproxy was not built
with prometheus plugin.
Today we have `REQUIRE_OPTIONS` in reg-tests which is based on `Feature
list` from `haproxy -vv`. Those options are coming from the Makefile
itself.
A plugin is build this way:
EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
It does register service actions through `service_keywords_register`.
Those are listed through `list_services` in `haproxy -vv`.
To facilitate parsing, I slightly changed the output to a single line
and integrate it in regtests shell script so that we can now specify a
dependency while writing a reg-test for prometheus, e.g:
#REQUIRE_SERVICE=prometheus-exporter
#REQUIRE_SERVICES=prometheus-exporter,foo
There might be other ways to handle this, but that's the cleanest I
found; I understand people might be concerned by this output change in
`haproxy -vv` which goes from:
Available services :
foo
bar
to:
Available services : foo bar
Signed-off-by: William Dauchy <wdauchy@gmail.com>
It takes so much time to write an announce message that sometimes it's
annoying not being able to start the work while a fix is being finished.
With the new "-p" argument, announce-release will allow to prepare the
announce message for the current HEAD and with no tag yet. It will
restart from the last tag and automatically increment the version using
the same algorithm as create-release so that everything is accurate. It
should then be easier at the last moment to just include the final entry
by hand when the last fix finally arrives. For convenience, this argument
also allows to create an announce from another branch than master.
By having three variables it will be easier to preset the version and
the tag separately. One contains the announced version, another one the
associated tag and the last one the final commit ID (used as the ending
point before the release). This initially allows to check for the HEAD
matching the tag only when the version was not forced, hence re-announce
already tagged versions after some extra commits were added for example.
Enabling memory poisonning is often pretty effective for detecting
uninitialized structure fields. Let's enable it by default and let
the user change the arguments at will (e.g. forcing some memory limits
or disabling a poller). This will work with the latest vtest version
to date (02a9bc1).
after 73b520b958 variables SSL_LIB, SSL_INC
are not set, but still used by BoringSSL builds. That leads to error
(I wish we could stop on such errors) and using stock openssl instead
of boringssl
It's cumbersome to copy-paste a commit ID into another window after having
typed "git cherry-pick -sx", so let's have the suggested output format of
git-show prepare this line just before the subject line, it remains at a
stable position on the terminal when searching for "/^commit". One just
has to copy-paste the line into another terminal will result in the commit
being properly picked.
We've never used the output of the rightmost branch with this tool,
and it systematically causes two identical outputs making the job
harder during backport sessions. Let's simply remove the right part
when it's identical to the left one. This also adds a few line feeds
to make the output more readable.
It just appeared that the tar.gz we put online are not reproducible
because a timestamp is put by default into the archive. Passing "-n"
to gzip is sufficient to remove this timestamp, so let's do it, and
also make the gzip command configurable for more flexibility. Now
issuing the commands multiple times finally results in the same
archives being produced.
This should be backported to supported stable branches.
with recent change, OpenSSL download URL was changed in
incompatiable way. i.e. only the most recent openssl version
might be downloaded using previous script.
older versions are available under different URLs. as we need
several openssl versions, let us adopt script accordingly.
bug was caught after travis-ci cache was purged for some reason.
Commit 0f5ce6014a ("SCRIPTS: announce-release: place the send command
in the mail's header") broke the announce-release script: by not having
to edit the message at all anymore, mutt does nothing when sending, but
it still does if the message is edited (which was the case before). With
some testing, it appears that mutt -H does work when there's no change,
so let's use this instead. This should be backported till 1.7.
I managed to mess up with the file's permission while using a temporary
one during last release, and to backport the non-exec version everywhere.
This can be backported as far as 1.7 now.
The condition was inverted. When the branch was the master, it was
harmless because it caused an extra "checkout master", but when it
was not the master, the commit could be applied to the wrong branch
and it could even possibly not match the name to stop on.
I'm fed up with having to scroll my terminals trying to look for the
mail send command printed 30 minutes before the release, let's have
it copied into the e-mail template itself, and replace the old headers
that used to be duplicated there and that are not needed anymore.
I find myself often getting trapped into calling "backport 2.0 HEAD" which
doesn't work because "HEAD" is passed as the argument to cherry-pick in
other repos. Let's resolve it first. And also let's shorten the commit IDs
to make the error messages more readable and to ease copy-paste.
Given that some OSes have bash in /usr/local/bin and in order not to
give too easy an excuse to Olivier for not backporting fixes, let's
make a few scripts rely on /usr/bin/env bash instead of /bin/bash :-)
The script is simply called from the repository holding the patch to
backport, with the last branch number and the commit(s) ID(s) to send
there and it then follows the chain of "down" repos to go down one step
until it meets the indicated last one. It basically automates what we do
by hand. Example:
./scripts/backport 1.9 1c7c0d6b97
Note that it does *not* push, which still has to be done by hand after
building and testing.
Implement #REQUIRE_BINARIES for vtc files.
The run-regtests.sh script will check if the binary is available in the
environment, if not, it wil disable the vtc.
This test checks that an HTTP message is properly processed when we failed to
add the HTX EOM block in an HTX message during the parsing because the buffer is
full. Some space must be released in the buffer to make it possible. This
requires an extra pass in the H1 multiplexer. Here, we must be sure the mux is
called while there is no more incoming data.
It is a "devel" test because conditions to run the test successfully is highly
dependent on the implementation. So if it fail, it is not necessarily a bug. It
may be due of an internal change. It relies on internal HTX sample fetches.
The changelog is empty when creating a dev0 version and this confuses
the commit message, let's clearly mention the exact copy when there are
no changes.
Willy Tarreau
Ilya Shipitsin
Tim Duesterhus
William Lallemand
William Dauchy
Thayne McCombs
William Lallemand
Christopher Faulet