1
0
mirror of http://git.haproxy.org/git/haproxy.git/ synced 2024-12-28 15:42:30 +00:00
Commit Graph

123 Commits

Author SHA1 Message Date
Willy Tarreau
7e943cdf27 CI: scripts: build vtest using multiple CPUs
Now that vtest supports make -j, let's use it to save a bit of time
(the build time is ~6s per test by default).
2024-05-27 12:15:50 +02:00
Willy Tarreau
01843c47a1 CI: scripts: fix build of vtest regarding option -C
On Linux, GNU make emits "w" at the beginning of the MAKEFLAGS
variable if -C is passed, which happens since vtest d6d228bcb3.
In fact it emits any of the command line flags without the leading
'-' in this case. gmake doesn't do that on BSD apparently. It's
documented under Options/Recursion in the GNU make doc. There's
also MFLAGS that could work but it does not contain the variables
definitions. So let's just avoid the -C that we don't really need.

This needs to be backported to stable versions.
2024-05-27 12:15:50 +02:00
William Lallemand
9c1fa3e411 CI: scripts/buil-ssl: cleanup the boringssl and quictls build
Put the quictls and boringssl build in their own function instead of
keeping it in the main part of the script.
2024-05-23 16:54:30 +02:00
William Lallemand
5d73643ca3 CI: scripts/build-ssl: add a DESTDIR and TMPDIR variable
Add a DESTDIR and TMPDIR variables so the build-ssl.sh script can be used as a
generic SSL lib installer outside the CI.

The varibles are prefixed with BUILDSSL so they doesn't collide with the
makefile one.

Ex:

  OPENSSL_VERSION=3.2.0 BUILDSSL_DESTDIR=/opt/openssl-3.2.0/ ./scripts/build-ssl.sh
  WOLFSSL_VERSION=5.7.0 BUILDSSL_DESTDIR=/opt/wolfssl-5.7.0/ ./scripts/build-ssl.sh
2024-05-23 15:34:59 +02:00
Ilia Shipitsin
4a968d9d27 CI: scripts/build-ssl.sh: loudly fail on unsupported platforms 2024-05-22 16:52:43 +02:00
William Lallemand
0182f6bbb6 REGTESTS: scripts: allow to change the vtest timeout
$ make reg-tests VTEST_TIMEOUT=5

Allow to change the timeout of the regtests with the VTEST_TIMEOUT
variable. The default value is still 10.
2024-05-22 15:43:53 +02:00
Willy Tarreau
b6ed749adc SCRIPTS: run-regtests: fix a few occurrences of extended regexes
Running run-regtests on OpenBSD failed to identify haproxy version and
the various build options because the backslash is not recognized in
grep expressions. One must only use -E for the extended regexes and
not use the slash.
2024-05-15 19:33:45 +02:00
William Lallemand
378d500caf REGTESTS: use -dI for insecure fork by default in the regtest scripts
Let's remove the CI HAPROXY_ARGS setting and set -dI for anything run
with the run-regtests.sh.
2024-04-22 16:15:57 +02:00
Ilya Shipitsin
76f56a39be CI: modernize macos matrix
let's stick to macos-13 for stable branches and macos-14 for development branches.
since macos-14 is available for Apple Silicon, some modifications are required
for VTest (should be ported to VTest later)

news: https://github.blog/changelog/2024-01-30-github-actions-macos-14-sonoma-is-now-available/
2024-04-19 17:37:24 +02:00
Ilya Shipitsin
a68a2891b3 CI: ssl: add yet another OpenSSL download fallback 2024-02-07 11:05:45 +01:00
Willy Tarreau
57c5ae10f6 SCRIPTS: mk-patch-list: produce a list of patches
There does not seem to be a convenient way to tell git-show-backports to
produce individual patches with numbers. That's what this script does by
calling git-format-patch for each specified commit ID, letting git do all
the painful work (formatting etc). This has been mostly used during
backport sessions but was apparently never committed!
2023-12-18 20:50:51 +01:00
William Lallemand
0be50f44f8 CI: ssl: add git id support for wolfssl download
Allow to download a git revision directly with the git ID.

WOLFSSL_VERSION=git-d83f2fa ./scripts/build-ssl.sh
2023-10-10 10:34:17 +02:00
William Lallemand
cfd72eb65f CI: ssl: add wolfssl to build-ssl.sh
Add wolfssl support to the build-ssl script.
2023-10-09 23:44:23 +02:00
Andrew Hopkins
c677bb840b CI: scripts: add support to build-ssl.sh to download and build AWS-LC
Relies on a new enviornment variable 'AWS_LC_VERSION' to be set to
the GitHub tag to download and build.
2023-09-06 13:41:36 +02:00
Willy Tarreau
f97db23b6d SCRIPTS: git-show-backports: automatic ref and base detection with -m
When running with -m (check for missing backports) we often have to fill
lots of information that can be determined automatically the vast majority
of the time:
  - restart point (last cherry-picked ID from one of the last commits)
  - current branch (HEAD)
  - reference branch (the one that contains most of the last commits)

These elements are not that hard to determine, so let's make sure we
can fall back to them when running in missing mode.

The reference branch is guessed by looking at the upstream branch that
most frequently contains some of the last 10 commits. It can be inaccurate
if multiple branches exist with these commits, or when upstream changes
due to a non-LTS branch disappearing in the middle of the series, in which
case passing "-r" will help. But most of the time it works OK. It also gives
precedence to local branches over remote ones for such choices. A test in
2.4 at commit 793a4b520 correctly shows 2.6/master as the upstream despite
2.5 having been used for the early ones of the tag.

For the restart point, we assume that the most recent commit that was
backported serves as a reference (and not the most recently backported
commit). This means that the usual case where an old commit was found
to be missing will not fool the analysis. Commits are inspected from
2 commits before the last tag, and reordered from the parent's tree
to see which one is the last one.

With this, it's sufficient to issue "git-show-backports -q -m" to get
the list of backports from the upstream branch, restarting from the
last backported one.
2023-08-14 13:12:56 +02:00
Willy Tarreau
f9b04bd48b SCRIPTS: publish-release: update the umask to keep group write access
This is to avoid the occasional error that arises when a release is
first done by another maintainer.
2023-05-24 22:49:12 +02:00
Ilia Shipitsin
6703653098 BUILD: ssl: switch LibreSSL to Fastly CDN
OpenBSD ftp is down, let us switch to CDN
2023-04-26 18:41:03 +02:00
Willy Tarreau
9d5e11682c SCRIPTS: run-regtests: add a version check
It happens from time to time while switching between branches and/or
updating after someone else's changes that regtests are run by accident
on the wrong binary, typically the one the tests were run on during
development and not with the latest adaptations. And obviously it's
when this happens that we break the CI. There are various causes to
this but they all come down to humans context-switching a lot, and
there's no real fix for this that doesn't add even more burden hence
increases the overhead. However we can help the human detect such
mistakes very easily.

This change here will compare the version of the haproxy binary to
the version of the tree, and will emit a warning in the regtest output
if they do not match, regardless of the outcome of the test. This is
sufficient in case of failures because these are quickly glanced over,
and is sufficient as well in case of accidental success because the
warning is the last message. E.g:

  ########################## Starting vtest ##########################
  Testing with haproxy version: 2.7-dev10-cfcdbc-38
  Warning: version does not match the current tree (2.7-dev10-111c78-39)
  0 tests failed, 0 tests skipped, 182 tests passed

This should not affect builds made out of a git tree because the version
is retrieved using "make version", or exactly the same way as it's passd
to the haproxy binary. We just need to know what "make" command to run,
so $MAKE is used primarily, falling back to "make" then to "gmake". In
case all of these fail, we just ignore the version check. This should be
sufficient to catch human mistakes without affecting the CI.
2022-11-30 18:44:33 +01:00
Willy Tarreau
e3a02d5e08 SCRIPTS: announce-release: add a link to the data plane API
Since Marko announced at HAProxyConf 2022 that the data plane API is
mostly complete and will now follow the same release cycle as haproxy
starting with 2.7, it's probably the right moment to encourage users
to start trying it so that we can hope to migrate all the painful
discovery stuff there in a not too distant future.

Let's just point to the latest release for now. We'll see in the future
if we need to adapt the link depending on the branch.
2022-11-28 07:31:54 +01:00
Ilya Shipitsin
4a689dad03 CLEANUP: assorted typo fixes in the code and comments
This is 32nd iteration of typo fixes
2022-10-30 17:17:56 +01:00
Ilya Shipitsin
47ddc89879 BUILD: scripts: disable tests build on QuicTLS build
during CI builds QuicTLS is not cached, let us speed it up by
disabling tests build. Doing so saves ~40s out of 3m40.
2022-10-17 05:51:18 +02:00
Willy Tarreau
68b3e135e3 SCRIPTS: announce-release: update some URLs to https
Some components like Discourse were already redirecting to https. Other
ones like docs and git are covered by the certificate, and finally
switching the advertised scheme for www should increase the ratio of
H2 and H3 in the stats (resp 8.9 and 1.9%) and possibly help spot new
issues.
2022-09-17 12:17:07 +02:00
Ilya Shipitsin
3f59ac5ce2 BUILD: SSL: allow to pass additional configure args to QUICTLS
this allows to pass QUICTLS_EXTRA_ARGS to QUICTLS builds. if no
 additional arg is passed, behaviour is kept unchanged

--
v2 indentation fixed
2022-08-06 17:10:04 +02:00
Willy Tarreau
8e9f915f8f SCRIPTS: make publish-release try to launch make-releases-json
Now when publishing a release, if make-releases-json is usable at the same
place, it will be called to regenerate the json index of available versions.
2022-05-30 15:44:10 +02:00
Willy Tarreau
f1c6ccfc6a SCRIPTS: add make-releases-json to recreate a releases.json file in download dirs
This will be used to rebuild a releases.json file in each download
directory. It only relies on existing files and sorts them by version,
appends known signatures (md5/sha256) and marks the most recent one as
the latest release.

This aims at addressing github issue .
2022-05-30 15:44:10 +02:00
Willy Tarreau
02df95e938 SCRIPTS: announce-release: add URL of dev packages
This is the shortened URL of the nightly builds maintained by William,
let's have them in announce messages.
2022-04-30 14:16:15 +02:00
Willy Tarreau
076b6cec0f SCRIPTS: announce-release: add shortened links to pending issues
The list of URLs now also adds pending bugs, reviewed bugs, and code
reports. The redirect is performed on haproxy.org since github URLs
are far too large here.
2022-04-16 12:06:07 +02:00
Willy Tarreau
f0089d86ce SCRIPTS: announce-release: update the doc's URL
Now that the doc is accessible on docs.haproxy.org via github pages,
let's update the URL.
2022-04-16 07:57:15 +02:00
Ilya Shipitsin
e9efc3a5be CI: refactor OpenTracing build script
re-use scripts/build-ot.sh in CI again. Bump opentracing-cpp to 1.6.0
2022-01-19 07:37:40 +01:00
Ilya Shipitsin
2091c7ca70 BUILD: SSL: add quictls build to scripts/build-ssl.sh
script/build-ssl.sh is used mostly in CI, let us introduce QUIC
OpenSSL fork support
2021-11-20 08:17:22 +01:00
Willy Tarreau
b7a28e7bc8 SCRIPT: run-regtests: avoid calling awk to compute the version
For each test, the version number is evaluated using a call to awk,
which can be slow to start depending on the versions and OS. This is
only needed for a printf() call to keep only leading digits of each
component, multiply them by 1000 and pad them to 3 digits, something
that's clearly doable in plain shell in a portable way. This is what
this patch does, and it saves yet another 400 ms here on the full
test sequence.
2021-11-18 17:54:49 +01:00
Willy Tarreau
0b3eb81c6b SCRIPT: run-regtests: avoid several calls to grep to test for features
grep is used in the arguments loops to check for features such as OPENSSL
or LUA or services like prometheus-exporter. Let's just look for the words
inside the list, which requires to prepend a delimitor at the beginning of
the list and add one at the end.
2021-11-18 17:54:49 +01:00
Willy Tarreau
969e7afa0d SCRIPTS: run-regtests: reduce the number of processes needed to check options
run-tegtests is starting to take a lot of time to spot which tests are
eligible, because for each test file a lot of "sed" sub-processes are
launched. This commit eliminates calls to sed by using the shell's
internal processing and parsing the VTC file only once. Instead of
extracting each option one by one from the file, all entries that look
like a valid option are passed to a single case/esac statement and their
value is extracted. Splitting into lists is simply done by adjusting the
IFS depending on the list's delimiter, which, contrary to the // pattern
modifier, is supported on every shell.

This was tested on both bash and dash, and the tests' execution time
dropped by 31% from 8.7 seconds to 6.0 seconds.
2021-11-18 17:54:49 +01:00
Willy Tarreau
0d026edaef SCRIPTS: git-show-backports: re-enable file-based filtering
The early version of the script used to support passing non-branch
arguments but as it evolved we lost that option. Let's use "--" as a
delimiter after the branch(es) to pass optional file names to filter
on. This is convenient to list missing patches on a specific set of
files.
2021-11-03 08:41:01 +01:00
Ilya Shipitsin
cd61e8383d BUILD: adopt script/build-ssl.sh for OpenSSL-3.0.0beta2
starting with
74b7f339aa,
libs are installed to "lib64", to get back required behaviour, let us
set libdir explicitly
2021-08-25 05:16:00 +02:00
Willy Tarreau
cbc0c232bf CI: ssl: keep the old method for ancient OpenSSL versions
I forgot about OpenSSL 1.0.2, which neither supports the build_sw target
to build only the software, nor reliably supports parallel builds. Given
that we're building 1.0.2 and 3.0.0, let's stay on the safe side and
keep 1.x sequential.
2021-06-17 15:40:53 +02:00
Willy Tarreau
f4b9801f05 CI: ssl: do not needlessly build the OpenSSL docs
1/4 of the OpenSSL build time is spent building the docs, let's just
build the software and not the doc, by replacing the "all" target
with "build_sw". With this my build time drops from 1'28 to 1'09.

Nothing was done for the other libs, as it's unknown whether they
provide specific build targets.
2021-06-17 15:16:57 +02:00
Willy Tarreau
c6f1498405 CI: ssl: enable parallel builds for OpenSSL on Linux
Running the "make all" phase on my machine with -j$(nproc) shrinks the
build time from 4'52 to 1'28. It will not be that big of a change in
the CI since it looks like two CPUs are exposed, but it should still
remain a net win. Let's enable it. The install phase obviously remains
sequential however.
2021-06-17 15:05:24 +02:00
Tim Duesterhus
8727614dc4 REGTESTS: Remove support for REQUIRE_BINARIES
This is no longer used since the migration to the native `feature cmd`
functionality.
2021-06-17 14:59:55 +02:00
Willy Tarreau
5ac9a393ef SCRIPTS: opentracing: enable parallel builds in build-ot.sh
The script didn't make use of parallel builds, which roughly cut the
build time in half with 4 cores. This can help a bit with the CI.
2021-06-10 07:35:15 +02:00
Tim Duesterhus
a9334df5a9 CLEANUP: reg-tests: Remove obsolete no-htx parameter for reg-tests
The legacy HTTP subsystem has been removed. HTX is always enabled.
2021-06-04 15:41:21 +02:00
Ilya Shipitsin
75c95aa649 CI: github actions: add OpenTracing builds
let us add OpenTracing module to "all features" builds
2021-05-18 18:37:37 +02:00
Ilya Shipitsin
0b03895620 CI: introduce scripts/build-vtest.sh for installing VTest
we install VTest for all CI systems, let us unify instalation
2021-05-18 10:48:30 +02:00
Willy Tarreau
5f3448f6de REGTESTS: run-regtests: match both "HAProxy" and "HA-Proxy" in the version
The script checks the HAProxy version and was looking for "HA-Proxy" which
is hard-coded in haproxy.c, let's relax the test to also accept "HAProxy".
2021-05-09 06:12:41 +02:00
William Lallemand
c1ddcafdf9 BUILD: Makefile: move REGTESTST_TYPE default setting
In patch 3bad3d5 ("BUILD: Makefile: exclude broken tests by default"),
the default setting of the REGTESTST_TYPE variable was set in the
Makefile instead of the run-regtests.sh script.

Doing it in the Makefile was breaking the use of this environment
varible with make ( REGTESTS_TYPES=slow,default make reg-tests )

This patch move the default setting from the Makefile to
run-regtests.sh. It also change the documentation in `make
reg-tests-help` about the default value.

This patch should be backported where 3bad3d5 is backported.
2021-02-05 11:41:16 +01:00
William Dauchy
4488434c97 BUG/MINOR: reg-tests: fix service dependency script
I badly tested my previous patch forgetting to remove the "+" testing
present in options, and not in services; the list of services do not
have any "+" at the beginning of each service

this patch is fixing commit aabde71332 ("MINOR:
reg-tests: add a way to add service dependency")

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-11 14:16:06 +01:00
William Dauchy
aabde71332 MINOR: reg-tests: add a way to add service dependency
I was looking at writing a simple first test for prometheus but I
realised there is no proper way to exclude it if haproxy was not built
with prometheus plugin.

Today we have `REQUIRE_OPTIONS` in reg-tests which is based on `Feature
list` from `haproxy -vv`. Those options are coming from the Makefile
itself.

A plugin is build this way:
  EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"

It does register service actions through `service_keywords_register`.
Those are listed through `list_services` in `haproxy -vv`.
To facilitate parsing, I slightly changed the output to a single line
and integrate it in regtests shell script so that we can now specify a
dependency while writing a reg-test for prometheus, e.g:

  #REQUIRE_SERVICE=prometheus-exporter
  #REQUIRE_SERVICES=prometheus-exporter,foo

There might be other ways to handle this, but that's the cleanest I
found; I understand people might be concerned by this output change in
`haproxy -vv` which goes from:

  Available services :
          foo
          bar

to:

  Available services : foo bar

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-10 07:42:33 +01:00
Thayne McCombs
478e5ddc8d SCRIPTS: announce-release: fix typo in help message
s/relase/release in -p help message.
2021-01-08 14:58:22 +01:00
Willy Tarreau
96d5368202 SCRIPTS: make announce release support preparing announces before tag exists
It takes so much time to write an announce message that sometimes it's
annoying not being able to start the work while a fix is being finished.
With the new "-p" argument, announce-release will allow to prepare the
announce message for the current HEAD and with no tag yet. It will
restart from the last tag and automatically increment the version using
the same algorithm as create-release so that everything is accurate. It
should then be easier at the last moment to just include the final entry
by hand when the last fix finally arrives. For convenience, this argument
also allows to create an announce from another branch than master.
2021-01-06 15:49:44 +01:00
Willy Tarreau
0c612936b2 SCRIPTS: improve announce-release to support different tag and versions
By having three variables it will be easier to preset the version and
the tag separately. One contains the announced version, another one the
associated tag and the last one the final commit ID (used as the ending
point before the release). This initially allows to check for the HEAD
matching the tag only when the version was not forced, hence re-announce
already tagged versions after some extra commits were added for example.
2021-01-06 15:45:15 +01:00