haproxy/.github/matrix.py

177 lines
4.4 KiB
Python
Raw Normal View History

CI: Expand use of GitHub Actions for CI Travis is becoming overall increasingly unreliable lately. We've already seen that the timing sensitive tests regularly fail and thus they were disabled. Additionally they recently announced a new pricing model that caps the number of minutes for Open Source projects: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing GitHub Actions VMs are working well, possibly allowing to use custom runners for special tasks in the future. In addition to this better performance its workflow configuration language is more expressive compared to the Travis CI one. Specifically the build matrix does not need to be specified in YAML. Instead it can be generated ad-hoc using a script. This allows us to cleanly define the various build configurations without having an unreadable 80 line mess where the flags are inconsistently activated. As an example in the current Travis CI configuration the prometheus exporter is tested together with LibreSSL 2.9.2 for whatever reason. In addition to all the previous points the UI of Travis is not that nice. On GitHub you are just seeing that "Travis failed" without any details which exact job failed. This requires you to visit the slow Travis page and look up the details there. GitHub Actions creates a single entry for each configuration that is tested, allowing you to see the details without needing to leave GitHub. This new GitHub Actions workflow aims to reproduce the configurations tested in Travis. It comes close, but is not completely there, yet. Consider this patch a proof of concept that will evolve in the future, ideally with Ilya's expertise. The current configurations are as follows. Each one is tested with both gcc and clang. - All features disabled (no USE flags) - All features enabled (all USE flags) - Standalone test of each of the supported compression libraries: - USE_ZLIB=1 - USE_SLZ=1 - Standalone test of various SSL libraries: - stock (the SSL installed by default on the VM) - OpenSSL 1.0.2u - LibreSSL 2.9.2, 3.0.2, 3.1.1 - All features enabled with ASAN (clang only) Future additions of new tests should take care to not test unrelated stuff. Instead a distinct configuration should be added. Additionally there is a Mac OS test with clang and all features disabled. Known issues: - Apparently the git commit is not properly detected during build. The HEAD currently shows as 2.4-dev0.
2020-07-28 21:00:35 +00:00
# Copyright 2019 Ilya Shipitsin <chipitsine@gmail.com>
# Copyright 2020 Tim Duesterhus <tim@bastelstu.be>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version
# 2 of the License, or (at your option) any later version.
import json
import sys
if len(sys.argv) == 2:
build_type = sys.argv[1]
else:
print("Usage: {} <build_type>".format(sys.argv[0]), file=sys.stderr)
sys.exit(1)
print("Generating matrix for type '{}'.".format(build_type))
CI: Expand use of GitHub Actions for CI Travis is becoming overall increasingly unreliable lately. We've already seen that the timing sensitive tests regularly fail and thus they were disabled. Additionally they recently announced a new pricing model that caps the number of minutes for Open Source projects: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing GitHub Actions VMs are working well, possibly allowing to use custom runners for special tasks in the future. In addition to this better performance its workflow configuration language is more expressive compared to the Travis CI one. Specifically the build matrix does not need to be specified in YAML. Instead it can be generated ad-hoc using a script. This allows us to cleanly define the various build configurations without having an unreadable 80 line mess where the flags are inconsistently activated. As an example in the current Travis CI configuration the prometheus exporter is tested together with LibreSSL 2.9.2 for whatever reason. In addition to all the previous points the UI of Travis is not that nice. On GitHub you are just seeing that "Travis failed" without any details which exact job failed. This requires you to visit the slow Travis page and look up the details there. GitHub Actions creates a single entry for each configuration that is tested, allowing you to see the details without needing to leave GitHub. This new GitHub Actions workflow aims to reproduce the configurations tested in Travis. It comes close, but is not completely there, yet. Consider this patch a proof of concept that will evolve in the future, ideally with Ilya's expertise. The current configurations are as follows. Each one is tested with both gcc and clang. - All features disabled (no USE flags) - All features enabled (all USE flags) - Standalone test of each of the supported compression libraries: - USE_ZLIB=1 - USE_SLZ=1 - Standalone test of various SSL libraries: - stock (the SSL installed by default on the VM) - OpenSSL 1.0.2u - LibreSSL 2.9.2, 3.0.2, 3.1.1 - All features enabled with ASAN (clang only) Future additions of new tests should take care to not test unrelated stuff. Instead a distinct configuration should be added. Additionally there is a Mac OS test with clang and all features disabled. Known issues: - Apparently the git commit is not properly detected during build. The HEAD currently shows as 2.4-dev0.
2020-07-28 21:00:35 +00:00
def clean_os(os):
if os == "ubuntu-latest":
return "Ubuntu"
elif os == "macos-latest":
return "macOS"
return os.replace("-latest", "")
def clean_ssl(ssl):
return ssl.replace("_VERSION", "").lower()
def clean_compression(compression):
return compression.replace("USE_", "").lower()
def get_asan_flags(cc):
if cc == "clang":
return [
"USE_OBSOLETE_LINKER=1",
'DEBUG_CFLAGS="-g -fsanitize=address"',
'LDFLAGS="-fsanitize=address"',
'CPU_CFLAGS.generic="-O1"',
]
raise ValueError("ASAN is only supported for clang")
matrix = []
# Ubuntu
os = "ubuntu-latest"
TARGET = "linux-glibc"
for CC in ["gcc", "clang"]:
matrix.append(
{
"name": "{}, {}, no features".format(clean_os(os), CC),
"os": os,
"TARGET": TARGET,
"CC": CC,
"FLAGS": [],
}
)
matrix.append(
{
"name": "{}, {}, all features".format(clean_os(os), CC),
"os": os,
"TARGET": TARGET,
"CC": CC,
"FLAGS": [
"USE_ZLIB=1",
"USE_PCRE=1",
"USE_PCRE_JIT=1",
"USE_LUA=1",
"USE_OPENSSL=1",
"USE_SYSTEMD=1",
"USE_WURFL=1",
"WURFL_INC=contrib/wurfl",
"WURFL_LIB=contrib/wurfl",
"USE_DEVICEATLAS=1",
"DEVICEATLAS_SRC=contrib/deviceatlas",
"EXTRA_OBJS=contrib/prometheus-exporter/service-prometheus.o",
"USE_51DEGREES=1",
"51DEGREES_SRC=contrib/51d/src/pattern",
CI: Expand use of GitHub Actions for CI Travis is becoming overall increasingly unreliable lately. We've already seen that the timing sensitive tests regularly fail and thus they were disabled. Additionally they recently announced a new pricing model that caps the number of minutes for Open Source projects: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing GitHub Actions VMs are working well, possibly allowing to use custom runners for special tasks in the future. In addition to this better performance its workflow configuration language is more expressive compared to the Travis CI one. Specifically the build matrix does not need to be specified in YAML. Instead it can be generated ad-hoc using a script. This allows us to cleanly define the various build configurations without having an unreadable 80 line mess where the flags are inconsistently activated. As an example in the current Travis CI configuration the prometheus exporter is tested together with LibreSSL 2.9.2 for whatever reason. In addition to all the previous points the UI of Travis is not that nice. On GitHub you are just seeing that "Travis failed" without any details which exact job failed. This requires you to visit the slow Travis page and look up the details there. GitHub Actions creates a single entry for each configuration that is tested, allowing you to see the details without needing to leave GitHub. This new GitHub Actions workflow aims to reproduce the configurations tested in Travis. It comes close, but is not completely there, yet. Consider this patch a proof of concept that will evolve in the future, ideally with Ilya's expertise. The current configurations are as follows. Each one is tested with both gcc and clang. - All features disabled (no USE flags) - All features enabled (all USE flags) - Standalone test of each of the supported compression libraries: - USE_ZLIB=1 - USE_SLZ=1 - Standalone test of various SSL libraries: - stock (the SSL installed by default on the VM) - OpenSSL 1.0.2u - LibreSSL 2.9.2, 3.0.2, 3.1.1 - All features enabled with ASAN (clang only) Future additions of new tests should take care to not test unrelated stuff. Instead a distinct configuration should be added. Additionally there is a Mac OS test with clang and all features disabled. Known issues: - Apparently the git commit is not properly detected during build. The HEAD currently shows as 2.4-dev0.
2020-07-28 21:00:35 +00:00
],
}
)
for compression in ["USE_SLZ=1", "USE_ZLIB=1"]:
matrix.append(
{
"name": "{}, {}, gz={}".format(
clean_os(os), CC, clean_compression(compression)
),
"os": os,
"TARGET": TARGET,
"CC": CC,
"FLAGS": [compression],
}
)
for ssl in [
"stock",
"OPENSSL_VERSION=1.0.2u",
"LIBRESSL_VERSION=2.9.2",
"LIBRESSL_VERSION=3.2.4",
"BORINGSSL=yes",
CI: Expand use of GitHub Actions for CI Travis is becoming overall increasingly unreliable lately. We've already seen that the timing sensitive tests regularly fail and thus they were disabled. Additionally they recently announced a new pricing model that caps the number of minutes for Open Source projects: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing GitHub Actions VMs are working well, possibly allowing to use custom runners for special tasks in the future. In addition to this better performance its workflow configuration language is more expressive compared to the Travis CI one. Specifically the build matrix does not need to be specified in YAML. Instead it can be generated ad-hoc using a script. This allows us to cleanly define the various build configurations without having an unreadable 80 line mess where the flags are inconsistently activated. As an example in the current Travis CI configuration the prometheus exporter is tested together with LibreSSL 2.9.2 for whatever reason. In addition to all the previous points the UI of Travis is not that nice. On GitHub you are just seeing that "Travis failed" without any details which exact job failed. This requires you to visit the slow Travis page and look up the details there. GitHub Actions creates a single entry for each configuration that is tested, allowing you to see the details without needing to leave GitHub. This new GitHub Actions workflow aims to reproduce the configurations tested in Travis. It comes close, but is not completely there, yet. Consider this patch a proof of concept that will evolve in the future, ideally with Ilya's expertise. The current configurations are as follows. Each one is tested with both gcc and clang. - All features disabled (no USE flags) - All features enabled (all USE flags) - Standalone test of each of the supported compression libraries: - USE_ZLIB=1 - USE_SLZ=1 - Standalone test of various SSL libraries: - stock (the SSL installed by default on the VM) - OpenSSL 1.0.2u - LibreSSL 2.9.2, 3.0.2, 3.1.1 - All features enabled with ASAN (clang only) Future additions of new tests should take care to not test unrelated stuff. Instead a distinct configuration should be added. Additionally there is a Mac OS test with clang and all features disabled. Known issues: - Apparently the git commit is not properly detected during build. The HEAD currently shows as 2.4-dev0.
2020-07-28 21:00:35 +00:00
]:
flags = ["USE_OPENSSL=1"]
if ssl != "stock":
flags.append("SSL_LIB=${HOME}/opt/lib")
flags.append("SSL_INC=${HOME}/opt/include")
matrix.append(
{
"name": "{}, {}, ssl={}".format(clean_os(os), CC, clean_ssl(ssl)),
"os": os,
"TARGET": TARGET,
"CC": CC,
"ssl": ssl,
"FLAGS": flags,
}
)
# ASAN
os = "ubuntu-latest"
CC = "clang"
TARGET = "linux-glibc"
matrix.append(
{
"name": "{}, {}, ASAN, all features".format(clean_os(os), CC),
"os": os,
"TARGET": TARGET,
"CC": CC,
"FLAGS": get_asan_flags(CC)
+ [
"USE_ZLIB=1",
"USE_PCRE=1",
"USE_PCRE_JIT=1",
"USE_LUA=1",
"USE_OPENSSL=1",
"USE_SYSTEMD=1",
"USE_WURFL=1",
"WURFL_INC=contrib/wurfl",
"WURFL_LIB=contrib/wurfl",
"USE_DEVICEATLAS=1",
"DEVICEATLAS_SRC=contrib/deviceatlas",
"EXTRA_OBJS=contrib/prometheus-exporter/service-prometheus.o",
"USE_51DEGREES=1",
"51DEGREES_SRC=contrib/51d/src/pattern",
CI: Expand use of GitHub Actions for CI Travis is becoming overall increasingly unreliable lately. We've already seen that the timing sensitive tests regularly fail and thus they were disabled. Additionally they recently announced a new pricing model that caps the number of minutes for Open Source projects: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing GitHub Actions VMs are working well, possibly allowing to use custom runners for special tasks in the future. In addition to this better performance its workflow configuration language is more expressive compared to the Travis CI one. Specifically the build matrix does not need to be specified in YAML. Instead it can be generated ad-hoc using a script. This allows us to cleanly define the various build configurations without having an unreadable 80 line mess where the flags are inconsistently activated. As an example in the current Travis CI configuration the prometheus exporter is tested together with LibreSSL 2.9.2 for whatever reason. In addition to all the previous points the UI of Travis is not that nice. On GitHub you are just seeing that "Travis failed" without any details which exact job failed. This requires you to visit the slow Travis page and look up the details there. GitHub Actions creates a single entry for each configuration that is tested, allowing you to see the details without needing to leave GitHub. This new GitHub Actions workflow aims to reproduce the configurations tested in Travis. It comes close, but is not completely there, yet. Consider this patch a proof of concept that will evolve in the future, ideally with Ilya's expertise. The current configurations are as follows. Each one is tested with both gcc and clang. - All features disabled (no USE flags) - All features enabled (all USE flags) - Standalone test of each of the supported compression libraries: - USE_ZLIB=1 - USE_SLZ=1 - Standalone test of various SSL libraries: - stock (the SSL installed by default on the VM) - OpenSSL 1.0.2u - LibreSSL 2.9.2, 3.0.2, 3.1.1 - All features enabled with ASAN (clang only) Future additions of new tests should take care to not test unrelated stuff. Instead a distinct configuration should be added. Additionally there is a Mac OS test with clang and all features disabled. Known issues: - Apparently the git commit is not properly detected during build. The HEAD currently shows as 2.4-dev0.
2020-07-28 21:00:35 +00:00
],
}
)
# macOS
os = "macos-latest"
TARGET = "osx"
for CC in ["clang"]:
matrix.append(
{
"name": "{}, {}, no features".format(clean_os(os), CC),
"os": os,
"TARGET": TARGET,
"CC": CC,
"FLAGS": [],
}
)
# Print matrix
print(json.dumps(matrix, indent=4, sort_keys=True))
print("::set-output name=matrix::{}".format(json.dumps({"include": matrix})))