Commit Graph

142 Commits

Author SHA1 Message Date
Willy Tarreau
bb7e62b98a DEV: patchbot: prepare for new version 3.1-dev
The bot will now load the prompt for the upcoming 3.1 version so we have
to rename the files and update their contents to match the current version.
2024-05-29 14:38:21 +02:00
Christopher Faulet
5df54f4796 DEV: flags/peers: Decode PEER and PEERS flags
Decode peer and peers flags via peer_show_flags() and peers_show_flags()
functions.
2024-04-25 18:29:58 +02:00
Willy Tarreau
eb3d5f464d MEDIUM: ring: use the topmost bit of the tail as a lock
We're now locking the tail while looking for some room in the ring. In
fact it's still while writing to it, but the goal definitely is to get
rid of the lock ASAP. For this we reserve the topmost bit of the tail
as a lock, which may have as a possible visible effect that buffers will
be limited to 2GB instead of 4GB on 32-bit machines (though in practise,
good luck for allocating more than 2GB contiguous on 32-bit), but in
practice since the size is read with atol() and some operating systems
limit it to LONG_MAX unless passing negative numbers, the limit is
already there.

For now the impact on x86_64 is significant (drop from 2.35 to 1.4M/s
on 48 threads on EPYC 24 cores) but this situation is only temporary
so that changes can be reviewable and bisectable.

Other approaches were attempted, such as using XCHG instead, which is
slightly faster on x86 with low thread counts (but causes more write
contention), and forces readers to stall under heavy traffic because
they can't access a valid value for the queue anymore. A CAS requires
preloading the value and is les good on ARMv8.1. XADD could also be
considered with 12-13 upper bits of the offset dedicated to locking,
but that looks overkill.
2024-03-25 17:34:19 +00:00
Willy Tarreau
dd8ea5d928 MEDIUM: ring: align the head and tail fields in the ring_storage structure
We really want to let the readers and writers act on different areas, so
we want to have the tail and the head on separate cache lines, themselves
separate from the rest of the ring. Doing so improves the performance from
2.15 to 2.35M msg/s at 48 threads on a 24-core EPYC.

This increases the header space from 32 to 192 bytes when threads are
enabled. But since we already have the header size available in the file,
haring remains able to detect the aligned vs unaligned formats and call
dump_v2a() when aligned is detected.
2024-03-25 17:34:19 +00:00
Willy Tarreau
bf3dead20c MEDIUM: ring: remove the struct buffer from the ring
The purpose is to store a head and a tail that are independent so that
we can further improve the API to update them independently from each
other.

The struct was arranged like the original one so that as long as a ring
has its head set to zero (i.e. no recycling) it will continue to work.
The new format is already detectable thanks to the "rsvd" field which
indicates the number of reserved bytes at the beginning. It's located
where the buffer's area pointer previously was, so that older versions
of haring can continue to open the ring in repair mode, and newer ones
can use the fact that the upper bits of that variable are zero to guess
that it's working with the new format instead of the old one. Also let's
keep in mind that the layout will further change to place some alignment
constraints.

The haring tool will thus updated based on this and it detects that the
rsvd field is smaller than a page and that the sum of it with the size
equals the mapped size, in which case it uses the new dump_v2() function
instead of dump_v1(). The new function also creates a buffer from the
ring's area, size, head and tail and calls the generic one so that no
other code had to be adapted.
2024-03-25 17:34:19 +00:00
Willy Tarreau
88e141b823 DEV: haring: automatically use the advertised ring header size
Instead of emitting a warning, since we don't need the ring struct
anymore, we can just read what we need, parse the buffer and use the
advertised offset. Thus for now -f is simply ignored.
2024-03-09 11:23:52 +01:00
Willy Tarreau
77d7c35243 DEV: haring: split the code between ring and buffer
By splitting the initialization and the parsing of the ring, we'll ease
the support for multiple ring sizes and get rid of the annoyances of the
optional lock.
2024-03-09 11:23:52 +01:00
Willy Tarreau
4dddbb63a0 DEV: haring: make haring not depend on the struct ring itself
haring needs to be self-sufficient about the ring format so that it continues
to build when the ring API changes. Let's import the struct ring definition
and call it "ring_v1".
2024-03-09 11:23:52 +01:00
Willy Tarreau
ab8928b9db BUILD: address a few remaining calloc(size, n) cases
In issue #2427 Ilya reports that gcc-14 rightfully complains about
sizeof() being placed in the left term of calloc(). There's no impact
but it's a bad pattern that gets copy-pasted over time. Let's fix the
few remaining occurrences (debug.c, halog, udp-perturb).

This can be backported to all branches, and the irrelevant parts dropped.
2024-02-10 11:37:27 +01:00
Willy Tarreau
f3a19e70e8 DEV: h2: support hex-encoded data sequences in mkhdr
For HPACK-encoded headers (particularly with huffman encoding), it's
really necessary to support hex sequences as they appear in RFC7541
examples, so let's support hex digit pairs with -R.

Now it's possible to do this to send GET https://www.example.com/ :

    (dev/h2/mkhdr.sh -t p; dev/h2/mkhdr.sh -t s;
     dev/h2/mkhdr.sh -t h -i 1 -f es,eh \
     -R '8286 8441 0f77 7777 2e65 7861 6d70 6c65 2e63 6f6d ') | nc 0 8080
2024-01-12 18:59:59 +01:00
Willy Tarreau
5bf19f8b57 DEV: h2: add support for multiple flags in mkhdr
The mkhdr script did not support passing multiple flags at once using
symbolic names. That's now done.
2024-01-12 18:59:59 +01:00
Willy Tarreau
3e31e2927c DEV: phash: add a trivial perfect hash generator for integers
We have a few places where we're checking many status codes. The sets
are so small that just a multiply, a shift and an optional xor are
sufficient to turn that into a smaller set. The program used to produce
them is hackish as it allows to easily fiddle with various operations
manually and experiment by brute-forcing a pair of integers for the
mul and the xor. It also supports producing an incomplete table that
gives an easier modulo operation. Let's commit it as-is so that it can
be reused later (e.g. if new status codes are introduced for example).
2024-01-11 15:10:08 +01:00
Willy Tarreau
b6c1f5c7d9 DEV: patchbot: produce a verdict for too long commit messages
Some rare commit messages area really too large because they contain
code excerpts in the message body or are release commits with their
changelog. In this case, instead of leaving an empty file that will
be silently ignored, let's produce an output message indicating that
the verdict is uncertain, with an explanation stating that there was
an error.
2024-01-09 14:46:04 +01:00
Willy Tarreau
57f8a4e879 DEV: h2: support passing raw data for a frame
With -r it's possible to pass raw data that will be interpreted by
printf so it even supports \x sequences. E.g. for a RST_STREAM, let's
just use \x00\x00\x00\x00.
2024-01-06 11:02:37 +01:00
Willy Tarreau
ec8ea73bc0 DEV: h2: add the preface as well in supported output types
It's annoying to always have to append it from a capture, let's also
support producing the preface.
2024-01-06 11:02:37 +01:00
Willy Tarreau
57b6c17099 DEV: h2: add the ability to emit literals in mkhdr
It's sometimes needed to be able to build a headers frame, let's
add this by permitting to encode key/value pairs as literal headers.
2024-01-06 11:02:37 +01:00
Ilya Shipitsin
8705e45964 CLEANUP: assorted typo fixes in the code and comments
This is 38th iteration of typo fixes
2024-01-02 10:19:48 +01:00
Willy Tarreau
71f626e3e2 DEV: patchbot: allow to show/hide backported patches
In order to spot old patches marked "wait" that have not yet been
backported, it's convenient to be able to click "all" to start the
review from the first patch, deselect "no", "uncertain" and "backported",
leaving only "wait" and "yes". This will reveal all pending patches that
have still not yet been backported, including those prior to the last
review, allowing to reconsider older patches marked "wait" that have
not yet been picked.
2023-12-19 17:01:35 +01:00
Willy Tarreau
896d452015 DEV: patchbot: use checked buttons as reference instead of internal table
The statuses[] table was pre-filled from the shell code during
initialization based on the evaluation and the buttons pre-checked
accordingly, but upon reload, the checked buttons are preserved and
the statuses reinitialized, leading to a different status and color
on lines that were changed.

In practice we don't need this table and we can directly check each
button's state. This makes sure that displayed state is consistent
with checked buttons and allows to preserve the statuses upon reloads
to benefit from updates. Only the start of the review is reset upon
reload now (this allows to consider latest backport state). Of course,
a full reload (shift-ctrl-R) continues to reset the form.
2023-12-19 16:22:04 +01:00
Willy Tarreau
693da29ab7 DEV: patchbot: add the AI-based bot to pre-select candidate patches to backport
This is a set of scripts, prompts and howtos to have an LLM read commit
messages and determine with great accuracy whether the patch's author
intended for the patch to be backported ASAP, backported after some time,
not backported, or unknown state. It provides all this in an interactive
interface making it easy to adjust choices and proceed with what was
selected. This has been improving over the last 9 months, as helped to
spot patches for a handful of backport sessions, and was only limited by
usability issues (UI). Now that these issues are solved, let's commit the
tool in its current working state. It currently runs every hour in a
crontab for me and started to prove useful since the last update, so it
should be considered in a usable state now, especially since this latest
update reaches close to 100% accuracy compared to a human choice, so it
saves precious development time and may allow stable releases to be
emitted more regularly.

There's detailed readme, please read it before complaining about the
ugliness of the UI :-)
2023-12-18 20:50:51 +01:00
Amaury Denoyelle
e1e6fbb3fd DEV: sslkeylogger: handle file opening error
Prevent a Lua error if output file cannot be opened when logging SSL
keys. Report a warning instead with the error description.
2023-10-03 15:23:35 +02:00
Willy Tarreau
cbbee15462 CLEANUP: ring: rename the ring lock "RING_LOCK" instead of "LOGSRV_LOCK"
The ring lock was initially mostly used for the logs and used to inherit
its name in lock stats. Now that it's exclusively used by rings, let's
rename it accordingly.
2023-09-20 21:38:33 +02:00
Willy Tarreau
67da85fa4c DEV: flags/show-sess-to-flags: properly decode fd.state
fd.state is reported without the "0x" prefix in show sess, let's support
this during decoding.

This may be backported to all versions supporting this utility.
2023-08-14 08:48:49 +02:00
Amaury Denoyelle
f1df006ffe DEV: add a Lua helper script for SSL keys logging
This script can be used through a http-request rules to log SSL keys for
traffic on a dedicated frontend. The resulting file can then be injected
into wireshark to decipher the corresponding network capture.
2023-05-24 16:08:23 +02:00
Willy Tarreau
2fab37eaf3 DEV: flags/show-sess-to-flags: add support for color output
Highlighting a few fields helps spot them, but only if there are not too
many. What is done here is the following:
  - the first line of each stream is highlighted in white (helps find
    beginning/end in long dumps
  - fields in the form name=value where value starts with upper case
    letters are considered as a state dump (e.g. stconn state) and are
    also highlighted. This results in ~20 pairs. In this case the name
    and value use two different colors (cyan vs yellow) to further help
    find what is being looked for

This is only done when the output is a terminal or when --color=always
is passed. It's also possible to disable it with --color=never or
--no-color.
2023-05-10 17:48:00 +02:00
Willy Tarreau
fd1047ae6e DEV: flags/show-sess-to-flags: only retrieve hex digits from hex fields
Some fields are followed by a comma or a closing parenthesis and we
take them because we read everything that's not a space. Better be
stricter, we're causing warnings about incorrect hex format when
they're passed to printf.
2023-05-10 17:46:10 +02:00
Willy Tarreau
058f246fc5 DEV: flags: add missing stream flags to show-sess-to-flags
I managed to miss strm.flg which appears at the top, in part because
there was no "stream" context. Let's create one and decode the flags.
2023-05-10 17:18:15 +02:00
Willy Tarreau
c147171d57 DEV: flags: add a script to decode most flags in the "show sess all" output
This ugly script decodes most flags in the "show sess all" output and
summarizes them after each stream in an aligned format that makes it
relatively easy to spot the ones of interest. It relies on "flags", and
if not found in various places, will replace its output with a copy-
pastable command that will produce it. h3/quic are not covered yet.

In order to compact the output format, the front and back stream conns
and connections are noted "f.sc", "f.co", "f.h1s", "b.co" etc. That's
sufficiently understandable and entries are grouped by context anyway.

Example of output:

0x7f51a43f7e30: [09/May/2023:20:28:57.183945] id=53704 proto=tcpv4 source=xx.xx.xx.xx:xxxx
  flags=0x100c4a, conn_retries=0, conn_exp=<NEVER> conn_et=0x000 srv_conn=0x464e4e0, pend_pos=(nil) waiting=0 epoch=0xa2
  frontend=public (id=2 mode=http), listener=SSL (id=4) addr=xx.xx.xx.xx:443
  backend=static (id=7 mode=http) addr=xx.xx.xx.xx:50916
  server=srv1 (id=1) addr=xx.xx.xx.xx:80
  task=0x7f51a42190a0 (state=0x00 nice=0 calls=2 rate=0 exp=59s tid=1(1/1) age=0s)
  txn=0x7f51a4397300 flags=0x43000 meth=1 status=200 req.st=MSG_DONE rsp.st=MSG_DATA req.f=0x4c rsp.f=0x0d
  scf=0x57af8b0 flags=0x00000482 state=EST endp=CONN,0x7f51a4269fa0,0x04005001 sub=3 rex=59s wex=59s
      h2s=0x7f51a4269fa0 h2s.id=1 .st=HCR .flg=0x207001 .rxbuf=0@(nil)+0/0
      .sc=0x57af8b0(.flg=0x00000482 .app=0x7f51a43f7e30) .sd=0x5832eb0(.flg=0x04005001)
      .subs=0x57af8c8(ev=3 tl=0x7f51a40dd390 tl.calls=4 tl.ctx=0x57af8b0 tl.fct=sc_conn_io_cb)
      h2c=0x7f51a44144d0 h2c.st0=FRH .err=0 .maxid=1 .lastid=-1 .flg=0x0200 .nbst=1 .nbsc=1
      .fctl_cnt=0 .send_cnt=1 .tree_cnt=1 .orph_cnt=0 .sub=3 .dsi=0 .dbuf=0@(nil)+0/0
      .mbuf=[3..4|32],h=[32730@0x57ec920+16431/32768],t=[32730@0x7f51a4322660+16431/32768] .task=0x5722d40 .exp=<NEVER>
      co0=0x5727120 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM target=LISTENER:0x1cc62d0
      flags=0x80000300 fd=30 fd.state=411 updt=0 fd.tmask=0x2
  scb=0x7f51a4064710 flags=0x00001211 state=EST endp=CONN,0x7f51a433eb50,0x011c0001 sub=0 rex=59s wex=<NEVER>
      h1s=0x7f51a433eb50 h1s.flg=0x4094 .sd.flg=0x11c0001 .req.state=MSG_DONE .res.state=MSG_DATA
      .meth=GET status=200 .sd.flg=0x011c0001 .sc.flg=0x00001211 .sc.app=0x7f51a43f7e30 .subs=(nil)
      h1c=0x7f51a4218b70 h1c.flg=0x80000020 .sub=0 .ibuf=32704@0x57d48c0+56/32768 .obuf=0@(nil)+0/0 .task=0x7f51a408ebb0 .exp=<NEVER>
      co1=0x57264e0 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM target=SERVER:0x464e4e0
      flags=0x00000300 fd=26 fd.state=10122 updt=0 fd.tmask=0x2
  req=0x7f51a43f7e50 (f=0x21840000 an=0x48000 pipe=0 tofwd=0 total=142)
      an_exp=<NEVER> buf=0x7f51a43f7e58 data=(nil) o=0 p=0 i=0 size=0
      htx=0x818e40 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
  res=0x7f51a43f7ea0 (f=0xa0040101 an=0x24000000 pipe=0 tofwd=-1 total=163472)
      an_exp=<NEVER> buf=0x7f51a43f7ea8 data=0x7f51a43c2f50 o=16328 p=16328 i=16440 size=32768
      htx=0x7f51a43c2f50 flags=0x0 size=32720 data=16328 used=1 wrap=NO extra=3852188
  -----------------------------------
  task.state            0  0
  txn.meth              1  GET
  txn.flg         0x43000  TX_NOT_FIRST TX_CACHE_COOK TX_CACHEABLE
  txn.req.flg        0x4c  HTTP_MSGF_BODYLESS HTTP_MSGF_VER_11 HTTP_MSGF_XFER_LEN
  txn.rsp.flg         0xd  HTTP_MSGF_VER_11 HTTP_MSGF_XFER_LEN HTTP_MSGF_CNT_LEN
  f.sc.flg          0x482  SC_FL_RCV_ONCE SC_FL_WONT_READ SC_FL_EOI
  f.sc.sd.flg   0x4005001  SE_FL_HAVE_NO_DATA SE_FL_EOI SE_FL_NOT_FIRST SE_FL_T_MUX
  f.h2s.flg      0x207001  H2_SF_MORE_HTX_DATA H2_SF_HEADERS_RCVD H2_SF_OUTGOING_DATA H2_SF_HEADERS_SENT H2_SF_ES_RCVD
  f.h2s.sd.flg  0x4005001  SE_FL_HAVE_NO_DATA SE_FL_EOI SE_FL_NOT_FIRST SE_FL_T_MUX
  f.h2c.flg         0x200  H2_CF_DEM_SHORT_READ
  f.co.flg     0x80000300  CO_FL_XPRT_TRACKED CO_FL_XPRT_READY CO_FL_CTRL_READY
  f.co.fd.st        0x19b  FD_POLL_IN FD_EV_ERR_RW FD_EV_READY_R FD_EV_ACTIVE_W FD_EV_ACTIVE_R 0x8
  b.sc.flg         0x1211  SC_FL_SND_NEVERWAIT SC_FL_NEED_ROOM SC_FL_NOHALF SC_FL_ISBACK
  b.sc.sd.flg   0x11c0001  SE_FL_WAIT_DATA SE_FL_WANT_ROOM SE_FL_RCV_MORE SE_FL_MAY_SPLICE SE_FL_T_MUX
  b.h1s.sd.flg  0x11c0001  SE_FL_WAIT_DATA SE_FL_WANT_ROOM SE_FL_RCV_MORE SE_FL_MAY_SPLICE SE_FL_T_MUX
  b.h1s.flg        0x4094  H1S_F_HAVE_O_CONN H1S_F_NOT_FIRST H1S_F_WANT_KAL H1S_F_RX_CONGESTED
  b.h1c.flg    0x80000020  H1C_F_IS_BACK H1C_F_IN_FULL
  b.co.flg          0x300  CO_FL_XPRT_READY CO_FL_CTRL_READY
  b.co.fd.st       0x278a  FD_POLL_OUT FD_POLL_PRI FD_POLL_IN FD_EV_ERR_RW FD_EV_READY_R 0x2008
  req.flg      0x21840000  CF_FLT_ANALYZE CF_DONT_READ CF_AUTO_CONNECT CF_WROTE_DATA
  req.ana         0x48000  AN_REQ_FLT_END AN_REQ_HTTP_XFER_BODY
  req.htx.flg           0  0
  res.flg      0xa0040101  CF_ISRESP CF_FLT_ANALYZE CF_WROTE_DATA CF_WRITE_EVENT CF_READ_EVENT
  res.ana      0x24000000  AN_RES_FLT_END AN_RES_HTTP_XFER_BODY
  res.htx.flg           0  0
  -----------------------------------
2023-05-09 20:38:33 +02:00
Willy Tarreau
b83bf68ec0 DEV: haring: update readme to suggest using the same build options for haring
It's not necessarily obvious so better suggest it there to use the same
build options, and indicate the tradeoffs (e.g. depend on more libs).
2023-05-04 08:13:44 +02:00
Willy Tarreau
46e0ea33e2 DEV: haring: automatically disable DEBUG_STRICT
Ideally haring should be compiled with the same options as haproxy so
that ring headers have the same size (e.g. with/without locks, with/
without lock debugging). But when enabling DEBUG_STRICT, BUG_ON() is
enabled and breaks the build by making references to complain() and
ha_backtrace_to_stderr().

Let's just disable DEBUG_STRICT before opening include files. This is
sufficient to address the problem.

This may be backorted to older versions that include haring.
2023-05-04 08:09:02 +02:00
Willy Tarreau
e8ae99b111 DEV: h2: support reading frame payload from a file
Now we can build a series of data frames by reading from a file and
chunking it into frames of requested length. It's mostly useful for
data frames (e.g. post). One way to announce these upfront is to
capture the output of curl use without content-length:

  $ nc -lp4446 > post-h2-nocl.bin
  $ curl -v --http2-prior-knowledge http://127.0.0.1:4446/url -H "content-length:" -d @/dev/null

Then just change the 5th byte from the end from 1 to 0 to remove the
end-of-stream bit, it will allow to chain a file, then to send an
empty DATA frame with ES set :

  $ (dev/h2/mkhdr.sh -i 1 -t data -d CHANGELOG;
     dev/h2/mkhdr.sh -i 1 -t data -l 0 -f es) > h2-data-changelog.bin

Then post that to the server:
  $ cat post-h2-nocl.bin h2-data-changelog.bin | nc 0 4446
2023-04-26 11:36:02 +02:00
Willy Tarreau
78bb934607 DEV: h2: add a script "mkhdr" to build h2 frames from scratch
It's a real pain to try to trigger certain edge cases in h2, so let's
write a simple tool aiming at creating frames.
2023-04-26 11:35:45 +02:00
Willy Tarreau
d9c7188633 MEDIUM: ring: make the offset relative to the head/tail instead of absolute
The ring's offset currently contains a perpetually growing custor which
is the number of bytes written from the start. It's used by readers to
know where to (re)start reading from. It was made absolute because both
the head and the tail can change during writes and we needed a fixed
position to know where the reader was attached. But this is complicated,
error-prone, and limits the ability to reduce the lock's coverage. In
fact what is needed is to know where the reader is currently waiting, if
at all. And this location is exactly where it stored its count, so the
absolute position in the buffer (the seek offset from the first storage
byte) does represent exactly this, as it doesn't move (we don't realign
the buffer), and is stable regardless of how head/tail changes with writes.

This patch modifies this so that the application code now uses this
representation instead. The most noticeable change is the initialization,
where we've kept ~0 as a marker to go to the end, and it's now set to
the tail offset instead of trying to resolve the current write offset
against the current ring's position.

The offset was also used at the end of the consuming loop, to detect
if a new write had happened between the lock being released and taken
again, so as to wake the consumer(s) up again. For this we used to
take a copy of the ring->ofs before unlocking and comparing with the
new value read in the next lock. Since it's not possible to write past
the current reader's location, there's no risk of complete rollover, so
it's sufficient to check if the tail has changed.

Note that the change also has an impact on the haring consumer which
needs to adapt as well. But that's good in fact because it will rely
on one less variable, and will use offsets relative to the buffer's
head, and the change remains backward-compatible.
2023-02-24 09:26:30 +01:00
Aurelien DARRAGON
532ebee38e DEV: hpack: fix trash build regression
Since 7d84439 ("BUILD: hpack: include global.h for the trash that is needed
in debug mode"), hpack decode tool fails to compile on targets that enable
USE_THREAD. (ie: linux-glibc target as reported by Christian Ruppert)

When building hpack devtool, we are including src/hpack-dec.c as a dependency.
src/hpack-dec.c relies on the global trash whe debug mode is enabled.
But as we're building hpack tool with a limited scope of haproxy
sources, global trash (which is declared in src/chunk.c) is not available.
Thus, src/hpack-dec.c relies on a local 'trash' variable declared within
dev/hpack/decode.c

This used to work fine until 7d84439.
But now that global.h is explicitely included in src/hpack-dec.c,
trash variable definition from decode.c conflicts with the one from global.h:

  In file included from include/../src/hpack-dec.c:35,
                   from dev/hpack/decode.c:87:
  include/haproxy/global.h:52:35: error: thread-local declaration of 'trash' follows non-thread-local declaration
     52 | extern THREAD_LOCAL struct buffer trash;

Adding THREAD_LOCAL attribute to 'decode.c' local trash variable definition
makes the compiler happy again.

This should fix GH issue #2009 and should be backported to 2.7.
2023-01-27 10:22:20 +01:00
Willy Tarreau
e06ba90318 DEV: haring: add a new option "-r" to automatically repair broken files
In case a file-backed ring was not properly synced before being dumped,
the output can look bogus due to the head pointer not being perfectly
up to date. In this case, passing "-r" will make haring automatically
skip entries not starting with a zero, and resynchronize with the rest
of the messages.

This should be backported to 2.6.
2023-01-24 12:13:14 +01:00
Willy Tarreau
b714e11aaa DEV: tcploop: add minimal support for unix sockets
Since the tool permits to pass an FD bound for listening, it's convenient
to test haproxy's "bind fd@". Let's add support for UNIX sockets the same
way. -U needs to be passed to change the default address family, and the
address must contain a "/".

E.g.
  $ dev/tcploop/tcploop -U /tmp/ux L Xi ./haproxy -f fd1.cfg
2023-01-11 11:27:20 +01:00
Willy Tarreau
d8a6a32ad7 DEV: tcploop: add optional support for epoll
When -e is passed, epoll is used instead of poll. The FD is added
then removed around the call to epoll_wait() so that we don't need
to track it. The only purpose is to compare events reported by each
syscall.
2022-11-25 17:08:06 +01:00
Willy Tarreau
fd0974cf3f DEV: tcploop: do not report an error on POLLERR
Actually this breaks certain client tests where the server closes with
RST, it prevents from reading the final data so better not abort on
that.
2022-11-25 17:07:22 +01:00
Willy Tarreau
e5f72b8754 DEV: tcploop: add support for POLLRDHUP when supported
This is just in order to closer match what haproxy does.
2022-11-25 17:07:22 +01:00
Willy Tarreau
45c2779456 DEV: tcploop: centralize the polling code into wait_for_fd()
There are multiple call places for poll(), let's first centralize them
to make it easier to enhance it. All callers now use the wait_for_fd()
function which was extended to take a timeout and which can return the
indication that an error was seen.
2022-11-25 17:07:22 +01:00
Willy Tarreau
2afa42599c DEV: poll: add support for epoll
When called with -e, epoll is used instead of poll. The poller does
very little in this code (just checks for any event without waiting) but
that's sufficient to see return values.
2022-11-25 17:07:22 +01:00
Willy Tarreau
946d370d22 BUILD: flags: really restrict the cases where flags are exposed
A number of internal flags started to be exposed to external programs
at the location of their definition since commit 77acaf5af ("MINOR:
flags: add a new file to host flag dumping macros"). This allowed the
"flags" utility to decode many more of them and always correctly. The
condition to expose them was to rely on the preliminary definition of
EOF that indicates that stdio is already included. But this was a
wrong approach. It only guarantees that snprintf() can safely be used
but still causes large functions to be built. But stdio is often
included before some of these includes, so these heavy inline functions
actually have to be compiled in many cases. The result is that the
build time significantly increased, especially with fast compilers
like gcc -O0 which took +50% or TCC which took +100%!

This patch addresses the problem by instead relying on an explicit
macro HA_EXPOSE_FLAGS that the calling program must explicitly define
before including these files. flags.c does this and that's all. The
previous build time is now restored with a speed up of 20 to 50%
depending on the build options.
2022-11-24 08:32:27 +01:00
Willy Tarreau
35402852bd DEV: poll: indicate the FD's side in front of its value
Some interleaved dumps were hard to follow. By indicating which side
we're talking about it's easier. E.g:

  $ dev/poll/poll -v -s pol,clo -c pol
  #### BEGIN ####
  cmd #1 stp #0: con(c=4): ret=0
  cmd #1 stp #0: acc(l=3): ret=5
  cmd #1 stp #1: pol(s=5): ret=1 ev=0x4 (OUT)
  cmd #1 stp #2: clo(s=5): ret=0
  cmd #2 stp #1: pol(c=4): ret=1 ev=0x2005 (IN OUT RDHUP)
2022-11-17 10:56:35 +01:00
Willy Tarreau
05ed7e849c DEV: poll: strip the "do_" prefix from reported function names
This doesn't bring anything and makes the output less readable. Let's
just keep the action name.
2022-11-17 10:56:35 +01:00
Willy Tarreau
e61064162b DEV: poll: make the connect() step an action as well
Now the connect() step becomes an action. It's still implicit before
any -c/-s but it allows the listener to close() before connect()
happens, showing the polling status for this condition:

  $ dev/poll/poll -v -l clo -c pol
  #### BEGIN ####
  cmd #1 stp #1: do_clo(3): ret=0
  cmd #2 stp #0: do_con(4): ret=-1 (Connection refused)
  cmd #2 stp #1: do_pol(4): ret=1 ev=0x14 (OUT HUP)
  #### END ####

which differs from a case where the server closes the just accepted
connection:

  $ dev/poll/poll -v -s clo -c pol
  #### BEGIN ####
  cmd #1 stp #0: do_con(4): ret=0
  cmd #1 stp #0: do_acc(3): ret=5
  cmd #1 stp #1: do_clo(5): ret=0
  cmd #2 stp #1: do_pol(4): ret=1 ev=0x2005 (IN OUT RDHUP)
  #### END ####

It's interesting to see OUT+HUP since HUP indicates that both directions
were closed, hence nothing may be written now, thus OUT just wants the
write handler to be notified.
2022-11-17 10:56:35 +01:00
Willy Tarreau
8dd672523f BUILD: makefile: move default verbosity settings to include/make/verbose.mk
The $(Q), $(V), $(cmd_xx) handling needs to be reused in sub-project
makefiles and it's a pain to maintain inside the main makefile. Let's
just move that into a new subdir include/make/ with a dedicated file
"verbose.mk". It slightly cleans up the makefile in addition.
2022-11-17 10:56:35 +01:00
Willy Tarreau
d575661d40 BUILD: makefile: properly pass CC to sub-projects
The "poll" and "tcploop" sub-projects have their own makefiles. But
since the cmd_* commands were migrated from "echo" to $(info) with
make 3.81, the command is confusingly displayed in the top-level
makefile before entering the directory, even making one think that
the build occurred.

Let's instead propagate the verbosity level through the sub-projects
and let them adapt their own cmd_CC. For now this peans a little bit
of duplication for poll and tcploop.
2022-11-17 10:56:35 +01:00
Christopher Faulet
c8db114afc MINOR: flags/mux-fcgi: Decode FCGI connection and stream flags
The new functions fconn_show_flags() and fstrm_show_flags() decode the flags
state into a string, and are used by dev/flags:

$ /dev/flags/flags fconn 0x3100
fconn->flags = FCGI_CF_GET_VALUES | FCGI_CF_KEEP_CONN | FCGI_CF_MPXS_CONNS

./dev/flags/flags fstrm  0x3300
fstrm->flags = FCGI_SF_WANT_SHUTW | FCGI_SF_WANT_SHUTR | FCGI_SF_OUTGOING_DATA | FCGI_SF_BEGIN_SENT
2022-10-12 17:10:41 +02:00
Christopher Faulet
7c4b2ec09d MINOR: flags/mux-h1: decode H1C and H1S flags
The new functions h1c_show_flags() and h1s_show_flags() decode the flags
state into a string, and are used by dev/flags:

$ /dev/flags/flags h1c 0x2200
h1c->flags = H1C_F_ST_READY | H1C_F_ST_ATTACHED

./dev/flags/flags h1s 0x190
h1s->flags = H1S_F_BODYLESS_RESP | H1S_F_NOT_FIRST | H1S_F_WANT_KAL
2022-09-15 11:01:59 +02:00
Willy Tarreau
439be5838d MINOR: flags/mux-h2: decode H2C and H2S flags
The new functions h2c_show_flags() and h2s_show_flags() decode the flags
state into a string, and are used by dev/flags:

  $ ./dev/flags/flags h2c 0x0600
  h2c->flags = H2_CF_DEM_IN_PROGRESS | H2_CF_DEM_SHORT_READ

  $ ./dev/flags/flags h2s 0x7003
  h2s->flags = H2_SF_HEADERS_RCVD | H2_SF_OUTGOING_DATA | H2_SF_HEADERS_SENT \
             | H2_SF_ES_SENT | H2_SF_ES_RCVD
2022-09-12 19:33:07 +02:00