HAProxy Enterprise Updates & Release Notes

Key changes in the HAProxy Enterprise 3.0r1 release include

HAProxy Enterprise WAF

Support for base64 decoding when defining custom rules.

WAF-related Prometheus exporter metrics.

Support for atomic transactions via the Runtime API.

Process request body without Content-Type, even in raw mode.

Validate incoming data to ensure it contains valid UTF-8 characters, filtering out potentially harmful characters that may attempt to exploit vulnerabilities.

Global Profiling Engine

New dynamic peer support and better memory management. It’s no longer mandatory to list peers in the Global Profiling Engine configuration file; they are learned dynamically as it runs. The dynamic peers result in a lower memory footprint from peer reuse and session reuse.

New global bind directive and global-exp argument on the from directive.

Support for the new HTTP/2 protocol glitch stick table counters, glitch_cnt and glitch_rate.

Bot Management module

New botmgmt-profile section, scoring algorithms, and sample fetches to tune the module and increase observability.

Export Prometheus metrics from Bot Management module data.

The Bot Management module emits a warning if the data-hapee data file is older than 4 weeks instead of 2 weeks.

UDP module

Basic logging can be enabled by specifying the log directive and its arguments within the udp-lb section. The log output format only contains the source and destination addresses, bytes received or sent, the instance name, and the server, if available.

Simplified configuration with the support of the default-server directive.

DeviceAtlas module

The DeviceAtlas module has been updated to support the DeviceAtlas v3 data format.

51Degrees module

The 51Degrees module now supports the 51Degrees V4 data format.

Security Assertion Markup Language (SAML) module

The new SAML module acts as a SAML Service Provider, providing single sign-on (SSO) to any web application located behind an HAProxy Enterprise server. This updated module simplifies the SSO configuration.

Captcha module

A new Captcha module supports Google reCAPTCHA v2, v3, and reCAPTCHA Enterprise, plus hCaptcha, Friendly Captcha, and Cloudflare Turnstile.

SSL/TLS Enhancements

SSL performance is improved as a result of switching from OpenSSL 3.X to OpenSSL 1.1.1 as the default.

The new crt-store configuration section provides a flexible way to store and consume SSL certificates by separating the certificate storage from their use in a frontend. This provides better visibility for certificate information by moving it from external files and placing it into the main HAProxy Enterprise configuration. The crt-store section allows you to individually specify the locations of each certificate component; for example certificates files, key files, and OCSP response files. Aliases provide support for human-friendly names for referencing the certificates more easily on bind lines.

A new global ssl-security-level directive allows you to set a global OpenSSL internal security level. This enforces the appropriate checks and restrictions per the level specified.

Explicitly set default TLS certificates.

You can now set multiple default certificates using the new default-crt argument on your bind line. You can also indicate a default certificate in a crt-list by marking it with an asterisk.

Decrypt TLS 1.3 packet captures to backend servers.

New to this version, you can now enable the logging of TLS secrets for TLS connections between HAProxy Enterprise and backend servers. This allows you to then use those TLS secrets to use Wireshark to decipher your encrypted packet captures.

Dependency on libsystemd removed

This version hardens HAProxy Enterprise’s security with the removal of the dependency on the libsystemd library. This libsystemd library has many additional dependencies of its own, including the library at fault for the XZ Utils backdoor vulnerability. Removing the dependency on libsystemd means that HAProxy Enterprise isn’t exposed.

Prevent HAProxy from accepting traffic on privileged ports

Two new global settings, harden.reject-privileged-ports.tcp for TCP connections and harden.reject-privileged-ports.quic for QUIC connections, enable HAProxy Enterprise to ignore traffic when the client uses a privileged port (0 - 1023) as their source port. Clients using this range of ports are suspicious and such behavior often indicates spoofing, such as to launch DNS/NTP amplification attacks. The benefit in using these settings is that during DNS/NTP amplification attacks, CPU is reserved as HAProxy Enterprise will drop the packets on these privileged ports instead of parsing them.

HTTP/2 tuning options

You can limit the number of total streams processed per incoming connection for HTTP/2 using the global directive tune.h2.fe-max-total-streams. Once this limit is reached, HAProxy Enterprise will send a graceful GOAWAY frame informing the client that it will close the connection after all pending streams have been closed. Usually, this prompts clients to reestablish their connections. This is helpful in situations where there is an imbalance in the load balancing due to clients maintaining long-lived connections.

HAProxy Enterprise now counts protocol “glitches” and allows you to set a limit on them. Protocol glitches are HTTP/2 requests that are valid but have potential to cause issues; for example, sending a single header as a large number of CONTINUATION frames could cause a denial of service. You can also track the glitches in a stick table to identify buggy applications or misbehaving clients.

Diagnostic mode

Diagnostic mode (-dD) will now report on likely problematic ACL pattern values that look like known ACL/sample fetch keywords in addition to its other warnings about suspicious configuration statements. It does not prevent startup. This is helpful for troubleshooting problematic configurations.

Consistent hash server mapping

When load balancing using a hash-based algorithm (balance hash), HAProxy Enterprise must keep track of which server is which. The new hash-key directive allows you to specify how the node keys, that is, the hash for each server, are calculated. This applies to node keys where you have set hash-type to consistent. What this means is that if you have multiple instances of HAProxy Enterprise, each with its own list of servers potentially listed in different orders, routing will be consistent across your load balancers when you use the hash-key directive. There are three options for hash-key:

• id: the server’s numeric id if set, or its position in the server list
• addr: the server’s IP address
• addr-port: the server’s IP address and port

Memory limit

The memory limiting mechanism now uses RLMIT_DATA instead of RLIMIT_AS, which prevents an issue where the processes were reaching out-of-memory conditions below the configured limit when using -m to limit the total allocatable memory to a specified number of megabytes across all processes. In master-worker mode, the memory limit is applied separately to the master and its forked worker process, as memory is not shared between processes.

Unix sockets and namespaces

This version supports the namespace argument for UNIX sockets specified on bind and server lines. If a permissions issue arises, HAProxy Enterprise will log a permissions error instead of “failed to create socket”, since the capability cap_sys_admin is required for using namespaces for sockets.

gRPC

Prior to this version, cancellations, which are represented as RST_STREAM frames in HTTP/2 or as STOP_SENDING frames in HTTP/3, were not being properly relayed to the server. This has been fixed. Furthermore, this version adds new fetch methods that indicate whether the stream was canceled (aka aborted) and the error code.

Silence warnings on deprecated directives

If you want to use deprecated directives and silence warnings, you must also use the new expose-deprecated-directives global directive. This directive applies to deprecated directives that don’t have alternative solutions.

Emergency memory allocation

The mechanism for emergency memory allocation in low memory conditions has been improved for HTTP/1 and applets. Forward progress in processing is guaranteed as tasks are now queued according to their criticality and which missing buffer they require.

Write HAProxy Enterprise logs as JSON or CBOR

You can now format log lines as JSON and CBOR. When configuring a custom log format, you will indicate which to use, and then in parentheses set the key for each field.

Virtual and optional ACL and Map files

HAProxy Enterprise now supports virtual ACL and virtual map files.

• A virtual file doesn’t exist on disk and exists only in memory. The load balancer creates it as an in-memory representation during startup. Virtual files will not persist after restarting the load balancer.
• When you mark a file as optional, the load balancer will check for it on the filesystem, but if it doesn’t find it, it will assume that the file is virtual. Optional files are useful for allowing startup without a file on disk, but saving the file to disk at a later time for long-term persistence.
• You can use virtual and optional files in cases where you’ll use the Runtime API to dynamically update the values.

Report the key that matched in a map file

New map_*_key converters return the key that was matched, rather than the associated value, making it easier to view the reason why the load balancer rerouted a request or took some other action.

Track specific HTTP errors

Two global directives http-err-codes and http-fail-codes let you specify which HTTP status codes to track when using the http_err_cnt, http_err_rate, http_fail_cnt, and http_fail_rate stick table data types.

Stick table pointers

Prior to this version, you could use the Runtime API to remove or update a record in a stick table based on its key. You can now specify the record’s unique ID (its pointer) instead of its key to update or delete that record. Use the show table command to retrieve the pointer.

Set fwmark on packets to clients and servers

HAProxy Enterprise now supports setting an fwmark on connections to backend servers as well as to clients connected on the frontend using the set-fc-mark and set-bc-mark actions. These replace the set-mark action, which had applied only to frontends. Setting an fwmark on an IP packet classifies it so that, for example, it can use a specific routing table.

Set traffic priority

New in this version, HAProxy Enterprise can modify the header of an IP packet to include the Differentiated Services (DS) field on connections to backend servers in addition to frontend connections to clients. Use the set-fc-tos and set-bc-tos actions (referring to the old Type of Service header field, which has been superseded by DS).

Weights in log backends

In this version, you can set weights on server lines in your mode log backends. mode log is used for load balancing the Syslog protocol.

Support for UUIDv7 identifiers

Generate universally unique identifiers that use the UUIDv7 format by using the unique-id-format directive with the uuid fetch.

OCSP

You can set an HTTP forward proxy to reach the Internet to connect to the OCSP server. Add the ocsp-update.httpproxy global directive to indicate the proxy’s address. This is useful in cases where you’re operating in an air-gapped environment where the HAProxy Enterprise server does not have direct access to the Internet, and therefore can’t connect to the OCSP server.

The tune.ssl.oscp-update global directive has been renamed oscp-update, as an OCSP update is unrelated to SSL tuning.

Reverse HTTP

A new directive named pool-conn-name provides more flexibility when using reverse HTTP, allowing you to set the name to match with an expression of fetch methods and converters, instead of matching via SNI.

Prometheus

When configuring the Prometheus exporter, you can include a URL parameter named extra-counters that enables additional counters that provide information related to the HTTP protocol, QUIC, and SSL.

Runtime API

The new wait command for the Runtime API will wait for some specified time before then performing the command that follows it. You could use this to collect metrics on a certain interval, for example, or in cases where you need to wait until a server becomes removable (the server has been drained of connections) before running additional commands, such as del server.

A reload will no longer reset the HAProxy Stats page, as long as you call the new Runtime API command dump stats-file first to save the current state to a file and then load that file with the stats-file global configuration directive. Upon reload, the new processes set their counters to the previous values (as they appear in the stats file). Only proxy counters are supported which includes values for frontends, backends, servers, and listeners. Ensure that you’ve set a GUID on each frontend, backend, listen, and server object by using the new guid directive. Only objects to which you have assigned a GUID will have their stats persisted across the reload.

Static cookies for session persistence are now supported for dynamically added servers. Dynamic servers refer to servers that don’t have an explicit entry within your HAProxy Enterprise configuration file. They are dynamic in the sense that you can add them programmatically using Runtime API commands. Dynamic servers are valuable in cases where you may have many servers that scale with traffic: when traffic loads are high, you add more servers, and when traffic loads diminish, you remove servers. You can now use the add server command to add the server and specify its static cookie using just one command.

Running multiple Runtime API commands separated by new lines is no longer supported.

Previously, it was occasionally possible to successfully issue multiple commands by separating them with newlines, which had the potential to produce unexpected results for long-running commands that may only partially complete. A warning will now be emitted when a \n is detected in a command and the command will not be accepted. This change has also been backported to ensure that user scripts that utilize this behavior can be remedied.

Run multiple Runtime API commands with semicolons instead of new lines.

Original source

Key changes in the HAProxy Enterprise 3.1r1 release include:

ADFSPIP module

The new ADFSPIP module enables HAProxy Enterprise to give external clients access to web applications running in a Windows corporate network. HAProxy Enterprise becomes a proxy in front of Microsoft Active Directory Federation Services and web applications running inside the corporate network. It can route external clients to an AD FS sign-in page and to the web applications that they would otherwise not be able to reach.

Web Application Firewall module

The Web Application Firewall (WAF) has added support for WAF profiles. Defined in the new waf-profile section, a profile specifies a set of WAF parameters that can be applied to any WAF in the configuration. You can define multiple WAF profiles in a configuration.

Captcha module

New Captcha module options give you better control over the cookie that the module creates in the user’s browser. You can set the cookie’s domain, expiration, max age, path, SameSite option, and Secure option.

Global Profiling Engine

This version updates the Global Profiling Engine, which typically runs on a server separate from HAProxy Enterprise, to have new logging capabilities. You can send log messages to a file, a local syslog UNIX domain socket, or a remote syslog server.

UDP module

When configuring the UDP module via its udp-lb configuration section, you can now set the hash-type and hash-balance-factor directives to control how hash-based load balancing behaves with UDP traffic.

Stick table aggregator

For customers who have not migrated to the Global Profiling Engine, HAProxy Enterprise 3.1 introduces the new Stick Table Aggregator package version 2.1. It has the following enhancements:

• show aggrs now supports multiple buffers, ensuring all data is visible instead of just the first chunk.
• Active connections to down peers are deactivated by default, reducing unnecessary overhead. The legacy behavior can still be enabled with the legacy-active-connect argument.
• A new no-ascend option prevents data from being sent to up peers in multilayer environments.
• Multiple from lines are now supported per aggregation, offering greater flexibility in defining aggregation sources.
• The module now properly handles previously unsupported stick table data types.

Logging

The way that HAProxy Enterprise emits its logs is more flexible now with the introduction of log profiles, which let you assign names to your log formats. By defining log formats with names, you can choose the one best suited for each log server and even emit logs to multiple servers at the same time, each with its own format. Log profiles also allow you to write log messages when different events occur, such as when accepting a connection, receiving a request, connecting to a backend server, and receiving a response.

With the new do-log action, you can emit custom log messages throughout the processing of a request or response, allowing you to add debug statements that help you troubleshoot issues. Add the do-log action at various points of your configuration.

The option tcplog directive now allows an optional argument: clf. When enabled, CLF (Common Log Format) sends the same information as the non-CLF option, but in a standardized format that CLF log servers can parse.

set-retries action

The tcp-request content and http-request directives have a new action named set-retries that dynamically changes the number of times HAProxy Enterprise will try to connect to a backend server if it fails to connect initially.

quic-initial directive

The new quic-initial directive, which you can add to frontend, listen, and named defaults sections, gives you a way to deny QUIC (Quick UDP Internet Connections) packets early in the pipeline to waste no resources on unwanted traffic.

SPOE

In your configuration file, set mode to spop in the backend that contains the SPOE agents. This mode is now mandatory and automatically set for backends referenced by SPOEs. Configuring your backend in this way means that you are no longer required to use a separate configuration file for SPOE. This new spop backend mode adds flexibility to SPOE, optimizes traffic distribution among servers, improves performance, and will ultimately make the entire system more reliable, as future changes to the SPOE engine will only affect pieces specific to SPOE.

You can now pass variables from the main stream that’s processing a request to the child stream of a Stream Processing Offload Agent (SPOA). When reading a variable in an spop backend, prefix the variable scope with the letter p for parent stream. For example, instead of req, use preq. This works for these scopes: psess, ptxn, preq, and pres.

The following SPOE parameters were removed in this version and are silently ignored when present in the SPOE configuration:

• maxconnrate
• maxerrrate
• max-waiting-frames
• timeout hello
• timeout idle

Health checks

When defining health checks, add the new init-state argument to a server directive or server-template directive to control how quickly each server can return to handling traffic after restarting, coming out of maintenance mode, or adding the server through service discovery.

The option httpchk directive supports a new parameter for passing a host HTTP header with each health check to backend servers. This avoids having to specify carriage return and newline characters to do it, unlike previous versions.

Address families

When setting an address on a bind line, you can now set these address family prefixes:

• abnsz@: Zero-terminated abstract namespace. This lets you interconnect with software that determines the length of the namespace’s name by the length of the string, terminated by a null byte.
• mptcp@, mptcp4@, mptcp6@: MultiPath Transmission Control Protocol (MPTCP). MPTCP is a TCP extension that improves resource utilization, increases throughput, and responds quicker to failures.

Fetch methods

This version introduces the following fetch methods:

• ssl_c_san: Returns a string of comma-separated Subject Alt Name fields contained in the client certificate.
• ssl_fc_sigalgs_bin: Returns the content of the signatures_algorithms (13) TLS extension presented during the Client Hello.
• ssl_fc_supported_versions_bin: Returns the content of the supported_versions (43) TLS extension presented during the Client Hello.
• last_entity: Returns the identity of the last entity that was evaluated during stream analysis.
• waiting_entity: Returns the identity of the entity that was waiting to continue its processing when an error or a timeout was encountered.

Converters

This version introduces the following converters:

• date: Converts an HTTP date string to a UNIX timestamp.
• rfc7239_nn: Converts an IPv4 or IPv6 address to a compliant address that you can use in the from field of a Forwarded header. The nn here stands for node name. You can use this converter to build a custom Forwarded header.
• rfc7239_np: Converts an integer into a compliant port that you can use in the from field of a Forwarded header. The np here stands for node port. You can use this converter to build a custom Forwarded header.
• when: Enables you to pass data, such as debugging information, only when a condition is met, such as an error condition.

Runtime API

This version makes the following changes to the Runtime API:

• The debug counters command shows all internal counters placed in the code. Primarily aimed at developers, these debug counters provide insight for analyzing glitch counters and counters placed in the code using the new COUNT_IF() macro. Developers can use this macro during development to place arbitrary event counters anywhere in the code and check the counters’ values at runtime using the Runtime API. For example, glitch counters can provide useful information when they are increasing even though no request is instantiated or no log is produced.
• The dump ssl cert command will display an SSL certificate directly in PEM format; useful for placing delimiters and saving a certificate when it was updated on the CLI and not on the filesystem yet. You can also dump a transaction by prefixing the filename with an asterisk. This command is restricted and can only be issued on sockets configured for level admin.
• The echo command with syntax echo will print what’s contained in to the console output; it’s useful for writing comments in between multiple commands.
• This version improves the show dev Runtime API command by printing more information about arguments provided on the command line as well as the Linux capabilities set at process start and the current capabilities (the ability to preserve capabilities was introduced in Version 2.9 and improved in Version 3.0). This information is crucial for engineers troubleshooting the product.
• The show env command, which dumps environment variables known to the process, can now show information for a specific environment variable.
• The show quic command produces more internal information about the internal state of the congestion control algorithm and other dynamic metrics (such as window size, bytes in flight, and counters).
• The show info command will now report the current and total number of streams. It can help to quickly detect if a slowdown is caused on the client side or the server side and facilitate the export of activity metrics.

Troubleshooting

This release includes a number of troubleshooting and debugging improvements:

• Starting in version 3.1, traces get a dedicated configuration section named traces, providing a better user experience compared to previous versions. Traces report more information than before, too.
• The new when converter adds a condition to the end of a fetch method to only fetch the data when the condition is true, such as an error condition.
• As of version 2.5, you can use the sample fetches fc_err_* for frontends and bc_err_* for backends to help determine the cause of an error on the current connection. In this release, these fetches have been enhanced to include connection-level errors that occur during data transfers.
• The system may produce a core dump on a fatal error or when the watchdog fires, which detects deadlocks. While crucial to diagnosing issues, sometimes these files are truncated or can be missing information vital to analysis. This release includes an internal post_mortem structure to be included in core dumps, which contains pointers to the most important internal structures. This structure, present in all core dumps, allows developers to more easily navigate the process’s memory, reducing analysis time, and prevents the user from needing to change their settings to produce different debug output. Additionally, more hints have been added to the crash output to help in decoding the core dump. To view this debugging information without producing a core dump, use the improved show dev command.
• In previous versions, sometimes stderr outputs of the thread backtraces in core dumps would be missing, or only the last one was present due to the reuse of the same output buffer for each thread. Core dumps now include backtraces for all threads, as each thread’s backtrace is now dumped in its own buffer. Also present in core dumps as of this version are the output messages for each thread, which assists developers in determining the causes of issues even when debug symbols are not present.
• This version includes improvements to HAProxy’s watchdog, which detects deadlocks and kills runaway processes. The watchdog will now watch for stuck threads more often, by default every 100ms, and it will emit warnings regarding a stuck thread’s backtrace before killing it. It will stop the thread if, after the first warning, the thread makes no progress for one second. In this way, you should see ten warnings about a stuck thread before the watchdog kills it. Note that you can adjust the time delay after which HAProxy Enterprise will emit a warning for a stuck thread using the global debugging directive warn-blocked-traffic-after. We do not advise that you change this value, but changing it may be necessary during a debugging session.
• This version enhances the accuracy of the memory profiler by improving the tracking of the association between memory allocations and releases and by intercepting more calls such as strdup() as well as non-portable calls such as strndup() and memalign(). This improvement in accuracy applies to the per-DSO (dynamic shared object) summary as well, and should fix some rare occurrences where it incorrectly appeared that there was more memory free than allocated. New to this version, a summary is provided per external dependency, which can help to determine if a particular library is leaking memory and where.

Performance improvements

HAProxy Enterprise 3.1r1 adds these performance improvements:

• The CPU binding configuration that HAProxy Enterprise determines automatically for your machine provides the best performance, and most users should see no difference in configuration requirements. However, if you are using a large system with many cores and multiple CCX, or a heterogeneous system with both “performance” and “efficiency” cores, some additional configuration tuning can lead to further performance gains. See Performance optimization for large systems.
• The H2 mux is significantly more performant in this version. This was accomplished by optimizing the H2 mux to wake up only when there are requests ready to process, saving CPU cycles, and resulting in using 30% fewer instructions on average when downloading. The POST upload performance has been increased up to 24x with default settings and it now also avoids head-of-line blocking when downloading from H2 servers. Two new global directives, tune.h2.be.rxbuf and tune.h2.fe.rxbuf allow for further tuning of this behavior. Specify a buffer size in bytes using tune.h2.fe.rxbuf for incoming connections and tune.h2.be.rxbuf for outgoing connections. For both uploads and for downloads, one buffer is granted to each stream and 7/8 of the unused buffers is shared between streams that are uploading / downloading, which is the mechanism that significantly improves performance.
• New to this version are two new global directives for tuning QUIC performance:
  • tune.quic.cc.cubic.min-losses takes a number that defines a threshold for how many packets must be missed before the Cubic congestion control algorithm determines that a loss has occurred. This setting allows the algorithm to be slightly more tolerant to false losses, though you should exercise caution when changing the value from the default value of 1. A value of 2 may prove to show some performance improvement, though we do not recommend running this way for extended periods of time, only for analysis, and you should avoid providing a value larger than 2.
  • tune.quic.frontend.default-max-window-size defines the default maximum window size for the congestion controller of a single QUIC connection, by specifying an integer value between 10k and 4g, with a suffix of “k”, “m” or “g”.
• This version sees an efficiency improvement in regards to the QUIC buffer allocator and using this tunable, you are able to vary the size of the memory required per-connection, thus reducing overallocation.
• The transmission path for QUIC has been significantly improved in this version so that it will now adapt to the current send window size and will use Generic Send Offload to let the kernel send multiple packets in a single system call. This offloads processing from HAProxy Enterprise and the kernel and places it onto the hardware. This is especially meaningful when used on virtual machines where system calls have potential to be expensive.
• Small frames for the QUIC buffer handling now use small buffers. This improves both the memory and CPU usage, as the buffers are now more appropriately sized and do not require realignment. QUIC will always send a NEW_TOKEN frame to new clients for reuse in the next connection. This behavior permits clients to reconnect after being validated without going through the address validation process again on the next connection. In other words, the next established connection will improve network performance when a listener is attacked or when dealing with a lossy network.
• To help improve performance in the case of large configurations that consume a lot of CPU on reload, two global configuration directives tune.renice.startup and tune.renice.runtime are new to this version. These global directives take a value between -20 and 19 to apply a scheduling priority to configuration parsing.
• TCP logs saw a 56% performance gain in this version thanks to the implementation of the line-by-line parser into the TCP log forwarder.
• In regards to log servers, the ring sending mechanism sees improvement in this version, as the load is better balanced across available threads, assigning new server connections to threads with the least load. You can now use the max-reuse directive for TCP connections served by rings. When used for this reason, the sink TCP connection processors will not reuse a server connection more times than the indicated maximum. This means that connections to the servers will be forcefully removed and re-created, which helps to better distribute the load across available threads, thus increasing performance. Make sure that when using this directive that the connections are not closed more than a couple of times per second.
• In previous versions, some users may have seen intense CPU usage by the pattern LRU cache when performing lookups with low cardinality. To remedy this, in this version the cache will be skipped for maps or expressions with patterns with low cardinality, that is, less than 5 for regular expressions, less than 20 for others. Depending on your setup, you could see a savings of 5-15% CPU in these cases.
• As of this version, configured servers for backends are now properly indexed, which saves time in detecting duplicate servers. As such, the startup time for a configuration with a large number of servers could see a reduction of up to a factor of 4.
• Variables have been moved from a list to a tree, resulting in a 67% global performance gain for a configuration including 100 variables.
• We saw a performance gain of, on average, 7% regarding arithmetic and string expressions by removing the need for trivial casts for samples and converters of the same types.
• The Lua function core.set_map() has doubled its performance in speed by avoiding duplicate lookups.
• This version includes a performance gain regarding smoother reloads for large systems, that is, systems requiring a large number of file descriptors and a large number of threads. This gain is due to how file descriptors are handled on boot, shortening initialization time from 1.6s to 10ms for a setup with 2M configured file descriptors.
• The master-worker mode was heavily reworked in this version to improve stability and maintainability. Its previous architecture model proved difficult in maintaining forward compatibility for seamless upgrades; the rework aims to remedy this problem. Per the new model, the master process does nothing after starting until it confirms the worker is ready, and it no longer re-executes itself to read the configuration, which greatly reduces the number of potential race conditions. The configuration is now buffered once for both the master and worker and as a result will be identical for both. As such, environment variables shared by both will be more consistent, and the worker will be isolated from variables applicable to the master only. This all improves the separation between the processes. An additional improvement is that this rework will reduce file descriptor leaks across the processes as they are now better separated. All of this to say: you should not notice anything as a result of this change except for improved reliability.

Deprecated features and breaking changes

• The program section is deprecated in HAProxy Enterprise 3.1 and will no longer be supported starting HAProxy Enterprise 3.3.
• The configuration options accept-invalid-http-request and accept-invalid-http-response are deprecated. Instead, use accept-unsafe-violations-in-http-request and accept-unsafe-violations-in-http-response.
• Duplicate names in various families of proxies, for example, frontend, listen, backend, defaults, and log-forward sections, and between servers, are detected and reported with a deprecation warning, specifying that the duplicate names will not be supported in HAProxy Enterprise 3.3.
• The legacy C-based mailers are deprecated and will be removed in HAProxy Enterprise 3.3. Set up mailers using Lua mailers instead.

Original source

HAProxy Enterprise 3.3r1

Key changes in the HAProxy Enterprise 3.3r1 release include:

HAProxy Enterprise WAF

HAProxy Enterprise 3.3r1 WAF introduces support for OWASP CRS version 4. This enhancement is backported to HAProxy Enterprise versions 3.0r1, 3.1r1, and 3.2r1.

Captcha module

Updates to the Captcha module include:

• Breaking change: Frontend configurations must include the new http-response captcha(<captcha_section>) action. See Actions.
• Breaking change: The filter directives in frontend configurations must append unless METH_POST { path /.well-known/haproxy/captcha_callback }. For example:

filter htmldom mode strict head-append '%[captcha.js(<captcha_section>)]' unless METH_POST { path /.well-known/haproxy/captcha_callback }

• New captcha section directives: callback-token <expr>, callback-token-lf <fmt>, callback-valid-time <sec>, cookie-name <name>, and cust-html-file-lf <file>. See Captcha section directives for details.
• New sample fetch: captcha.callback_token(<captcha_section>). See Sample fetches for details.

HTTP/3 over QUIC

Updates to HTTP/3 support are:

• An experimental feature lets you use HTTP/3 over QUIC with backend servers.
• The global directives that were prefixed with tune.quic.frontend are deprecated in favor of the directives prefixed with tune.quic.fe. A few other QUIC directives were also deprecated, as the naming is consolidated.
• The no-quic global directive has been renamed tune.quic.listen, which you can set to on or off to enable and disable the QUIC protocol.
• New QUIC-related global directives are available:
  • tune.quic.be.cc.cubic-min-losses: Defines how many lost packets are needed for the Cubic congestion control algorithm to really consider a loss event.
  • tune.quic.be.cc.hystart: Enables or disables the HyStart++ (RFC 9406) algorithm for QUIC connections used as a replacement for the slow start phase of congestion control algorithms, which may cause high packet loss. It’s disabled by default.
  • tune.quic.be.cc.max-frame-loss: Sets the limit for which a single QUIC frame can be marked as lost. If exceeded, the connection is considered as failing and is closed immediately.
  • tune.quic.be.cc.max-win-size: Sets the default, maximum window size for the congestion controller of a single QUIC connection either on the frontend or backend side.
  • tune.quic.be.cc.reorder-ratio: Sets the ratio applied to the packet reordering threshold calculated.
  • tune.quic.be.max-idle-timeout: Sets the QUIC max_idle_timeout transport parameters on either the frontend or backend side.
  • tune.quic.be.sec.glitches-threshold: Sets the threshold for the number of glitches per connection either on the frontend or backend side, where that connection will automatically be killed.
  • tune.quic.be.stream.data-ratio: Allows you to configure the hard limit of the number of data bytes in flight over each stream.
  • tune.quic.be.stream.max-concurrent: Sets the QUIC initial_max_streams_bidi transport parameter either on frontend or backend side.
  • tune.quic.be.stream.rxbuf: Sets the hard limit for the number of data bytes in flight over a QUIC frontend connection.
  • tune.quic.be.tx.pacing: Enables or disables pacing support for QUIC emission. By default, it’s enabled.
  • tune.quic.be.tx.udp-gso: Enables or disables UDP GSO support for QUIC emission. By default, it’s enabled.
  • tune.quic.listen: Disables the QUIC transport protocol on the frontend side.
  • tune.quic.mem.tx-max: Sets the maximum amount of memory usable by the QUIC stack at the transport layer for emission.

Load balancing

The default load balancing algorithm in a backend is now random instead of roundrobin. When benchmarked, the random algorithm performed better and balanced traffic more fairly across servers. This algorithm, unless otherwise set, behaves as a power-of-two algorithm: it randomly selects two servers from the list and chooses the least loaded one.

Statistics

An experimental feature lets you persist statistics after a reload of the load balancer process. It works by using Linux shared memory.

Traces

Version 3.3r1 adds new sources for traces, acme, and ssl, giving insight into TLS-related events.

SSL/TLS

Changes to SSL/TLS in this version include:

• When you configure HAProxy Enterprise to connect to backend servers over TLS, you can send an SNI (Server Name Indication) value. In this version, HAProxy Enterprise sets this value for you automatically to match the value of the Host header it received from the client. This is a common pattern, so making it the default simplifies the configuration.
• A new global directive, ssl-passphrase-cmd, lets you unlock password-protected TLS private keys.
• An experimental feature in which you add ktls on to a bind line enables support for Kernel TLS, which offloads to the Linux kernel the symmetric cryptography part of the TLS processing. For even greater efficiency, you can combine this with splicing by adding option splice-auto to the frontend. With splicing, you transfer data directly from the client socket to the server socket, bypassing the userland process.
• A new, experimental bind argument, ech, enables HAProxy Enterprise to use TLS ECH (Encrypted Client Hello). This new feature encrypts the ClientHello message sent to the load balancer, protecting sensitive fields including the SNI field, so they remain private and only decryptable by the target server. Using this argument requires the global directive expose-experimental-directives. Additionally, ECH requires clients to retrieve the public key from DNS, so first add your public key to your DNS configuration.
• New preprocessor conditions return true or false depending on the TLS library that HAProxy Enterprise uses. With preprocessor conditions, you can enable or disable portions of your configuration depending on the result. The new conditions are:
  • awslc_api_atleast(<ver>): True if the current AWS-LC API number is at least as recent as ver, otherwise false.
  • awslc_api_before(<ver>): True if the current AWS-LC API number is strictly older than ver, otherwise false.
  • ssllib_name_startswith(<name>): True if the SSL library name HAProxy was linked with starts with name.

OAuth

When using OAuth authentication, the new jwt_verify_cert fetch method will automatically extract the key from the certificate used to sign the JWT, which had previously been a manual step.

Fetch methods

New fetch methods in this version are:

• req.bytes_in: An alias for bytes_in, this returns the number of bytes received from the client.
• req.bytes_out: Returns the number of bytes sent to the server.
• res.bytes_in: An alias for bytes_out, this returns the number of bytes received from the server.
• res.bytes_out: Returns the number of bytes sent to the client.

Converters

New converters in this release are:

• base2: Converts a binary input sample to a binary string containing eight binary digits per input byte.
• le2dec: Converts a little-endian binary input sample to a string containing an unsigned integer number per a given chunk size of input bytes.

Prometheus

Prometheus metrics introduced in this version are:

• haproxy_process_patterns_added_total: Total number of patterns added (ACL and map entries)
• haproxy_process_patterns_freed_total: Total number of patterns freed (ACL and map entries)
• haproxy_frontend_req_bytes_in_total: Total number of request bytes received since process started. Label: proxy=frontend name.
• haproxy_frontend_req_bytes_out_total: Total number of request bytes sent since process started. Label: proxy=frontend name.
• haproxy_frontend_res_bytes_in_total: Total number of response bytes received since process started. Label: proxy=frontend name.
• haproxy_frontend_res_bytes_out_total: Total number of response bytes sent since process started. Label: proxy=frontend name.
• haproxy_backend_req_bytes_in_total: Total number of request bytes received since process started. Label: proxy=backend name.
• haproxy_backend_req_bytes_out_total: Total number of request bytes sent since process started. Label: proxy=backend name.
• haproxy_backend_res_bytes_in_total: Total number of response bytes received since process started. Label: proxy=backend name.
• haproxy_backend_res_bytes_out_total: Total number of response bytes sent since process started. Label: proxy=backend name.
• haproxy_server_private_idle_connections_current: Current number of private idle connections. Labels: proxy=backend name, server=server name.
• haproxy_server_req_bytes_in_total: Total number of request bytes received since process started. Labels: proxy=backend name, server=server name.
• haproxy_server_req_bytes_out_total: Total number of request bytes sent since process started. Labels: proxy=backend name, server=server name.
• haproxy_server_res_bytes_in_total: Total number of response bytes received since process started. Labels: proxy=backend name, server=server name.
• haproxy_server_res_bytes_out_total: Total number of response bytes sent since process started. Labels: proxy=backend name, server=server name.

Runtime API

New commands for managing ECH (Encrypted Client Hello) keys are:

• The add ssl ech command adds an ECH (Encrypted Client Hello) key to a bind line.
• The del ssl ech command deletes an ECH key from a bind line.
• The set ssl ech command replaces an ECH key on a bind line.
• The show ssl ech command displays the list of ECH keys loaded in the HAProxy Enterprise process.

New commands for managing TLS certificates for JWT (JSON Web Token) validation are:

• The add ssl jwt command adds a TLS certificate for use in JWT validation.
• The del ssl jwt command deletes a TLS certificate from the list of those used in JWT validation.
• The show ssl jwt command displays certificates that can be used for JWT validation.

Other changes are:

• The add ssl crt-list command no longer verifies that the certificate’s path matches its name in memory, which better supports setting aliases for your certificates via crt-store and crt-list.
• The show dev command now displays information on thread-to-CPU bindings.
• The show info command now reports the number of added and removed lines in map and ACL files, which helps detect scripts that add entries but may not remove them as expected.
• The show quic command has new options for listing only backend connections or closing frontend connections.
• When calling the show stat command with the typed argument, it now shows next to each metric the letter V for volatile or P for persistent. This is part of HAProxy Enterprise’s new support for persisting metrics across reloads.

Performance tuning

Starting in version 3.3r1, the default behavior is to use all available CPU cores and threads, even on massively-multicore systems. HAProxy Enterprise leverages optimized multithreading and provides automatic CPU binding to deliver both unparalleled efficiency and speed, ensuring your load balancing is faster than ever. Read our blog post for more information.

HAProxy Enterprise’s cpu-policy now defaults to performance, meaning that on systems with a heterogeneous mix of efficiency and performance core types, the load balancer will run on only the performance cores. Also, the maximum number of threads is no longer limited to 64, which matters for machines that have more than 64 cores, particularly on non-NUMA systems. The increased processing capacity benefits CPU-intensive use cases such as TLS, Lua, and regex.

Frontend-, backend-, and server-related event counters are now stored per thread group, which reduces bottlenecks when accessing the counters.

Backends with mode http now set option abortonclose by default. This setting tries to stop processing a request before it’s been sent to a server if the client aborted on their end, such as by closing the tab or refreshing. Also, you can now set option abortonclose in a frontend, which wasn’t allowed before, and HAProxy will avoid computing the TLS handshake on connections that are already closed.

HAProxy Enterprise saves resources by no longer allocating memory for a default-server directive unless you declare one in your configuration. Also, after parsing the directive, the load balancer releases the memory associated with it.

When using a use-server directive or track argument in a backend, startup will be faster now that HAProxy Enterprise uses a more efficient algorithm for finding the server.

You can now choose a different TCP congestion algorithm by setting it with the new cc argument on a bind or server line.

This version relaxes the amount of locking between stick tables and peers by batching the updates and delaying work, leading to a smoother traffic flow and better overall performance.

Some multi-threaded tasks that caused a lot of contention on servers with many CPUs, such as stick table expirations and resolvers connections, were changed to be single threaded.

All applets, including the DNS, http-client, Lua, logs, peers, and Prometheus applets which were updated with this release, now maintain their own buffers rather than share buffers with the stream, which is used for socket reads and writes. Each applet having its own buffers requires less locking and improves synchronization, which reduces contention across the applets, improving the applets’ scalability.

Security

Updates to security features include:

• You’ll now get a warning at startup if you’re running HAProxy Enterprise as root and your configuration is missing the global directive user, which sets a Linux user account that HAProxy Enterprise should run as.
• The bind and server directives support a new argument named tcp-md5sig that adds support for Protection of BGP Sessions via the TCP MD5 Signature Option (RCF 2385), which many routers require when placing a TCP proxy like HAProxy Enterprise between them.

Usability

Usability improvements in this version include:

• You can get the HAProxy Enterprise version in different formats. Pass the command-line arguments -vq for version, -vqs for the short version, or -vqb for the branch.
• If you’ve set the expose-experimental-directives global directive, but all of the experimental features you were using are no longer experimental, you’ll get a reminder to remove the directive. That should help users in avoiding having experimental features enabled unintentionally.
• The global directive dns-accept-family that was introduced in the previous version now defaults to the value auto. This directive lets you disable IPv4 or IPv6 DNS resolution. A value of auto will enable IPv4 DNS resolution and check for IPv6 connectivity at startup, then again every 30 seconds to determine whether to enable IPv6 resolution.
• Setting the global directive nbthreads to a total number of threads on which HAProxy Enterprise should run, while also declaring a thread-groups directive with a range of threads that exceeds that number, will now emit a warning and the missing threads will be removed. If a thread group is left with no threads at all, it causes a startup error.
• The tune.disable-fast-forward directive is no longer experimental, so you don’t need to set expose-experimental-directives to use it. This directive was introduced in version 2.8r1 and disables data fast-forwarding.
• For debugging, you may want to prevent all workers from being killed when a segfault occurs. You can use the global directive master-worker no-exit-on-failure.
• The default number of reloads defined by mworker-max-reloads is now 50.

Deprecated features

These features are now deprecated, meaning they’ll be removed in a future version:

• The backend directives dispatch and option transparent are deprecated and will emit a warning to replace them if used.
• Global directives prefixed with tune.quic.frontend are deprecated. Use the same directives prefixed with tune.quic.fe instead.
• The master-worker global directive has been deprecated. Use the command-line arguments -W or -Ws instead.

Breaking changes

This version has the following breaking changes:

• The WAF Offloader has been removed from this and future versions.
• The minimum, default Linux kernel version, the one corresponding to the build target linux-glibc, has been updated to 4.17, which is a version older than all of the currently maintained LTS distros. This version was needed to support the new Kernel TLS feature.
• The program configuration section, which allows you to start and run an external program as a child process, was deprecated in version 3.1r1 and is now removed.
• Using the same name for more than one frontend, backend, listen, defaults, or log-forward section is no longer allowed. Duplicated names, which have emitted a warning since version 3.1r1, will now emit an error at startup.
• Using the same name for more than one server in a backend isn’t allowed.
• When configuring email alerts, you must enable the Lua implementation. If you add a mailers configuration section, but forget to load the Lua file, you’ll get a warning.
• The backend directive http-send-name-header, which lets you send the name of the server HAProxy Enterprise is connecting to as an HTTP request header, had always let you decide which HTTP header to use for that purpose. But now, it won’t allow you to choose the headers connection, content-length, host, or transfer-encoding. Overwriting those headers would only cause an invalid request.
• When declaring an ACL, you can set the match type via the -m flag to explicitly compare the input value as a boolean, string, integer, and others. Specifying more than one match type after this flag is no longer allowed. Previously, HAProxy Enterprise had silently used the last match type. Also, HAProxy Enterprise will emit a warning when the match type is ambiguous, such as in path_beg -m reg, which is ambiguous as to whether it matches the beginning of the path or matches paths with a regular expression.
• This version renames the no-quic global directive to tune.quic.listen, which lets you enable or disable the QUIC transport protocol on all frontend listeners.

Original source

Key changes in the HAProxy Enterprise 3.2r1 release include:

Bot Management module

The Bot Management module now has better integration with third-party proxies. Proxies in front of HAProxy Enterprise often remove some client information before relaying the request, which can impact the accuracy of the module’s scoring. For instance, they sometimes remove the client’s source IP address or information about the client’s TLS stack. To solve that, you can configure the proxy to pass that information as HTTP request headers and then pass it to the Bot Management module. Specifically in this version, you can send the client’s TLS stack profile via a JA4 fingerprint.

The Bot Management module can now tell which type of threat was posed by a suspicious bot. When you enable threat detection, the module will analyze the bot’s behavior and attempt to classify it. Classifications include DDoS, login brute forcing, vulnerability scanning, and web scraping.

You can now use the botmgmt-evaluate directive in a defaults section.

Web Application Firewall module

The Web Application Firewall (WAF) has added support for the http-request waf-evaluate action, which allows you to apply WAF Intelligent WAF Engine (IWE) rules on-demand in a frontend, backend, or listen proxy.

The WAF has added support for WAF profiles. Defined in the new waf-profile section, a profile specifies a set of WAF parameters that can be applied to any WAF in the configuration, whether via a filter waf or an http-request waf-evaluate directive. You can define multiple WAF profiles in a configuration. This enhancement will be backported to HAProxy Enterprise 3.1r1.

Captcha module

When using the Captcha module, you can set the on-error directive to allow users to access your application when the Captcha provider’s verification server is unreachable. See the Captcha module reference.

The cookie-path directive, which indicates whether the login cookie applies to the entire site or only a portion of the site, now defaults to /, the entire site.

51Degrees module

The 51Degrees modules makes it easier to set variables from 51Degrees properties with the one-liner, global directive 51d-set-property-vars.

MaxMind module

When using the MaxMind module, you can now specify any unique string as the database type, not only the strings ANONYMOUS, CITY, CONNTYPE, COUNTRY, DOMAIN, ISP, and ANY. This freedom to use any string as the database type will make it easier to write a valid configuration.

OpenID Connect module

The new OpenID Connect module enables single sign-on by integrating with identity providers such as Microsoft EntraID and Okta. Use it to secure access to your applications, while offloading the integration work to HAProxy Enterprise.

SAML module

The SAML module has a new option, idp_ca_file, for validating the identity provider’s digital signature. Also, the list of supported signing algorithms has been updated. Additionally, you can now use the saml-sso action with the http-after-response directive.

Route Health Injection module

The Route Health Injection module has been redesigned to be simpler to configure and to work without the third-party component BIRD. Previous versions had you configure settings across multiple files. The new version places configuration in a single file.

GSLB module

The GSLB module now supports an HTTPS proxy mode for sending health checks over HTTPS. When run alongside HAProxy Enterprise, it sends health checks to HAProxy Enterprise, which will forward them over a secure connection.

The GSLB module can now aggregate many, smaller GSLB templates into one large file that is then converted all at once into the corresponding GDNSD configuration. These options are available in its service environment file (located at /etc/default/hapee-extras-gslb by default).

Global Profiling Engine

The Global Profiling Engine embeds an API that you can use to inspect stick table data and troubleshoot issues. Now, its show table command supports filtering. You can return stick table records that have a matching key, data value, or memory pointer.

AWS-LC TLS library

In HAProxy Enterprise 3.2, we’ve strategically replaced the default OpenSSL library with AWS-LC. AWS-LC is a general-purpose cryptographic library maintained by the AWS Cryptography team, based on code from Google BoringSSL and the OpenSSL project. While this enhancement improves security and performance, it may constitute a breaking change in environments relying on older cipher suites not supported by AWS-LC. For details, see Cipher suite support.

ACME protocol

This version adds experimental support for the ACME protocol, enabling you to obtain and renew TLS certificates from issuers like Let’s Encrypt.

Runtime API

The add server command’s new help argument shows which server arguments it supports for your version of HAProxy Enterprise.

The clear table command has a new syntax that lets you access data that was configured as an array, such as gpc().

The debug counters command has filters for enabling and disabling counters for COUNT_IF statements.

The prompt command now has an interactive mode.

The set table command has a new syntax that lets you access data that was configured as an array, such as gpc().

The show events command accepts the -0 flag to display the lines of the output with terminating null bytes instead of newlines.

The show pools command has a detailed argument that provides more information about the pools.

The show quic command has a new argument, stream, for showing only active streams and excluding connections that are closing or draining.

The show sess command now supports filtering the output to a given backend and server.

The show table command has a new syntax that lets you access data that was configured as an array, such as gpc().

Prometheus

This version adds new Prometheus metrics:

• haproxy_frontend_current_session_rate
• haproxy_frontend_ssl_ocsp_staple
• haproxy_frontend_ssl_failed_ocsp_staple
• haproxy_backend_current_session_rate
• haproxy_backend_ssl_ocsp_staple
• haproxy_backend_ssl_failed_ocsp_staple
• haproxy_server_current_session_rate
• haproxy_server_http_requests_total
• haproxy_server_ssl_ocsp_staple
• haproxy_server_ssl_failed_ocsp_staple

Traffic policing

You can now delay the processing of a request with the pause response policy, giving you another option for slowing down clients that exceed your rate limit.

Health checks

To send health checks to backend servers over existing, idle connections, use the check-reuse-pool argument on a server directive.

DNS resolver

Use the global directive dns-accept-family to indicate whether to request and accept IPv4 and/or IPv6 records from nameservers.

Compression

You can now set a minimum file size to control which files to compress.

HTTP/3 over QUIC

When you set the QUIC congestion control algorithm with the quic-cc-algo directive, it now automatically enables pacing on top of the chosen algorithm. It had been an opt-in, experimental feature before. Pacing smooths the emission of data to reduce network losses and has shown performance increases of approximately 10-20 fold over lossy networks or when communicating with slow clients at the expense of a higher CPU usage in HAProxy. On a related note, you can set the Bottleneck Bandwidth and Round-trip Propagation Time algorithm, which relies on pacing, without enabling experimental features. Set the quic-cc-algo directive’s bbr argument. Or if you don’t want pacing, disable it completely with tune.quic.disable-tx-pacing.

This version improves QUIC upload performance. Previous versions only supported the equivalent of a single buffer in flight, which would limit the upload bandwidth to about 1.4 Mbps per stream, which was slow for users attempting to upload large images or videos. Starting with 3.2r1, uploading streams can use up to 90% (by default) of the memory allocated to the connection, allowing them to use the full bandwidth even with a single stream. You can adjust this ratio by using the global directive tune.quic.frontend.stream-data-ratio, allowing you to prioritize fairness (small values) or throughput (higher values). The default setting should suit common, web scenarios by striking a balance.

A new, global setting, tune.quic.frontend.max-tx-mem, caps the total memory that the QUIC tx buffers can consume, helping to moderate the congestion window so that the sum of the connections don’t allocate more than that. By default, there’s no limitation.

HTTP protocol

The accept-unsafe-violations-in-http-request and accept-unsafe-violations-in-http-response backend directives introduced in in the previous version allow HTTP messages to violate some rules of the protocol. In version 3.2r1, they allow HAProxy Enterprise to accept WebSocket requests that are missing the Sec-WebSocket-Key HTTP header and responses missing the Sec-WebSocket-Accept HTTP header.

You can now set the HTTP response header Content-Length to 0, which some non-compliant applications need with HTTP 101 and 204 responses.

HAProxy Enterprise has become stricter about not permitting some characters in the Authority and Host HTTP headers.

Two new directives let you drop trailers from HTTP requests or responses, which is useful for removing sensitive information that shouldn’t be exposed to clients:

• option http-drop-request-trailers
• option http-drop-response-trailers

To more efficiently close idle HTTP/2 connections, set the idle-ping argument on a bind or server directive.

Syslog load balancing

New options in the log-forward section let you control how the load balancer parses and validates Syslog messages.

SSL/TLS

Use the new ssl-f-use directive to specify the certificates to load in a frontend. This directive simplifies setting TLS-related properties such as ALPN fields, ciphers, signature algorithms, and CAs for client certificate authentication.

Consistent hashing

When using the balance hash algorithm for consistent-hash load balancing, you can now set the directive hash-preserve-affinity to indicate what to do when servers become maxed out or have full queues. Consistent hashing configures the load balancer to maintain server affinity, but when a server is overwhelmed, blindly preserving that affinity can lead to issues. With hash-preserve-affinity, you can now reroute traffic to available servers while still maintaining affinity.

Overload protection

When setting maxconn on a server line, you can set strict-maxconn to apply the limit to TCP connections instead of HTTP requests.

Layer 7 retries

The retry-on directive, which was introduced in version 2.0r1, allows you to set which errors should trigger the retry of a failed request. It now supports the HTTP response status 421 Misdirected Request as a reason.

Fetch methods

This version introduces the following fetch methods:

• bc_reused: Returns true if the transfer was performed via a reused backend connection.
• req.ssl_cipherlist: Returns the binary form of the list of symmetric cipher options supported by the client as reported in the TLS ClientHello.
• req.ssl_keyshare_groups: Returns the binary format of the list of cryptographic parameters for key exchange supported by the client as reported in the TLS ClientHello.
• req.ssl_sigalgs: Returns the binary form of the list of signature algorithms supported by the client as reported in the TLS ClientHello.
• req.ssl_supported_groups: Returns the binary form of the list of groups supported by the client as reported in the TLS ClientHello and used for key exchange, which can include both elliptic and non-elliptic key exchange.
• sc_key(): Returns the key used to match the currently tracked counter.
• table_clr_gpc([,]): Clears the General Purpose Counter at index of the array and returns its previous value.
• table_inc_gpc([,]): Increments the General Purpose Counter at index of the array and returns its new value.
• term_events: Returns a series of comma-separated values that indicate the states of a request as its flowed through the load balancer. Clone the HAProxy GitHub repository, compile the term_events program, then run it to decode the values.

The accept_date and request_date fetch methods now fall back to using the session’s date if not otherwise set, which can happen when logging SSL handshake errors that occur prior to creating a stream.

Converters

It was discovered that a risk of buffer overflow can occur when using the regsub converter to replace patterns multiple times at once (multi-reference) with longer patterns. Although the risk is low, it has been fixed. CVE-2025-32464 was filed. The vulnerability affects all versions, so the fix will be backported.

Lua

The new patref class gives you a way to modify ACL and Map files from your Lua code and is an improvement over the older core.add_acl function. It makes it easier to dynamically change Map and ACL files from your Lua code, such as to build modules that cache responses only for URLs that have a certain URL parameter attached to them. The patref class offers other features too:

• Manipulate both ACL and Map files.
• For Map files, replace the values of matching keys.
• Add new patterns via bulk entry with the patref.add_bulk function.
• Use prepare() and commit() functions to replace the entire ACL file at once with a new set of data.
• Subscribe to events related to manipulating pattern files with callback functions.

A new global directive, tune.lua.bool-sample-conversion, allows you to opt in to proper handling of booleans returned by fetch methods. The default behavior has been that when the Lua code calls a fetch method that returns a boolean, that return value is converted to an integer 0 or 1. Setting the new global directive to normal enables the correct behavior of treating booleans as booleans.

The AppletTCP class’s receive function now accepts a timeout parameter to limit how long it will wait for data from the client. This makes it easier to design services that take in varying lengths of data, such as interactive utilities that read user input, as opposed to expecting fixed-length data.

This version introduces new Lua functions:

• AppletTCP.try_receive: Reads available data from the TCP stream and returns immediately.
• core.wait: Waits for an event to wake the task. It takes an optional delay after which it will awake even if no event fired.
• HTTPMessage.set_body_len: Changes the expected payload length of the HTTP message.

Other performance updates

By fixing the fairness of the lock that the scheduler uses for shared tasks, heavily loaded machines (64 cores NUMA) will see less latency, typically 8x lower, and 300x fewer occurrences of latencies 32ms or above.

HAProxy Enterprise will now interrupt the processing of TCP and HTTP rules in the configuration at every 50 rules, a number that’s configurable, to perform other concurrent tasks. This will help keep latencies low for configurations that have hundreds of rules.

HAProxy Enterprise servers with many CPU cores will see significantly better performance of queues in regards to CPU usage. Queues were refined to be thread group aware, favoring pending requests in the same group when a stream finishes, which reduces data sharing between CPU cores.

QUIC now supports a larger Rx window to significantly speed up uploads (POST requests).

The HAProxy Runtime API’s wait command has been optimized to consume far less CPU while waiting for a server to be removable if you’ve set the srv-removable argument, which will be especially relevant for users that add and remove many servers dynamically through the Runtime API.

On a server with a 128-thread EPYC microprocessor, watchdog warnings were emitted occasionally under extreme contention on the mt_lists, indicating that some CPUs were blocked for at least 100ms. To solve this issue, we shortened the exponential back-off, which seemed too high for these CPUs.

Memory pools have been optimized. Previously, HAProxy Enterprise merged similar pools that were the same size. Now, pools with less than 16 bytes of difference or 1% of their size will be merged. During a test of 1-million requests, this reduced pools from 48 to 36 and saved 3MB or RAM. The HAProxy Runtime API command show pools detailed will now show which pools have been merged.

The leastconn load balancing algorithm, which is more sensitive to thread contention because it must use locking when moving the server’s position after getting and releasing connections, shows a lower peak CPU usage in this version. By moving the server less often, we observed a performance improvement of 60% on x86 machines with 48 threads. On an Arm server with 64 threads, we saw a 260% improvement. While faster, the algorithm also became more fair than previous versions, which had to sacrifice fairness to maintain a decent level of performance. There’s now much less difference between the most and least loaded servers.

The roundrobin load balancing algorithm now scales better on systems with many threads. Testing on a 64-core EPYC server with cpu-policy performance showed a 150% performance increase thanks to no longer accessing variables and locks from distant cores.

The deadlock watchdog and thread dump signal handlers were reworked to address some of the remaining deadlock cases that users reported in version 3.1 and 3.2. The new approach minimizes inter-thread synchronization, resulting in much less CPU overhead when displaying a warning.

Performance for stick tables that sync updates from peers got a boost by changing the code to use a single, dedicated thread to update the tree, which reduces thread locking. On a server with 128 threads, speed increased from 500k-1M updates/second to 5-8M updates/second.

The default limit on the number of threads was raised from 256 to 1024.

The default limit on the number of thread groups was raised from 16 to 32.

New global directives, tune.notsent-lowat.client and tune.notsent-lowat.server, allow you to limit the amount of kernel-side socket buffers to the strict minimum required and for the non-acknowledged bytes, lowering memory consumption.

A new global directive tune.glitches.kill.cpu-usage takes a number between 0 and 100 to indicate the minimum CPU usage at which HAProxy Enterprise should begin to kill connections showing too many protocol glitches. In other words, kill connections that have reached the glitches-threshold limit, once the server gets too busy. The default is 0, where a connection reaching the threshold will be killed automatically, regardless of CPU usage. Consider setting this directive to twice the normally observed CPU usage, or the normal usage plus half the idle one. This setting requires that you also set tune.h2.fe.glitches-threshold or tune.quic.frontend.glitches-threshold.

Original source