问题[haproxy](server)

我有一个 HAProxy 2.8.4，用于代理多个 URL 路径和多个不同后端上的多个 HTTPS 服务，还有一个基于 TCP 的 PostgreSQL 集群。这是完整的 haproxy -vvv 输出：

HAProxy version 2.8.4-a4ebf9d 2023/11/17 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.4.html
Running on: Linux 4.18.0-513.5.1.el8_9.x86_64 #1 SMP Fri Sep 29 05:21:10 EDT 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_THREAD=1 USE_LINUX_TPROXY=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT +PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1k  FIPS 25 Mar 2021
Running on OpenSSL version : OpenSSL 1.1.1k  FIPS 25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.4
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.5.0 20210514 (Red Hat 8.5.0-20)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

这是我的配置，省略了不相关的 Postgre 内容和其他 HTTPS URL 路径：

lobal
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/haproxy.sock mode 600 level admin
    pidfile /var/run/haproxy.pid
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  dontlognull
    retries 2
    timeout connect 4s
    timeout client  30m
    timeout server  30m
    timeout check   5s

# Temporary detailed logging
   log-format "Client IP:port = [%ci:%cp], Start Time = [%tr], Frontend Name = [%ft], Backend Name = [%b], Backend Server = [%s], Time to receive full request = [%TR ms], Response time = [%Tr ms], Status Code = [%ST], Bytes Read = [%B], Request = [%{+Q}r], Request Body = [%[capture.req.hdr(0)]]"

    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend main
    timeout client 86400000
    bind :443 ssl crt /etc/haproxy/haproxy.crt

    option http-buffer-request
    declare capture request len 40000
    http-request capture req.body id 0

    capture request header origin len 128

    # Many other URL mappings and other use_backend directives omitted here

    acl url_apigee path_beg -i /apigee-connector
    use_backend voda-apigee-conn-be if url_apigee

    default_backend deny_be

# Many other backend definitions omitted here

backend voda-apigee-conn-be
    balance roundrobin

    option httpchk
    http-check send meth GET uri /actuator/health

    server api1 x.x.x.x:8002 check inter 10s fall 3 rise 2 ssl verify none
    server api2 y.y.y.y:8002 check inter 10s fall 3 rise 2 ssl verify none

backend deny_be
    http-request deny

当我直接使用 cURL 调用后端时，我得到 HTTP 200 响应，并且它也在内部执行预期的操作：

curl -vvv -k -w "@curl-format.txt" -X POST -H "X-API-Key: my-api-key-1" -H "Content-Type: application/json" -d @apigee-conn-email3.json https://x.x.x.x:8002/apigee-connector/outbound-communication
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying x.x.x.x...
* TCP_NODELAY set
* Connected to x.x.x.x (x.x.x.x) port 8002 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: <omitted>
*  start date: Dec  8 07:51:33 2023 GMT
*  expire date: Dec  7 07:51:32 2025 GMT
*  issuer: <omitted>
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> POST /apigee-connector/outbound-communication HTTP/1.1
> Host: x.x.x.x:8002
> User-Agent: curl/7.61.1
> Accept: */*
> X-API-Key: my-api-key-1
> Content-Type: application/json
> Content-Length: 344
>
* upload completely sent off: 344 out of 344 bytes
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 200
< Server: nginx
< Date: Wed, 28 Feb 2024 16:38:26 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Expires: 0
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Set-Cookie: JSESSIONID=YWLvqrZAAMHBgbOLK9Q-nz2vrxhs9mKw_Yt92lQ5.cgwcsmfuatapp1; path=/; secure; HttpOnly
< X-XSS-Protection: 1; mode=block
< Pragma: no-cache
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< RequestId: 59d32aad-4294-4811-a385-a4e65d68065f
< Quota-Reset: 1709139600925
< Quota-Allowed: 10000
< Quota-Available: 9998
< Content-Type: application/json
< Transfer-Encoding: chunked
<
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection #0 to host x.x.x.x left intact
{"id":"94440760","attachment":{"id":{"value":"DOC-20240228-173826-WNHPC"}}}

     time_namelookup:  0.000076s
        time_connect:  0.000414s
     time_appconnect:  0.026246s
    time_pretransfer:  0.026375s
       time_redirect:  0.000000s
  time_starttransfer:  2.132503s
                     ----------
          time_total:  2.133572s

当我通过 HAProxy 调用相同的方法时：

curl -vvv -k -w "@curl-format.txt" -X POST -H "X-API-Key: my-api-key-1" -H "Content-Type: application/json" -d @apigee-conn-email3.json https://z.z.z.z/apigee-connector/outbound-communication
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying z.z.z.z...
* TCP_NODELAY set
* Connected to z.z.z.z (z.z.z.z) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: <omitted>
*  start date: Jan 31 20:50:09 2024 GMT
*  expire date: Jan 30 20:50:08 2026 GMT
*  issuer: <omitted>
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> POST /apigee-connector/outbound-communication HTTP/1.1
> Host: z.z.z.z
> User-Agent: curl/7.61.1
> Accept: */*
> X-API-Key: my-api-key-1
> Content-Type: application/json
> Content-Length: 344
>
* upload completely sent off: 344 out of 344 bytes
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* HTTP 1.0, assume close after body
< HTTP/1.0 502 Bad Gateway
< cache-control: no-cache
< content-type: text/html
<
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>

* TLSv1.3 (IN), TLS alert, [no content] (0):
* TLSv1.3 (IN), TLS alert, close notify (256):
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, [no content] (0):
* TLSv1.3 (OUT), TLS alert, close notify (256):

     time_namelookup:  0.000073s
        time_connect:  0.001454s
     time_appconnect:  0.023841s
    time_pretransfer:  0.023926s
       time_redirect:  0.000000s
  time_starttransfer:  2.589714s
                     ----------
          time_total:  2.589899s

HAProxy 日志：

Feb 28 18:01:38 localhost haproxy[3223599]: Client IP:port = [a.a.a.a:34660], Start Time = [28/Feb/2024:18:01:36.068], Frontend Name = [main~], Backend Name = [voda-apigee-conn-be], Backend Server = [api2], Time to receive full request = [0 ms], Response time = [-1 ms], Status Code = [502], Bytes Read = [208], Request = ["POST https://z.z.z.z/apigee-connector/outbound-communication HTTP/2.0"], Request Body = [<JSON body omitted>]

我总是得到 502 Bad Gateway，但是，后端仍然按预期完全执行请求，并且它说它生成了响应......至少我在这两种情况下在应用程序级别上看到完全相同的日志消息。

我注意到 HAProxy 切换到 HTTP/2，我尝试使用 alpn 指令在 HAProxy 配置中强制使用 HTTP 1.1，并在 cURL 命令行中使用 --http1.1 标志强制使用 HTTP 1.1，然后它是 HTTP 1.1，但仍然是 502 Bad Gateway 。

这里可能出了什么问题？

- - - 更新 - - -

在 AlexD 发表评论后，我修改了我的日志记录，我基本上添加了我能找到的所有 %Tx 参数，只是添加了人类可读的名称，因为我永远不会知道什么是 Tr 和 TR 和 Th 等：

log-format "Client IP:port = [%ci:%cp], Start Time = [%tr], Frontend Name = [%ft], Backend Name = [%b], Backend Server = [%s], Active time of the request = [%Ta ms], Time to establish TCP connection to the server = [%Tc ms], SSL handshake time = [%Th ms], Idle time before the HTTP request = [%Ti ms], Time to get the client's request = [%Tq ms], Time to receive full request = [%TR ms], Response time = [%Tr ms], Total session duration time = [%Tt ms], Status Code = [%ST], Bytes Read = [%B], Termination state = [%ts], Request = [%{+Q}r], Request Body = [%[capture.req.hdr(0)]]"

此后的新日志：

Feb 29 08:46:33 localhost haproxy[3388761]: Client IP:port = [10.215.30.29:37666], Start Time = [29/Feb/2024:08:46:31.067], Frontend Name = [main~], Backend Name = [voda-apigee-conn-be], Backend Server = [api1], Active time of the request = [2092 ms], Time to establish TCP connection to the server = [8 ms], SSL handshake time = [20 ms], Idle time before the HTTP request = [0 ms], Time to get the client's request = [20 ms], Time to receive full request = [0 ms], Response time = [-1 ms], Total session duration time = [2112 ms], Status Code = [502], Bytes Read = [208], Termination state = [PH], Request = ["POST /apigee-connector/outbound-communication HTTP/1.1"], Request Body = [<omitted>]

这似乎是有用的信息。终止状态是“PH”，这似乎意味着响应在标头处理期间被阻塞。这很奇怪，我不认为我们的后端应用程序返回任何无效的标头，您可以看到上面的直接 cURL 请求响应。我尝试在前端设置 http-response strict-mode 关闭，但它并没有改变行为。我也没有找到有关此“PH”终止状态的更多信息。

----- 更新 2 -----

我设法通过调用 systemctl reload haproxy 来启用管理套接字，但不知道为什么需要这样做。但查询“显示错误”后，它显示的是：

[admin@ccaas1t-postgres-t1 ~]$ echo "show errors" | sudo socat stdio /run/haproxy/haproxy.sock
Total events captured on [29/Feb/2024:11:24:00.722] : 1

[29/Feb/2024:11:23:53.236] backend voda-apigee-conn-be (#13): invalid response
  frontend main (#5), server api1 (#1), event #0, src x.x.x.x:60930
  buffer starts at 0 (including 0 out), 15627 free,
  len 757, wraps at 16336, error at position 665
  H1 connection flags 0x80000000, H1 stream flags 0x00004810
  H1 msg state MSG_HDR_L2_LWS(24), H1 msg flags 0x00011654
  H1 chunk len 0 bytes, H1 body len 0 bytes :

  00000  HTTP/1.1 200 \r\n
  00015  Server: nginx\r\n
  00030  Date: Thu, 29 Feb 2024 10:23:53 GMT\r\n
  00067  Transfer-Encoding: chunked\r\n
  00095  Connection: keep-alive\r\n
  00119  Expires: 0\r\n
  00131  Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\n
  00194  Set-Cookie: JSESSIONID=nGfILjPozmcVahj831vf21a6BXIxgqElGlE8zxqA.cgwcsm
  00264+ fuatapp1; path=/; secure; HttpOnly\r\n
  00300  X-XSS-Protection: 1; mode=block\r\n
  00333  Pragma: no-cache\r\n
  00351  X-Frame-Options: DENY\r\n
  00374  X-Content-Type-Options: nosniff\r\n
  00407  Strict-Transport-Security: max-age=31536000 ; includeSubDomains\r\n
  00472  VFHU-RequestId: 20681405-102a-4eed-8eee-00ed8f3c37c8\r\n
  00526  VFHU-Quota-Reset: 1709204400244\r\n
  00559  VFHU-Quota-Allowed: 10000\r\n
  00586  VFHU-Quota-Available: 9999\r\n
  00614  Content-Type: application/json\r\n
  00646  Transfer-Encoding: chunked\r\n
  00674  \r\n
  00676  4b\r\n
  00680  {"id":"94464170","attachment":{"id":{"value":"DOC-20240229-112353-OTXV
  00750+ J"}}}\r\n

“位置 665 处出错”。位置665是“Transfer-Encoding：chunked”头中的冒号(:)。这对我来说看起来没问题，块的大小以 4b 十六进制形式发送，即十进制 75，这是 JSON 的大小。另外，我们不是手动构建此 HTTP 响应，它是由 Java 库完成的，所以我很确定这应该没问题。

我的文档中有一个使用 uwsgi 的 Django 应用程序。由于我们有多个 Docker 在不同的机器上运行，因此我们使用 haproxy 作为负载均衡器。

我的问题是：我应该将 uwsgi 与httpor 一起使用http-socket吗？

从这里https://uwsgi-docs.readthedocs.io/en/latest/HTTP.html

它似乎http

http-socket 选项将使 uWSGI 原生使用 HTTP。如果您的 Web 服务器不支持 uwsgi 协议，但能够与上游 HTTP 代理通信，或者如果您使用 Webfaction 或 Heroku 等服务来托管您的应用程序，则可以使用 http-socket。如果您打算仅使用 uWSGI 向世界公开您的应用程序，请改用 http 选项，因为路由器/代理/负载均衡器将成为您的盾牌。

最后一部分建议http如果有负载均衡器该怎么办

但这里https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

http 和 http-socket 选项是完全不同的东西。第一个进程产生一个额外的进程，将请求转发给一系列工作进程（将其视为一种屏蔽形式，与 apache 或 nginx 处于同一级别），而第二个进程则设置工作进程以本地方式使用 http 协议。TL/DR：如果你打算直接向公众公开uWSGI，请使用--http，如果你想在后端使用http的网络服务器后面代理它，请使用--http-socket。

似乎推断，http-socket如果不直接暴露在公众面前

那么，用 haproxy 该怎么做呢？

我有一台 Munin 服务器（2.72）可以很好地监控很多服务器。

我有一个 haproxy 服务器，其中 munin-node （2.0.73）对于每个插件都运行良好，除了 haproxy_* （haproxy_sessions_frontend、haproxy_bytes_frontend、haproxy_rate_frontend 等），它给我 NaN 值。

当我在 haproxy 服务器上本地测试插件时，我得到相关值：

# Skipping systemd properties simulation due to required systemd version (999)
# Processing plugin configuration from /etc/munin/plugin-conf.d/00-default
# Processing plugin configuration from /etc/munin/plugin-conf.d/haproxy
# Processing plugin configuration from /etc/munin/plugin-conf.d/munin-node
# Setting /rgid/ruid/ to /982/0/
# Setting /egid/euid/ to /982 982/0/
# Setting up environment
# Environment frontend = https_front haproxy-monitoring fe-letsencrypt
# Environment url = http://127.0.0.1:8080/haproxy_stats;csv;norefresh
# Environment backend = app-main haproxy-monitoring letsencrypt-backend
# About to run '/etc/munin/plugins/haproxy_sessions_frontend'
scurea490f35148af7011c03b86177948440.value 3
smaxea490f35148af7011c03b86177948440.value 25
scur4dc7a8e588e9bb787fe6e39c1c96957a.value 1
smax4dc7a8e588e9bb787fe6e39c1c96957a.value 2
scura92191189150289a82c2c4464351178c.value 0
smaxa92191189150289a82c2c4464351178c.value 5

当我从 Munin 服务器远程连接时，我得到的是：

bash-4.4$ /usr/share/munin/munin-update --debug --nofork --host servxxx.domain.tld --service haproxy_session_frontend
2023/10/05 17:12:57 [DEBUG] Creating new lock file /var/run/munin/munin-update.lock
2023/10/05 17:12:57 [DEBUG] Creating lock : /var/run/munin/munin-update.lock succeeded
2023/10/05 17:12:57 [INFO]: Starting munin-update
2023/10/05 17:12:58 [DEBUG] Creating new lock file /var/run/munin/munin-datafile.lock
2023/10/05 17:12:58 [DEBUG] Creating lock : /var/run/munin/munin-datafile.lock succeeded
2023/10/05 17:12:58 [DEBUG] Writing state to /var/lib/munin/datafile.storable
2023/10/05 17:12:58 [DEBUG] about to write '/var/lib/munin/datafile.storable'
2023/10/05 17:12:58 [INFO]: Munin-update finished (1.47 sec)

但我的 rrd 文件充满了 NaN，并且在 munin-update.log 中，每次 Munin 尝试从节点更新数据时我都会收到此警告：

2023/10/05 17:15:13 [WARNING] 6 lines had errors while 0 lines were correct (100.00%) in data from 'fetch haproxy_sessions_frontend' on haproxy.captcha/x.x.x.x:4949

我已经尝试了此页面https://guide.munin-monitoring.org/en/latest/tutorial/troubleshooting.html中的几乎所有内容，但我一无所知。相同的插件在另一个 haproxy 服务器上运行良好。主要区别是另一台服务器运行的是 Rocky Linux 8，而有问题的服务器运行的是 Rocky Linux 9

我该怎么做才能让这个插件运行？

我有以下 haproxy 配置，它access-control-allow-origin使用以下配置在成功的 200 个请求上添加标头。我的问题是，当我遇到超时或 haproxy 本身（不是我的上游服务器）因其他原因引发错误时，不会添加此标头。如何在例如 504 响应中添加此标头？

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA>
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5s
        timeout client  30s
        timeout server  30s
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend www-https
    bind *:80
    bind *:443 ssl crt /etc/ssl/cert.io.pem

    http-response add-header access-control-allow-origin "myCoolWebsite.com"

    redirect scheme https code 301 if !{ ssl_fc }
    mode http

    default_backend myBackend


backend myBackend
  server myServer 123.456.789.101:2345

编辑：我主要通过为 504 添加自定义错误文件来解决这个问题，如下所示。唯一的问题是我无法弄清楚如何将 动态设置access-control-allow-origin为仅当前请求的域而不是*. 这是一个问题，因为某些浏览器在接收*. 我有什么想法可以做到这一点吗？

// 504.http
HTTP/1.0 504 Gateway Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods:\ GET,\ HEAD,\ OPTIONS,\ POST,\ PUT
Access-Control-Allow-Credentials:\ true
Access-Control-Allow-Headers:\ Origin,\ Accept,\ X-Requested-With,\ Content-Type,\ Access-Control-Request-Method,\ Access-Control-Request-Headers,\ Authorization

{
  "html": {
    "body": {
      "h1": "504 Gateway Time-out",
      "#text": "The server didn't respond in time."
    }
  }
}

我们在 HA (Wildcard.pem)上使用 Godaddy 通配符证书，在后端使用自定义 CA 证书。内部 CA 受到 HA 和所有服务器的信任。我可以毫无问题地单独访问所有后端服务器，但是当我们启用 SSL 验证时，健康检查几乎立即失败并出现 503 错误。

高可用性配置

frontend api.qa.domain.com_esil_HTTPS_443_VS_172.20.25.69
    mode http
    bind 172.20.25.69:80
    bind 172.20.25.69:443 ssl crt /etc/hapee-2.4/ssl/Wildcard.pem ciphers ECDHE-RSA-AES128-CBC-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-CBC-SHA:AES128-SHA:AES256-SHA:ECDHE-ECDSA-AES128-SHA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS ssl-min-ver TLSv1.2
    http-after-response replace-header set-cookie "^(my.session=.*)" "\1; SameSite=Strict" 
    http-request set-var(txn.path) path
    http-request set-var(req.backend) str(api.qa.domain.com_loggingservices_http_33313_POOL) if { path -i -m beg /loggingservice/ }
    
    use_backend %[var(req.backend)]

    backend api.qa.domain.com_loggingservice_http_333_POOL
    mode http
    http-request return status 503 content-type "text/html; charset=utf-8" lf-file /etc/hapee-2.4/apps/app_offline.htm if { nbsrv() le 0 }

    balance roundrobin
    dynamic-cookie-key Av3ryS3curek3y
    option httpchk GET /loggingservice/v1/_healthcheck HTTP/1.1
    http-check send hdr host api.qa.domain.com
    http-check expect status 200
    server QA_SERVER1_Node 10.133.20.24:333 check ssl verify none

我是否需要将我们使用 CA 创建的后端证书与 GoDaddy pem 文件合并？如果是，正确的顺序是什么？

Haproxy专家，

我无法让 haproxy 统计页面在后端使用 TCP 模式。

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    tcp
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
    stats enable
    stats hide-version
    stats refresh 30s
    stats show-node
    stats auth admin:xxxxxxxxxxxxxxxxx
    stats uri /haproxy?stats

frontend main
    listen stats
    bind *:5000
    mode tcp
    default_backend             app

backend app
    balance     roundrobin
#backends
    server   server1  xxx.xxx.xxx.xxx:10222 check
    server   server2  xxx.xxx.xxx.xxx:34013 check

我的 haproxy 配置如下，后端有 pg_autoctl 集群，另一个 VM (bunty4) 托管监视器并安装了 haproxy。

global
    maxconn 100

defaults
    log global
    mode tcp
    retries 2
    timeout client 30m
    timeout connect 4s
    timeout server 30m
    timeout check 5s

listen stats
    mode tcp
    bind *:7000
    stats enable
    stats uri /

listen ReadWrite
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server bunty1 bunty1:6001 maxconn 100 check port 23267
    server bunty2 bunty2:6002 maxconn 100 check port 23267
    server bunty3 bunty3:6003 maxconn 100 check port 23267

listen ReadOnly
    bind *:5001
    option httpchk
    http-check expect status 206
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server bunty1 bunty1:6001 maxconn 100 check port 23267
    server bunty2 bunty2:6002 maxconn 100 check port 23267
    server bunty3 bunty3:6003 maxconn 100 check port 23267

这运行良好：

postgres@bunty4:~$ psql -h bunty2 -p 6002
psql (14.6 (Ubuntu 14.6-1.pgdg22.04+1))
Type "help" for help.

postgres=# \q
postgres@bunty4:~$ psql -h bunty1 -p 6001
psql (14.6 (Ubuntu 14.6-1.pgdg22.04+1))
Type "help" for help.

当我尝试连接前端端口 7000（通过 pgadmin 或 cli）时，出现错误：

postgres@bunty4:~$ psql -h localhost -p 7000
psql: error: connection to server at "localhost" (127.0.0.1), port 7000 failed: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

psql -h 192.168.5.129 -p 7000   ## this also fails with same error.

不知道为什么，haproxy.log 没有抛出任何东西。

我最近尝试在 HAProxy 上为 mongodbmongodb+srv协议连接设置基于 SNI 的路由。

我让它工作，但直到我把它

tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }

在我的前端配置中它开始正常工作

如果没有这些（或只有其中之一），我的大约 70% 的请求会不断收到零星的连接重置。请注意，有时连接确实能够建立。

根据http://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4-tcp-request%20content

基于内容的规则按照它们的确切声明顺序进行评估。如果没有规则匹配或没有规则，则默认操作是接受内容。可插入的规则数量没有具体限制。

所以它应该默认工作。

要么是我遗漏了一些基本的 TCP 概念，要么是一些特定的 Mongo 连接细节。但我注意到几乎每个人在选择基于 SNI 的路由时都定义了相同的规则，因此它对大多数人来说一定是有意义的。

工作 HAProxy 配置：

    frontend fe_mongo-nonprod
      bind :27017
      tcp-request inspect-delay 5s
      tcp-request content accept if { req_ssl_hello_type 1 }

      acl mongoAtlas-01 req_ssl_sni -i host1.mongodb.net
      acl mongoAtlas-02 req_ssl_sni -i host2.mongodb.net
      acl mongoAtlas-03 req_ssl_sni -i host3.mongodb.net
      use_backend be_mongo-nonprod if mongoAtlas-01 || mongoAtlas-02 || mongoAtlas-03

    backend be_mongo-nonprod
      mode tcp

      acl mongoAtlas-01 req_ssl_sni -i host1.mongodb.net
      acl mongoAtlas-02 req_ssl_sni -i host2.mongodb.net
      acl mongoAtlas-03 req_ssl_sni -i host3.mongodb.net

      use-server server-01 if mongoAtlas-01
      use-server server-02 if mongoAtlas-02
      use-server server-03 if mongoAtlas-03

      server server-01 host1.mongodb.net:27017
      server server-02 host2.mongodb.net:27017
      server server-03 host3.mongodb.net:27017

还值得一提的是，HAProxy 部署在 Kubernetes（GCP 中由 Google 管理的集群）中，具有 istio-sidecar 并通过 Internal LoadBalancer k8s 服务公开。

如上所述，上述配置运行良好。我感兴趣的是为什么tcp-request content accept绝对需要并且这里默认情况下 [always] 不接受连接。

我创建了一个简单的应用程序，并在 4 个不同的容器中运行它。

我设法将所有端口代理到绑定 9991，但是当我创建 acl 以使用不同的后端时它不起作用，只有默认后端。

配置：

frontend tests
  bind *:9991

  timeout client 60s
  mode http

  acl is_servers_2 path_end /app2
  use_backend servers_2 if is_servers_2

  acl is_servers_3 path_end /app3
  use_backend servers_3 if is_servers_3

  default_backend servers_2

backend servers_2
  timeout connect 10s
  timeout server 60s
  mode http
  server server3001 127.0.0.1:3001
  server server3002 127.0.0.1:3002

backend servers_3
  timeout connect 10s
  timeout server 60s
  mode http
  server server3003 127.0.0.1:3003
  server server3004 127.0.0.1:3004

代理版本：2.4.4

Ubuntu的：18.04

我向 haproxy 发出相同的请求（POST /user/oauth），但有时我得到 503，直接向后端发送请求总是有效

这是日志：

1. local_server~ app/s2 0/0/0/100/100 200 813 - - --VN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
2. local_server~ app/s1 0/0/1/107/108 200 820 - - --NI 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
3. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 - - SCNN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
4. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 - - SCNN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
5. local_server~ app/s2 0/0/1/106/107 200 831 - - --NI 2/2/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
6. local_server~ app/s1 0/0/1/3/6206 101 337 - - --NI 3/3/1/1/0 0/0 “GET / HTTP/1.1”
7. local_server~ app/s1 0/0/1/83/84 200 258 - - --VN 2/2/0/0/0 0/0 "POST /transaction/concludeTransaction HTTP/1.1"
8. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 - - SCNN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
9. local_server~ app/s2 0/0/1/107/108 200 831 - - --NI 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"

如您所见，POST /user/oauth 有时会转到 s1，有时会转到 s2，有时会转到 NOSRV。

有时它连续通过 10 次，有时它连续失败 10 次。

这是我的配置：

defaults
log     global
mode    http
option  httplog
option  dontlognull
timeout connect 10000
timeout client  50000
timeout server  50000
retries 3
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend local_server
bind *:8080 ssl crt /etc/letsencrypt/live/xxx/sslcert.pem
mode http
default_backend app

backend app
balance roundrobin
cookie SERVERID insert indirect nocache
http-send-name-header Host
server s1 172.26.13.134:3000 check cookie s1
server s2 172.26.13.134:3000 check cookie s2
http-request add-header X-Server %b

编辑：

我确实将请求重定向到在本地主机上运行的快速 python 服务器，但问题仍然存在。所以我确实编辑了 haproxy 配置文件，只有服务器 s1 127.0.0.1:8086 检查 cookie s1

并运行 python 脚本： python3 -m http.server 8086 --bind 127.0.0.1&

当我从邮递员那里击中时，我连续获得了 3 次成功，而不是 503。