我正在使用 Prometheus Blackbox Exporter http_2xx 检查监控各种 Web 服务器。然后,Web 托管提供商从 cPanel 迁移到 Stack CP。
从那时起,所有 http_2xx 都只返回 403(禁止)错误,我无法弄清楚为什么,因为我可以成功运行来自同一个 Prometheus 服务器/IP 的命令wget
。curl
我似乎无法使用其他命令重新创建 403 错误。
任何人都可以提出任何建议,为什么会出现这种情况或如何进一步调试?
我检查了 StackCP 配置,似乎没有任何内容会禁止这些监控检查。
我在 Web 服务器日志中看不到任何有关 403 错误的信息。
blackbox.yml
配置开始:
modules:
http_2xx:
prober: http
timeout: 10s
http:
method: GET
我也尝试过这个,认为这可能是 SSL/证书问题,但这没有什么区别:
modules:
http_2xx:
prober: http
timeout: 10s
http:
method: GET
tls_config:
insecure_skip_verify: true
以及普罗米修斯服务器的系统日志摘录:
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Beginning probe" probe=http timeout_seconds=9.5
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip6
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip4
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolved target address" ip=185.151.30.208
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Making HTTP request" url=https://185.151.30.208 host=southcoastgroup.co.uk
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Received HTTP response" status_code=403
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Invalid HTTP response status code, wanted 2xx" status_code=403
Ubuntu 22.04.3 LTS / prometheus,版本 2.31.2+ds1 / blackbox_exporter,版本 0.19.0
按照@AlexD的建议编辑1,尽管输出对我来说意义不大。在 中strace
,我可以看到它连接到网络服务器(185.151.30.208),但我看不到任何 403 错误或任何明显的问题原因。
tcpdump
:
12:38:35.206371 IP (tos 0x0, ttl 64, id 10246, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.50.42780 > 185.151.30.208.443: Flags [S], cksum 0x9a70 (incorrect -> 0xce3d), seq 709527684, win 64240, options [mss 1460,sackOK,TS val 1145455779 ecr 0,nop,wscale 7], length 0
12:38:35.212490 IP (tos 0x0, ttl 64, id 10247, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xd0f0), ack 1090810164, win 502, options [nop,nop,TS val 1145455785 ecr 1774061831], length 0
12:38:35.213068 IP (tos 0x0, ttl 64, id 10248, offset 0, flags [DF], proto TCP (6), length 351)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9b93 (incorrect -> 0x362b), seq 0:299, ack 1, win 502, options [nop,nop,TS val 1145455786 ecr 1774061831], length 299
12:38:35.219896 IP (tos 0x0, ttl 64, id 10249, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc71e), ack 2207, win 496, options [nop,nop,TS val 1145455792 ecr 1774061839], length 0
12:38:35.221063 IP (tos 0x0, ttl 64, id 10250, offset 0, flags [DF], proto TCP (6), length 116)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9aa8 (incorrect -> 0x5ff2), seq 299:363, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 64
12:38:35.221177 IP (tos 0x0, ttl 64, id 10251, offset 0, flags [DF], proto TCP (6), length 138)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9abe (incorrect -> 0xe1f8), seq 363:449, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 86
12:38:35.221291 IP (tos 0x0, ttl 64, id 10252, offset 0, flags [DF], proto TCP (6), length 125)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9ab1 (incorrect -> 0xdb93), seq 449:522, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 73
12:38:35.227011 IP (tos 0x0, ttl 64, id 10253, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc543), ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 0
12:38:35.227070 IP (tos 0x0, ttl 64, id 10254, offset 0, flags [DF], proto TCP (6), length 83)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a87 (incorrect -> 0x9012), seq 522:553, ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 31
12:38:35.229169 IP (tos 0x0, ttl 64, id 10255, offset 0, flags [DF], proto TCP (6), length 76)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a80 (incorrect -> 0xa0f1), seq 553:577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 24
12:38:35.229208 IP (tos 0x0, ttl 64, id 10256, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [F.], cksum 0x9a68 (incorrect -> 0xc458), seq 577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 0
12:38:35.234976 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0
12:38:35.235003 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0
12:38:35.235026 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b5 (correct), seq 709528263, win 0, length 0
摘录自sudo strace -f -o file -p 3043635
:
3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043960 <... futex resumed>) = 1
3043638 <... futex resumed>) = 0
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
3043960 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043960 <... socket resumed>) = 8
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043960 connect(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, 16) = -1 EINPROGRESS (Operation now in progress)
3043636 <... nanosleep resumed>NULL) = 0
3043960 epoll_ctl(5, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=2636876800, u64=140362168110080}} <unfinished ...>
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043960 <... epoll_ctl resumed>) = 0
3043960 write(7, "\0", 1) = 1
3043636 <... nanosleep resumed>NULL) = 0
3043958 <... epoll_pwait resumed>[{events=EPOLLIN, data={u32=15774768, u64=15774768}}], 128, 9499, NULL, 2321375387197960) = 1
3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043960 <... futex resumed>) = 1
3043958 read(6, <unfinished ...>
3043638 <... futex resumed>) = 0
3043960 futex(0xc000081148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
3043958 <... read resumed>"\0", 16) = 1
3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043958 epoll_pwait(5, <unfinished ...>
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043958 <... epoll_pwait resumed>[], 128, 0, NULL, 2321375387197960) = 0
3043958 epoll_pwait(5, <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043636 futex(0xedd178, FUTEX_WAIT_PRIVATE, 0, {tv_sec=9, tv_nsec=463217921} <unfinished ...>
3043958 <... epoll_pwait resumed>[{events=EPOLLOUT, data={u32=2636876800, u64=140362168110080}}], 128, 9463, NULL, 2321375387197773) = 1
3043958 futex(0xedd178, FUTEX_WAKE_PRIVATE, 1) = 1
3043958 getsockopt(8, SOL_SOCKET, SO_ERROR, <unfinished ...>
3043636 <... futex resumed>) = 0
3043958 <... getsockopt resumed>[0], [4]) = 0
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043958 getpeername(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, [112 => 16]) = 0
3043636 <... nanosleep resumed>NULL) = 0
3043958 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1) = 1
3043638 <... futex resumed>) = 0
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043958 getsockname(8, <unfinished ...>
3043638 epoll_pwait(5, <unfinished ...>
@DazWilkin 注意到的编辑 2openssl
命令在我的服务器上出现错误(Verify return code: 21 (unable to verify the first certificate)
。在检查了网站后,https://www.ssllabs.com
证书路径中似乎有额外的下载,并且我的服务器上丢失了几个中间证书。所以我修复了该问题现在这些openssl
命令可以正常工作,没有任何证书错误。
但是,Blackbox http_2xx 仍然返回 403
好吧,我终于明白了。由于某种原因,迁移的网络服务器基于用户代理而拒绝(或者可能缺少用户代理,因为我不确定 http2_xx 探针发送什么或是否发送用户代理)。但我需要做的就是在我的例子中添加一个类似浏览器的假用户代理,如下所示: