Início / server / Perguntas / 1152680
Accepted
codlord
Asked: 2024-02-01 02:16:39 +0800 CST

Prometheus Blackbox http_2xx retornando 403 desde que o servidor web migrou para StackCP

  • 772

Eu estava trabalhando com verificações http_2xx do Prometheus Blackbox Exporter monitorando vários servidores da web. Em seguida, o provedor de hospedagem migrou do cPanel para o Stack CP.

Desde então, todos os http_2xx retornam apenas erros 403 (Proibido) e não consigo entender o porquê, pois posso executar comandos wgete curlcomandos com êxito do mesmo servidor/IP Prometheus. Não consigo recriar um erro 403 com outros comandos.

Alguém pode fazer alguma sugestão de por que esse pode ser o caso ou como depurar ainda mais?

Eu verifiquei a configuração do StackCP e não parece haver nada que proíba essas verificações de monitoramento.

Não consigo ver nada nos logs do servidor web sobre erros 403.

blackbox.ymla configuração começa:

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      method: GET

E também tentei pensar que poderia ser um problema de SSL/certificado, mas isso não faz diferença:

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      method: GET
      tls_config:
        insecure_skip_verify: true

E trecho do syslog do servidor prometheus:

Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Beginning probe" probe=http timeout_seconds=9.5
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip6
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip4
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolved target address" ip=185.151.30.208
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Making HTTP request" url=https://185.151.30.208 host=southcoastgroup.co.uk
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Received HTTP response" status_code=403
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Invalid HTTP response status code, wanted 2xx" status_code=403

Ubuntu 22.04.3 LTS/prometheus, versão 2.31.2+ds1/blackbox_exporter, versão 0.19.0

EDIT 1 conforme sugerido por @AlexD, embora os resultados não signifiquem muito para mim. Posso stracevê-lo conectando-se ao servidor web (185.151.30.208), mas não consigo ver nenhum erro 403 ou qualquer motivo óbvio para o problema.

tcpdump:

12:38:35.206371 IP (tos 0x0, ttl 64, id 10246, offset 0, flags [DF], proto TCP (6), length 60)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [S], cksum 0x9a70 (incorrect -> 0xce3d), seq 709527684, win 64240, options [mss 1460,sackOK,TS val 1145455779 ecr 0,nop,wscale 7], length 0

12:38:35.212490 IP (tos 0x0, ttl 64, id 10247, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xd0f0), ack 1090810164, win 502, options [nop,nop,TS val 1145455785 ecr 1774061831], length 0

12:38:35.213068 IP (tos 0x0, ttl 64, id 10248, offset 0, flags [DF], proto TCP (6), length 351)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9b93 (incorrect -> 0x362b), seq 0:299, ack 1, win 502, options [nop,nop,TS val 1145455786 ecr 1774061831], length 299

12:38:35.219896 IP (tos 0x0, ttl 64, id 10249, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc71e), ack 2207, win 496, options [nop,nop,TS val 1145455792 ecr 1774061839], length 0

12:38:35.221063 IP (tos 0x0, ttl 64, id 10250, offset 0, flags [DF], proto TCP (6), length 116)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9aa8 (incorrect -> 0x5ff2), seq 299:363, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 64

12:38:35.221177 IP (tos 0x0, ttl 64, id 10251, offset 0, flags [DF], proto TCP (6), length 138)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9abe (incorrect -> 0xe1f8), seq 363:449, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 86

12:38:35.221291 IP (tos 0x0, ttl 64, id 10252, offset 0, flags [DF], proto TCP (6), length 125)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9ab1 (incorrect -> 0xdb93), seq 449:522, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 73

12:38:35.227011 IP (tos 0x0, ttl 64, id 10253, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc543), ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 0

12:38:35.227070 IP (tos 0x0, ttl 64, id 10254, offset 0, flags [DF], proto TCP (6), length 83)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a87 (incorrect -> 0x9012), seq 522:553, ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 31

12:38:35.229169 IP (tos 0x0, ttl 64, id 10255, offset 0, flags [DF], proto TCP (6), length 76)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a80 (incorrect -> 0xa0f1), seq 553:577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 24

12:38:35.229208 IP (tos 0x0, ttl 64, id 10256, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [F.], cksum 0x9a68 (incorrect -> 0xc458), seq 577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 0

12:38:35.234976 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0

12:38:35.235003 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0

12:38:35.235026 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b5 (correct), seq 709528263, win 0, length 0

Trecho de sudo strace -f -o file -p 3043635:

3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043960 <... futex resumed>)            = 1

3043638 <... futex resumed>)            = 0

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

3043960 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043960 <... socket resumed>)           = 8

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043960 connect(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, 16) = -1 EINPROGRESS (Operation now in progress)

3043636 <... nanosleep resumed>NULL)    = 0

3043960 epoll_ctl(5, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=2636876800, u64=140362168110080}} <unfinished ...>

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043960 <... epoll_ctl resumed>)        = 0

3043960 write(7, "\0", 1)               = 1

3043636 <... nanosleep resumed>NULL)    = 0

3043958 <... epoll_pwait resumed>[{events=EPOLLIN, data={u32=15774768, u64=15774768}}], 128, 9499, NULL, 2321375387197960) = 1

3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043960 <... futex resumed>)            = 1

3043958 read(6,  <unfinished ...>

3043638 <... futex resumed>)            = 0

3043960 futex(0xc000081148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

3043958 <... read resumed>"\0", 16)     = 1

3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043958 epoll_pwait(5,  <unfinished ...>

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043958 <... epoll_pwait resumed>[], 128, 0, NULL, 2321375387197960) = 0

3043958 epoll_pwait(5,  <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043636 futex(0xedd178, FUTEX_WAIT_PRIVATE, 0, {tv_sec=9, tv_nsec=463217921} <unfinished ...>

3043958 <... epoll_pwait resumed>[{events=EPOLLOUT, data={u32=2636876800, u64=140362168110080}}], 128, 9463, NULL, 2321375387197773) = 1

3043958 futex(0xedd178, FUTEX_WAKE_PRIVATE, 1) = 1

3043958 getsockopt(8, SOL_SOCKET, SO_ERROR,  <unfinished ...>

3043636 <... futex resumed>)            = 0

3043958 <... getsockopt resumed>[0], [4]) = 0

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043958 getpeername(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, [112 => 16]) = 0

3043636 <... nanosleep resumed>NULL)    = 0

3043958 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1) = 1

3043638 <... futex resumed>)            = 0

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043958 getsockname(8,  <unfinished ...>

3043638 epoll_pwait(5,  <unfinished ...>

EDIT 2 , conforme observado pelos opensslcomandos @DazWilkin, estavam apresentando um erro no meu servidor ( Verify return code: 21 (unable to verify the first certificate). Depois de verificar o site, https://www.ssllabs.comparecia que havia downloads extras nos caminhos de certificação e estavam faltando alguns certificados intermediários no meu servidor. Então, consertei isso e agora os opensslcomandos funcionam sem erros de certificado.

No entanto, Blackbox http_2xx ainda retorna 403

ubuntu

1 respostas

  1. Best Answer

    ok, então finalmente descobri. Por algum motivo, o servidor da web migrado foi rejeitado com base no User-Agent (ou talvez na falta de um User-Agent, pois não tenho certeza do que ou se o probe http2_xx envia um User-Agent). Mas tudo que eu precisava fazer era adicionar um User-Agent falso, semelhante a um navegador, no meu caso, assim:

    modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      method: GET
      headers:
        User-Agent: "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
    • 0

relate perguntas