主页 / coding / 问题 / 78833889
Accepted
Julio640
Asked: 2024-08-05 18:02:56 +0800 CST

在 R 中进行网页抓取，页面没有响应请求

  • 772

我大多数时候无法向以下网站发出请求：

https://www.adondevivir.com/proyectos-etapa-pre-venta-en-construccion.html

library(rvest);library(tibble);library(httr2)

base_url <- "https://www.adondevivir.com/proyectos-etapa-pre-venta-en-construccion.html"

parsed_base_url <- base_url |> 
  read_html()  # This works sometimes and I get the underlying html

# THIS NEVER WORKS
pagina_parsed <- base_url |> 
  request() |> 
  req_user_agent(
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
  ) |> 
  req_headers(
    Referer = "https://www.adondevivir.com/",
    Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    `Accept-Language` = "es-419,es;q=0.6",
    `Accept-Encoding` = "gzip, deflate, br, zstd",
    `Cache-Control` = "max-age=0",
    `Sec-Ch-Ua` = '"Brave";v="125", "Chromium";v="125", "Not.A/Brand";v="24"',
    Priority = "u=0, i"
  ) |> 
  req_perform()

为什么大多数时候我都无法向页面发出请求（更不用说它不适用于上面提供的标头的 httr2）？有没有办法克服这个“问题” httr2？这与 cookie 有关，还是页面保护自己不被抓取的方式有关？

我想我可以重试很多次该请求直到它起作用，但我认为我不会了解太多关于它不起作用的原因。

1 个回答

  1. Best Answer

    这是否与 cookie 有关，或者是页面保护自身不被抓取的方式？

    两者都有。它受 Cloudflare 的保护，Cloudflare 通过一系列诡计来混淆自动化工具和无头浏览器。如果您在新会话或浏览器的隐身模式下打开该网站，您应该会看到 Cloudflare Javascript 挑战的实际效果。如果您打开了 DevTools 的网络选项卡（启用了保留日志，也许还启用了节流以减慢速度），您应该会看到一些有关探测内容的其他提示。如果 Cloudflare 发现您的请求合法，则会设置 cookie 以在您的会话期间授予通过权。您可以使用 重用它们httr2。

    半手动方法可能是这样的：

    • 在浏览器中打开页面，通过Cloudflare
    • 打开 DevTools 并再次导航到相同的 URL
    • 将请求复制为 cURL（在 DevTools 的网络选项卡中左键单击请求）
    • 传递给httr2::curl_translate()
    library(rvest)
library(httr2)

# translate curl to httr2:
curl_translate(r"(curl 'https://www.adondevivir.com/proyectos-etapa-pre-venta-en-construccion.html' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8' \
  -H 'accept-language: en-GB,en;q=0.9' \
  -H 'cookie: __cf_bm=MbgHRAOsR8nrNwVtUAXzb0HMRrBSK4hbiiNtO3mg41A-1722860033-1.0.1.1-WvLWPer9d5s3PyuFjRZeCwyIRmiyELE5bs40JWH4Txc4OZXWvFUaSgqUbvbf_xYKnpePSYWv5GY4btcLBR_vvc6pt01F1sLPXt0QVrPolJk; sessionId=b5a7a506-4689-4188-88e7-fe005fc154ab; cf_clearance=vchooYPfzwbG.fmcao_YheRm7DPILRmr8xhNSWLUabQ-1722860102-1.0.1.1-tYQZHx6sTEzzRUS.UHz1rjZNq1a1VcSoOcn7l0EjRqbFeHNUwzsHyhTjW2R0RU_Tnv.6L5WbxxEy8m3xxcepaw' \
  -H 'priority: u=0, i' \
  -H 'sec-ch-ua: "Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-model: ""' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-ch-ua-platform-version: "15.0.0"' \
  -H 'sec-fetch-dest: document' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-site: none' \
  -H 'sec-fetch-user: ?1' \
  -H 'sec-gpc: 1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36')")
#> request("https://www.adondevivir.com/proyectos-etapa-pre-venta-en-construccion.html") |> 
#>   req_headers(
#>     accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
#>     `accept-language` = "en-GB,en;q=0.9",
#>     cookie = "__cf_bm=MbgHRAOsR8nrNwVtUAXzb0HMRrBSK4hbiiNtO3mg41A-1722860033-1.0.1.1-WvLWPer9d5s3PyuFjRZeCwyIRmiyELE5bs40JWH4Txc4OZXWvFUaSgqUbvbf_xYKnpePSYWv5GY4btcLBR_vvc6pt01F1sLPXt0QVrPolJk; sessionId=b5a7a506-4689-4188-88e7-fe005fc154ab; cf_clearance=vchooYPfzwbG.fmcao_YheRm7DPILRmr8xhNSWLUabQ-1722860102-1.0.1.1-tYQZHx6sTEzzRUS.UHz1rjZNq1a1VcSoOcn7l0EjRqbFeHNUwzsHyhTjW2R0RU_Tnv.6L5WbxxEy8m3xxcepaw",
#>     priority = "u=0, i",
#>     `sec-gpc` = "1",
#>     `upgrade-insecure-requests` = "1",
#>     `user-agent` = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
#>   ) |> 
#>   req_perform()

# make request, parse html response 
request("https://www.adondevivir.com/proyectos-etapa-pre-venta-en-construccion.html") |> 
  req_headers(
    accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    `accept-language` = "en-GB,en;q=0.9",
    cookie = "__cf_bm=MbgHRAOsR8nrNwVtUAXzb0HMRrBSK4hbiiNtO3mg41A-1722860033-1.0.1.1-WvLWPer9d5s3PyuFjRZeCwyIRmiyELE5bs40JWH4Txc4OZXWvFUaSgqUbvbf_xYKnpePSYWv5GY4btcLBR_vvc6pt01F1sLPXt0QVrPolJk; sessionId=b5a7a506-4689-4188-88e7-fe005fc154ab; cf_clearance=vchooYPfzwbG.fmcao_YheRm7DPILRmr8xhNSWLUabQ-1722860102-1.0.1.1-tYQZHx6sTEzzRUS.UHz1rjZNq1a1VcSoOcn7l0EjRqbFeHNUwzsHyhTjW2R0RU_Tnv.6L5WbxxEy8m3xxcepaw",
    priority = "u=0, i",
    `sec-gpc` = "1",
    `upgrade-insecure-requests` = "1",
    `user-agent` = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
  ) |> 
  req_perform() |> 
  resp_body_html() |> 
  html_elements("h3[data-qa='POSTING_CARD_DESCRIPTION']") |> 
  html_text() |> 
  head() |> 
  stringr::str_trunc(80)
#> [1] "Vive en el distrito patriota de Lima, hogar de libertadores. Disfruta de una ..."  
#> [2] "Obra en curso - 47% vendido! Un proyecto inigualable ubicado en la zona monum..."  
#> [3] "¡Vive frente al Campo de Marte en Jesús María! Presentamos \"Salaverry 571\", u..."
#> [4] "¡Vive en la mejor zona de Surquillo! Lobby, Sala de niños, Sala de Usos Multi..."  
#> [5] "Grupo Lar, única inmobiliaria en Perú, con presencia en 5 países en simultáne..."  
#> [6] "Proyecto exclusivo en la Nueva Santa Catalina, a 5 minutos del centro financi..."

    创建于 2024-08-05，使用reprex v2.1.0

    • 4

相关问题