主页 / coding / 问题 / 79012310
Accepted
Sharif
Asked: 2024-09-23 02:26:38 +0800 CST

来自 BeautifulSoup 对象的数据框

  • 772

我想从 BeautifulSoup 对象创建一个数据框 -

import pandas as pd
from requests import get
from bs4 import BeautifulSoup
import re

# Fetch the web page
url = 'https://carbondale.craigslist.org/search/apa#search=1~gallery~0~0'
response = get(url) # link exlcudes posts with no picures
page = response.text

# Parse the HTML content
soup = BeautifulSoup(page, 'html.parser')

# Information I need 
list_url = []
title = []
location = []
price = []

# I run the following 
list_url = [a['href'] for a in soup.select('a[href^="https"]')]
title = [x.text for x in soup.find_all(class_="title")]
location = [x.text for x in soup.find_all(class_="location")]
price = [x.text for x in soup.find_all(class_="price")]

但我面临的问题是，对于某些类别（例如，标题或位置），缺少一些元素，因此，当我尝试创建数据框时，它会显示错误，因为None值不相等，因为所有列表的大小都不相等。您可以使用该len()函数检查列表的大小。实际上，我想在数据框的列中包含“无”一词来表示缺失的元素。

python

1 个回答

  1. Best Answer

    您需要遍历页面中的每个列表，并逐一将值添加到list_url、list_location和list_title。list_price如果其中任何一个值缺失，则将 None 添加到相应的列表中。然后，您可以使用列表创建 DataFrame。

    要迭代列表，我必须查看行的结构，并注意到li class="cl-static-search-result"使用了。然后，您可以迭代此列表以找到所需的值，而不是find_all在整个页面上使用，因为这样做不会考虑列表中项目之间的关系。

    尝试一下：

    import pandas as pd
from requests import get
from bs4 import BeautifulSoup


# Fetch the web page
url = 'https://carbondale.craigslist.org/search/apa#search=1~gallery~0~0'
response = get(url) # link exlcudes posts with no picures
page = response.text

# Parse the HTML content
soup = BeautifulSoup(page, 'html.parser')

# Extract listings from the page
listings = soup.find_all('li', class_='cl-static-search-result')

for listing in listings:
    # Extract URL
    url = listing.find('a')
    list_url.append(url['href'] if url else None)

    # Extract Title
    title_text = listing.find('div', class_='title')
    title.append(title_text.text if title_text else None)
    
    # Extract Location
    location_text = listing.find('div', class_='location')
    location.append(location_text.text.strip() if location_text else None)
    
    # Extract Price
    price_text = listing.find('div', class_='price')
    price.append(price_text.text if price_text else None)

# Create DataFrame
df = pd.DataFrame({
    'URL': list_url,
    'Title': title,
    'Location': location,
    'Price': price
})

    打印出前 5 行

                                                     URL                                              Title        Location Price
0  https://carbondale.craigslist.org/apa/d/johnst...                Almost New, 2 BR APT for Rent in JC   Johnston City  $900
1  https://carbondale.craigslist.org/apa/d/northb...  Enjoy 2 Bed/2 Bath/2 Car Ranch Home With Great...  Northbrook, IL  $945
2  https://carbondale.craigslist.org/apa/d/mount-...  Love where you live! Beautiful senior communit...    Mount Vernon  $848
3  https://carbondale.craigslist.org/apa/d/mount-...  Your dream 1 bed, 1 bath is closer than you th...    Mount Vernon  $848
4  https://carbondale.craigslist.org/apa/d/mount-...  Be at the center of it all: 1 BR, 1 BA, 553 Sq...    Mount Vernon  $848

    对于任何列表，如果值不可用或我们找不到它，我们会附加 None。缺少值的数据框将如下所示：

                                                      URL                                              Title Location   Price
16  https://carbondale.craigslist.org/apa/d/marion...  Cfd housing Houses for rent or purchase southe...     None    $500
17  https://carbondale.craigslist.org/apa/d/herrin...                         4 Bedroom/2 bathroom House     None  $1,561
    • 1

相关问题