在Python中创建了web抓取程序，需要将web链接保存到csv并删除重复项

Question

我已经能够成功地刮网站，并有一些困难，作为一个csv保存，需要帮助，看看我在哪里搞砸了。这是我的代码，我还包括了一段代码：

import bs4 as BeautifulSoup
import CSV
import re
import urllib.request
from IPython.display import HTML

# Program that scraps the website for 
r= urllib.request.urlopen('https://www.census.gov/programs- 
                           surveys/popest.html').read()
soup = BeautifulSoup(r,"html.parser")


for link in soup.find_all('a'):
    print(link.get('href'))

with open("Giles_C996.csv","w") as csv_file:
    writer = csv.writer(csv_file,delimiter="/n")
    writer.writerow(Links)

Close()

错误消息：

回溯（最近一次呼叫最后一次）： 文件“C:\Users\epiph\Giles_C996 Project 2.txt”，第2行，在 导入CSV ModuleNotFoundError:没有名为“CSV”的模块

My Code

Answer 1

您错误地导入了csv和bs4模块。而且Close()是不正确的。您可以使用到set的转换来消除重复项

import csv
import urllib.request
from bs4 import BeautifulSoup

r = urllib.request.urlopen('https://www.census.gov/programs-surveys/popest.html').read()    
soup = BeautifulSoup(r, "html.parser")
links = set([a['href'] for a in soup.find_all('a', href=True)])

with open("Giles_C996.csv", "w", newline='') as f:
    writer = csv.writer(f)
    writer.writerows([link] for link in links)

输出为：

https://www.census.gov/programs-surveys/cps.html
/newsroom/press-releases/2020/65-older-population-grows/65-older-population-grows-spanish.html
https://www.census.gov/businessandeconomy
https://www.census.gov/data
/programs-surveys/popest/library.html

等等