用Python获取“真实”的Google搜索结果

Question

我想做一件很简单的事情，就是获取谷歌搜索的第一条结果链接。例如，当我搜索 LMFAO Yes at Pandora（这里的 at 是故意加上的）时，我希望能得到这个链接 www.pandora.com/music/song/lmfao/yes。其实，所有主要的搜索引擎都能轻松返回这个结果。

在StackOverflow上，大家通常建议使用 Google AJAX API，但说实话，这个东西真是糟糕透了。它返回的结果完全是奇怪的，比如：

http://lyrics.wikia.com/LMFAO:Yes
http://www.pandora.com/music/artist/lmfao
http://www.pandora.com/music/song/goo%2Bgoo%2Bdolls/black%2Bballoon
http://new.music.yahoo.com/lmfao/

可惜的是，无论我怎么调整查询，结果都没有改善，对于一些比较冷门的搜索，比如 Andrea Bauer at ilike，情况更糟。

Bing和Yahoo也提供API，但它们需要你先获取一个开发者ID，也就是说不能随便分发程序。

我还发现了这个建议，它解析标准的谷歌页面，但似乎现在也不再有效了。

在我浏览器中查看的谷歌返回页面的源代码看起来完全无法搜索。

有没有什么办法可以让我获取到“真实”的谷歌搜索结果呢？

Answer 1

我之前在Groovy和Java上做过这个，重点是通过一个网址向谷歌请求信息，然后构建一个代表HTML的DOM文档（Java中用的是TagSoup，Python中有类似的工具吗？）。接下来，使用XPath从你的DOM文档中提取链接，这样你就可以根据谷歌结果页面来“排名”你想要提升的网站。

欢迎来到黑帽搜索引擎优化的世界？

Answer 2

这个库可能对你有帮助：http://www.catonmat.net/blog/python-library-for-google-search/

>>> from xgoogle.search import GoogleSearch, SearchError
>>> try:
...   gs = GoogleSearch("LMFAO Yes at Pandora")
...   gs.results_per_page = 50
...   results = gs.get_results()
...   for res in results:
...     print res.title.encode("utf8")
...     print res.desc.encode("utf8")
...     print res.url.encode("utf8")
...     print
... except SearchError, e:
...   print "Search failed: %s" % e
... 
Yes - LMFAO - Pandora Internet Radio
Information about Yes - LMFAO at Pandora.com. Pandora is the Internet radio service that helps you find new music based on your old and current favorites.
http://www.pandora.com/music/song/lmfao/yes

LMFAO - Pandora Internet Radio
Listen and find out more about LMFAO at Pandora.com. Pandora is the Internet ...
http://www.pandora.com/music/artist/lmfao

search engine - Getting the 'real' Google results with Python ...
I'm trying to do something very simple, get the top result URL of a Google search, f.e. for the search LMFAO Yes at Pandora ( at is intentional here) I want ...
http://stackoverflow.com/questions/5361735/getting-the-real-google-results-with-python

LMFAO:Yes Lyrics - LyricWiki - Music lyrics from songs and albums
This song is performed by LMFAO and appears on the album Party Rock (2009).LMFAO:Yes ... Pandora: search for… LMFAO • Yes. Wikipedia: search for… ...
http://lyrics.wikia.com/LMFAO:Yes

YouTube - LMFAO - YES
Oct 6, 2007 ... JUST ANOTHER DAY IN THE CAR 4 LMFAO... OFFICIAL LMFAO Myspace  - http://lmfaomusic.com http://partyrocklife.com/ for OFFICIAL Party Rock gear ...
http://www.youtube.com/watch?v=nXPT8sw_FjU

Yes Lyrics - Lmfao
The group LMFAO goes double platnium hayyy. I got a party man that's how I live. So I take my elevator to the club in my crib like [Chorus:] Yes it's on and ...
http://www.6lyrics.com/yes10-lyrics-lmfao.aspx

LMFAO - YES LYRICS
1 post - Last post: Jun 3, 2010Lmfao Yes lyrics in the Party Rock Album. These Yes lyrics are performed by Lmfao Get the music video and song lyrics here.
http://www.metrolyrics.com/yes-lyrics-lmfao.html

Easy ChickHEN CFW installer (Without Pandora)
2 posts - 1 author - Last post: Jul 9, 2009Nice post! Smile I have the... Motherboard: TA-079v1, Model: Phat 100x, Hackable: yes, and Creates Pandora: yes. LMFAO! lol! Razz Very Happy ...
http://pspcoding.darkbb.com/t379-easy-chickhen-cfw-installer-without-pandora

Links on "Party Rock" | Facebook
EXCLUSIVE: Jamie Foxx Interview at LMFAO's "YES" Video Shoot - BVTV "Band of the ... Pandora Hau fuck yes party people! lmfao madness ...
http://www.facebook.com/posted.php?id=139241478710&share_id=145095215520613&comments=1

Pandora stays hot with investors - YEA!!! LMFAO!!! - ba.broadcast ...
Aug 31, 2010 ... Subject: Re: Pandora stays hot with investors - YEA!!! LMFAO! ... said that yes, Pandora was running an ad about every half hour. ...
http://groups.google.com/group/ba.broadcast/browse_thread/thread/947fb52042cd8211/41f25665629133cc?show_docid=41f25665629133cc

>>>

Answer 3

很遗憾，Xgoogle对我来说已经不管用了，不过我发现Bing并不介意你抓取他们的搜索结果，只要你使用他们的RSS源，链接在这里。

所以现在我只是通过urlopen获取的结果页面来用正则表达式处理数据。