Python 3 robotparser错误

2024-06-16 13:34:25 发布

您现在位置:Python中文网/ 问答频道 /正文

我有下一个机器人.txt地址:

User-agent: *
Disallow: /*/feed
Disallow: /*/trackback
Disallow: /category/
Disallow: /forum/
Disallow: /program/
Disallow: /wp-content/
Disallow: /trafficsystem/
Disallow: /wp-admin/
Disallow: /*?
Disallow: /*.css$
Disallow: /author/
Disallow: /*/?replytocom
Disallow: /privacy/
Disallow: /terms/
Disallow: /copyright/
Disallow: /*/users/
Disallow: /*/topic-tag/
Disallow: /quick-share/

我使用python3.4.3和robotparser 当我调用can\u fetch to disallowed page时,它返回True:

can_fetch("*", "http://example.com/2008/04/10/8-reasons-i-am-successful-and-you-are-not/?replytocom=154237")

为什么?你知道吗


Tags: txt地址feed机器人forumfetchprogramcan