如何处理 urllib2.urlopen 中的 ® 符号？

Question

我收到了一个网址：https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp®-75-desktop-virtualization-solutions；这个网址是通过BeautifulSoup获取的。

url=u'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp\xae-75-desktop-virtualization-solutions'

我想再次用urllib2.urlopen来处理这个网址。

import urllib2
source = urllib2.urlopen(url).read()

我遇到的错误是：

UnicodeEncodeError: 'gbk' codec can't encode character u'\xae' in position 43: illegal multibyte sequence

所以，我尝试了：

source = urllib2.urlopen(url.encode("utf-8")).read()

它获取了页面的源代码，但和原始网址的内容不一样。

originalUrl = 'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp®-75-desktop-virtualization-solutions'
originalSource = urllib2.urlopen(originalUrl).read()
originalSource == source

结果是False。有没有什么办法可以修复这个网址？怎么把u'\xae'转换回原来的®？

Answer 1

网址必须是有效的字节串，里面的非ASCII字符要正确编码。你需要先把网址的路径编码成UTF-8格式，然后再进行网址编码：

import urllib
import urllib2
import urlparse

originalUrl = u'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp\xae-75-desktop-virtualization-solutions'
parsed_link = urlparse.urlsplit(originalUrl.encode('utf8'))
parsed_link = parsed_link._replace(path=urllib.quote(parsed_link.path))
encoded_link = parsed_link.geturl()
source = urllib2.urlopen(encoded_link).read()

示例：

>>> import urllib
>>> import urllib2 
>>> import urlparse
>>> originalUrl = u'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp\xae-75-desktop-virtualization-solutions'
>>> parsed_link = urlparse.urlsplit(originalUrl.encode('utf8'))
>>> parsed_link = parsed_link._replace(path=urllib.quote(parsed_link.path))
>>> encoded_link = parsed_link.geturl()
>>> encoded_link
'https://www.packtpub.com/virtualization-and-cloud/citrix-xenapp%C2%AE-75-desktop-virtualization-solutions'
>>> source = urllib2.urlopen(encoded_link).read()
>>> len(source)
68758