将UTF-8的二进制0和1字符串转换为代码点

Question

如果你有一串由0和1组成的比特串，怎么把它转换成正确的编码点，前提是这串比特是用utf-8编码的。

举个例子：

accented_a_part1 = "100001"
accented_a_part2 = "00011"

accented_a_int = int(accented_a_part2 + accented_a_part1, 2)
print(accented_a_int),          # => 225
print(unichr(accented_a_int))   # => á    http://www.unicode.org/charts/PDF/U0080.pdf

accented_a_in_utf8 = "110" + accented_a_part2 + "10" + accented_a_part1
accented_a_in_utf8_as_raw_int = int(accented_a_in_utf8, 2)
print(accented_a_in_utf8_as_raw_int),        # => 50081 (not the codepoint you want)
print(unichr(accented_a_in_utf8_as_raw_int)) # => 쎡    (and therefore not the character you want)

UTF-8 规范

Answer 1

要把十六进制的字面值转换成Unicode：

>>> h = '16 03 01 00 e3 01'
>>> h.replace(' ','').decode('hex')
'\x16\x03\x01\x00\xe3\x01'
>>> h.replace(' ','').decode('hex').decode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe3 in position 4: unexpected end of data
>>>

如果这些十六进制数实际上是utf8格式的，你最终会得到一个Unicode字符串：

>>> h = 'c3 a1'
>>> u = h.replace(' ','').decode('hex').decode('utf8')
>>> u
u'\xe1'
>>> print u
á