Python判斷字元串、檔案字元編碼

本段工具代碼用于判斷字元串或者文本檔案的字元編碼類型，可以識别常用的UTF-8，UTF-8-SIG，UTF-16，GBK，GB2312 ，GB18030 ，ASCII字元編碼格式，如果有特殊字元集需求，可以擴充字元編碼清單。

代碼如下：

[

charset.py

]

# this util is used to detect file or string bytes encoding
# author: lixk
# email: [email protected]

# info: UTF includes ISO8859-1，GB18030 includes GBK，GBK includes GB2312，GB2312 includes ASCII
CODES = ['UTF-8', 'UTF-16', 'GB18030', 'BIG5']
# UTF-8 BOM prefix
UTF_8_BOM = b'\xef\xbb\xbf'


def detect(s):
    """
    get file encoding or bytes charset

    :param s: file path or bytes data

    :return: encoding charset
    """

    if type(s) == str:
        with open(s, 'rb') as f:
            data = f.read()
    elif type(s) == bytes:
        data = s
    else:
        raise TypeError('unsupported argument type!')

    # iterator charset
    for code in CODES:
        try:
            data.decode(encoding=code)
            if 'UTF-8' == code and data.startswith(UTF_8_BOM):
                return 'UTF-8-SIG'
            return code
        except UnicodeDecodeError:
            continue
    raise UnicodeDecodeError('unknown charset!')
           

detect

方法用于判斷檔案或者位元組清單編碼類型，參數為檔案路徑或者位元組清單bytes

使用示例：

import charset

print(charset.detect('abc哈哈'.encode('gbk')))
print(charset.detect('test.txt'))