universal character encoding detector

Find a file

PyYoshi 64c9146714 add pypi badge		2016-10-17 18:50:45 +09:00
dockerfiles	update 3.4-slim	2015-09-07 17:57:39 +09:00
src	fix test error	2016-10-17 14:19:56 +09:00
.gitignore	use tox	2016-10-17 12:19:24 +09:00
.travis.yml	unsupport 3.3	2016-10-17 15:04:31 +09:00
appveyor.yml	unsupport 3.3	2016-10-17 15:04:31 +09:00
build.cmd	add build.cmd	2016-10-17 14:55:31 +09:00
Makefile	update make commands	2016-10-17 15:32:41 +09:00
MANIFEST.in	move tests	2016-10-17 11:43:08 +09:00
README.markdown	add pypi badge	2016-10-17 18:50:45 +09:00
setup.cfg	use tox	2016-10-17 12:19:24 +09:00
setup.py	fix email addr	2016-10-17 15:18:10 +09:00
tox.ini	unsupport py33	2016-10-17 15:38:02 +09:00

README.markdown

cChardet

cChardet is high speed universal character encoding detector. - binding to charsetdetect.

Support codecs

Big5
EUC-JP
EUC-KR
GB18030
HZ-GB-2312
IBM855
IBM866
ISO-2022-CN
ISO-2022-JP
ISO-2022-KR
ISO-8859-2
ISO-8859-5
ISO-8859-7
ISO-8859-8
KOI8-R
Shift_JIS
TIS-620
UTF-8
UTF-16BE
UTF-16LE
UTF-32BE
UTF-32LE
WINDOWS-1250
WINDOWS-1251
WINDOWS-1252
WINDOWS-1253
WINDOWS-1255
EUC-TW
X-ISO-10646-UCS-4-2143
X-ISO-10646-UCS-4-3412
x-mac-cyrillic

Requires

Cython: http://www.cython.org/

Installation

$ cd /tmp
$ git clone git://github.com/PyYoshi/cChardet.git
$ cd cChardet
$ python setup.py install

$ pip install -U cchardet

Example

# -*- coding: utf-8 -*-
import cchardet as chardet
with open(r"src/tests/testdata/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt", "rb") as f:
    msg = f.read()
    result = chardet.detect(msg)
    print(result)

Benchmark

$ cd src/
$ pip install chardet
$ python tests/bench.py

Performance

CPU: Intel(R) Core(TM) i3-4170 CPU @ 3.70GHz

RAM: DDR3 1600Mhz 16GB

Platform: Ubuntu 16.04 amd64

Python 2.7.12

	Request (call/s)
chardet	0.26
cchardet	1408.73

Python 3.5.2

	Request (call/s)
chardet	0.28
cchardet	1380.40