universal character encoding detector

Find a file

PyYoshi 83a7cf4f6a update		2012-07-07 11:38:39 +09:00
src	deleted warnings because there is confidence.	2012-07-07 11:35:00 +09:00
test	add "cchardet.detect_with_confidence" method.	2012-07-05 12:05:11 +09:00
.gitignore	update	2012-07-07 11:38:39 +09:00
ez_setup.py	add ezsetup	2012-06-23 12:27:38 +09:00
MANIFEST.in	update build method	2012-06-26 16:00:39 +09:00
pandoc_markdown2rst.bat	add md2rst_converter	2012-06-26 11:02:21 +09:00
readme.md	add "cchardet.detect_with_confidence" method.	2012-07-05 12:05:11 +09:00
readme.rst	update	2012-06-26 18:13:53 +09:00
setup.py	add "cchardet.detect_with_confidence" method.	2012-07-05 12:05:11 +09:00

readme.md

cChardet

This library is high speed universal character encoding detector. - binding to charsetdetect.

This library is faster than chardet.

Support codecs

Big5
EUC-JP
EUC-KR
GB18030
gb18030
HZ-GB-2312
IBM855
IBM866
ISO-2022-CN
ISO-2022-JP
ISO-2022-KR
ISO-8859-2
ISO-8859-5
ISO-8859-7
ISO-8859-8
KOI8-R
Shift_JIS
TIS-620
UTF-8
UTF-16BE
UTF-16LE
UTF-32BE
UTF-32LE
windows-1250
windows-1251
windows-1252
windows-1253
windows-1255
x-euc-tw
X-ISO-10646-UCS-4-2143
X-ISO-10646-UCS-4-3412
x-mac-cyrillic

Requires

Cython: http://www.cython.org/

Install

$cd /tmp
$git clone git://github.com/PyYoshi/cChardet.git
$cd cChardet
$python setup.py build
$sudo python setup.py install

Example

# coding: utf8
import cchardet
msg = file(r"test/testdata/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt").read()
result = cchardet.detect(msg)
print(result)
result2 = cchardet.detect_with_confidence(msg)
print(result2)

Test

$sudo easy_install or pip install -U chardet nose
$nosetests --nocapture tests.py

Benchmark

see tests.TestCchardetSpeed

Sample(shift_jis):

test/testdata/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt

PC Spec.:

CPU: Intel Core i7 860 2.8GHz
RAM: DDR3-1333 16GB
Platform: Windows 7 HP x64, Python 2.7.3 32-bit

Result:

chardet: 4.009999990463257s, shift_jis
cchardet: 0.0009999275207519531s, shift_jis

License

This library files("cchardet.pyx","setup.py","tests.py") are "The MIT License".
Other Library License: Please, look at the "ext" directory.

Thanks

Contact

My blog

Sorry for my poor English :)