2012-07-07 03:30:02 +00:00
|
|
|
|
2012-06-26 01:54:57 +00:00
|
|
|
cChardet
|
|
|
|
|
========
|
|
|
|
|
|
|
|
|
|
This library is high speed universal character encoding detector. -
|
2012-07-07 03:30:02 +00:00
|
|
|
binding to
|
|
|
|
|
`charsetdetect <https://bitbucket.org/medoc/uchardet-enhanced/overview>`_.
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
This library is faster than
|
|
|
|
|
`chardet <http://pypi.python.org/pypi/chardet>`_.
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
Support codecs
|
|
|
|
|
==============
|
|
|
|
|
|
|
|
|
|
- Big5
|
|
|
|
|
- EUC-JP
|
|
|
|
|
- EUC-KR
|
|
|
|
|
- GB18030
|
|
|
|
|
- gb18030
|
|
|
|
|
- HZ-GB-2312
|
|
|
|
|
- IBM855
|
|
|
|
|
- IBM866
|
|
|
|
|
- ISO-2022-CN
|
|
|
|
|
- ISO-2022-JP
|
|
|
|
|
- ISO-2022-KR
|
|
|
|
|
- ISO-8859-2
|
|
|
|
|
- ISO-8859-5
|
|
|
|
|
- ISO-8859-7
|
|
|
|
|
- ISO-8859-8
|
|
|
|
|
- KOI8-R
|
|
|
|
|
- Shift\_JIS
|
|
|
|
|
- TIS-620
|
|
|
|
|
- UTF-8
|
|
|
|
|
- UTF-16BE
|
|
|
|
|
- UTF-16LE
|
|
|
|
|
- UTF-32BE
|
|
|
|
|
- UTF-32LE
|
2012-07-07 03:30:02 +00:00
|
|
|
- WINDOWS-1250
|
|
|
|
|
- WINDOWS-1251
|
|
|
|
|
- WINDOWS-1252
|
|
|
|
|
- WINDOWS-1253
|
|
|
|
|
- WINDOWS-1255
|
|
|
|
|
- EUC-TW
|
2012-06-26 01:54:57 +00:00
|
|
|
- X-ISO-10646-UCS-4-2143
|
|
|
|
|
- X-ISO-10646-UCS-4-3412
|
|
|
|
|
- x-mac-cyrillic
|
|
|
|
|
|
|
|
|
|
Requires
|
|
|
|
|
========
|
|
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
- Cython: `http://www.cython.org/ <http://www.cython.org/>`_
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
e.g.) Ubuntu 12.04
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
$sudo apt-get install build-essential python-dev cython
|
|
|
|
|
|
|
|
|
|
Installation
|
|
|
|
|
============
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
$cd /tmp
|
|
|
|
|
|
|
|
|
|
$git clone git://github.com/PyYoshi/cChardet.git
|
|
|
|
|
|
|
|
|
|
$cd cChardet
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
$python setup.py build
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
$sudo python setup.py install
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
or
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
::
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
$sudo easy_install cchardet
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
Test
|
|
|
|
|
====
|
|
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
::
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
$sudo easy_install or pip install -U chardet nose
|
|
|
|
|
|
|
|
|
|
$cd test
|
|
|
|
|
|
|
|
|
|
$nosetests --nocapture tests.py
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
Benchmark
|
|
|
|
|
=========
|
|
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
code:
|
|
|
|
|
`tests.TestCchardetSpeed <https://github.com/PyYoshi/cChardet/blob/master/test/tests.py#L415>`_
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
sample:
|
|
|
|
|
`test/testdata/wikipediaJa\_One\_Thousand\_and\_One\_Nights\_SJIS.txt <https://github.com/PyYoshi/cChardet/blob/master/test/testdata/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt>`_
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
Performance:
|
|
|
|
|
~~~~~~~~~~~~
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
CPU: Intel Core i7 860 2.8GHz
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
RAM: DDR3-1333 16GB
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
Platform: Windows 7 HP x64, Python 2.7.3 32-bit
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
Result:
|
|
|
|
|
~~~~~~~
|
|
|
|
|
|
2012-07-07 05:37:11 +00:00
|
|
|
::
|
2012-07-07 03:30:02 +00:00
|
|
|
|
2012-07-07 05:37:11 +00:00
|
|
|
chardet: 0.25 (call/s)
|
2012-07-07 03:30:02 +00:00
|
|
|
|
2012-07-07 05:37:11 +00:00
|
|
|
cchardet: 500.03 (call/s)
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
License
|
|
|
|
|
=======
|
|
|
|
|
|
2012-07-07 05:37:11 +00:00
|
|
|
- This library files("cchardet.pyx","setup.py","tests.py") are "The MIT License".
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
- Other Libraries License: Please, look at the
|
|
|
|
|
`ext <https://github.com/PyYoshi/cChardet/tree/master/src/ext>`_
|
|
|
|
|
directory.
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
Thanks
|
|
|
|
|
======
|
|
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
- `https://bitbucket.org/medoc/uchardet-enhanced/overview <https://bitbucket.org/medoc/uchardet-enhanced/overview>`_
|
2012-06-26 01:54:57 +00:00
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
- `http://www.cython.org/ <http://www.cython.org/>`_
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
Contact
|
|
|
|
|
=======
|
|
|
|
|
|
2012-07-07 03:30:02 +00:00
|
|
|
`My blog <http://blog.remu.biz>`_
|
2012-06-26 01:54:57 +00:00
|
|
|
|
|
|
|
|
Sorry for my poor English :)
|
