Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cchardetect CLI script #30

Merged
merged 2 commits into from
May 15, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ dist
build
eggs
parts
bin
var
sdist
develop-eggs
Expand Down
1 change: 0 additions & 1 deletion TODO.md
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
- Implement cli tool (like chardet cli)
- Improve docs
32 changes: 32 additions & 0 deletions bin/cchardetect
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/usr/bin/env python
from __future__ import print_function, unicode_literals
import argparse
import cchardet


def read_chunks(f, chunk_size):
chunk = f.read(chunk_size)
while chunk:
yield chunk
chunk = f.read(chunk_size)


def main():
parser = argparse.ArgumentParser()
parser.add_argument('files', nargs='+', help="Files to detect encoding of", type=argparse.FileType('rb'))
parser.add_argument('--chunk-size', type=int, default=(256 * 1024))
args = parser.parse_args()

for f in args.files:
detector = cchardet.UniversalDetector()
for chunk in read_chunks(f, args.chunk_size):
detector.feed(chunk)
detector.close()
print('{file.name}: {result[encoding]} with confidence {result[confidence]}'.format(
file=f,
result=detector.result
))


if __name__ == '__main__':
main()
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ def read(f):
cmdclass={'build_ext': build_ext},
package_dir={'': 'src'},
packages=['cchardet', ],
scripts=['bin/cchardetect'],
ext_modules=[
cchardet_module
],
Expand Down