Unicode tests fail in python3.3 when using a non-UTF-8 locale #344
Milestone
Comments
|
we all love encodings ;) i'm somewhat sure that setting the encoding in open isn't correct for all rdf serialization formats, some are plain ascii based, some utf-8... also are you sure that the test is wrong? will have a closer look when i find some time |
|
This changeset 98fc6b3 actually breaks several of our Plone-built web applications that use RDFLib for our tripple-store storage backend(s). See Plone traceback below: I've bisected this down to the changeset mentioned above but haven't had the time yet to investigate what's going wrong. |
|
I think this solves only part of the issue. Other options like reading in the response and converting it manually, or re-wrapping it into something with a mode attribute properly set (or even setting mode on a file like object) are somewhat suboptimal. IMO the default should assume 'utf-8' encoding if not already unicode or stated otherwise. |
|
Actually - after doing I read Armin Ronacher guide to unicode wrangling in py2/3 - and I think a better way is possible, rather than checking the |
|
Interesting article. Thanks for sharing. After reading it, I think the py3 model only works if you really separate strings and bytes. Strings are something decoded and they just work, and bytes are just bytes but can be decoded to strings if required. I don't have enough experience with py3 but so far I believe this model can work nicely, and developers have to be explicit when to en/decode strings and bytes, which should help a lot to avoid UnicodeDecodeErrors. (yes it is annoying to do it manually, but I don't see another way around, and it kinda follows the principle: rather explicit than implicit). Maybe it would be best to require that the input data is unicode or the stream has to produce unicode. If it's not unicode, then the library uses a default decoder, which could be either system default, utf-8 or even mimetype dependent if that information is available. The user of the library can easily wrap a decoder around the input data (shouldn't be more than one line I guess). This would make the parser maybe a bit simpler, and follows the py3 idea of being explicit with bytes and strings. (May well be that I have missed some use-cases here). |
mamash
pushed a commit
to TritonDataCenter/pkgsrc-wip
that referenced
this issue
Feb 15, 2014
2013/12/31 RELEASE 4.1 ====================== This is a new minor version RDFLib, which includes a handful of new features: * A TriG parser was added (we already had a serializer) - it is up-to-date wrt. to the newest spec from: http://www.w3.org/TR/trig/ * The Turtle parser was made up to date wrt. to the latest Turtle spec. * Many more tests have been added - RDFLib now has over 2000 (passing!) tests. This is mainly thanks to the NT, Turtle, TriG, NQuads and SPARQL test-suites from W3C. This also included many fixes to the nt and nquad parsers. * ```ConjunctiveGraph``` and ```Dataset``` now support directly adding/removing quads with ```add/addN/remove``` methods. * ```rdfpipe``` command now supports datasets, and reading/writing context sensitive formats. * Optional graph-tracking was added to the Store interface, allowing empty graphs to be tracked for Datasets. The DataSet class also saw a general clean-up, see: RDFLib/rdflib#309 * After long deprecation, ```BackwardCompatibleGraph``` was removed. Minor enhancements/bugs fixed: ------------------------------ * Many code samples in the documentation were fixed thanks to @PuckCh * The new ```IOMemory``` store was optimised a bit * ```SPARQL(Update)Store``` has been made more generic. * MD5 sums were never reinitialized in ```rdflib.compare``` * Correct default value for empty prefix in N3 [#312]RDFLib/rdflib#312 * Fixed tests when running in a non UTF-8 locale [#344]RDFLib/rdflib#344 * Prefix in the original turtle have an impact on SPARQL query resolution [#313]RDFLib/rdflib#313 * Duplicate BNode IDs from N3 Parser [#305]RDFLib/rdflib#305 * Use QNames for TriG graph names [#330]RDFLib/rdflib#330 * \uXXXX escapes in Turtle/N3 were fixed [#335]RDFLib/rdflib#335 * A way to limit the number of triples retrieved from the ```SPARQLStore``` was added [#346]RDFLib/rdflib#346 * Dots in localnames in Turtle [#345]RDFLib/rdflib#345 [#336]RDFLib/rdflib#336 * ```BNode``` as Graph's public ID [#300]RDFLib/rdflib#300 * Introduced ordering of ```QuotedGraphs``` [#291]RDFLib/rdflib#291 2013/05/22 RELEASE 4.0.1 ======================== Following RDFLib tradition, some bugs snuck into the 4.0 release. This is a bug-fixing release: * the new URI validation caused lots of problems, but is nescessary to avoid ''RDF injection'' vulnerabilities. In the spirit of ''be liberal in what you accept, but conservative in what you produce", we moved validation to serialisation time. * the ```rdflib.tools``` package was missing from the ```setup.py``` script, and was therefore not included in the PYPI tarballs. * RDF parser choked on empty namespace URI [#288](RDFLib/rdflib#288) * Parsing from ```sys.stdin``` was broken [#285](RDFLib/rdflib#285) * The new IO store had problems with concurrent modifications if several graphs used the same store [#286](RDFLib/rdflib#286) * Moved HTML5Lib dependency to the recently released 1.0b1 which support python3 2013/05/16 RELEASE 4.0 ====================== This release includes several major changes: * The new SPARQL 1.1 engine (rdflib-sparql) has been included in the core distribution. SPARQL 1.1 queries and updates should work out of the box. * SPARQL paths are exposed as operators on ```URIRefs```, these can then be be used with graph.triples and friends: ```py # List names of friends of Bob: g.triples(( bob, FOAF.knows/FOAF.name , None )) # All super-classes: g.triples(( cls, RDFS.subClassOf * '+', None )) ``` * a new ```graph.update``` method will apply SPARQL update statements * Several RDF 1.1 features are available: * A new ```DataSet``` class * ```XMLLiteral``` and ```HTMLLiterals``` * ```BNode``` (de)skolemization is supported through ```BNode.skolemize```, ```URIRef.de_skolemize```, ```Graph.skolemize``` and ```Graph.de_skolemize``` * Handled of Literal equality was split into lexical comparison (for normal ```==``` operator) and value space (using new ```Node.eq``` methods). This introduces some slight backwards incomaptible changes, but was necessary, as the old version had inconsisten hash and equality methods that could lead the literals not working correctly in dicts/sets. The new way is more in line with how SPARQL 1.1 works. For the full details, see: https://github.com/RDFLib/rdflib/wiki/Literal-reworking * Iterating over ```QueryResults``` will generate ```ResultRow``` objects, these allow access to variable bindings as attributes or as a dict. I.e. ```py for row in graph.query('select ... ') : print row.age, row["name"] ``` * "Slicing" of Graphs and Resources as syntactic sugar: ([#271](RDFLib/rdflib#271)) ```py graph[bob : FOAF.knows/FOAF.name] -> generator over the names of Bobs friends ``` * The ```SPARQLStore``` and ```SPARQLUpdateStore``` are now included in the RDFLib core * The documentation has been given a major overhaul, and examples for most features have been added. Minor Changes: -------------- * String operations on URIRefs return new URIRefs: ([#258](RDFLib/rdflib#258)) ```py >>> URIRef('http://example.org/')+'test rdflib.term.URIRef('http://example.org/test') ``` * Parser/Serializer plugins are also found by mime-type, not just by plugin name: ([#277](RDFLib/rdflib#277)) * ```Namespace``` is no longer a subclass of ```URIRef``` * URIRefs and Literal language tags are validated on construction, avoiding some "RDF-injection" issues ([#266](RDFLib/rdflib#266)) * A new memory store needs much less memory when loading large graphs ([#268](RDFLib/rdflib#268)) * Turtle/N3 serializer now supports the base keyword correctly ([#248](RDFLib/rdflib#248)) * py2exe support was fixed ([#257](RDFLib/rdflib#257)) * Several bugs in the TriG serializer were fixed * Several bugs in the NQuads parser were fixed
This was referenced Mar 3, 2014
This was referenced Jan 16, 2017
This was referenced Mar 16, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For example, run
(Note: this is somewhat important for linux packagers, because most linux packaging runs in a "clean" environment with posix locale set, and we'd like to run tests where possible.)
You will get several errors like:
The trivial fix of setting the character encoding on the open() statements in test_dawg.py allows the tests to work in python3.3; but if you set suitably, one test in python2.7 will fail with this output:
The text was updated successfully, but these errors were encountered: