Newest html5lib 0.999999999 breaks rendering

[OktarinTentakel]

Tested on OS X according to install instructions with Python 2.7 and 3.4

Every rendering exits with:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/weasyprint/navigator.py", line 143, in app
    return make_response(render_template(url))
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/weasyprint/navigator.py", line 65, in render_template
    html = HTML(url)
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/weasyprint/__init__.py", line 92, in __init__
    namespaceHTMLElements=False)
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/html5lib/html5parser.py", line 35, in parse
    return p.parse(doc, **kwargs)
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/html5lib/html5parser.py", line 235, in parse
    self._parse(stream, False, None, *args, **kwargs)
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/html5lib/html5parser.py", line 85, in _parse
    self.tokenizer = _tokenizer.HTMLTokenizer(stream, parser=self, **kwargs)
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/html5lib/_tokenizer.py", line 36, in __init__
    self.stream = HTMLInputStream(stream, **kwargs)
  File "/Users/aw-sebastianschlapkohl/own-projects/weasyprint-test/venv/lib/python2.7/site-packages/html5lib/_inputstream.py", line 151, in HTMLInputStream
    return HTMLBinaryInputStream(source, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'encoding'

Going back two versions (seven 9s after the dot oO) fixes this.

[SimonSapin]

This might be

https://github.com/html5lib/html5lib-python/blob/master/CHANGES.rst#user-content-09999999910b9

Replace the charset keyword argument on parse and related methods with a set of keyword arguments: override_encoding, transport_encoding, same_origin_parent_encoding, likely_encoding, and default_encoding.

Although that says charset an the backtrack says encoding.

Anyway, I think WeasyPrint should be updated for the newer html5lib and the requirements in setup.py changed accordingly. CC @liZe

[liZe]

The argument was encoding, and we now have to pick one from (override | transport | same_origin_parent | likely | default)_encoding. We should read the documentation, but, well, there's no documentation, of course. So, it's default_encoding now, and if it breaks anything in WeasyPrint, we'll randomly change that 😉.

[liZe]

It doesn't work, because it crashes when input is unicode (we can't blame them for that).

[SimonSapin]

The terminology is based on https://html.spec.whatwg.org/multipage/#determining-the-character-encoding

• override_encoding is probably appropriate for the encoding parameter of WeasyPrint’s HTML class.
• transport_encoding is the one from HTTP Content-Type. (The encoding key in "URL fetchers" return values.)
• same_origin_parent refers to parent is the sense that an <iframe> element can introduced a child HTML document. This can be ignored since we don’t implement <iframe>. Or the cross-origin policy.
• likely_encoding, the spec says “if the user agent has information on the likely encoding for this page”. Safe to declare we don’t.
• default_encoding, the spec suggests UTF-8 in "controlled environment", but I don’t think WeasyPrint can make that kind of assumption about how it’s used. Then “In other environments, the default encoding is typically dependent on the user's locale” which is kinda terrible since it can lead to “works on my machine” sites broken for users with a different locale. But that’s what many browsers do today. I’ve heard of ideas/experiments to use the site’s top-level-domain instead but I don’t know where that’s at. Let’s not bother with any of this.

So I think we should override and transport instead of default.

@gsnedders How does this sound?

[liZe]

I've used override for our encoding parameter and transport for our protocol (given by the URL fetcher). Tests pass, but I'm not really sure…

[gsnedders]

@SimonSapin that sounds right

And yes, @liZe, docs are one of the two big things blocking 1.0 now…

[liZe]

@gsnedders No offense, that's a problem for other projects too 😉. Thanks for your hard work!

[mwangikinuthia]

so what should i do if this error arises?

[liZe]

so what should i do if this error arises?

Use the latest versions of WeasyPrint and html5lib.