web-ext build/run command does not work with manifest.json in utf-8 with BOM encoding

[xiaoyechen]

Is this a feature request or a bug?

bug

What is the current behavior?

When running the command "web-ext build" or "web-ext run" in an extension folder where the manifest.json file has the encoding of utf-8 with BOM, it returns an error saying
InvalidManifest: Error parsing manifest.json at manifest.json: JSONError: Unexpected token '?' at 1:1 in manifest.json
?{
^
( I have omitted the absolute paths in the error message)

The commands work as normal when the manifest.json has ANSI encoding.

What is the expected or desired behavior?

I think "web-ext build" and "web-ext run" should work as normal when the manifest.json is in utf-8 with BOM encoding.

Version information (for bug reports)

• Firefox version: 54.0.1 (64-bit)
• Your OS and version: Windows 7 Home Premium SP1
• Paste the output of these commands:

node --version && npm --version && web-ext --version

v8.2.1
5.3.0
master-19bdd384191dbeb51b7376feb3ce47e01347977e

[xiaoyechen]

I dug into the source code, looks like when the parse-json library processes a utf8 encoded json, it adds a question mark at the very beginning of the json string.
I tested web-ext build after programmatically eliminating the question mark inside the parse-json method, and then it worked.
Hope this finding helps.

[kumar303]

Hi. Sorry about that. Rather than a problem with the encoding, it sounds like web-ext can't handle the BOM at the beginning of your file. Can you upload your manifest.json to this issue so I can take a closer look?

[xiaoyechen]

Hi. I think I was wrong about web-ext can't work with utf8 encoding. It should be that web-ext can't work with utf8 with BOM encoding. I have edited the issue for it.

My manifest.json is here. It does contain some BOM characters but there's no BOM at the beginning. I think the problem is more generic with all the manifest files in utf8 with BOM encoding.
manifest.zip

I also tried web-ext with a couple add-on examples from https://github.com/mdn/webextensions-examples by saving their manifest files as utf8 with BOM, then web-ext returned the same error as in the issue.

[ExE-Boss]

This is because according to the JSON specification, the BOM is an illegal character, so this is correct behaviour.

[kumar303]

Yes but using a BOM is common on Windows so we should support it.

[hiikezoe]

FWIW, the spec clearly says about BOM.

In the interests of interoperability, implementations
that parse JSON texts MAY ignore the presence of a byte order mark
rather than treating it as an error.

https://tools.ietf.org/html/rfc7159#section-8.1

[Rob--W]

Test case with BOM the manifest.json file from: https://bugzilla.mozilla.org/show_bug.cgi?id=1583159#c2

The extension loads well in Firefox (when extracted and repackaged as an XPI file), so web-ext should handle it just fine.