Package Manager: File "cannot be stored" (when contains Unicode?)

[TurekBot]

I ran into this error when I tried to upload an app I made to a new eXist install.

What is the problem

When I try to upload the .xar archive via the Package Manager, everything works except it won't store just one certain file.

After some testing, I narrowed the culprit in that file to an em dash (—) inside of a comment.

When the em dash is there, I get the error; when there's no em dash, no error.

Here's the part of the file in question. (Scroll over to see the em dash.)

    <!-- This block replaces the `email_addr` atribute of the Email element with the given email. -->
    <xsl:template match="/Message/Email/@email_addr">
        <!-- The name attribute is required, so we just specify the same name as before. `name()` uses the same name the attribute had previously—we don't want to change the attribute name, just its value. -->
        <xsl:attribute name="{name()}"> 
            <xsl:value-of select="$emailReplacement"/>
        </xsl:attribute>
    </xsl:template>

I think this is the line that throws the exception: https://github.com/eXist-db/exist/blob/1095e49769c5edfefdde022678e65755d5f00e8c/exist-core/src/main/java/org/exist/repo/Deployment.java#L807

Here's the important section of exist.log when the error is thrown.

What did you expect

I expected any unicode symbol to be valid XML, since XML's default encoding is UTF-8, but maybe it's not related to unicode at all. I checked the XSL file against various validators, and it appears to be well-formed.

Describe how to reproduce or add a test

Try to upload this package with an em dash (with the file that has an em dash)
Notice the error. ("Deployment incomplete, 1 issues found: replace-email.xsl: replace-email.xsl cannot be stored")
Try to upload this package without an em dash (everything the same, but no em dash)
Notice that everything is stored as expected.

Context information

• eXist-db version + Git Revision hash: eXist 5.0.0 5a27d10ba1710c1df751344b53ac6e4cb4778e3b 20190902160649
• Java version: "1.8.0_221"
• Operating system: Windows 7 Professional
• 64 bit
• How is eXist-db installed? JAR installer
• Any custom changes in e.g. conf.xml? None

[duncdrum]

I think the em-dash is illegal inside inline comments, but maybe we could improve the error message. To hint that an illegal character at line xx is the problem.

[dizzzz]

In the past we had another issue where a XAR generated by exude contained an illegal XML construct making it impossible to reinstall the XAR file. Not sure which ticket however...

[TurekBot]

@duncdrum, thanks for commenting.

I looked, but can't find any evidence that the em dash is illegal in XML comments. Are you able to find any?

[duncdrum]

@TurekBot I was wrong, U+2014 — is allowed. I can think of two things that would trip this up, either a) package-manager is doing something funky when uploading and re-encoding the code point on the way or b) the file is not properly UTF-8, i remember a special case about Windows encoding which involved em-dash being wrongly translated into some control code point in Unicode.

In any case a clearer error message when dealing with perceived well-formedness still sounds the way to go.