Osmosis: Unable to parse xml file

[VShkaberda]

I can't make updates work. Obviously there is an error with osmosis part, but I can't figure out what caused it.

configuration of osmosis

# The URL of the directory containing change files.
baseUrl=https://planet.openstreetmap.org/replication/minute

# Defines the maximum time interval in seconds to download in a single invocation.
# Setting to 0 disables this feature.
maxInterval = 3600

contents of run.log where the error occures

[2020-04-21 11:36:02] 28670 [error] Osmosis error

contents of osmosis.log where the error occures

Apr 21, 2020 11:36:02 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-replication-interval failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file /tmp/change12498295845671173843.tmp.  publicId=(null), sy5, columnNumber=22.

full log

Apr 21, 2020 11:36:01 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.46
Apr 21, 2020 11:36:01 AM org.java.plugin.registry.xml.ManifestParser <init>
INFO: got SAX parser factory - org.apache.xerces.jaxp.SAXParserFactoryImpl@72d818d1
Apr 21, 2020 11:36:01 AM org.java.plugin.registry.xml.PluginRegistryImpl configure
INFO: configured, stopOnError=false, isValidating=true
Apr 21, 2020 11:36:01 AM org.java.plugin.registry.xml.PluginRegistryImpl register
INFO: plug-in and fragment descriptors registered - 1
Apr 21, 2020 11:36:01 AM org.java.plugin.standard.StandardPluginManager activatePlugin
INFO: plug-in started - org.openstreetmap.osmosis.core.plugin.Core@0.46.0 (active/total: 1 of 1)
Apr 21, 2020 11:36:01 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Apr 21, 2020 11:36:01 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Apr 21, 2020 11:36:01 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Apr 21, 2020 11:36:02 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-replication-interval failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file /tmp/change11543893959405443609.tmp.  publicId=(null), systemId=(null), lineNumber=1775, columnNumber=22.
        at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:100)
        at org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloader.processChangeset(ReplicationDownloader.java:107)
        at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.processReplicationFile(BaseReplicationDownloader.java:133)
        at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.download(BaseReplicationDownloader.java:235)
        at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:271)
        at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:350)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1775; columnNumber: 22; Invalid byte 2 of 4-byte UTF-8 sequence.
        at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
        at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
        at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
        at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReader.run(XmlChangeReader.java:90)
        ... 6 more
Caused by: org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of 4-byte UTF-8 sequence.
        at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown Source)
        at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
        at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
        at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown Source)
        at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
        ... 16 more

Apr 21, 2020 11:36:02 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

So it is raised in this part of openstreetmap-tiles-update-expire but I am stuck with figuring out if it caused by osmosis:

openstreetmap-tile-server/openstreetmap-tiles-update-expire

Lines 147 to 149 in e838310

	if ! $OSMOSIS_BIN --read-replication-interval workingDirectory=$WORKOSM_DIR --simplify-change --write-xml-change $CHANGE_FILE 1>&2 2> "$OSMOSISLOG"; then
	m_error "Osmosis error"
	fi

[VShkaberda]

I noticed that updates worked but stop working at some point. Log from the server where one country has been imported:

[2020-04-11 17:38:01] 29296 start import from seq-nr 3973309, replag is 1 minute(s)
[2020-04-11 17:38:01] 29296 downloading diff
[2020-04-11 17:38:03] 29296 filtering diff
[2020-04-11 17:38:04] 29296 importing diff
[2020-04-11 17:38:04] 29296 expiring tiles
[2020-04-11 17:38:04] 29296 Done with import
[2020-04-11 17:39:01] 29380 start import from seq-nr 3973310, replag is 2 minute(s)
[2020-04-11 17:39:01] 29380 downloading diff
[2020-04-11 17:39:03] 29380 [error] Osmosis error

On the server with the full planet planet the same problem occures from the sequence 3973308.
Could it be that some sequences are broken? Is there a way to skip those sequences?

[VShkaberda]

So, I guess it is a bug caused by apache which has been fixed in Osmosis with commit simonpoole:gzip_fix2 and is live since osmosis release 0.47.2. Osmosis used in this project is 0.46. I understand that osmium release 0.47.2 is pretty recent and even unverified. But please, can you tell your thoughts about switching to newest osmosis version.

[Istador]

0.46-2 is the most recent version in the ubuntu 18.04 sources, so it's not that easy to update right now, as it's installed over apt in the Dockerfile currently.

I don't know if there are any plans to bring the 0.47.2 osmosis update to ubuntu 18.04 (bionic) or backport it soon.

Ubuntu 19.04, 19.10 and 20.04 already have a osmosis version with a fix for this. Ubuntu 20.04 is the next LTS version since 18.04 and should release tomorrow. Upgrading this project to 20.04 should be a goal for the next days/weeks/months.

You could try to use newer ubuntu sources just for the osmosis package, by apt pinning everything to 18.04 and just use newer versions for osmosis, but there might be dependency issues preventing this approach to work.