SQL IMPORT DATABASE Cannot Restore Backup

[gramian]

ArcadeDB Version: v22.10.1 (build e692f4c/1667424782661/main)

JDK Version: openjdk 11.0.16.1 2022-08-12

OS: MacOS 12.6

Dear Developers,

I unsuccesfully tried to remotely import a backup. I was expecting that IMPORT DATABASE not only imports exported databases but also backups, as there seems to be no other remote way to restore a backup. The restore.sh script requires to be logged into a terminal on the server's machine / container.

Expected behavior

SQL IMPORT DATABASE imports backup made with BACKUP DATABASE

Actual behavior

Import technically succeeds but does not import anything

Server Output

Recognized format JSON (parsingLimitBytes=9,54MB parsingLimitEntries=0)- Status update: parsed 0 (0/sec) - 0 documents (0/sec) - 0 vertices (0/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)

Steps to reproduce

BACKUP DATABASE;
IMPORT DATABASE file://backups/test/backup-test-20221128-140440531.zip

[lvca]

Right now the only way to restore a database is by using API or the restore.sh command on the server. The reason why we cannot use the IMPORT DATABASE command is that in that case, the database must exist, while with the restore database the database is created by the restore API. The restore database, if any, should be supported by a new server command (that requires server authentication).

[gramian]

OK, so I should rather use EXPORT DATABASE in my use-case if I want to import it remotely.

[gramian]

The reason why we cannot use the IMPORT DATABASE command is that in that case, the database must exist, ...

Hmm, but if I use the HTTP API, I would have to use the command endpoint, which in turn requires a database in its API path, as in: api/v1/{db}/command, right? So a database has to exist. Or would that be the source of the problem then?

[lvca]

That's right, you need to create an empty database first, and then import from a previous export.

[gramian]

@lvca The IMPORT DATABASE function seems to have a general problem:
I tried to import the databases:

• https://github.com/ArcadeData/arcadedb-datasets/blob/main/arcadedb/Movies-backup-20211005-134613862.zip
• https://github.com/ArcadeData/arcadedb-datasets/blob/main/orientdb/GratefulDeadConcerts.gz
• https://github.com/ArcadeData/arcadedb-datasets/blob/main/orientdb/OpenBeer.gz
• https://github.com/ArcadeData/arcadedb-datasets/blob/main/neo4j/movies.graphml.tgz
• https://github.com/ArcadeData/arcadedb-datasets/blob/main/neo4j/movies.graphson.tgz

None of them work.

[lvca]

By checking our test cases I can see it's tested a minimal database exported from OrientDB:

https://github.com/ArcadeData/arcadedb/blob/5d260bd1a4bd1d32897aecb0707c3a0bfcf553a9/server/src/test/resources/orientdb-export-small.gz

[lvca]

I was checking this issue, is the issue still there?

[gramian]

Yes,
using (in Studio):

IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/blob/main/arcadedb/Movies-backup-20211005-134613862.zip

(or others listed above) yields:

INFO  [SourceDiscovery] <ArcadeDB_0> Analyzing url: https://github.com/ArcadeData/arcadedb-datasets/blob/main/arcadedb/Movies-backup-20211005-134613862.zip...- Status update: parsed 0 (0/sec) - 0 documents (0/sec) - 0 vertices (0/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)
<ArcadeDB_0> Error on command execution (PostCommandHandler)
com.arcadedb.exception.CommandExecutionException: Error on importing database
	at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:70)
	at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:95)
	at com.arcadedb.query.sql.parser.SimpleExecStatement.execute(SimpleExecStatement.java:67)
	at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:92)
	at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:76)
	at com.arcadedb.query.sql.SQLQueryEngine.command(SQLQueryEngine.java:106)
	at com.arcadedb.database.EmbeddedDatabase.command(EmbeddedDatabase.java:1247)
	at com.arcadedb.server.ServerDatabase.command(ServerDatabase.java:416)
	at com.arcadedb.server.http.handler.PostCommandHandler.executeCommand(PostCommandHandler.java:130)
	at com.arcadedb.server.http.handler.PostCommandHandler.execute(PostCommandHandler.java:90)
	at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:85)
	at com.arcadedb.server.http.handler.AbstractHandler.handleRequest(AbstractHandler.java:111)
	at io.undertow.server.RoutingHandler.handleRequest(RoutingHandler.java:93)
	at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:104)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
	at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:256)
	at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
	at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
Caused by: com.arcadedb.integration.importer.ImportException: Error on parsing source 'null'
	at com.arcadedb.integration.importer.Importer.load(Importer.java:62)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:65)
	... 21 more
Caused by: com.arcadedb.integration.importer.ImportException: Cannot determine the file type. If it is a CSV file, please specify the header via settings
	at com.arcadedb.integration.importer.SourceDiscovery.analyzeSourceContent(SourceDiscovery.java:317)
	at com.arcadedb.integration.importer.SourceDiscovery.getSchema(SourceDiscovery.java:60)
	at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:78)
	at com.arcadedb.integration.importer.Importer.load(Importer.java:53)
	... 26 more

[lvca]

Did you figure out a way to get this working?

[gramian]

No, I still get: Cannot execute command Error on parsing source 'null'

Trace

[SourceDiscovery] <ArcadeDB_0> Analyzing url: https://github.com/ArcadeData/arcadedb-datasets/blob/main/orientdb/OpenBeer.gz...- Status update: parsed 0 (0/sec) - 0 documents (0/sec) - 0 vertices (0/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)
<ArcadeDB_0> Error on command execution (PostCommandHandler)
com.arcadedb.exception.CommandExecutionException: Error on importing database
	at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:69)
	at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:91)
	at com.arcadedb.query.sql.parser.SimpleExecStatement.execute(SimpleExecStatement.java:63)
	at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:85)
	at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:69)
	at com.arcadedb.query.sql.SQLQueryEngine.command(SQLQueryEngine.java:97)
	at com.arcadedb.database.EmbeddedDatabase.command(EmbeddedDatabase.java:1264)
	at com.arcadedb.server.ServerDatabase.command(ServerDatabase.java:421)
	at com.arcadedb.server.http.handler.PostCommandHandler.executeCommand(PostCommandHandler.java:127)
	at com.arcadedb.server.http.handler.PostCommandHandler.execute(PostCommandHandler.java:88)
	at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:92)
	at com.arcadedb.server.http.handler.AbstractHandler.handleRequest(AbstractHandler.java:127)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.arcadedb.integration.importer.ImportException: Error on parsing source 'null'
	at com.arcadedb.integration.importer.Importer.load(Importer.java:63)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:61)
	... 18 more
Caused by: com.arcadedb.integration.importer.ImportException: Cannot determine the file type. If it is a CSV file, please specify the header via settings
	at com.arcadedb.integration.importer.SourceDiscovery.analyzeSourceContent(SourceDiscovery.java:294)
	at com.arcadedb.integration.importer.SourceDiscovery.getSchema(SourceDiscovery.java:60)
	at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:81)
	at com.arcadedb.integration.importer.Importer.load(Importer.java:54)
	... 23 more

[lvca]

Back on this, the original issue was that you cannot import a database from a backup and this is still correct. In order to import a database you have to create it first.

Have you tried importing an exported database?

[gramian]

Yes, and it also does not work.
I did:

EXPORT DATABASE file://myexport.tgz FORMAT JSONL

and after dropping and creating a database I tried:

IMPORT DATABASE file://myexport.tgz
IMPORT DATABASE file:///myexport.tgz
IMPORT DATABASE file://exports/myexport.tgz
IMPORT DATABASE file:///exports/myexport.tgz

(and a few others) each resulting in:

Internal error Database 'null' is not available

 Error on command execution (PostCommandHandler)
com.arcadedb.exception.DatabaseIsClosedException: Database 'null' is not available
	at com.arcadedb.server.ArcadeDBServer.getDatabase(ArcadeDBServer.java:366)
	at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:58)
	at com.arcadedb.server.http.handler.AbstractHandler.handleRequest(AbstractHandler.java:128)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.base/java.lang.Thread.run(Thread.java:829)
</details>

If this is supposed to work, then maybe my import path is not correct?

[lvca]

Ok, working on this.