Interface conversion error while live loading exported JSON data

[prashant-shahi]

What version of Dgraph are you using?

master

Steps to reproduce the issue (command/config used to run Dgraph).

• Exporting database in JSON.

$ curl 'localhost:8080/admin/export?format=json'
{"code": "Success", "message": "Export completed."}

• Fast load the data using Live Loader.

$ dgraph live -f g01.json.gz -s g01.schema --alpha=localhost:9080 --zero=localhost:5080
...

Running transaction with dgraph endpoint: localhost:9080

Processing schema file "g01.schema"
Processed schema file "g01.schema"

Found 1 data file(s) to process
Processing data file "g01.json.gz"
panic: interface conversion: interface {} is *time.Time, not time.Time

goroutine 45 [running]:
github.com/dgraph-io/dgraph/tok.HourTokenizer.Tokens(0x1818880, 0xc0027920a0, 0x10, 0xc004074020, 0xa, 0xc00328c003, 0xc000131880)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/tok/tok.go:262 +0x3ca
github.com/dgraph-io/dgraph/tok.BuildTokens(0x1818880, 0xc0027920a0, 0x1a230e0, 0x261d298, 0x1a230e0, 0x261d298, 0x0, 0x1, 0xc004074010)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/tok/tok.go:104 +0x63
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).conflictKeysForNQuad(0xc0002e8000, 0xc00348c510, 0x3e8, 0xc004072000, 0xc000045380, 0x0, 0xc00009db00)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/batch.go:301 +0x448
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).conflictKeysForReq(0xc0002e8000, 0xc000459f88, 0xc0007e2001, 0x7d0, 0x800)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/batch.go:322 +0xe9
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).makeRequests(0xc0002e8000)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/batch.go:384 +0x1db
created by github.com/dgraph-io/dgraph/dgraph/cmd/live.setup
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:356 +0x2f0

Expected behaviour and actual result.

This issue is seen only while live loading JSON data and not for RDFs.

Also, I was able to bulk load the exported data in both JSON and RDFs without any problem.

Note: Dataset used here was from JS Flock.

[abhimanyusinghgaur]

It is happening with RDF dataset as well.
I had tried loading the 1million.rdf.gz in tutorial with following command:
./dgraph live -f ~/dgraph/data/1million.rdf.gz -s ~/dgraph/data/1million.schema

[JimWen]

It is happening with RDF dataset as well with the shell below:

nohup ./bin/dgraph live -f ./data --format rdf -s ./schema.txt -a "172.21.46.35:9080,172.21.46.39:9080,172.21.46.40:9080" > live.log 2>&1 &

[MichelDiz]

@sleto-it quick note about this issue. it is not related to #3610

After reading this https://discuss.dgraph.io/t/panic-when-importing-a-bigger-dataset-in-tour/5969 I noticed the "YearTokenizer" error msg. So this is related to Datetime Tokenizers.

If I removed the indexing from the schema it loads normally.

To test just create an empty RDF file(no need to gzip it) and add:

<15113317135919419286> <initial_release_date> "2001-01-01T00:00:00Z" .

and load it:

dgraph live -f 1million.rdf --alpha localhost:9080 --zero localhost:5080 -c 1

if you add @index(year) it will fail.

initial_release_date: dateTime @index(year) .

So, the workaround, for now, is to remove the indexing just in the moment of the load. And then you can add again the indexing that will be fine.

Also, we need to make sure that we have small test cases for each case of a load(or even just for Tokenizers). Thus making sure that won't happen again.

The test was done with RDF in this version bellow

Dgraph version   : v2.0.0-beta1
Dgraph SHA-256   : 178663a98a3d59879a3d5c42928c89eb5f83afc2bfc0093272941e7a53515847
Commit SHA-1     : 6fac5d7c4
Commit timestamp : 2020-01-30 14:45:54 +1100
Branch           : HEAD
Go version       : go1.13.7

[LoganLee1991]

Something more about this: the @index(float) and @index(int) have the same problem

[zbioe]

For fix the load in tour https://tour.dgraph.io/moredata/1/:

Change this:

# Define Directives and index

director.film: [uid] @reverse .
genre: [uid] @reverse .
initial_release_date: dateTime @index(year) .
name: string @index(exact, term) @lang .

To this:

# Define Directives and index

director.film: [uid] @reverse .
genre: [uid] @reverse .
initial_release_date: dateTime year .
name: string @index(exact, term) @lang .

press Run. wait for Load the data and after finish change back again and press Run again with initial config (press Reset and after reset press Run)

[MichelDiz]

@iuryfukuda we can't edit the tour for that. We have to fix the issue. But thanks.

[zbioe]

@MichelDiz Yeah, no problem, I only putted a resolution description of the tour in this issue if anyone needs.

[ghost]

initial_release_date: dateTime year .

The line above should have year removed, then the live import runs as expected.