Import data with foreign key

[DayDotMe]

Hello and first of all, thanks for this package !

I'm having issue when trying to import data with a foreign key field. The code doesnt raise any error, but nothing is saved in database. I've tried using custom initialize as in the documentation, however though I obtain correct values, no object is saved in database. When trying the same code with entities without foreign keys, everything goes fine.

Please find attached my function code. Thanks in advance for any help.
ps : here is what print(row) prints : (types and values are as expected)
['SCHNEIDER ELECTRIC', 'A9L16577', 'v8', 'Z130', 'B', 'aaa', 1, 'dummy', <Groupe: Serveurs>]
`
def import_xls(item, form):

    if form.is_valid():
        if request.FILES['file'].content_type.find('spreadsheet') == -1:
            messages.add_message(request, messages.ERROR, 'Le fichier n\'est pas un tableur !')
            return
        fields = []
        temp = item._meta.get_fields()
        for field in temp:
            if not field.name.startswith("article") and not field.name == 'id' and not isinstance(field,ManyToManyField)and not isinstance(field, ManyToManyRel):
                fields.append(field.name)
        if item == ArticleHW or item == ArticleSW: #no foreign keys, everything goes fine
            try:
                request.FILES['file'].save_to_database(model=item, mapdict=fields)
                messages.add_message(request, messages.SUCCESS, 'Enregistrement : Succès')
            except ValueError:
                messages.add_message(request, messages.ERROR, 'Erreur : Le nombre de champs ne correspond pas !')
        elif item == Equipement:
            try:
                def set_group(row):
                    group = Groupe.objects.get(id=int(row[8]))
                    row [8] = group
                    print(row)
                    return row
                #this is not saving 
                request.FILES['file'].save_to_database(model=item, mapdict=fields, initializer=set_group)           
                messages.add_message(request, messages.SUCCESS, 'Enregistrement : Succès')
            except ValueError as e:
                print(e)
                messages.add_message(request, messages.ERROR, 'Erreur : Le nombre de champs ne correspond pas !') `

[chfw]

Are you able to check your log file see what cause it to fail?

pyexcel-io v0.4.x line:
https://github.com/pyexcel/pyexcel-io/blob/master/pyexcel_io/database/importers/django.py#L51

and pyexcel-io v0.3.x line:

https://github.com/pyexcel/pyexcel-io/blob/v0.3.x/pyexcel_io/database/django.py#L68

[DayDotMe]

God thank you for answer, it was a problem with the excel file formatting. However it's a bit of a shame to have to go through pyexcel logs to find this kind of error. Anyway thanks again for your work and your answer.

[chfw]

Hmm... what if the save_to_database throw a exception in the first place? would it help you find the problem early?

[DayDotMe]

Yes to me it would definitely be a great feature.
For instance I added a length check to save_as() in core.py and the corresponding error message to warn the user if he tries to import files with more fields than in the model and I'll probably add importers messages.

[DayDotMe]

Hi again. I still have trouble with these imports and this time nothing is logged by django.py
Actually, when I try to save in database, django throws a value error with this message :
"<Equipement: SIEMENS>" needs to have a value for field "id" before this many-to-many relationship can be used
Below is my code:

           try:
                def set_group(row):
                    try: # if group exists its assigned, if not i create a new one, no issue there
                        group = Groupe.objects.filter(name__contains=str(row[8]))[0]
                    except IndexError:
                        group = Groupe(name=str(row[8]))
                        group.save()
                    row[8] = group
                    return row
                request.FILES['file'].save_to_database(model=item, mapdict=fields, initializer=set_group)
                messages.add_message(request, messages.SUCCESS, 'Enregistrement : Succès')
            except ValueError as e:
                messages.add_message(request, messages.ERROR, 'Erreur : ' + str(e) )

Thanks in advance for any help.

[chfw]

This means that equipment need to be saved in the database before the database give an id back.

[DayDotMe]

Yes I know, but equipment should be saved by save_to_database (item is of type Equipment).

[chfw]

Alright, in this case where nothing is left, it is assumed that the item is created. However, the item was not there in the database.

I would suggest that to set a break point in the django.py, right after the close() method. And then try upload items again. step through the code and see.

[chfw]

pyexcel-io v0.4.1 with explicit exception is released.

[DayDotMe]

I was in a rush so I have written a workaround but I will debug this, try v0.4.1 and update

[DayDotMe]

New issue with pyexcel-io, here is my debugger content if it can be of any use to you.

I tried commenting this line, but then it'll throw an attribute error: "module 'pyexcel' has no attribute 'save_as' ". In fact, a coworker already faced this error with v0.4.0 of pyexcel though we installed it and django-excel exactly in the same way.

[chfw]

You are reproducing this problem here. Are you able to try logging it? At the meantime, I checked the package files pyexcel 0.5.0 and pyexcel-io 0.4.x and the packages are not having the same issue as this one: pyexcel/pyexcel-io#37, where an extra folder 'pyexcel' accidentally appeared in pyexcel-io.

[chfw]

The screenshot you had is telling the uploaded file is empty hence an IOError will be thrown.

[DayDotMe]

I tried to use trace-pyexcel.py but it's not recognised. I've included logging lines in django_excel init since it's throwing the exception but I dunno where I can find this log file.

By the way, my file is obviously not empty, actually data are read and I can see it in the debugger, but it throws an error anyway.

[chfw]

the read() returned empty content, it could mean either no content uploaded or the file pointer was pointing at the end of the stream. So the latter case may apply to your exception.

[chfw]

as to django_excel log, I guess it will go to your django log and you should be able to find out the log setting in settings.py.

[chfw]

What kind of problem did you have here?

I tried to use trace-pyexcel.py but it's not recognised.

[chfw]

error: "module 'pyexcel' has no attribute 'save_as' " was transferred to #90.

[chfw]

please try pyexcel-io v0.4.2. I think it should fix the problem: no attribute 'save_as'

[DayDotMe]

Sorry I was a bit in a rush for the project and debugging gave me no clue. I'll try and test this today if I have time and let you know.

[DayDotMe]

Hi again, so I've just installed and tried v0.4.2.
I still have a no content error even with a brand new file. I also tried again commenting the error raise but then I got the no attribute 'save_as' error.
I dont get why this IOError is raised, below is Django "debugger" content.

[chfw]

What if you insert an extra line into django-excel? In past, the django-excel developer has passed a file which has been read and get unexpected errors. That's why an IOError was raised up-front. The reason why a file could have been read is down to the other middleware in django. But this depends on the middleware.

def get_params(self, **keywords):
    extension = self.name.split(".")[-1]
    keywords['file_type'] = extension
    self.file.seek(0) # <--
    content = self.file.read()
    if content:
        keywords['file_content'] = content
    else:
        raise IOError("No content was uploaded.")
    return keywords

[chfw]

I also tried again commenting the error raise but then I got the no attribute 'save_as' error.

How would you comment the error and get 'save_as' error?

[DayDotMe]

I tried adding this extra line, now I got AttributeError: "module 'pyexcel' has no attribute 'save_as'" l°50

How would you comment the error and get 'save_as' error?

Well I just commented the else in if content:

[chfw]

Hmm, this means that you still have pyexcel/pyexcel#89. Could you quickly try this?

$ python
> import pyexcel as p
> p.save_as()

And I would recommend that you uninstall all previous versions of pyexcel-io, v0.4.0, v0.4.1 and pyexcel v0.5.0. And then re-install latest pyexcel and pyexcel-io.

There is a un-wanted record in pyexcel-io 0.4.0 and v0.4.1 which removes pyexcel/init.py when you do uninstall.

[DayDotMe]

Indeed I tried in python console and pyexcel had no attribute save_as. I reinstalled pyexcel and with that extra line of your it seems to work !

[chfw]

And that also means some of your middleware has read the content. But I think I could put the extra line as a fix for un-wanted read by other django middleware.