Correct issue with corpus loading process #69
Conversation
Previously, if tests failed, randomly named test databases could be left behind. This change makes it so that all test data files are created within a directory which is created and removed after tests have finished. In the case on an exception, a directory is easier to manually clean up than many data files.
| @@ -1,4 +1,5 @@ | |||
| import importlib | |||
| from importlib import import_module as import_string | |||
| from inspect import isclass | |||
There was a problem hiding this comment.
Is this being used anywhere?
There was a problem hiding this comment.
I don't believe so. I will make sure to remove them before this is merged. I'm currently getting better, results with these changes (basically attempting to use python modules for data instead of json files) but I'm still hitting an issue where importlib is returning a different sub-module depending on how it is called (works in tests, fails in application).
There was a problem hiding this comment.
If you aren't hitting the issue on Python 3, it might be an issue with relative imports vs absolute imports.
There was a problem hiding this comment.
It appears to be both python ~2.7 and 3. I am planning on scrapping importing data files as python modules in favor of importing them as regular files. The advantage is that it is easier to traverse a directory and find all of the contained .json files than it is to traverse a python module to find all of the sub-modules.
Previously, corpus data modules were being treated as python modules. This caused some strange issues with relative imports. The problem has been fixed by redesigning the corpus loader to use a data directory and traverse subdirectories to load any data files contained within.
Correct issue with corpus loading process
This makes corrections to the loading process for loading corpus modules where an import error is occurring.