Database becoming corrupt

[springjools]

I'm using tinyDB for simple storage in a dababase that isn't that big, it consists of 2 tables with < 100 elements each. However, after operation for a while, I've noticed that I run into db corruption issues, either that the db becomes empty, or that it contains "extra data": http://pastebin.com/BuY0XAuR

I'm using one python module to insert data, and another one to read data. It's a telegram bot so I suppose it can have many worker threads at the same time, however there are not many users, maybe 5.

I've noticed, when examining the db file in notepad++ and using a json formatter, that it adds data right in the middle of previous data, or it adds both tables twice, instead of updating them and keeping them unique.

It looks like this:

{
"_default" : {
"users" : {
}{
"_default" : {
"users" : {
}

Is the problem that I'm using tinyDB for something that should be thread safe?

Here is the problem again:

userlist = users.search(User.notify_list.test(test_func))
File "C:\bin\Python27\lib\site-packages\tinydb\database.py", line 406, in search
elements = [element for element in self.all() if cond(element)]
File "C:\bin\Python27\lib\site-packages\tinydb\database.py", line 303, in all
return list(itervalues(self._read()))
File "C:\bin\Python27\lib\site-packages\tinydb\database.py", line 276, in _read
return self.storage.read()
File "C:\bin\Python27\lib\site-packages\tinydb\database.py", line 31, in read
raw_data = (self.storage.read() or {})[self.table_name]
File "C:\bin\Python27\lib\site-packages\tinydb\storages.py", line 101, in read
return json.load(self.handle)
File "C:\bin\Python27\lib\json__init.py", line 291, in load
**kw)
File "C:\bin\Python27\lib\json__init.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\bin\Python27\lib\json\decoder.py", line 367, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))

[eugene-eeo]

Hi, TinyDB is not multiprocess safe. It is meant to be written to and read from only one thread. :)

On the other hand if you need something more advanced and supports multiprpcess readers and writers maybe something like MongoDB would help.

[eugene-eeo]

Or you could write an SQLite backend with the BLOB type.

[msiemens]

@springjools Just for clarification: are you using multiple threads or multiple processes?

[springjools]

I'm not that expert with python: I use two processes, where only 1 alters the db. But I think telegram starts multiple threads when you make a bot, so I suppose it would be both multi thread and multi process.

[msiemens]

Closing this. Multithreaded/multiprocess usage is documented to be unsupported.

For multithreaded use one can resort to TinyRecord, multiprocess usage is unsolved (altough one could write a storage with file based locking).