Move blockchain out of memory

[cosme12]

As blockchain gets bigger, it will be necessary to be moved out of memory. Since this is a simplified version of a blockchain, saving it in a text file may be enough.

[kennethgoodman]

feather and pandas should do it if we only allow one input to one output transactions.

For more complex transactions, json with a fast json library: http://artem.krylysov.com/blog/2015/09/29/benchmark-python-json-libraries/

[rdenadai]

Perhaps TinyDB could be a option? If you don't need concurrent access to the file...

http://tinydb.readthedocs.io/en/latest/

Just store the blockchain in it as a json format...

[Aareon]

I would suggest a combination of SQLite/PostgreSQL and SQLAlchemy. Preferably Postgres in production.

I’ve never personally used Feather or Pandas, but I have seen the combo that I’ve suggested used in production by other block chains, and it works more than well.

In my opinion, I don’t believe TinyDB or any flat file option would be efficient enough.

[rdenadai]

@Aareon as i said, if concurrent access is not necessary...

I use TinyDB in a project, but since then i need concurrent access to my database so i change it to rethinkdb.

It was a simple migration between both of the, by simple rewriting my DAO class...

Postgresql + SQLAlchemy are always welcome but it brings a new layer of complexy to the project which main goal claims to be simple...

[Aareon]

Neither one of those solutions are ACID compliant. I don’t think we want to use something that has a chance of losing data. It may be simpler, but does not by any means mean better. My solution may be more complicated, but it will be much more reliable. If you’re okay with losing data in your ledger, be my guest.

[HourGlss]

Right now I can mine between 33-36 blocks, depending on how I choose to handle the Pipes. The limiting factor is the Pipe though, not the size of the blockchain in memory. do a test and don't send information around in pipes. I can handle thousands of them, with pipes, less than 40.

[HourGlss]

I fixed it using Queue and restructuring the way that multiprocessing works

[HourGlss]

Implemented a blockchain class which utilizes a sqldatabase https://github.com/hourglss/hourcrypto WORK IS ONGOING, PROJECT IS CURRENTLY nonfunctional.