Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

cmd/swarm: new localstore database migration #1297

Merged
merged 18 commits into from
Apr 10, 2019

Conversation

acud
Copy link
Member

@acud acud commented Mar 13, 2019

This PR handles and tests the flow of migrating from the old localstore to the new one.
The update sequence is as follows:

  1. User updates the Swarm binary once this gets released
  2. Swarm binary checks for old schema name and identifies when the old chunk store format is present and quits the process with exit code 1. User gets alerted. The binary won't start before datastore is migrated
  3. User should create a DB export using swarm db export
  4. User should delete/move old chunk store (/chunks)
  5. User starts the Swarm binary, new empty chunkstore gets created, user should quit the Swarm binary at this point
  6. User should run the DB import command with --legacy flag
  7. Datastore migration should be now complete

At a minimum the disk space toll is @ x2 and x3 at most. We have with @janos decided we do not want to automate this process and we also don't want to add an automatic migration flag to the binary because this will require users to change the way they run Swarm and this change might not be reverted in some cases and can cause trouble in the future (i.e. a User adds a --migrate flag to her runswarm.sh script and forgets to remove it, cause future migrations to potentially be executed automatically).

RTFM: https://hackmd.io/UAV2uybkQlSerVIX5xUF0g?view

@janos janos mentioned this pull request Mar 15, 2019
@acud acud marked this pull request as ready for review March 19, 2019 09:07
@acud acud force-pushed the ls-integration-migration branch 3 times, most recently from 896d485 to a4df16e Compare March 19, 2019 09:34
@zelig zelig mentioned this pull request Mar 20, 2019
26 tasks
swarm/storage/localstore/schema.go Outdated Show resolved Hide resolved
swarm/swarm.go Outdated Show resolved Hide resolved
@@ -0,0 +1,1390 @@
package testdata
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this be shorter?

cmd/swarm/flags.go Show resolved Hide resolved
acud added 15 commits March 28, 2019 20:41
cmd/swarm: add ldb fixture for integration testing, add gzip deflate

cmd/swarm: pull deflate to helper function

cmd/swarm: start swarm with old datastore

cmd/swarm: test assert that process exits with non-zero exit code on old db

cmd/swarm: assert export size

cmd/swarm: remove datadir

cmd/swarm: change export verbosity

cmd/swarm: fix TestSomething

cmd/swarm: rename migration test

cmd/swarm: wip e2e test

cmd/swarm: wip e2e test

swarm: legacy schema detection
…t, remove unnecessary error log line that might confuse users, simplify legacy db check
@acud
Copy link
Member Author

acud commented Mar 28, 2019

@nonsense I have the results for a dry run migration on my local machine. I'm running an i5 CPU and my SSD is a SATA III SSD, so performance is not so high.
Datastore size was at around 5.5gb.
I added a resource pooling to throttle the db.Puts. I tried 100 and 200 for pool size without a big difference, so I'm pushing 100 at the moment.
tl;dr: export takes 1 minute for 5 gigs, import takes about 15m

export
time ./swarm_mig db export /home/growlster/migrationTest/sw/swarm/bzz-9fae04300e275782afafb52f8a988e23d98b85d9/chunks /home/growlster/migrationTest/exphttps://github.com/ethersphere/go-ethereum/pull/1297/commits/33e1bb277f193c34e68cb93091762ab59385f85dort.tar bd83051b5639cadaba5821a57efa131304a2465b522c19d45e5087eece9dc321                                                                                    
INFO [03-28|21:45:56.713] successfully exported 1375147 chunks 
./swarm_mig db export  /home/growlster/migrationTest/export.tar   24.01s user 19.84s system 78% cpu 56.153 total

import (token pool @ 100)
time ./swarm_mig db import /home/growlster/migrationTest/sw/swarm/bzz-9fae04300e275782afafb52f8a988e23d98b85d9/chunks /home/growlster/migrationTest/export.tar bd83051b5639cadaba5821a57efa131304a2465b522c19d45e5087eece9dc321
INFO [03-28|22:08:13.592] successfully imported 1375147 chunks 
./swarm_mig db import  /home/growlster/migrationTest/export.tar   1134.67s user 131.82s system 164% cpu 12:50.79 total

import (token pool @ 200)
time ./swarm_mig db import /home/growlster/migrationTest/sw/swarm/bzz-9fae04300e275782afafb52f8a988e23d98b85d9/chunks /home/growlster/migrationTest/export.tar bd83051b5639cadaba5821a57efa131304a2465b522c19d45e5087eece9dc321
INFO [03-28|22:29:11.723] successfully imported 1375147 chunks 
./swarm_mig db import  /home/growlster/migrationTest/export.tar   1134.91s user 132.98s system 165% cpu 12:47.45 total

@nonsense nonsense merged commit 904df7a into localstore-storage-integration Apr 10, 2019
@acud acud deleted the ls-integration-migration branch April 11, 2019 03:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants