IPFS_PATH Folder Compatibility

[aphelionz]

Inside my go-ipfs generated ~/.ipfs folder, which is conected to a private IPFS network, I have the following item. I'd like to make sure we go through these one at a time with @vmx and/or @Stebalien's guidance and determine which files or folders need to be compatible with js <-> go <-> rust.

Should be compatible:

Note: asterisks signify "as compatible as possible, with different nomenclature denoting any departure from spec"

• api
• config*
• repo.lock*

Does not need to be compatible

• version
• datastore_spec
• blocks
• datastore

Out of grant scope

• keystore
• swarm.key

We should consider storing the repo in a different directory ~/.rs-ipfs or ~/.rust-ipfs to avoid confusion / mismanagement errors.

[aphelionz]

I'll get the ball rolling with my initial thoughts, which are:

The following feel like they're easy to make compatible, so we should:

• api
• version
• swarm.key

The following probably doesnt need to be compatible:

• datastore_spec
• blocks
• datastore

Special considerations:

• keystore: should probably be compatible
• repo.lock: may not be necessary, but if it is then its easy
• config: as compatible as possible, considering fairly minor things like go-ipfs/go-libp2p protobuf encoded base64'd privatekey

[Stebalien]

That sounds right. Definitely keep api. Note:

• go-ipfs is renaming the keys in the keystore, base32 encoding the names to avoid platform specific issues.
• repo.lock: I'd definitely include this one.
• config: depends on how much compatibility you want.

Also note, if you're planning on majorly breaking repo compatibility, I'd store your repo in a different directory (i.e., not IPFS_PATH/~/.ipfs).

[Stebalien]

Also note, if you're planning on majorly breaking repo compatibility, I'd store your repo in a different directory (i.e., not IPFS_PATH/~/.ipfs).

Alternatively, make sure it's possible to re-implement your changes in other languages. E.g., if you're going to change how something is encoded/formatted, call it a different thing so another IPFS implementation doesn't get confused.

[aphelionz]

@Stebalien Thank you for this insight! Super helpful and validating to know that we're mostly on the right track here.

When you say "call it a different thing" do you mean internally or at the public API level e.g. ipfs dag put should be ipfs dag putSomethingElse?

Maybe it's more case by case...

[Stebalien]

It depends. If you change how private keys are encoded in the config, call it something else (or, even better, don't store them in the config). If you decide to use toml/cbor to encode the config, call it config.{toml,cbor}.

[aphelionz]

Makes sense to me

[koivunej]

Started looking at json compatible config in #95 which can be made to work, but I currently have no clue on how to get the openssl working on all platforms. The config crate, if we ever want to migrate to any other encodings as it might handle those quite transparently. Currently all of this is in the http crate so it exists mostly for the tests. Implementing the PrivateKey support was quite a chore.

Collecting the files from above under $IPFS_PATH (again, in the order of "easy to impl"):

• api is filled in and truncated already
• config as above, trying, generating rsa keys require openssl at the moment
• repo.lock not set/checked yet, could easily do, same as api
• version not set/checked yet
• swarm.key nothing yet for this or private networks
• keystore nothing yet, unsure if there are tests
• datastore_spec, blocks, datastore nothing yet

swarm.key and keystore are completly out of grant scope, I think.

There is also consideration if the we mostly aim to provide a library to be used, could the configuration be provided all in rust and keep the "sort of compatible" stuff under "http". Even if going mostly-rust configuration, the repo.lock checking and setting might be good to catch misuse early when running conformance tests.

[dvc94ch]

making the stores compatible seems like more effort than it's worth, and I don't see that it's within the grant scope as the grant is about http api compatibility if I understand correctly.

[aphelionz]

@dvc94ch You are correct. @Stebalien and @koivunej I've updated the issue description based on the consensus in this thread.

[koivunej]

The updated description looks ok except for the version. I am hesitant to put anything in that file until (if ever) we are compatible with any of the repo versions.

This deliberation is valid currently only in the scope of rust-ipfs-http, perhaps we could even remove/rename the default impls in the library to pub fn from_env() or something like that and so that while an end developer can have the rust-ipfs point to a $IPFS_PATH this needs to happen deliberately. I'd see this being in line or an alternative solution to earlier:

[...] if you're planning on majorly breaking repo compatibility, I'd store your repo in a different directory (i.e., not IPFS_PATH/~/.ipfs).

[aphelionz]

Good point @koivunej. Moving that to "Does not need to be compatble"

I do think having a way to report the datastore version may come in handy if other applications are doing reads on what's inside the IPFS_PATH folder. It doesn't need to match what Go or JS uses, it can be whatever and could may likely include rust or some signifier. e.g. 1.2-rust for the version

[aphelionz]

Closing this as I believe we have a good understanding here. Feel free to re-open if there are new thoughts, or open a new specific issue.

[koivunej]

No cause to reopen but to document:

• the /api/v0/version already returns empty string for repo, something I thought up above
• I've already tried to connect two instances using the same keypair which resulted in difficult to debug situation with secio rejecting communications so we should get that repo.lock going :)

[aphelionz]

Indeed!