async/await: complete rewrite of foundationdb

[Speedy37]

I don't know if this commit can be the future of this project or if I should publish it on my own with another name. I changed the API and logic quite a lot.

The most important change to me is how transactions are shared across threads. They can't be cloned anymore, so you share them by reference, Transaction implements Send/Sync, but commit/cancel/reset api requires mutable access to a Transaction. This protect against undefined behavior that was previously possible (cancel/reset) data races.

• no more indirection within FdbFuture
• no race condition/undefined behavior possible (Transaction can't be cloned anymore) and commit, cancel, reset are protected
• fdb api 610+
• option generation is now indented and the code is simpler
• RangeOption and KeySelector can be either be Owned or Borrowed
• KeySelector offset can be negative (there is a test of this in the binding checker, this was not found due to casting luck)
• some int options can be negative
• fix boot api safety (undefined behavior was possible)
• simple boot process
• use rustc stable in 1.39 async/await feature
• foundationdb 510, 520, 600 support with common Database::new_compat api

I'm marking this as WIP as I still need to polish/document it.
Here is a small/incomplete TODO of what I want to do:

• check every single integer casts
• threaded bindingtester
• write doc
• continuous integration
• directory / hca

I've not focused much on the tuple/subspace/directory API as I'm not using them in production. I built a strictly typed API that enforce database layout.

Fixes #111
Fixes #117
Fixes #123
Fixes #124
Fixes #125
Fixes #128
Fixes #130
Fixes #132
Fixes #135

[bluejekyll]

@brndnmtthws, thoughts? I currently don’t have skin in the game, but this is a significant rewrite and change in direction.

[Speedy37]

Well serde was a bad idea to implement the tuple layer, I'm converting it back.

[Speedy37]

This latest version passes the following bindingtester tests 100 times (might still miss some low occurrences bugs).

./bindingtester.py --num-ops 1000 --test-name api --api-version 610
./bindingtester.py --num-ops 1000 --concurrency 5 --test-name api --api-version 610
./bindingtester.py --test-name scripted

Most bugs were in the tuple pack/unpack code and in the bindingtester code itself.
It was quite hard to keep the state of the bindingtester ok due to the new safety measures of the new api. I had to simulated get committed version on a non committed transaction for example (last commit).

[bluejekyll]

Ok, I went through at the given state. Given it's such a massive rewrite, I'm sure I missed something, but otherwise it all looks reasonable.

I'm not sure about the serde depedency and the value add there, but it does look to have simplified the serialization quite a bit.

@yjh0502 do you have any issues with this? @brndnmtthws? @dae, this would supersede your async/await support PR if we decide to go this direction.

@Speedy37 assuming everyone else is ok with this rewrite, I'd say let's merge this. Rather than forking, I'd prefer to make you collaborator or transfer ownership (whatever we think is best), and then I'll add you on crates.io to be able to push new versions.

[dae]

I haven't reviewed the code properly, but have no objections to my PR being discarded if this is a better long term solution.

@Speedy37 : Is the retry/timeout code in transact() necessary? I was under the impression FDB handles that itself when on_error is called.
https://apple.github.io/foundationdb/api-c.html#c.fdb_transaction_on_error

[Speedy37]

I dropped the serde approach to ser/deserialization as some types (Versionstamp, Uuid) were quite hard to support. It might be possible to add it back as impl TuplePack/TupleUnPack for T where T: Serialize/Deserialize which would make "named" tuple possible by using structs.

I had like to not fork too. I'm also open to suggestion and would recommend not merging this until rustc 1.39 is out. I still need to finish porting my app whole stack to this version to validate the usability of the API.

In the current state, I'm not sure about the signature of Database::transact.

Fdb does handle the retry count itself if specified, so this limit is a duplicate.

[Speedy37]

Ok this is only missing the directory layer API now and I don't know if I can spend the time to write it soon. Probably next month.

[Speedy37]

Here is a link to the latest generated documentation: https://clikengo.github.io/foundationdb-rs/foundationdb/index.html

[Speedy37]

Ok for now, the biggest downside of this version is that Database::transact is not the easiest to use.
For me there is two way to approach how to use foundationdb with this branch:

• share a transaction reference, this is easy, but might require some lifetimes hints to the compiler which might be hard for new to rust users. On the other side, doing it this way, force you to prove to the compiler that what you are doing makes sense.
• share a reference counted transaction (either Arc or Rc). It's is easier to do, but, also there is no compiler proof at commit/abort/reset point that there is only one reference to the transaction alive. It's something that is done at runtime.

[josephg]

This is awesome - thankyou so much for getting this working! 😍

[josephg]

FWIW I think sharing a transaction reference is the right API - getting static lifetime checks is good rust code, even if it means users need lifetime qualifiers for the transaction object.

[pH14]

🎉 this is such a huge step forward, amazing work