Port TorchCraft communication frames from txt to binary

[syhw]

Run all TC tests + test with the bot.

[ebetica]

The test we usually do for anything related to the replay format is here: https://github.com/TorchCraft/TorchCraft/blob/master/examples/dump_replay.lua

[dgant]

Seconding @jgehring 's suggestion of FlatBuffers. It has the third-fastest round-trip serialization among these benchmarked libraries: https://github.com/thekvs/cpp-serializers (accepting the usual disclaimers about the utility of benchmarks).

• python client #1 Cap'n Proto isn't supported by Visual Studio: Why flatbuffers instead of capnp? google/flatbuffers#2
• Fix bwapi on linux documentation #2 YAS I haven't seen much adoption for. It'd be a dart throw
• Install script fails on Mac #3 is flatbuffers.

For FlatBuffers, that'd entail:

• Adding the FlatBuffers library
• Adding a schema for the Replayer
• Adding a script for compiling the schema
• Including the schema compilation in the TC build script
• Replacing the DSV serialization with the FlatBuffers serialization

Thumbs, @syhw @jgehring @ebetica ?

[syhw]

We already use flatbuffers for some stuff https://github.com/TorchCraft/TorchCraft/tree/master/BWEnv/fbs I think that's the point of the remark of @jgehring . My point of view: "Yay go FBs!"

[jgehring]

Yup, the current wire protocol is using fbs for message serialization, so it's a natural choice. One particular advantage (in general) is its zero-overhead deserialization which makes it a great option for e.g. ML datasets. At the path that @syhw is pointing to, there's already a small script which prepares the fbs header file. We keep that one in git directly (and also include the fbs header) so that we don't introduce another build-time dependency.

The frame-related serialization methods are located in replayer/frame.cpp. We can use fbs for the replayer as well, but as a first step it might be easier to use fbs-serialized frames for client-server communication only (basically, having meaningful structs for Frame.data in messages.fbs and doing serialization in BWEnv and deserialization in TC client). As you said, we can then also port all of replayer, including frame diffs, to fbs. Note that the current struct-based setup of replayer::Frame requires copying the data from the fbs struct if we don't want to lose compatibility (which is what we should aim for at this stage).

One caveat (if you're on macOS) is that IIRC @syhw mentioned that the recent fbs version from Homebrew is somehow borked; I can't remember the details though.

[dgant]

What are we hoping to maintain compatibility with? StarData?

[ebetica]

StarData already doesn't work on develop. I think it's fine to just replace everything, but I might be missing something obvious.

[jgehring]

What are we hoping to maintain compatibility with? StarData?

I wanted to point out that changing the actual data-structures as opposed to just the serialization format is a bigger effort and affects library users, scripting frontends, etc.