ARROW-7003: [Rust] Generate flatbuffers files in docker build image

This PR updates the build docker image to generate IPC source based on the flatbuffer specification.

Note that the generated files are currently not used in the build (the generated files have a `_generated.rs` suffix so don't overwrite the checked in versions of the files).

We cannot fully automate this until google/flatbuffers#5589 is resolved, or we figure out a workaround, but I suggest we merge this current PR since it lays the groundwork.

Closes #5738 from andygrove/ARROW-7003 and squashes the following commits:

80abddb <Andy Grove> lint
43f3e89 <Andy Grove> reset workdir
0a186b5 <Andy Grove> lint
6a9c6f0 <Andy Grove> fix docker lint issue
b2c9173 <Andy Grove> Generate flatbuffers files in docker build image

Authored-by: Andy Grove <andygrove73@gmail.com>
Signed-off-by: Andy Grove <andygrove73@gmail.com>

ci/docker_build_rust.sh

@@ -26,6 +26,13 @@ rustup show
 # clean first
 cargo clean
 
+# flatbuffer codegen
+echo Generating IPC
+flatc --rust -o arrow/src/ipc/gen/ ../format/*.fbs
+
+# work around some bugs in flatbuffers
+find arrow/src/ipc/gen/ -name "*_generated.rs" -exec sed -i 's/type__type/type_type/g' {} \;
+
 # raises on any formatting errors
 echo "Running formatting checks ..."
 cargo +stable fmt --all -- --check

rust/Dockerfile

@@ -30,6 +30,23 @@ RUN rustup default "$(cat /tmp/rust-toolchain)"
 # Enable stable rustfmt for nightly Rust
 RUN rustup component add rustfmt --toolchain stable-x86_64-unknown-linux-gnu
 
+# Install pre-requisites for building flatbuffers
+RUN apt-get update \
+    && apt-get install -y build-essential cmake \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install flatbuffers
+RUN wget https://github.com/google/flatbuffers/archive/v1.11.0.tar.gz \
+    && tar xzf v1.11.0.tar.gz
+
+WORKDIR /flatbuffers-1.11.0
+
+RUN cmake -G "Unix Makefiles" \
+    && make install
+
+WORKDIR /
+
 # Set environment variables for location of test data required by unit and integration tests
 ENV ARROW_TEST_DATA=/arrow/testing/data
 ENV PARQUET_TEST_DATA=/arrow/cpp/submodules/parquet-testing/data

rust/README.md

@@ -65,3 +65,19 @@ and check for lint issues:
 ```bash
 cargo +stable fmt --all -- --check
 ```
+
+## CI and Dockerized builds
+
+There are currently multiple CI systems that build the project and they all use the same docker image. It is possible to run the same build locally.
+
+From the root of the Arrow project, run the following command to build the Docker image that the CI system uses to build the project.
+
+```bash
+docker-compose build rust
+```
+
+Run the following command to build the project in the same way that the CI system will build the project. Note that this currently does cause some files to be written to your local workspace.
+
+```bash
+docker run -v `pwd`:/arrow -it arrowdev/arrow-rust
+```

rust/rustfmt.toml

@@ -15,4 +15,9 @@
 # specific language governing permissions and limitations
 # under the License.
 
-max_width = 90
+max_width = 90
+
+# ignore generated files
+ignore = [
+    "arrow/src/ipc/gen",
+]