Skip to content

Running the tests on S3

KiterLuc edited this page Apr 12, 2021 · 7 revisions

Minio (local)

Within the TileDB directory, create a build directory and move to it.
$ mkdir build && cd build

Build with S3 enabled:
$ ../bootstrap --enable-verbose --enable-s3 && make -j4

If this is your first time running minio, you will need to install it. For OS X, you must have homebrew installed.
$ source ../scripts/install-minio.sh

Start the Minio server. It is important that this is run from the TileDB/build directory because the following script contains relative paths.
$ source ../scripts/run-minio.sh

Run the unit tests:
make check

Alternatively, you may wish to just run the [s3] tests to quickly verify connection with minio.
make -C tiledb tiledb_unit && ./tiledb/test/tiledb_unit "[s3]"

AWS S3

  1. cd TileDB/build
  2. ../bootstrap --enable-verbose --enable-s3 && make -j4
  3. cmake -DTILEDB_TESTS_AWS_S3_CONFIG=ON .. && make -j4
  4. export AWS_SECRET_ACCESS_KEY=... export AWS_ACCESS_KEY_ID=...
  5. make -j4 check

The only difference here is the cmake step.

Note: If any tests fail, the created buckets may not be removed. Remember to remove them manually after test failures.

AWS S3 (Assuming ARN)

Alternatively you may wish to run without aws configure or access credentials. Note: In any case, you need to have an IAM role!

  1. cd TileDB/build
  2. ../bootstrap --enable-verbose --enable-s3 && make -j4
  3. cmake -DTILEDB_TESTS_AWS_S3_CONFIG=ON .. && make -j4
  4. List the roles of resources that you want/need to assume with $ aws iam list-roles. In UTs' context: "RoleName": "S3Access"
  5. Make sure that your role belongs to the trusted relationship. Available inside the Principal section of "Action": "sts:AssumeRole".
  6. You can test preemptively from CLI if you are authorized to assume a role with $ aws sts assume-role --role-arn [role-to-assume].
  7. Set the config variable VFS_S3_AWS_ROLE_ARN inside sm/config/config.cc to "[role-to-assume]".
  8. make -j4 check

Note: If any tests fail, the created buckets may not be removed. Remember to remove them manually after test failures.