HDDS-8450. Dedicated acceptance test suite for s3a #6458

adoroszlai · 2024-03-29T19:30:16Z

What changes were proposed in this pull request?

Run some S3A contract tests as part of acceptance check. Tests are executed via Maven against Docker Compose-based Ozone cluster, using Hadoop 3.3.6 source. JUnit test results and output for failed tests are included in the artifact, similar to integration check.

Limitations:

~~only works in unsecure env. for now~~
tests that currently fail for various reasons are excluded (created Jira issues for each)

https://issues.apache.org/jira/browse/HDDS-8450

How was this patch tested?

CI:
https://github.com/adoroszlai/ozone/actions/runs/8484286662

Also tried some failing tests locally to verify behavior of the check.

errose28

Thanks for adding these tests @adoroszlai

hadoop-ozone/dist/src/main/compose/testlib.sh

hadoop-ozone/dev-support/checks/acceptance.sh

hadoop-ozone/dist/src/main/compose/common/s3a-test.sh

errose28 · 2024-04-02T22:56:44Z

hadoop-ozone/dist/src/main/compose/test-all.sh

-generate_report "acceptance" "${ALL_RESULT_DIR}" "${XUNIT_RESULT_DIR}"
-
+if [[ "${OZONE_ACCEPTANCE_SUITE:-}" != "s3a" ]]; then
+  generate_report "acceptance" "${ALL_RESULT_DIR}" "${XUNIT_RESULT_DIR}"


Maybe add a comment or rename this method to clarify that this method only applies to tests run with Robot.

Generalized s3a-specific condition to check acceptance test type (robot or maven). We may be adding more Maven-based tests in the future.

Although the overall job passes, there is still a failure because generate_report is being called (not sure why this doesn't fail the s3a job):

Robot framework is not installed, the reports cannot be generated (sudo pip install robotframework). ... Error: Process completed with exit code 1.

Looks like OZONE_ACCEPTANCE_TEST_TYPE is not exported in acceptance.sh and not visible in this subshell.

Side note, will this also be a problem if devs run test-all.sh manually to run all the tests locally?

Thanks for pointing out the problem with OZONE_ACCEPTANCE_TEST_TYPE.

will this also be a problem if devs run test-all.sh manually to run all the tests locally?

When running test-all.sh directly, installing Robot is the developer's responsibility (it was already required before this change).

Right, my question is more generally: what is the expected way to run all acceptance tests locally and manually? Ideally running test-all.sh (or maybeacceptance.sh) should kick off all the acceptance tests locally. After this change, running either of these will only run the robot tests, unless the user knows which variables to export to their shell before running.

Maybe that's ok for now and we can handle that in a follow up task, since there's probably not many people running all the tests this way.

what is the expected way to run all acceptance tests locally and manually?
...
After this change, running either of these will only run the robot tests

Actually, they will also try to run s3a tests, but that part will fail, since Hadoop sources will be missing.

We can avoid that by further separating the two kinds of tests.

I'm ok to handle this in a follow up task if you want. It works fine in CI and I don't think many people are running the tests this way.

Slightly changed s3a-test.sh to not fail, rather skip the test if HADOOP_AWS_DIR is missing. This way current behavior is retained when running acceptance.sh or test-all.sh without any test filters.

hadoop-ozone/dist/src/main/compose/testlib.sh

… auth-keys.xml inline

adoroszlai · 2024-04-09T06:41:55Z

@ivandika3 Could you please review, too?

ivandika3

Thank you for the S3A acceptance test integration. Mostly LGTM.

Not terribly familiar with S3A, but I left some comments.

ivandika3 · 2024-04-09T11:21:17Z

hadoop-ozone/dist/src/main/compose/common/s3a-test.sh

+  pushd "${HADOOP_AWS_DIR}"
+
+  # S3A contract tests are enabled by presence of `auth-keys.xml`.
+  # https://hadoop.apache.org/docs/3.3.6/hadoop-aws/tools/hadoop-aws/testing.html#Setting_up_the_tests


Nit: I can't seem to access this link.

Do you mean r3.3.6 instead of 3.3.6? (i.e. https://hadoop.apache.org/docs/r3.3.6/hadoop-aws/tools/hadoop-aws/testing.html#Setting_up_the_tests)

ivandika3 · 2024-04-09T11:33:41Z

hadoop-ozone/dist/src/main/compose/common/s3a-test.sh

+    <property>
+      <name>fs.s3a.path.style.access</name>
+      <value>true</value>
+    </property>


It might be nice to also test for virtual host style pattern to uncover some AWS Java SDK problems with it (e.g. HDDS-7035) since our S3G implementation is built on top of path style pattern assumption (i.e. S3G needs to use VirtualHostStyleFilter to convert virtual host style -> path style).

This can be addressed in future patches if needed.

ivandika3 · 2024-04-09T12:01:24Z

@adoroszlai @errose28 I have a question regarding S3A in general. In what case does using S3A protocol more beneficial rather than using the OFS protocol? My understanding is that S3A aims to provide HDFS semantic to non hadoop-compatible storage (like using S3Guard when AWS S3 was still eventually consistent), whereas Ozone already provides OFS as a direct implementation of HDFS filesystem.

From the performance point of view, it seems using s3a adds more network overhead since the s3a client send request to S3G, and the S3G to OM whereas OFS will directly connect to OM.

Edit: One advantage I can think of is that the S3A client (just like any other S3 client) does not need to configure for OM services (or multiple OM services if federation OMs are supported in the future).

Thanks in advance.

adoroszlai · 2024-04-09T15:30:46Z

In what case does using S3A protocol more beneficial rather than using the OFS protocol?

Another case may be if client already has jars required for S3A, but classpath cannot be changed to add Ozone FS jar. Not sure if that's a common situation.

ivandika3

Thanks for updating the patch and the clarification. LGTM +1.

adoroszlai · 2024-04-10T04:35:30Z

Thanks @errose28 for the review. Patch is updated, please take another look.

errose28

Thanks for the updates @adoroszlai. Looks like there is still one issue remaining and then this should be good to go.

adoroszlai · 2024-04-12T10:40:51Z

Thanks @errose28, @ivandika3 for the review.

(cherry picked from commit a523fd9)

HDDS-8450. Dedicated acceptance test suite for s3a

4e645cd

adoroszlai added s3 S3 Gateway CI labels Mar 29, 2024

adoroszlai self-assigned this Mar 29, 2024

reject empty target dir in download_hadoop_aws

f0fc477

adoroszlai requested a review from errose28 March 30, 2024 09:51

fix setup for ITestS3AContractVectoredRead (see HADOOP-19133)

a34ecd9

errose28 reviewed Apr 2, 2024

View reviewed changes

move execute_s3a_tests(); add comments; implement secure test; create…

5e0146d

… auth-keys.xml inline

adoroszlai requested a review from errose28 April 8, 2024 18:00

ivandika3 reviewed Apr 9, 2024

View reviewed changes

fix typo in link

add9ce2

ivandika3 approved these changes Apr 9, 2024

View reviewed changes

errose28 reviewed Apr 11, 2024

View reviewed changes

adoroszlai added 4 commits April 11, 2024 08:43

export OZONE_ACCEPTANCE_TEST_TYPE

3ffb95c

avoid early exit due to test failures

288fcd2

Merge remote-tracking branch 'origin/master' into HDDS-8450

da2dc75

skip instead of fail if HADOOP_AWS_DIR is missing

45be98e

adoroszlai merged commit a523fd9 into apache:master Apr 12, 2024
41 checks passed

adoroszlai deleted the HDDS-8450 branch April 12, 2024 10:40

Tejaskriya pushed a commit to Tejaskriya/ozone that referenced this pull request Apr 17, 2024

HDDS-8450. Dedicated acceptance test suite for s3a (apache#6458)

7ce9e12

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request May 29, 2024

HDDS-8450. Dedicated acceptance test suite for s3a (apache#6458)

0bf35e3

(cherry picked from commit a523fd9)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Aug 14, 2024

HDDS-8450. Dedicated acceptance test suite for s3a (apache#6458)

615ce2a

(cherry picked from commit a523fd9)

xichen01 mentioned this pull request Aug 15, 2024

[DO NOT MERGE] Backport some fixes and compatibility commits from master to ozone-1.4 #7077

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-8450. Dedicated acceptance test suite for s3a #6458

HDDS-8450. Dedicated acceptance test suite for s3a #6458

adoroszlai commented Mar 29, 2024 •

edited

Loading

errose28 left a comment

errose28 Apr 2, 2024

adoroszlai Apr 8, 2024

errose28 Apr 11, 2024

errose28 Apr 11, 2024

adoroszlai Apr 11, 2024

errose28 Apr 11, 2024

adoroszlai Apr 11, 2024

errose28 Apr 11, 2024

adoroszlai Apr 12, 2024

adoroszlai commented Apr 9, 2024

ivandika3 left a comment •

edited

Loading

ivandika3 Apr 9, 2024 •

edited

Loading

ivandika3 Apr 9, 2024 •

edited

Loading

ivandika3 commented Apr 9, 2024 •

edited

Loading

adoroszlai commented Apr 9, 2024

ivandika3 left a comment

adoroszlai commented Apr 10, 2024

errose28 left a comment

adoroszlai commented Apr 12, 2024

HDDS-8450. Dedicated acceptance test suite for s3a #6458

HDDS-8450. Dedicated acceptance test suite for s3a #6458

Conversation

adoroszlai commented Mar 29, 2024 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

errose28 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adoroszlai commented Apr 9, 2024

ivandika3 left a comment • edited Loading

Choose a reason for hiding this comment

ivandika3 Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

ivandika3 Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

ivandika3 commented Apr 9, 2024 • edited Loading

adoroszlai commented Apr 9, 2024

ivandika3 left a comment

Choose a reason for hiding this comment

adoroszlai commented Apr 10, 2024

errose28 left a comment

Choose a reason for hiding this comment

adoroszlai commented Apr 12, 2024

adoroszlai commented Mar 29, 2024 •

edited

Loading

ivandika3 left a comment •

edited

Loading

ivandika3 Apr 9, 2024 •

edited

Loading

ivandika3 Apr 9, 2024 •

edited

Loading

ivandika3 commented Apr 9, 2024 •

edited

Loading