Functional test: minio-py while running in mint should exit on failure

[kannappanr]

Continuation of issue #708
If minio-py tests fail,

1. Check if minio-py on failure returns 1 to the shell script that checks it's return value
2. If minio-py is not returning 1, we need to do so
3. If minio-py is returning 1, then we need to see why shell script is ignoring that

[Praveenrajmani]

I checked the minio-py functional test cases, It returns with proper exit code 1 if it catches an exception https://github.com/minio/minio-py/blob/26982067494f425a4235a21f12d4c8767438a3d0/tests/functional/tests.py#L1975.
I ran the functional test cases and inferred the following discrepancies from both s3 and minio

• s3
  The test cases test_put_object and test_stat_object were failing because of the metadata non-canonicalization. s3 does not canonicalize the metadata in response. So few tests like https://github.com/minio/minio-py/blob/26982067494f425a4235a21f12d4c8767438a3d0/tests/functional/tests.py#L604 and https://github.com/minio/minio-py/blob/26982067494f425a4235a21f12d4c8767438a3d0/tests/functional/tests.py#L598 were failing.
  And test_make_bucket_recreate was failing. which will be covered by Anis' PR
• minio
  regional make bucket failures as reported by Functional tests: Fail against play/ #710

Will send a PR which covers the s3 failures for now.

[kannappanr]

@Praveenrajmani Did you check if the shell script in mint is ignoring the return value from minio-py?

[Praveenrajmani]

@Praveenrajmani Did you check if the shell script in mint is ignoring the return value from minio-py?

yes @kannappanr , mint's generic functionrun_test() https://github.com/minio/mint/blob/c5035ea2fa7f979a98f6a961d8ecc1ce77611672/mint.sh#L72 executes the tests for all the sdks. It checks for the error code $? https://github.com/minio/mint/blob/c5035ea2fa7f979a98f6a961d8ecc1ce77611672/mint.sh#L83 in all the runs.

[kannappanr]

@Praveenrajmani This issue does not seem to be fixed yet. I ran it now against the latest staging build in Gateway s3 mode

`docker run -e "SERVER_ENDPOINT=192.168.1.70:9000" -e "ACCESS_KEY=ACCESSKEY" -e "SECRET_KEY=AWSSECRETKEY" -e "ENABLE_HTTPS=0" -e "MINT_MODE=full" minio/mint:edge minio-py`
Running with
SERVER_ENDPOINT:      192.168.1.70:9000
ACCESS_KEY:           ACCESSKEY
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 478b9e02c82d:/mint/log /tmp/mint-logs'

(1/1) Running minio-py tests ... done in 4 seconds

All tests ran successfully

$ docker cp 478b9e02c82d:/mint/log/log.json /tmp/log.json
$ cat /tmp/log.json | jq
{
  "duration": 1332,
  "function": "make_bucket(bucket_name, location)",
  "args": {
    "location": "default value ('us-east-1')",
    "bucket_name": "minio-py-test-a8883eb2-b4f8-4960-b74b-677e2f6ef674"
  },
  "name": "minio-py:test_make_bucket_default_region",
  "status": "PASS"
}
{
  "duration": 1488,
  "function": "make_bucket(bucket_name, location)",
  "args": {
    "location": "us-west-1",
    "bucket_name": "minio-py-test-fb150438-94c3-456b-b6f3-dbadcd096f0d"
  },
  "name": "minio-py:test_make_bucket_with_region",
  "status": "PASS"
}
{
  "function": "make_bucket(bucket_name, location)",
  "args": {
    "location": "default value ('us-east-1')",
    "bucket_name": [
      "minio-py-test-bccd6161-c298-461b-952a-395aa1c89e72.",
      ".minio-py-test-bccd6161-c298-461b-952a-395aa1c89e72",
      "minio-py-test-bccd6161-c298-461b-952a-395aa1c89e72...abcd"
    ]
  },
  "name": "minio-py:test_negative_make_bucket_invalid_name",
  "status": "PASS"
}
{
  "function": "make_bucket(bucket_name, location)",
  "args": {
    "bucket_name": "minio-py-test-d037a265-897e-4308-b3af-88b38e0472fe"
  },
  "message": "Recreating the same bucket SHOULD have failed!",
  "error": "NoneType\n",
  "duration": 917,
  "name": "minio-py:test_make_bucket_recreate",
  "status": "FAIL"
}

The mint says success but there is a failure that is logged.

[Praveenrajmani]

Hi @kannappanr,

This was one such test case which was not actively raising an exception on failure rather just logging it. So mint will not treat it as an error since the error code will not be 1.

BTW, this particular test is removed in this commit cbcf28e1849dda3c159248f91163cf38822ce4e3.

Mint pulls from the latest release tag from https://api.github.com/repos/minio/minio-py/releases/latest which did not cover the changes yet. We will no longer see this test running after the release (the release tag gets updated).

[kannappanr]

@Praveenrajmani I wanted to make sure that was the only test that was causing this behavior. Will close this issue now.