fixed timeout issue in UT

[harshbafna]

Description

• The issue was observed because of the race condition while reusing the management/inference channel in negative test cases.

• The channel was closed by another test case asynchronously and actually gets closed when while other test case is in middle of executing the API call.

• Refactored the code to always use a fresh connection in case of negative test cases.

• Also simplified a few assert statements.

Fixes #492

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

Feature/Issue validation/testing

Successfully executed the frontend build 100 times using following code snippet on m4.large EC2 instance :

#!/bin/bash
END=100
COUNT=0
for ((i=1;i<=END;i++)); do
    if frontend/gradlew -p frontend clean build;
    then
        ((COUNT=COUNT+1))
    fi
done

echo "Frontend build succeeded $COUNT times out of $END"

Note: The log file generated by above code exceeds the GH limit of 10 MB.

Checklist:

• Have you added tests that prove your fix is effective or that this feature works?
• New and existing unit tests pass locally with these changes?

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-cpu
• Commit ID: 2afd71f
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-gpu
• Commit ID: 2afd71f
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-cpu
• Commit ID: 1bfecb9
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-gpu
• Commit ID: 1bfecb9
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-cpu
• Commit ID: cf72878
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-gpu
• Commit ID: cf72878
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-cpu
• Commit ID: 3f1a828
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-gpu
• Commit ID: 3f1a828
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[maaquib]

I am not sure if I understand this issue. I was under the impression that the UTs were executed sequentially. If not, aren't there other test cases where race conditions would make this non-deterministic.

[harshbafna]

I am not sure if I understand this issue. I was under the impression that the UTs were executed sequentially. If not, aren't there other test cases where race conditions would make this non-deterministic.

Yes, the UTs are executed sequentially. However, in most of the places we use channel.close() which immediately returns takes its own sweet time in the background to actually close the channel. There is an option to make these channel.close() calls synchronous. However, that increases the overall build time almost to double.

Further, we always create a new channel for negative test cases, but these were a few missed test cases during refactoring where we reused the existing channel instead of creating a new one.

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-cpu
• Commit ID: df4213e
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-gpu
• Commit ID: df4213e
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-cpu
• Commit ID: 0db75c7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-gpu
• Commit ID: 0db75c7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-cpu
• Commit ID: 0db75c7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[sagemaker-neo-ci-bot]

AWS CodeBuild CI Report

• CodeBuild project: torch-serve-build-gpu
• Commit ID: 0db75c7
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository