Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

integration test(ticdc): Fix Tar Wildcard Handling in download-integration-test-binaries.sh #11513

Conversation

wlwilliamx
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #11512

What is changed and how it works?

Summary:

This PR fixes an issue related to wildcard handling in the scripts/download-integration-test-binaries.sh script, which is used to download and extract required binaries for integration tests in TiCDC. The problem affects systems with certain tar implementations where wildcard matching is not enabled by default, causing the extraction of PD binaries to fail and integration tests to break.

Issue Explanation:

In the current implementation, the following command is used to extract the pd-server.tar.gz package:

tar -xz -C third_bin 'bin/*' -f tmp/pd-server.tar.gz && mv third_bin/bin/* third_bin/

This command uses a wildcard pattern 'bin/*' to match and extract all files within the bin directory of the compressed pd-server.tar.gz archive.

However, not all versions of tar handle wildcard patterns by default. In particular:

  • Older or non-GNU versions of tar: These versions do not enable wildcard support unless the --wildcards option is explicitly specified.
  • GNU tar: Newer versions of GNU tar generally support wildcards by default, but it's not guaranteed across all environments.

When the script is executed on systems where tar does not natively support wildcard patterns, the command fails to extract the necessary PD executables. As a result:

  • The third_bin directory remains empty of PD-related binaries.
  • The subsequent integration tests fail due to missing dependencies.

Proposed Solution:

To resolve this issue, I have added the --wildcards option to the tar command in the script, ensuring that wildcard matching works correctly across all versions of tar. Here is the updated command:

tar -xz --wildcards -C third_bin 'bin/*' -f tmp/pd-server.tar.gz && mv third_bin/bin/* third_bin/

By explicitly adding the --wildcards option:

  • We ensure consistent behavior of the script across different systems and tar implementations.
  • The PD binaries are correctly extracted into the third_bin directory, allowing the integration tests to run successfully.

Steps Taken:

  • Updated the tar command in the scripts/download-integration-test-binaries.sh script to include the --wildcards option.
  • Verified that the script runs successfully on systems with both GNU tar and non-GNU tar versions.
  • Confirmed that the PD binaries are correctly extracted, and the integration tests complete without issues.

Impact:

This fix ensures that the integration test binaries are reliably downloaded and extracted across different environments, preventing failures caused by missing PD executables. This improves the stability and portability of the TiCDC integration tests.

No other parts of the system are affected by this change, and no additional dependencies are introduced.

Check List

Tests

  • Manual test (add detailed scripts or steps below)
make prepare_test_binaries

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

No

Release note

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 23, 2024
Copy link

codecov bot commented Aug 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 57.4887%. Comparing base (691380d) to head (082911a).
Report is 7 commits behind head on master.

Additional details and impacted files
Components Coverage Δ
cdc 61.2314% <ø> (+0.0412%) ⬆️
dm 51.0636% <100.0000%> (+0.0341%) ⬆️
engine 63.4161% <ø> (+0.0282%) ⬆️
Flag Coverage Δ
unit 57.4887% <ø> (+0.0376%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11513        +/-   ##
================================================
+ Coverage   57.4511%   57.4887%   +0.0376%     
================================================
  Files           851        851                
  Lines        126431     126463        +32     
================================================
+ Hits          72636      72702        +66     
+ Misses        48384      48361        -23     
+ Partials       5411       5400        -11     

@wlwilliamx
Copy link
Contributor Author

/retest

@wlwilliamx
Copy link
Contributor Author

/check-issue-triage-complete

Copy link
Contributor

ti-chi-bot bot commented Aug 27, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongyunyan, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [hongyunyan,lance6716]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link
Contributor

ti-chi-bot bot commented Aug 27, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-08-23 06:12:18.410408564 +0000 UTC m=+505133.544858685: ☑️ agreed by hongyunyan.
  • 2024-08-27 05:27:23.538584493 +0000 UTC m=+848038.673034614: ☑️ agreed by lance6716.

@wlwilliamx
Copy link
Contributor Author

/test cdc-integration-kafka-test

@ti-chi-bot ti-chi-bot bot merged commit 6f697c4 into pingcap:master Aug 27, 2024
28 checks passed
@wlwilliamx wlwilliamx deleted the fix/tar-wildcard-handling-integration-test branch September 5, 2024 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ticdc: tar Extraction Failure in scripts/download-integration-test-binaries.sh Due to Wildcard Handling
3 participants