Improved URL Parsing

[brendan-matroid]

• Use shellquote to split lines in commands.txt files so filenames with spaces can be used
• Escape regex characters in prefixes when initializing ObjectURLs. These may be a part of valid S3 prefixes.

[igungor]

Hey Brendan, thanks for the pull request.

Changes are looking good. I've a few requests though:

• Could you add tests please? You can find sample tests at e2e/run_test.go.
• We vendor dependencies. Could you run go mod tidy && go mod vendor and commit the changes?

Thanks

[brendan-matroid]

Hey @igungor, thanks for the review.

Are there any steps I need to take before running the tests locally? I'm seeing a bunch of errors similar to this:

--- FAIL: TestCopyMultipleFlatS3ObjectsToLocal (0.05s)
    cp_test.go:138: assertion failed: directory /tmp/TestCopyMultipleFlatS3ObjectsToLocal-726849334/workdir does not match expected:
        /another_test_file.txt
          mode: expected -rw-r--r-- got -rw-rw-r--
        /filename-with-hypen.gz
          mode: expected -rw-r--r-- got -rw-rw-r--
        /readme.md
          mode: expected -rw-r--r-- got -rw-rw-r--
        /testfile1.txt
          mode: expected -rw-r--r-- got -rw-rw-r--

[igungor]

Are there any steps I need to take before running the tests locally? I'm seeing a bunch of errors similar to this:

--- FAIL: TestCopyMultipleFlatS3ObjectsToLocal (0.05s)
    cp_test.go:138: assertion failed: directory /tmp/TestCopyMultipleFlatS3ObjectsToLocal-726849334/workdir does not match expected:
        /another_test_file.txt
          mode: expected -rw-r--r-- got -rw-rw-r--
        /filename-with-hypen.gz
          mode: expected -rw-r--r-- got -rw-rw-r--
        /readme.md
          mode: expected -rw-r--r-- got -rw-rw-r--
        /testfile1.txt
          mode: expected -rw-r--r-- got -rw-rw-r--

Some of the tests are failing due to the new o.Prefix = regexp.QuoteMeta(o.Prefix) line. The above test has an assertion that file modbits are expected to be 0644 but got 0664.

New run test is failing because of the prefix change as well.

--- FAIL: TestRunSpecialCharactersInPrefix (0.20s)
    run_test.go:247: line 0: equals: (-want +got):
          string(
        - 	"cp s3://test-run-special-characters-in-prefix/special-chars_!@#$%!^(MISSING)&_()_+{[_%!C(MISSING)äè| __;'_,_._-中文 =/_!@#$%!^(MISSING)&_()_+{[_%!C(MISSING)äè| __;'_,_._-中文 =image.jpg",
        + 	"",
          )
    run_test.go:247: expected a comparison function for line "cp s3://test-run-special-characters-in-prefixspecial-chars_!@#$%^&_()_+{[_%5Cäè| __;'_,_._-中文 =/_!@#$%^&_()_+{[_%5Cäè| __;'_,_._-中文 =image.jpg" (lineno: 1)
    run_test.go:247: cp s3://test-run-special-characters-in-prefixspecial-chars_!@#$%^&_()_+{[_%5Cäè| __;'_,_._-中文 =/_!@#$%^&_()_+{[_%5Cäè| __;'_,_._-中文 =image.jpg

That'd be very helpful if you could provide a concrete use-case to demonstrate the problem. We can debug the issue as well.

[igungor]

That'd be very helpful if you could provide a concrete use-case to demonstrate the problem. We can debug the issue as well.

Ping @brendan-matroid

[brendan-matroid]

Hey @igungor, sorry for the delay.
There are two separate use cases addressed by this PR:

1. Some AWS S3 prefixes may contain spaces. E.g. s3://some-bucket/some prefix/some file is a valid S3 path. Splitting commands by whitespace in s5cmd run does not permit copying to/from such files.

2. Escaping regex metacharacters for the prefix portion of the wildcard is necessary in cases where the path contains regex metacharacters before the wildcard. This is also problematic in cases where there is no wildcard (since the entire path is treated as the prefix).

At objurl.go:220: filterRegex = o.Prefix + filterRegex which is then compiled in the next line.

If o.Prefix contains unescaped regex metacharacters then they will be compiled (incorrectly), which may either result in some unexpected, valid regex (and silently be used in addition to any regexes resulting from wildcard expressions), or (in the case of the s3 prefix used in the added testcase), it will result in an invalid regex and compilation of said regex will throw an error.

[igungor]

There are two separate use cases addressed by this PR:

1. Some AWS S3 prefixes may contain spaces. E.g. `s3://some-bucket/some prefix/some file` is a valid S3 path. Splitting commands by whitespace in `s5cmd run` does not permit copying to/from such files.

I'm OK with this change. I think it is necessary.

2. Escaping regex metacharacters for the `prefix` portion of the wildcard is necessary in cases where the path contains regex metacharacters before the wildcard. This is also problematic in cases where there is no wildcard (since the entire path is treated as the `prefix`).

At objurl.go:220: filterRegex = o.Prefix + filterRegex which is then compiled in the next line.

If o.Prefix contains unescaped regex metacharacters then they will be compiled (incorrectly), which may either result in some unexpected, valid regex (and silently be used in addition to any regexes resulting from wildcard expressions), or (in the case of the s3 prefix used in the added testcase), it will result in an invalid regex and compilation of said regex will throw an error.

I understand the problem now, thanks for the detailed answer.

The problem with this change is that u.Prefix needs to stay as is since it's the canonical prefix information. The change should be, as you said earlier, at objurl.go:220: filterRegex = o.Prefix + filterRegex. It should be filterRegex = regexp.QuoteMeta(o.Prefix) + filterRegex.

Could you try and check if it's working for your use case? If so, and if you're still interested, could you rebase your branch please, so that we can progress.

[sonmezonur]

Thanks for the PR again. Mostly LGTM with minor comments.

[ilkinulas]

will re-open to trigger circle-ci

[igungor]

will re-open to trigger circle-ci

let's try again now.