Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to custom handle redirects #5678

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions api/internal/loader/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,12 @@ var (
ErrHTTP = errors.Errorf("HTTP Error")
ErrRtNotDir = errors.Errorf("must build at directory")
)

type RedirectionError struct {
NewPath string
Msg string
}

func (e *RedirectionError) Error() string {
return e.Msg
}
19 changes: 15 additions & 4 deletions api/internal/loader/fileloader.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@
cleaner func() error
}

// This redirect code does not process automaticali by http client and we can process it manualy

Check failure on line 109 in api/internal/loader/fileloader.go

View workflow job for this annotation

GitHub Actions / Lint

`manualy` is a misspelling of `manually` (misspell)
const MULTIPLE_CHOICES_REDIRECT_CODE = 300

Check failure on line 110 in api/internal/loader/fileloader.go

View workflow job for this annotation

GitHub Actions / Lint

var-naming: don't use ALL_CAPS in Go names; use CamelCase (revive)

// Repo returns the absolute path to the repo that contains Root if this fileLoader was created from a url
// or the empty string otherwise.
func (fl *FileLoader) Repo() string {
Expand Down Expand Up @@ -318,11 +321,19 @@
defer resp.Body.Close()
// response unsuccessful
if resp.StatusCode < 200 || resp.StatusCode > 299 {
_, err = git.NewRepoSpecFromURL(path)
if err == nil {
return nil, errors.Errorf("URL is a git repository")
if resp.StatusCode == MULTIPLE_CHOICES_REDIRECT_CODE {
var newPath string = resp.Header.Get("Location")
return nil, &RedirectionError{
Msg: "Response is redirect",
NewPath: newPath,
}
Comment on lines +325 to +329
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe That is better to solve redirects here than return RedirectionError and a new path.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! It does not work in all cases. For example if I return redirect to git. In that case httpClient has noe credentials for download content

} else {
_, err = git.NewRepoSpecFromURL(path)
if err == nil {
return nil, errors.Errorf("URL is a git repository")
}
return nil, fmt.Errorf("%w: status code %d (%s)", ErrHTTP, resp.StatusCode, http.StatusText(resp.StatusCode))
}
return nil, fmt.Errorf("%w: status code %d (%s)", ErrHTTP, resp.StatusCode, http.StatusText(resp.StatusCode))
}
content, err := io.ReadAll(resp.Body)
return content, errors.Wrap(err)
Expand Down
30 changes: 30 additions & 0 deletions api/internal/loader/fileloader_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
"sigs.k8s.io/kustomize/api/ifc"
"sigs.k8s.io/kustomize/api/internal/git"
"sigs.k8s.io/kustomize/api/konfig"
"sigs.k8s.io/kustomize/kyaml/errors"
"sigs.k8s.io/kustomize/kyaml/filesys"
)

Expand Down Expand Up @@ -659,6 +660,35 @@
_, err := l2.Load(x.path)
require.Error(err)
}

var testCaseRedirect = []testData{
{
path: "https://example.com/resource.yaml",
expectedContent: "https content",
},
Comment on lines +664 to +668
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no plans to add another tests.

}
for _, x := range testCaseRedirect {
expectedLocation := "https://redirect.com/resource.yaml"
Copy link
Member

@koba1t koba1t Jun 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like redirect.com is owned by a company.
I understand this fakeHttpClient won't access the global internet, but I prefer to use a reserved example domain instead of something that someone else owns.


   Domain Name: REDIRECT.COM
   Registry Domain ID: 32687398_DOMAIN_COM-VRSN
   Registrar WHOIS Server: Whois.bigrock.com
   Registrar URL: http://www.bigrock.com
   Updated Date: 2023-06-20T20:24:31Z
   Creation Date: 2000-08-10T10:33:16Z
   Registry Expiry Date: 2024-08-10T10:33:15Z
   Registrar: BigRock Solutions Ltd
   Registrar IANA ID: 1495
   Registrar Abuse Contact Email: abuse@bigrock.com
   Registrar Abuse Contact Phone: +1.832-295-1535
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Name Server: NS-1118.AWSDNS-11.ORG
   Name Server: NS-1717.AWSDNS-22.CO.UK
   Name Server: NS-320.AWSDNS-40.COM
   Name Server: NS-918.AWSDNS-50.NET
   DNSSEC: unsigned
   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2024-06-30T18:22:52Z <<<

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ок. I'll fix it!

hc := makeFakeHTTPClient(func(req *http.Request) *http.Response {
response := &http.Response{
StatusCode: 300,

Check failure on line 674 in api/internal/loader/fileloader_test.go

View workflow job for this annotation

GitHub Actions / Lint

"300" can be replaced by http.StatusMultipleChoices (usestdlibvars)
Body: io.NopCloser(bytes.NewBufferString("")),
Header: make(http.Header),
}
response.Header.Add("Location", expectedLocation)
return response
})
l2 := l1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does l1 and l2 mean?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is loaders. I use solution from testcase from

l2.http = hc
_, err := l2.Load(x.path)
require.Error(err)
var redErr *RedirectionError
var path string = ""
if errors.As(err, &redErr) {
path = redErr.NewPath
}
require.Equal(expectedLocation, path)
}
}

// setupOnDisk sets up a file system on disk and directory that is cleaned after
Expand Down
5 changes: 5 additions & 0 deletions api/internal/localizer/localizer.go
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,11 @@ func (lc *localizer) localizeResource(path string) (string, error) {
}
}
if fileErr != nil {
var redErr *loader.RedirectionError
if errors.As(fileErr, &redErr) {
path = redErr.NewPath
}

var rootErr error
locPath, rootErr = lc.localizeRoot(path)
if rootErr != nil {
Expand Down
5 changes: 5 additions & 0 deletions api/internal/target/kusttarget.go
Original file line number Diff line number Diff line change
Expand Up @@ -423,7 +423,12 @@ func (kt *KustTarget) accumulateResources(
if errors.Is(errF, load.ErrHTTP) {
return nil, errF
}
var redErr *load.RedirectionError
if errors.As(errF, &redErr) {
path = redErr.NewPath
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code is using redirect Location path directly.
I am worried that newPath will be used by no check.
like the redirected path requires redirecting more, or the redirected path is missing contents.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I investigated in these scenarious

  1. new loader works only with directory and git repo because of that all checks will be done in common way in
    func (fl *FileLoader) New(path string) (ifc.Loader, error) {
  2. If newPath will be a simple url in new loader it will be treated as directory and kustomize finish work with error in common way
  3. If some redirects (301 to 309 http codes) will happen in loading process
    resp, err := hc.Get(path)
    They will be processed by http client out of the box
    Following this points I think that all possible checks have been done

}
ldr, err := kt.ldr.New(path)

if err != nil {
// If accumulateFile found malformed YAML and there was a failure
// loading the resource as a base, then the resource is likely a
Expand Down
Loading