Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix windows race condition when writing image with duplicate layers #1921

Merged

Conversation

dgannon991
Copy link
Contributor

This is a potential solution for #1920, any feedback would be greatly appreciated.

Signed-off-by: David Gannon <19214156+dgannon991@users.noreply.github.com>
@slonopotamus
Copy link

I'd say a more robust solution is "try to rename, if it fails, but destination exists, just remove the source and pretend you're fine". This would also cover the case when multiple crane processes stomp on the same file.

@slonopotamus
Copy link

I had a very similar issue in git-lfs in the past. Here's the analysis: git-lfs/git-lfs#3880 (comment) and here's the final fix that solved the issue: git-lfs/git-lfs@662a624

@dgannon991
Copy link
Contributor Author

I'll give that a go, thank you for the feedback.

@dgannon991
Copy link
Contributor Author

@slonopotamus I've written this up and given it a test, and I can confirm it works:

	err = os.Rename(w.Name(), renamePath)
	var badInputErr *os.LinkError
	if err != nil && errors.As(err, &badInputErr) && badInputErr.Err == syscall.ERROR_ACCESS_DENIED {
		if _, err := os.Stat(renamePath); err == nil {
			// windows os.Rename() issue #1920
			// the destination already exists, just remove the target and continue
			return os.Remove(w.Name())
		}
	}
	return err

I do have to say though of the two solutions I think I'm leaning towards the first one, as it makes os.Rename behave more like linux (in that it's an atomic transaction.) Would you mind if I left this up for a second opinion? I'm more than happy to swap over to this approach, but I worry it would mask other issues outside of the scope of this ticket (as you mentioned, running two crane processes at the same time, it got me thinking that realistically that /should/ be an error!)

@slonopotamus
Copy link

Sure, I'm just passing by, and simply gave my thoughts.

@dgannon991
Copy link
Contributor Author

@slonopotamus - thank you for taking the time, the thoughts were definitely appreciated, and it was great to have a confirm that it was an issue in other places!

@dgannon991
Copy link
Contributor Author

Hi Guys,
Is there anything more I need to do on this one? Just aware it will auto close at some point!
Cheers,
David

@imjasonh imjasonh enabled auto-merge (squash) May 30, 2024 17:25
@imjasonh imjasonh merged commit 3764db2 into google:main May 30, 2024
18 checks passed
@dgannon991 dgannon991 deleted the bug/1794/windows-rename-access-denied branch May 30, 2024 20:26
andrewpollock referenced this pull request in google/osv-scanner Jun 18, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | Type |
Update |
|---|---|---|---|---|---|---|---|
| [deps.dev/api/v3](https://github.com/google/deps.dev) |
`v3.0.0-20240611045547-af20eef0f1eb` ->
`v3.0.0-20240617015216-b147e04533eb` |
[![age](https://developer.mend.io/api/mc/badges/age/go/deps.dev%2fapi%2fv3/v3.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/deps.dev%2fapi%2fv3/v3.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/deps.dev%2fapi%2fv3/v3.0.0-20240611045547-af20eef0f1eb/v3.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/deps.dev%2fapi%2fv3/v3.0.0-20240611045547-af20eef0f1eb/v3.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
| require | patch |
| [deps.dev/util/maven](https://github.com/google/deps.dev) |
`af20eef` -> `b147e04` |
[![age](https://developer.mend.io/api/mc/badges/age/go/deps.dev%2futil%2fmaven/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/deps.dev%2futil%2fmaven/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/deps.dev%2futil%2fmaven/v0.0.0-20240611045547-af20eef0f1eb/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/deps.dev%2futil%2fmaven/v0.0.0-20240611045547-af20eef0f1eb/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
| require | digest |
| [deps.dev/util/resolve](https://github.com/google/deps.dev) |
`af20eef` -> `b147e04` |
[![age](https://developer.mend.io/api/mc/badges/age/go/deps.dev%2futil%2fresolve/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/deps.dev%2futil%2fresolve/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/deps.dev%2futil%2fresolve/v0.0.0-20240611045547-af20eef0f1eb/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/deps.dev%2futil%2fresolve/v0.0.0-20240611045547-af20eef0f1eb/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
| require | digest |
| [deps.dev/util/semver](https://github.com/google/deps.dev) |
`af20eef` -> `b147e04` |
[![age](https://developer.mend.io/api/mc/badges/age/go/deps.dev%2futil%2fsemver/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/deps.dev%2futil%2fsemver/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/deps.dev%2futil%2fsemver/v0.0.0-20240611045547-af20eef0f1eb/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/deps.dev%2futil%2fsemver/v0.0.0-20240611045547-af20eef0f1eb/v0.0.0-20240617015216-b147e04533eb?slim=true)](https://docs.renovatebot.com/merge-confidence/)
| require | digest |
|
[github.com/google/go-containerregistry](https://github.com/google/go-containerregistry)
| `v0.19.1` -> `v0.19.2` |
[![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fgoogle%2fgo-containerregistry/v0.19.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fgoogle%2fgo-containerregistry/v0.19.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fgoogle%2fgo-containerregistry/v0.19.1/v0.19.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fgoogle%2fgo-containerregistry/v0.19.1/v0.19.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
| require | patch |
| golang.org/x/exp | `fc45aab` -> `7f521ea` |
[![age](https://developer.mend.io/api/mc/badges/age/go/golang.org%2fx%2fexp/v0.0.0-20240613232115-7f521ea00fb8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/golang.org%2fx%2fexp/v0.0.0-20240613232115-7f521ea00fb8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/golang.org%2fx%2fexp/v0.0.0-20240604190554-fc45aab8b7f8/v0.0.0-20240613232115-7f521ea00fb8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/golang.org%2fx%2fexp/v0.0.0-20240604190554-fc45aab8b7f8/v0.0.0-20240613232115-7f521ea00fb8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
| require | digest |
|
[google.golang.org/protobuf](https://github.com/protocolbuffers/protobuf-go)
| `v1.34.1` -> `v1.34.2` |
[![age](https://developer.mend.io/api/mc/badges/age/go/google.golang.org%2fprotobuf/v1.34.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/google.golang.org%2fprotobuf/v1.34.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/google.golang.org%2fprotobuf/v1.34.1/v1.34.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/google.golang.org%2fprotobuf/v1.34.1/v1.34.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
| require | patch |

---

### Release Notes

<details>
<summary>google/go-containerregistry
(github.com/google/go-containerregistry)</summary>

###
[`v0.19.2`](https://github.com/google/go-containerregistry/releases/tag/v0.19.2)

[Compare
Source](https://github.com/google/go-containerregistry/compare/v0.19.1...v0.19.2)

#### What's Changed

- Add JSON marshalling funcs for Digest. by
[@&#8203;wlynch](https://github.com/wlynch) in
[https://github.com/google/go-containerregistry/pull/1915](https://github.com/google/go-containerregistry/pull/1915)
- registry: Implement Range requests for blobs by
[@&#8203;jonjohnsonjr](https://github.com/jonjohnsonjr) in
[https://github.com/google/go-containerregistry/pull/1917](https://github.com/google/go-containerregistry/pull/1917)
- Support podman auth file REGISTRY_AUTH_FILE. by
[@&#8203;zhaoyonghe](https://github.com/zhaoyonghe) in
[https://github.com/google/go-containerregistry/pull/1914](https://github.com/google/go-containerregistry/pull/1914)
- feat: crane mutate platform by
[@&#8203;joshwlewis](https://github.com/joshwlewis) in
[https://github.com/google/go-containerregistry/pull/1919](https://github.com/google/go-containerregistry/pull/1919)
- Add Context support to auth methods by
[@&#8203;jonjohnsonjr](https://github.com/jonjohnsonjr) in
[https://github.com/google/go-containerregistry/pull/1949](https://github.com/google/go-containerregistry/pull/1949)
- Fix windows race condition when writing image with duplicate layers by
[@&#8203;dgannon991](https://github.com/dgannon991) in
[https://github.com/google/go-containerregistry/pull/1921](https://github.com/google/go-containerregistry/pull/1921)
- Add -O shorthand for --omit-digest-tags to crane. by
[@&#8203;smoser](https://github.com/smoser) in
[https://github.com/google/go-containerregistry/pull/1958](https://github.com/google/go-containerregistry/pull/1958)

#### New Contributors

- [@&#8203;wlynch](https://github.com/wlynch) made their first
contribution in
[https://github.com/google/go-containerregistry/pull/1915](https://github.com/google/go-containerregistry/pull/1915)
- [@&#8203;zhaoyonghe](https://github.com/zhaoyonghe) made their first
contribution in
[https://github.com/google/go-containerregistry/pull/1914](https://github.com/google/go-containerregistry/pull/1914)
- [@&#8203;joshwlewis](https://github.com/joshwlewis) made their first
contribution in
[https://github.com/google/go-containerregistry/pull/1919](https://github.com/google/go-containerregistry/pull/1919)
- [@&#8203;dgannon991](https://github.com/dgannon991) made their first
contribution in
[https://github.com/google/go-containerregistry/pull/1921](https://github.com/google/go-containerregistry/pull/1921)
- [@&#8203;smoser](https://github.com/smoser) made their first
contribution in
[https://github.com/google/go-containerregistry/pull/1958](https://github.com/google/go-containerregistry/pull/1958)

**Full Changelog**:
google/go-containerregistry@v0.19.1...v0.19.2

</details>

<details>
<summary>protocolbuffers/protobuf-go
(google.golang.org/protobuf)</summary>

###
[`v1.34.2`](https://github.com/protocolbuffers/protobuf-go/releases/tag/v1.34.2)

[Compare
Source](https://github.com/protocolbuffers/protobuf-go/compare/v1.34.1...v1.34.2)

Minor feature:

- [CL/589336](https://go.dev/cl/589336): gofeatures: allow setting
legacy_unmarshal_json_enum feature at file level

Minor bug fixes:

- [CL/588875](https://go.dev/cl/588875): types/descriptorpb: regenerate
using latest protobuf v27.0 release
- [CL/586396](https://go.dev/cl/586396): internal/impl: fix size cache
semantics with lazy decoding
- [CL/585736](https://go.dev/cl/585736): reflect/protodesc: remove
obsolete JSON name check from desc validator
- [CL/588976](https://go.dev/cl/588976): reflect/protoreflect:
FieldDescriptor.Kind should never be GroupKind for maps or fields of map
entry

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 6am on monday" in timezone
Australia/Sydney, Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config help](https://github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/google/osv-scanner).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5My4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants