Can't run any podman command anymore - panic: runtime error: invalid memory address or nil pointer dereference #2184
Assignees
Labels
Comments
Sounds like your storage is corrupted, so the easiest thing would be to remove the container storage and start from an empty storage again and recreate your containers. Of course it would be good to understand what happend there and certainly it should never panic. Are you running rootless? If so what is the output of |
|
Oh I see the issue in the c/storage code. While iterating r.layers r.deleteInternal() while remove the element from r.layers as such the slice is modified while we iterate over it which is not safe as we access the last element which no longer exists and we skip one layer. Lines 1928 to 1930 in 3021f6a |
|
Thank you very much for your time looking into this issue, I really appreciated it. This is the requested output (rather long, so I posted there not pollute this thread too much) What I'm supposed to do exactly to get podman working again? |
|
If you don't need need the data delete Otherwise you need to wait for a patch from us to make it work again I think, or try to manually delete the incomplete layer ID from the json file but I don't know if this causes other bad things so just deleting all data seems safest to avoid any storage corruption. |
|
good analysis! If you've a patch, please tag me and I'll review it |
Luap99
added a commit
to Luap99/storage
that referenced
this issue
Dec 3, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
Thanks everyone involved in this. I tried inspecting the just a I'm unsure now if I want to wait for the patch or delete what needs to be deleted :) |
Luap99
added a commit
to Luap99/storage
that referenced
this issue
Dec 3, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99
added a commit
to Luap99/storage
that referenced
this issue
Dec 3, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
|
@Luap99 I see that the proposed MR is already merged which is great and it was super fast! |
|
Yes you need to wait for the next version. |
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 4, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99
added a commit
to Luap99/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
(cherry picked from commit 99b0d2d)
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 9, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 10, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 10, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 10, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 10, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
kwilczynski
pushed a commit
to kwilczynski/storage
that referenced
this issue
Dec 11, 2024
In go one should never modify a slice while also iterating over it at
the same time. This causes weird side effects as the underlying array
elements are shifted around without the range loop index knowing.
So if you delete a element the loop will then actually skip the next one
and theoretically access out of bounds on the last element which does
not panic but rather return the default zero type, nil here which then
causes the panic on layer.Flags == nil.
Here is a simple example to show the behavior:
func main() {
slice := []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
for _, num := range slice {
if num == 5 {
slice = slices.DeleteFunc(slice, func(n int) bool {
return n == 5
})
}
fmt.Println(num)
}
}
The loop will not print 6, but then as last number it prints 0 (the
default zero type for an int).
Fixes containers#2184
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Discussed in containers/podman#24736
Originally posted by dozenposture December 2, 2024
I've used to have about 20 containers running without issues.
While trying to update an image, I've found podman can't execute any command anymore.
It really doesn't matter which command I try to run, I always get the following error message:
I've browsed for similar issues and I've found something about lingering. The podman user has lingering enabled (since the beginning)
Can someone please help me out understanding what went wrong and how to fix this?
Thanks!
The text was updated successfully, but these errors were encountered: