Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ecs-agent crashes randomly with slice index out of range #570

Closed
jamesongithub opened this issue Oct 27, 2016 · 7 comments
Closed

ecs-agent crashes randomly with slice index out of range #570

jamesongithub opened this issue Oct 27, 2016 · 7 comments
Labels
kind/bug
Milestone
1.14.1

Comments

@jamesongithub
Copy link

jamesongithub commented Oct 27, 2016 •
edited
Loading

happens intermittently

ecs-agent

image

            "Env": [
                "ECS_DISABLE_PRIVILEGED=true",
                "ECS_IMAGE_CLEANUP_INTERVAL=30m",
                "ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=1m",
                "ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true",
                "ECS_APPARMOR_CAPABLE=true",
                "ECS_DOCKER_GRAPHPATH=${path}",
                "ECS_DISABLE_IMAGE_CLEANUP=False",
                "ECS_IMAGE_MINIMUM_CLEANUP_AGE=30m",
                "ECS_CLUSTER=${cluster}",
                "ECS_DISABLE_METRICS=false",
                "ECS_AVAILABLE_LOGGING_DRIVERS=[\"awslogs\", \"fluentd\", \"json-file\", \"syslog\"]",
                "ECS_ENABLE_TASK_IAM_ROLE=true",
                "ECS_DATADIR=/data",
                "ECS_LOGFILE=/log/ecs-agent.log",
                "ECS_LOG_LEVEL=debug",
                "ECS_NUM_IMAGES_DELETE_PER_CYCLE=5",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],

docker info

Containers: 18
 Running: 8
 Paused: 0
 Stopped: 10
Images: 22
Server Version: 1.12.2
Storage Driver: devicemapper
 Pool Name: ${poolName}
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: 
 Metadata file: 
 Data Space Used: 13.45 GB
 Data Space Total: 219.9 GB
 Data Space Available: 206.5 GB
 Metadata Space Used: 15.71 MB
 Metadata Space Total: 2.751 GB
 Metadata Space Available: 2.736 GB
 Thin Pool Minimum Free Space: 21.99 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-96-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 60 GiB
Name: ${name}
ID: ${ID}
Docker Root Dir: {$rootDir}
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 50
 Goroutines: 72
 System Time: 2016-10-27T17:55:11.802302881Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

logs

panic: reflect: slice index out of range [recovered]
        panic: reflect: slice index out of range [recovered]
        panic: reflect: slice index out of range

goroutine 935146 [running]:
panic(0x8e12c0, 0xc820236da0)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
encoding/json.(*encodeState).marshal.func1(0xc8204cdad8)
        /usr/local/go/src/encoding/json/encode.go:269 +0x11f
panic(0x8e12c0, 0xc820236d90)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
encoding/json.(*encodeState).marshal.func1(0xc8204cd380)
        /usr/local/go/src/encoding/json/encode.go:269 +0x11f
panic(0x8e12c0, 0xc820236d80)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
reflect.Value.Index(0x8c7a40, 0xc820965658, 0x197, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:854 +0x151
encoding/json.(*arrayEncoder).encode(0xc8201721c0, 0xc8201f6420, 0x8c7a40, 0xc820965658, 0x197, 0xf00)
        /usr/local/go/src/encoding/json/encode.go:690 +0xd2
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0x8c7a40, 0xc820965658, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:697 +0x51
encoding/json.(*sliceEncoder).encode(0xc8201721c8, 0xc8201f6420, 0x8c7a40, 0xc820965658, 0x197, 0xf00)
        /usr/local/go/src/encoding/json/encode.go:667 +0xb0
encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0x8c7a40, 0xc820965658, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:676 +0x51
encoding/json.(*structEncoder).encode(0xc8203e4e70, 0xc8201f6420, 0xa92620, 0xc820965500, 0x199, 0x750f00)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0xa92620, 0xc820965500, 0x199, 0xc820965500)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*ptrEncoder).encode(0xc820172310, 0xc8201f6420, 0xa665c0, 0xc820a265e8, 0x196, 0x900)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0xa665c0, 0xc820a265e8, 0x196, 0xa66500)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*arrayEncoder).encode(0xc820172318, 0xc8201f6420, 0x8c3f60, 0xc8205b9c50, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:690 +0x11b
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0x8c3f60, 0xc8205b9c50, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:697 +0x51
encoding/json.(*sliceEncoder).encode(0xc820172320, 0xc8201f6420, 0x8c3f60, 0xc8205b9c50, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:667 +0xb0
encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0x8c3f60, 0xc8205b9c50, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:676 +0x51
encoding/json.(*structEncoder).encode(0xc8203e4d80, 0xc8201f6420, 0xa87860, 0xc8205b9c20, 0x199, 0x0)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0xa87860, 0xc8205b9c20, 0x199, 0xc8205b9c00)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*ptrEncoder).encode(0xc820172388, 0xc8201f6420, 0xaa5ea0, 0xc8209b8d20, 0x196, 0x0)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0xaa5ea0, 0xc8209b8d20, 0x196, 0xaa5e00)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*arrayEncoder).encode(0xc820172390, 0xc8201f6420, 0x8c4020, 0xc820840100, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:690 +0x11b
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0x8c4020, 0xc820840100, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:697 +0x51
encoding/json.(*sliceEncoder).encode(0xc820172398, 0xc8201f6420, 0x8c4020, 0xc820840100, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:667 +0xb0
encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0x8c4020, 0xc820840100, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:676 +0x51
encoding/json.(*structEncoder).encode(0xc8203e4c60, 0xc8201f6420, 0xa1f6e0, 0xc820840100, 0x99, 0x0)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8201f6420, 0xa1f6e0, 0xc820840100, 0x99, 0xc820840100)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*encodeState).reflectValue(0xc8201f6420, 0xa1f6e0, 0xc820840100, 0x99)
        /usr/local/go/src/encoding/json/encode.go:301 +0x6b
encoding/json.(*encodeState).marshal(0xc8201f6420, 0xa1f6e0, 0xc820840100, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:274 +0xa9
encoding/json.Marshal(0xa1f6e0, 0xc820840100, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:139 +0x84
github.com/aws/amazon-ecs-agent/agent/engine/dockerstate.(*DockerTaskEngineState).MarshalJSON(0xc8200b8340, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/dockerstate/json.go:43 +0x20d
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).MarshalJSON(0xc82017e1c0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:128 +0x42
encoding/json.marshalerEncoder(0xc8201f6160, 0xaa65a0, 0xc82017e1c0, 0x16, 0x0)
        /usr/local/go/src/encoding/json/encode.go:418 +0x159
encoding/json.(*encodeState).reflectValue(0xc8201f6160, 0xaa65a0, 0xc82017e1c0, 0x16)
        /usr/local/go/src/encoding/json/encode.go:301 +0x6b
encoding/json.interfaceEncoder(0xc8201f6160, 0x8f03c0, 0xc82014c430, 0x194, 0xc82014c400)
        /usr/local/go/src/encoding/json/encode.go:560 +0xd8
encoding/json.(*ptrEncoder).encode(0xc820172038, 0xc8201f6160, 0x8b0c40, 0xc82014c430, 0x16, 0x0)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc8201f6160, 0x8b0c40, 0xc82014c430, 0x16, 0xc820236800)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*mapEncoder).encode(0xc820172040, 0xc8201f6160, 0x988f40, 0xc82014c490, 0x195, 0x0)
        /usr/local/go/src/encoding/json/encode.go:622 +0x344
encoding/json.(*mapEncoder).(encoding/json.encode)-fm(0xc8201f6160, 0x988f40, 0xc82014c490, 0x195, 0x0)
        /usr/local/go/src/encoding/json/encode.go:632 +0x51
encoding/json.(*structEncoder).encode(0xc8203e4b70, 0xc8201f6160, 0x9f21c0, 0xc82014c490, 0x199, 0xc800000000)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8201f6160, 0x9f21c0, 0xc82014c490, 0x199, 0xc82014c400)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*ptrEncoder).encode(0xc820172050, 0xc8201f6160, 0x8b0e80, 0xc82014c490, 0x16, 0x0)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc8201f6160, 0x8b0e80, 0xc82014c490, 0x16, 0xc82014c400)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*encodeState).reflectValue(0xc8201f6160, 0x8b0e80, 0xc82014c490, 0x16)
        /usr/local/go/src/encoding/json/encode.go:301 +0x6b
encoding/json.(*encodeState).marshal(0xc8201f6160, 0x8b0e80, 0xc82014c490, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:274 +0xa9
encoding/json.Marshal(0x8b0e80, 0xc82014c490, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:139 +0x84
github.com/aws/amazon-ecs-agent/agent/statemanager.(*basicStateManager).ForceSave(0xc820065e00, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/statemanager/state_manager.go:198 +0x103
github.com/aws/amazon-ecs-agent/agent/statemanager.(*basicStateManager).Save(0xc820065e00, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/statemanager/state_manager.go:166 +0xef
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).applyContainerState(0xc82017e1c0, 0xc8205b9c20, 0xc820965500, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:649 +0x909
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).transitionContainer(0xc82017e1c0, 0xc8205b9c20, 0xc820965500, 0x4)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:657 +0x58
github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).progressContainers.func1(0xc820896f50, 0xc820966cb0, 0xc82073c780, 0xc820965500, 0x4)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:399 +0x43
created by github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).progressContainers
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:402 +0x11c2
@jamesongithub jamesongithub changed the title ecs-agent crashed randomly with slice index out of range ecs-agent crashes randomly with slice index out of range Oct 27, 2016
@jamesongithub
Copy link
Author

think this is different than #501

@richardpen
Copy link

@jamesongithub Thanks for your reporting, we have found one bug in agent code that could cause the error "index out of range". This could happen when there are tasks running/updated with the same image name but different image id. Are your services running into this scenario?

@jamesongithub
Copy link
Author

It might, although we haven't run tasks manually for awhile. They are generally spawned by the service.

@jamesongithub
Copy link
Author

another if it helps

panic: reflect: slice index out of range [recovered]
        panic: reflect: slice index out of range [recovered]
        panic: reflect: slice index out of range

goroutine 2724902 [running]:
panic(0x8e12c0, 0xc8206d7c70)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
encoding/json.(*encodeState).marshal.func1(0xc8208bbac8)
        /usr/local/go/src/encoding/json/encode.go:269 +0x11f
panic(0x8e12c0, 0xc8206d7c60)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
encoding/json.(*encodeState).marshal.func1(0xc8208bb370)
        /usr/local/go/src/encoding/json/encode.go:269 +0x11f
panic(0x8e12c0, 0xc8206d7c50)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
reflect.Value.Index(0x8c7a40, 0xc8208577d8, 0x197, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:854 +0x151
encoding/json.(*arrayEncoder).encode(0xc82014a1e8, 0xc82048f1e0, 0x8c7a40, 0xc8208577d8, 0x197, 0x6900)
        /usr/local/go/src/encoding/json/encode.go:690 +0xd2
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0x8c7a40, 0xc8208577d8, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:697 +0x51
encoding/json.(*sliceEncoder).encode(0xc82014a1f0, 0xc82048f1e0, 0x8c7a40, 0xc8208577d8, 0x197, 0x6900)
        /usr/local/go/src/encoding/json/encode.go:667 +0xb0
encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0x8c7a40, 0xc8208577d8, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:676 +0x51
encoding/json.(*structEncoder).encode(0xc820451920, 0xc82048f1e0, 0xa92620, 0xc820857680, 0x199, 0x750f00)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0xa92620, 0xc820857680, 0x199, 0xc820857600)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*ptrEncoder).encode(0xc82014a338, 0xc82048f1e0, 0xa665c0, 0xc820fb4a50, 0x196, 0x6300)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0xa665c0, 0xc820fb4a50, 0x196, 0xa66500)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*arrayEncoder).encode(0xc82014a340, 0xc82048f1e0, 0x8c3f60, 0xc820cd84b0, 0x197, 0x5100)
        /usr/local/go/src/encoding/json/encode.go:690 +0x11b
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0x8c3f60, 0xc820cd84b0, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:697 +0x51
encoding/json.(*sliceEncoder).encode(0xc82014a348, 0xc82048f1e0, 0x8c3f60, 0xc820cd84b0, 0x197, 0x5100)
        /usr/local/go/src/encoding/json/encode.go:667 +0xb0
encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0x8c3f60, 0xc820cd84b0, 0x197, 0x0)
        /usr/local/go/src/encoding/json/encode.go:676 +0x51
encoding/json.(*structEncoder).encode(0xc8204518f0, 0xc82048f1e0, 0xa87860, 0xc820cd8480, 0x199, 0x0)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0xa87860, 0xc820cd8480, 0x199, 0xc820cd8400)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*ptrEncoder).encode(0xc82014a3b0, 0xc82048f1e0, 0xaa5ea0, 0xc820f38ca8, 0x196, 0x5000)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0xaa5ea0, 0xc820f38ca8, 0x196, 0xaa5e00)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*arrayEncoder).encode(0xc82014a3b8, 0xc82048f1e0, 0x8c4020, 0xc820829640, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:690 +0x11b
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0x8c4020, 0xc820829640, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:697 +0x51
encoding/json.(*sliceEncoder).encode(0xc82014a3c0, 0xc82048f1e0, 0x8c4020, 0xc820829640, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:667 +0xb0
encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0x8c4020, 0xc820829640, 0x97, 0x0)
        /usr/local/go/src/encoding/json/encode.go:676 +0x51
encoding/json.(*structEncoder).encode(0xc8204518c0, 0xc82048f1e0, 0xa1f6e0, 0xc820829640, 0x99, 0x0)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc82048f1e0, 0xa1f6e0, 0xc820829640, 0x99, 0xc820829600)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*encodeState).reflectValue(0xc82048f1e0, 0xa1f6e0, 0xc820829640, 0x99)
        /usr/local/go/src/encoding/json/encode.go:301 +0x6b
encoding/json.(*encodeState).marshal(0xc82048f1e0, 0xa1f6e0, 0xc820829640, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:274 +0xa9
encoding/json.Marshal(0xa1f6e0, 0xc820829640, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:139 +0x84
github.com/aws/amazon-ecs-agent/agent/engine/dockerstate.(*DockerTaskEngineState).MarshalJSON(0xc8201a0240, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/dockerstate/json.go:43 +0x20d
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).MarshalJSON(0xc8201c2000, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:128 +0x42
encoding/json.marshalerEncoder(0xc82048f080, 0xaa65a0, 0xc8201c2000, 0x16, 0x0)
        /usr/local/go/src/encoding/json/encode.go:418 +0x159
encoding/json.(*encodeState).reflectValue(0xc82048f080, 0xaa65a0, 0xc8201c2000, 0x16)
        /usr/local/go/src/encoding/json/encode.go:301 +0x6b
encoding/json.interfaceEncoder(0xc82048f080, 0x8f03c0, 0xc820109a30, 0x194, 0xc820109a00)
        /usr/local/go/src/encoding/json/encode.go:560 +0xd8
encoding/json.(*ptrEncoder).encode(0xc82014a058, 0xc82048f080, 0x8b0c40, 0xc820109a30, 0x16, 0x0)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc82048f080, 0x8b0c40, 0xc820109a30, 0x16, 0xc820143600)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*mapEncoder).encode(0xc82014a060, 0xc82048f080, 0x988f40, 0xc820109a90, 0x195, 0x0)
        /usr/local/go/src/encoding/json/encode.go:622 +0x344
encoding/json.(*mapEncoder).(encoding/json.encode)-fm(0xc82048f080, 0x988f40, 0xc820109a90, 0x195, 0x0)
        /usr/local/go/src/encoding/json/encode.go:632 +0x51
encoding/json.(*structEncoder).encode(0xc820451860, 0xc82048f080, 0x9f21c0, 0xc820109a90, 0x199, 0xc800000000)
        /usr/local/go/src/encoding/json/encode.go:587 +0x2c4
encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc82048f080, 0x9f21c0, 0xc820109a90, 0x199, 0xc820109a00)
        /usr/local/go/src/encoding/json/encode.go:601 +0x51
encoding/json.(*ptrEncoder).encode(0xc82014a070, 0xc82048f080, 0x8b0e80, 0xc820109a90, 0x16, 0x0)
        /usr/local/go/src/encoding/json/encode.go:709 +0xea
encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc82048f080, 0x8b0e80, 0xc820109a90, 0x16, 0xc820109a00)
        /usr/local/go/src/encoding/json/encode.go:714 +0x51
encoding/json.(*encodeState).reflectValue(0xc82048f080, 0x8b0e80, 0xc820109a90, 0x16)
        /usr/local/go/src/encoding/json/encode.go:301 +0x6b
encoding/json.(*encodeState).marshal(0xc82048f080, 0x8b0e80, 0xc820109a90, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:274 +0xa9
encoding/json.Marshal(0x8b0e80, 0xc820109a90, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:139 +0x84
github.com/aws/amazon-ecs-agent/agent/statemanager.(*basicStateManager).ForceSave(0xc820065860, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/statemanager/state_manager.go:198 +0x103
github.com/aws/amazon-ecs-agent/agent/statemanager.(*basicStateManager).Save(0xc820065860, 0x0, 0x0)
        /go/src/github.com/aws/amazon-ecs-agent/agent/statemanager/state_manager.go:166 +0xef
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).applyContainerState(0xc8201c2000, 0xc820cd8480, 0xc820857680, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:649 +0x909
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).transitionContainer(0xc8201c2000, 0xc820cd8480, 0xc820857680, 0x4)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:657 +0x58
github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).progressContainers.func1(0xc82061ac30, 0xc820111f80, 0xc8208d5260, 0xc820857680, 0x4)
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:399 +0x43
created by github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).progressContainers
        /go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:402 +0x11c2

@richardpen
Copy link

@jamesongithub thanks for your reporting, we have found one bug in ecs agent that may cause this problem, and we are working on a fix. Will let you know when we have an update.

thanks,
Peng

@richardpen richardpen mentioned this issue Nov 7, 2016
Closed
@jamesongithub
Copy link
Author 

panic: reflect: slice index out of range [recovered]
        panic: reflect: slice index out of range [recovered]
        panic: reflect: slice index out of range

goroutine 1156493 [running]:
panic(0x865ca0, 0xc4209149b0)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
encoding/json.(*encodeState).marshal.func1(0xc420b27d28)
        /usr/local/go/src/encoding/json/encode.go:275 +0x19f
panic(0x865ca0, 0xc4209149a0)
        /usr/local/go/src/runtime/panic.go:458 +0x243
encoding/json.(*encodeState).marshal.func1(0xc420b27620)
        /usr/local/go/src/encoding/json/encode.go:275 +0x19f
panic(0x865ca0, 0xc420914990)
        /usr/local/go/src/runtime/panic.go:458 +0x243
reflect.Value.Index(0x85a200, 0xc42074f828, 0x197, 0x1, 0x8d0100, 0xc4200b1500, 0x199)
        /usr/local/go/src/reflect/value.go:853 +0x1aa
encoding/json.(*arrayEncoder).encode(0xc420196238, 0xc420264e70, 0x85a200, 0xc42074f828, 0x197, 0x100)
        /usr/local/go/src/encoding/json/encode.go:723 +0xa2
encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc420264e70, 0x85a200, 0xc42074f828, 0x197, 0x820100)

@richardpen richardpen mentioned this issue Feb 1, 2017
Merged
8 tasks
@samuelkarp samuelkarp added this to the 1.14.1 milestone Feb 11, 2017
@samuelkarp
Copy link
Contributor

Fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
1.14.1
Development

No branches or pull requests
3 participants
@samuelkarp @jamesongithub @richardpen