Executor is killed after client cannot fetch stats #1719

kaskavalci · 2016-09-20T12:07:44Z

Nomad version

v0.4.0

Operating system and Environment details

Linux (Not tested on other platforms)

Issue

Nomad client sends SIGKILL to executor and its spawned process when driver cannot fetch stats for the task. We have a cluster in which we experienced some slow network communications. We assume ~~problem may arise from executor_linux.go which is the only place Stats returns an error. (pidStats does not return an error effectively)~~ We have disabled that part and problem still occurs. ~~Currently, not returning an error from Stats#L281 seems working. We'll test further.~~

Reproduction steps

We don't have an exact reproduction steps since it only happens in ~~one cluster~~ docker, AWS and custom clusters, after some time.

Nomad Client logs

Sep 16 14:18:50 node nomad[25981]: client: error fetching stats of task mytask: unexpected EOF
Sep 16 14:18:50 node nomad[25981]: driver.raw_exec: failed to deregister services: connection is shut down
Sep 16 14:18:50 node nomad[25981]: driver.raw_exec: error killing executor: connection is shut down
Sep 16 14:18:50 node nomad[25981]: client: task "mytask" for alloc "7ceb4e5a-3da2-2c61-8e29-8fb5dbf87c4c" failed: Wait returned exit code 0, signal 0, and error unexpected EOF
Sep 16 14:18:50 node nomad[25981]: client: Restarting task "mytask" for alloc "7ceb4e5a-3da2-2c61-8e29-8fb5dbf87c4c" in 1m10.70563676s

The text was updated successfully, but these errors were encountered:

dadgar · 2016-09-20T17:28:15Z

Hey,

I don't think this has anything to do with stats. It looks like the executor is exiting and you are seeing the error first with stats. driver.raw_exec: error killing executor: connection is shut down means that the connection is gone.

Are you guys killing the executor?

tugbabodrumlu · 2016-09-20T19:28:53Z

Hi,
We thought Nomad gets an unexpected EOF error while fetching stats of the process and then sends a SIGKILL to its spawned process. However, could it be the other way around? Do you think is it possible that the spawned process is already dead(?) so we get "failed to deregister services" and "error killing executor"?

kaskavalci · 2016-09-22T19:02:09Z

Our job is wrapped by a python script. Here we see nomad client sends a kill signal to executor and our script since it's child process of executor.

auditd traces:

## Nomad processes:

root     12357     1  0 07:47 ?        00:00:02 /var/bin/nomad executor /var/local/client/alloc/9eeb4fef-194a-1c10-8fdd-3925c9ceda37/mytask/mytask-executor.out
root     27427 13132  4 07:52 pts/5    00:00:05 /var/bin/nomad agent -config /etc/server_config.json
root     28878 17780  0 07:53 pts/8    00:00:00 /var/bin/nomad agent -config /etc/client_config.json


## mytask processes:

root     12357     1  0 07:47 ?        00:00:02 /var/bin/nomad executor /var/local/client/alloc/9eeb4fef-194a-1c10-8fdd-3925c9ceda37/mytask/mytask-executor.out
root     12383 12357  0 07:47 ?        00:00:00 python /var/lib/ambari-agent/cache/stacks/HDP/2.2/service/package/mytask/control.py start_mytask_server
halil    12426 12383  0 07:47 ?        00:00:00 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/mytask/bin/task serve


## kill_signals:

type=SYSCALL msg=audit(1474523908.286:30775): arch=c000003e syscall=62 success=yes exit=0 a0=305f a1=9 a2=0 a3=0 items=0 ppid=17780 pid=28878 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts8 ses=12945 comm="nomad" exe="/var/bin/nomad" key="kill_signals"
type=SYSCALL msg=audit(1474523908.286:30776): arch=c000003e syscall=62 success=yes exit=0 a0=3045 a1=9 a2=0 a3=0 items=0 ppid=17780 pid=28878 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts8 ses=12945 comm="nomad" exe="/var/bin/nomad" key="kill_signals"

process 28878 sends SIGKILL(a1=9) to process 12383(a1=305f hex)
nomad client sends SIGKILL to control.py since PR_SET_PDEATHSIG, signal.SIGKILL and executor is parent of control.py

process 28878 sends SIGKILL(a1=9) to process 12357(a1=3045 hex)
nomad client sends SIGKILL to executor

kaskavalci · 2016-09-23T07:55:24Z

I've patched raw_exec.go as @diptanu suggested:

func (h *rawExecHandle) run() {
    ps, err := h.executor.Wait()
    close(h.doneCh)
    if ps.ExitCode == 0 && err != nil {
        if e := killProcess(h.userPid); e != nil {
            h.logger.Printf("[ERR] driver.raw_exec: error killing user process: %v", e)
        }
        if e := h.allocDir.UnmountAll(); e != nil {
            h.logger.Printf("[ERR] driver.raw_exec: unmounting dev,proc and alloc dirs failed: %v", e)
        }
    }
    h.waitCh <- &dstructs.WaitResult{ExitCode: ps.ExitCode, Signal: ps.Signal, Err: err}
    close(h.waitCh)
    // Remove services
    if err := h.executor.DeregisterServices(); err != nil {
        h.logger.Printf("[ERR] driver.raw_exec: failed to deregister services: %v", err)
    }

    if err := h.executor.Exit(); err != nil {
        h.logger.Printf("[ERR] driver.raw_exec: error killing executor: %v", err)
    }
    if err == nil {
        h.pluginClient.Kill()
    }
}

We expect if nomad executor is unreachable due to unreliable network, it should still be running after this change. In other words, executor did not panic but only was unreachable. Results confirmed that assumption.

After some time when bug occured multiple times.

$ ps -ef | grep nomad
root     11678 30500  0 09:32 ?        00:00:00 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/lib/bin/nomad executor /var/local/client/alloc/0237870e-392b-832b-a67c-f5d073af16cf/mytask/mytask-executor.out
root     15202  1494  0 09:33 pts/4    00:00:00 grep --color=auto nomad
root     16388 30500 20 09:22 ?        00:02:21 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/lib/bin/nomad executor /var/local/client/alloc/0237870e-392b-832b-a67c-f5d073af16cf/mytask/mytask-executor.out
root     30487 29814  0 00:58 ?        00:03:37 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/../../package//lib/bin/nomad agent -config /etc/server_config.json
root     30500 29814  0 00:58 ?        00:01:40 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/../../package//lib/bin/nomad agent -config /etc/client_config.json

Bug occurs on 09:44:23

Sep 23 09:44:23 coehvc001 nomad[30500]: client: error fetching stats of task mytask: unexpected EOF

Processes after the bug

$ ps -ef | grep nomad
root     11678 30500 17 09:32 ?        00:02:20 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/lib/bin/nomad executor /var/local/client/alloc/0237870e-392b-832b-a67c-f5d073af16cf/mytask/mytask-executor.out
root     14430 30500  0 09:45 ?        00:00:00 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/lib/bin/nomad executor /var/local/client/alloc/0237870e-392b-832b-a67c-f5d073af16cf/mytask/mytask-executor.out
root     15642  1494  0 09:45 pts/4    00:00:00 grep --color=auto nomad
root     30487 29814  0 00:58 ?        00:03:42 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/../../package//lib/bin/nomad agent -config /etc/server_config.json
root     30500 29814  0 00:58 ?        00:01:43 /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/package/../../package//lib/bin/nomad agent -config /etc/client_config.json

Executor with pid 11678 is still running. I appreciate your comments on this.

kaskavalci · 2016-09-23T08:00:38Z

After 10 minutes both executors got killed by a panic. I noticed that change causes a memory leak in Nomad. Probably not related to bug.

$ top
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12063 root      20   0 17.1g  16g 6920 S   65 53.1   3:15.88 nomad

Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: fatal error: runtime: out of memory
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime stack:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.throw(0x1037ec6, 0x16)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/panic.go:566 +0x95
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.sysMap(0xca8fc60000, 0x2acf60000, 0x1767d00, 0x1781a98)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/mem_linux.go:219 +0x1d0
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.(*mheap).sysAlloc(0x1767da0, 0x2acf60000, 0xdf00000001)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/malloc.go:407 +0x37a
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.(*mheap).grow(0x1767da0, 0x1567ae, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/mheap.go:726 +0x62
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.(*mheap).allocSpanLocked(0x1767da0, 0x1567ae, 0x45a58a)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/mheap.go:630 +0x4f2
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.(*mheap).alloc_m(0x1767da0, 0x1567ae, 0x7f0100000000, 0x7f8710afada0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/mheap.go:515 +0xe0
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.(*mheap).alloc.func1()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/mheap.go:579 +0x4b
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.systemstack(0x7f8710afada8)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/asm_amd64.s:314 +0xab
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.(*mheap).alloc(0x1767da0, 0x1567ae, 0x10100000000, 0x653500afae10)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/mheap.go:580 +0x73
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.largeAlloc(0x2acf5c000, 0x1760201, 0xc420434001)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/malloc.go:774 +0x93
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.mallocgc.func1()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/malloc.go:669 +0x3e
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.systemstack(0xc42001d500)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/asm_amd64.s:298 +0x79
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.mstart()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/proc.go:1079
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 36 [running]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.systemstack_switch()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc4202369d0 sp=0xc4202369c8
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.mallocgc(0x2acf5c000, 0xef6e80, 0x416001, 0xc420236a98)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/malloc.go:670 +0x903 fp=0xc420236a70 sp=0xc4202369d0
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.growslice(0xef6e80, 0xc42054c000, 0x223f7c00, 0x223f7c00, 0x223f7c01, 0xc42054c000, 0x1b65fc00, 0x223f7c00)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/slice.go:131 +0x15e fp=0xc420236b00 sp=0xc420236a70
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/client/driver/executor.(*UniversalExecutor).scanPids(0xc420122480, 0x2d9e, 0xc420466800, 0x13600001, 0x1a0, 0x2, 0xc420236f90, 0x2)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/executor/executor.go:791 +0x279 fp=0xc420236cd0 sp=0xc420236b00
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/client/driver/executor.(*UniversalExecutor).getAllPids(0xc420122480, 0xc42005f020, 0x0, 0xc4200a0301)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/executor/executor_linux.go:294 +0x4d6 fp=0xc420236dc0 sp=0xc420236cd0
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/client/driver/executor.(*UniversalExecutor).collectPids(0xc420122480)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/executor/executor.go:744 +0x11c fp=0xc420236fa8 sp=0xc420236dc0
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.goexit()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420236fb0 sp=0xc420236fa8
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/client/driver/executor.(*UniversalExecutor).LaunchCmd
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/executor/executor.go:304 +0x5b1
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 1 [chan receive, 23 minutes]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.Serve(0xc4204ddc78)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/server.go:144 +0x781
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/command.(*ExecutorPluginCommand).Run(0xc420157380, 0xc4200660e0, 0x1, 0x1, 0xc420156e80)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/command/executor_plugin.go:41 +0x3ec
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc420124480, 0xc420124480, 0x17, 0xc42014b3a0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:153 +0x330
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: main.RunCustom(0xc4200660d0, 0x2, 0x2, 0xc42012b6e0, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/main.go:55 +0x7a1
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: main.Run(0xc4200660d0, 0x2, 0x2, 0xc4200001a0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/main.go:20 +0x56
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: main.main()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/main.go:16 +0x66
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 17 [syscall, 23 minutes, locked to thread]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.goexit()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 20 [syscall, 23 minutes]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: os/signal.signal_recv(0xc420020fa8)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/sigqueue.go:116 +0x157
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: os/signal.loop()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by os/signal.init.1
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 1004 [IO wait]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.runtime_pollWait(0x7f87153c2128, 0x72, 0xc)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/netpoll.go:160 +0x59
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*pollDesc).wait(0xc420150300, 0x72, 0xc4202399d0, 0xc420062080)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*pollDesc).waitRead(0xc420150300, 0x173d580, 0xc420062080)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*netFD).Read(0xc4201502a0, 0xc4201dc000, 0x1000, 0x1000, 0x0, 0x173d580, 0xc420062080)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/fd_unix.go:243 +0x1a1
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*conn).Read(0xc42006e038, 0xc4201dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/net.go:173 +0x70
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net/http.(*persistConn).Read(0xc4200d6500, 0xc4201dc000, 0x1000, 0x1000, 0x805d30, 0xc420239b58, 0x40946d)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/http/transport.go:1261 +0x154
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: bufio.(*Reader).fill(0xc420408d80)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/bufio/bufio.go:97 +0x10c
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: bufio.(*Reader).Peek(0xc420408d80, 0x1, 0xc420239bbd, 0x1, 0x0, 0xc4204093e0, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/bufio/bufio.go:129 +0x62
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net/http.(*persistConn).readLoop(0xc4200d6500)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/http/transport.go:1418 +0x1a1
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by net/http.(*Transport).dialConn
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/http/transport.go:1062 +0x4e9
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 24 [select, 23 minutes, locked to thread]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.gopark(0x10c1068, 0x0, 0x10229fd, 0x6, 0x18, 0x2)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/proc.go:259 +0x13a
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.selectgoImpl(0xc420022f30, 0x0, 0x18)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/select.go:423 +0x11d9
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.selectgo(0xc420022f30)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/select.go:238 +0x1c
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.ensureSigM.func1()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: runtime.goexit()
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 25 [chan receive, 23 minutes]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.Serve.func1(0xc42005ec60)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/server.go:128 +0x6c
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.Serve
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/server.go:134 +0x6eb
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 26 [IO wait, 23 minutes]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.runtime_pollWait(0x7f87153c21e8, 0x72, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/runtime/netpoll.go:160 +0x59
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*pollDesc).wait(0xc420151330, 0x72, 0xc420039e20, 0xc420062080)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*pollDesc).waitRead(0xc420151330, 0x173d580, 0xc420062080)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*netFD).accept(0xc4201512d0, 0x0, 0x173ba80, 0xc420010fe0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/fd_unix.go:419 +0x238
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*UnixListener).accept(0xc42014bd80, 0xc4200280e0, 0x0, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/unixsock_posix.go:158 +0x32
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net.(*UnixListener).Accept(0xc42014bd80, 0xc420039f58, 0xc420039f60, 0xc420039f50, 0x5aebfe)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/unixsock.go:229 +0x49
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*rmListener).Accept(0xc42014bda0, 0x10bec48, 0xc42012bda0, 0x7f8715375050, 0xc4200280e0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011<autogenerated>:5 +0x49
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCServer).Accept(0xc42012bda0, 0x1741d80, 0xc42014bda0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_server.go:36 +0x35
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.Serve
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/server.go:141 +0x75b
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 9 [syscall, 23 minutes]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: syscall.Syscall(0x0, 0x6, 0xc42021e000, 0x8000, 0x42, 0x7f87153bf7c8, 0x458880)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: syscall.read(0x6, 0xc42021e000, 0x8000, 0x8000, 0xc420023d01, 0x10100000041196f, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: syscall.Read(0x6, 0xc42021e000, 0x8000, 0x8000, 0x0, 0xc420023fa0, 0x2)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/syscall/syscall_unix.go:161 +0x49
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: os.(*File).read(0xc42006e328, 0xc42021e000, 0x8000, 0x8000, 0x7f8715375101, 0xc42021e000, 0x4131e9)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/os/file_unix.go:228 +0x4f
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: os.(*File).Read(0xc42006e328, 0xc42021e000, 0x8000, 0x8000, 0x8000, 0x8000, 0xa49f6d)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/os/file.go:101 +0x59
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: io.copyBuffer(0x173ab40, 0xc420177110, 0x173b940, 0xc42006e328, 0xc42021e000, 0x8000, 0x8000, 0xc420023f18, 0xc4200174f0, 0xc)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/io/io.go:390 +0x147
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: io.Copy(0x173ab40, 0xc420177110, 0x173b940, 0xc42006e328, 0x0, 0x0, 0x0)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/io/io.go:360 +0x68
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x1022abd, 0x6, 0x173ab40, 0xc420177110, 0x173b940, 0xc42006e328)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x79
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCServer).ServeConn
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_server.go:81 +0x1f4
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 30 [select, 23 minutes]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/client/driver/logging.(*FileRotator).purgeOldFiles(0xc4200a0870)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:218 +0x736
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/client/driver/logging.NewFileRotator
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:64 +0x1b8
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 31 [chan receive]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/client/driver/logging.(*FileRotator).flushPeriodically(0xc4200a0870)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:193 +0x6b
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/client/driver/logging.NewFileRotator
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:65 +0x1da
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 32 [select, 23 minutes]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/client/driver/logging.(*FileRotator).purgeOldFiles(0xc4200a0900)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:218 +0x736
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/client/driver/logging.NewFileRotator
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:64 +0x1b8
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 33 [chan receive]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/client/driver/logging.(*FileRotator).flushPeriodically(0xc4200a0900)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:193 +0x6b
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/client/driver/logging.NewFileRotator
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/logging/rotator.go:65 +0x1da
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 39 [select]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: github.com/hashicorp/nomad/command/agent/consul.(*Syncer).Run(0xc420404820)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/command/agent/consul/syncer.go:786 +0x39b
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by github.com/hashicorp/nomad/client/driver/executor.(*UniversalExecutor).SyncServices
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/home/i325753/go/src/github.com/hashicorp/nomad/client/driver/executor/executor.go:513 +0x44f
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: goroutine 1005 [select]:
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: net/http.(*persistConn).writeLoop(0xc4200d6500)
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/http/transport.go:1646 +0x3bd
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: created by net/http.(*Transport).dialConn
Sep 23 09:56:11 coehvc001 nomad[30500]: plugin: nomad: #011/usr/local/go/src/net/http/transport.go:1063 +0x50e

kaskavalci · 2016-09-23T11:18:17Z

I also see this in the logs. [ERR] plugin: stream copy 'stderr' error: session shutdown

Sep 23 13:05:39 coehvc001 nomad[17035]: client: error fetching stats of task mytask: unexpected EOF
Sep 23 13:05:39 coehvc001 nomad[17035]: #012runtime/debug.Stack(0xc4207e8a20, 0xc420803e00, 0x1736cc0)#012#011/usr/local/go/src/runtime/debug/stack.go:24 +0x79#012git.luolix.top/hashicorp/nomad/client/driver.(*rawExecHandle).run(0xc4207e8ae0)#012#011/home/halil/go/src/github.com/hashicorp/nomad/client/driver/raw_exec.go:291 +0x2fa#012created by github.com/hashicorp/nomad/client/driver.(*RawExecDriver).Start#012#011/home/halil/go/src/github.com/hashicorp/nomad/client/driver/raw_exec.go:176 +0xca8
Sep 23 13:05:39 coehvc001 nomad[17035]: executor exited with error: unexpected EOF
Sep 23 13:05:39 coehvc001 nomad[17035]: driver.raw_exec: failed to deregister services: connection is shut down
Sep 23 13:05:39 coehvc001 nomad[17035]: driver.raw_exec: error killing executor: connection is shut down
Sep 23 13:05:39 coehvc001 nomad[17035]: client: task "mytask" for alloc "ace2c14c-f336-fa2a-0f03-357d04ffc9a4" failed: Wait returned exit code 0, signal 0, and error unexpected EOF
Sep 23 13:05:39 coehvc001 nomad[17035]: client: Restarting task "mytask" for alloc "ace2c14c-f336-fa2a-0f03-357d04ffc9a4" in 1m3.029845568s
Sep 23 13:05:39 coehvc001 nomad[17035]: plugin: nomad: 2016/09/23 13:05:39 [ERR] plugin: stream copy 'stderr' error: session shutdown
Sep 23 13:05:39 coehvc001 nomad[17035]: client: updated allocations at index 5212 (pulled 0) (filtered 1)
Sep 23 13:05:39 coehvc001 nomad[17035]: client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)

dadgar · 2016-09-26T18:59:29Z

@kaskavalci Was the nomad client memory leaking or was it the executors, and this only happened after your patch?

kaskavalci · 2016-09-26T19:20:40Z

@dadgar It's after the patch, I didn't encounter a memory leak rather than that.

kaskavalci · 2016-09-27T07:16:25Z

I changed unix sockets into good-old TCP connection to test if it is from the unix sockets but now it's just a different connection error. Here is the log output with more traces I added.

2016/09/27 08:04:02 Recieved config
2016/09/27 08:04:02 &{HandshakeConfig:{ProtocolVersion:1 MagicCookieKey:NOMAD_PLUGIN_MAGIC_COOKIE MagicCookieValue:e4327c2e01eabfd75a8a67adb114fb34a757d57eee7728d857a8cec6e91a7255} Plugins:map[executor:0xc4202f19c0 syslogcollector:0xc4202f19d0] Cmd:0xc420125760 Reattach:<nil> Managed:false MinPort:14000 MaxPort:14512 StartTimeout:0s Stderr:<nil> SyncStdout:<nil> SyncStderr:<nil>}
2016/09/27 08:04:02 [DEBUG] plugin: starting plugin: /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/bin/nomad []string{"/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/bin/nomad", "executor", "/var/local/client/alloc/d2437335-9c1d-a60e-e0b0-231ffb406240/mytask/mytask-executor.out"}
2016/09/27 08:04:02 [DEBUG] plugin: waiting for RPC address for: /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/bin/nomad
2016/09/27 08:04:02 [DEBUG] plugin: nomad: 2016/09/27 08:04:02 [DEBUG] plugin: plugin address: tcp 127.0.0.1:14000
2016/09/27 08:04:02 [DEBUG] plugin: Received config for plugin: 1|1|tcp|127.0.0.1:14000
2016/09/27 08:04:02 [DEBUG] plugin: Address to dial. 127.0.0.1:14000
2016/09/27 08:04:02.257272 [DEBUG] driver.raw_exec: started process with pid: 12144
2016/09/27 08:04:02.387864 [DEBUG] client: updated allocations at index 435 (pulled 0) (filtered 1)
2016/09/27 08:04:02.388130 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2016/09/27 08:04:09 [DEBUG] http: Request /v1/agent/servers (3.071048ms)
2016/09/27 08:04:19 [DEBUG] http: Request /v1/agent/servers (258.22µs)
2016/09/27 08:04:22 [ERR] yamux: Failed to read header: read tcp 127.0.0.1:48227->127.0.0.1:14000: i/o timeout
2016/09/27 08:04:22.238032 goroutine 124 [running]:
runtime/debug.Stack(0xc42036d860, 0xc4202e6800, 0x16e3280)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/hashicorp/nomad/client/driver.(*rawExecHandle).run(0xc42036d980)
    /home/i325753/go/src/github.com/hashicorp/nomad/client/driver/raw_exec.go:291 +0x2e9
created by github.com/hashicorp/nomad/client/driver.(*RawExecDriver).Start
    /home/i325753/go/src/github.com/hashicorp/nomad/client/driver/raw_exec.go:176 +0xca5
2016/09/27 08:04:22.238192 [ERR] executor exited with error: unexpected EOF
2016/09/27 08:04:22.238523 [ERR] driver.raw_exec: failed to deregister services: connection is shut down
2016/09/27 08:04:22.238565 [ERR] driver.raw_exec: error killing executor: connection is shut down
2016/09/27 08:04:22 [DEBUG] plugin: Killing process 12096
2016/09/27 08:04:22 goroutine 124 [running]:
runtime/debug.Stack(0x100c678, 0x28, 0xc420348ea0)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Kill(0xc420361740)
    /home/i325753/go/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:263 +0xb7
github.com/hashicorp/nomad/client/driver.(*rawExecHandle).run(0xc42036d980)
    /home/i325753/go/src/github.com/hashicorp/nomad/client/driver/raw_exec.go:310 +0x1b5
created by github.com/hashicorp/nomad/client/driver.(*RawExecDriver).Start
    /home/i325753/go/src/github.com/hashicorp/nomad/client/driver/raw_exec.go:176 +0xca5

dadgar · 2016-09-27T16:45:58Z

@kaskavalci How are you reproducing this so consistently? It would be great if you could share reproduction steps, so that we can debug more easily

dadgar · 2016-09-27T23:42:09Z

@kaskavalci Thanks so much for the dump. There is definitely a memory leak:

940.83MB   940.83MB (flat, cum) 99.95% of Total
         .          .    792:                   if childPid {
         .          .    793:                           processFamily[pid.Pid()] = struct{}{}
         .          .    794:                           foundNewPid = true
         .          .    795:                   } else {
         .          .    796:                           // if it is not, then we add the pid to the buffer
  940.83MB   940.83MB    797:                           pidsRemaining = append(pidsRemaining, pid)
         .          .    798:                   }
         .          .    799:                   // scan only the pids which are left in the buffer
         .          .    800:                   allPids = pidsRemaining
         .          .    801:           }
         .          .    802:

github-actions · 2022-12-19T02:12:19Z

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

dadgar added type/bug theme/client stage/needs-investigation labels Sep 20, 2016

dadgar mentioned this issue Sep 27, 2016

Constant size space tracking of pids #1762

Merged

dadgar closed this as completed in #1762 Sep 28, 2016

github-actions bot locked as resolved and limited conversation to collaborators Dec 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Executor is killed after client cannot fetch stats #1719

Executor is killed after client cannot fetch stats #1719

kaskavalci commented Sep 20, 2016 •

edited

Loading

dadgar commented Sep 20, 2016

tugbabodrumlu commented Sep 20, 2016

kaskavalci commented Sep 22, 2016

kaskavalci commented Sep 23, 2016

kaskavalci commented Sep 23, 2016 •

edited

Loading

kaskavalci commented Sep 23, 2016

dadgar commented Sep 26, 2016

kaskavalci commented Sep 26, 2016

kaskavalci commented Sep 27, 2016

dadgar commented Sep 27, 2016

dadgar commented Sep 27, 2016

github-actions bot commented Dec 19, 2022

Executor is killed after client cannot fetch stats #1719

Executor is killed after client cannot fetch stats #1719

Comments

kaskavalci commented Sep 20, 2016 • edited Loading

Nomad version

Operating system and Environment details

Issue

Reproduction steps

Nomad Client logs

dadgar commented Sep 20, 2016

tugbabodrumlu commented Sep 20, 2016

kaskavalci commented Sep 22, 2016

kaskavalci commented Sep 23, 2016

kaskavalci commented Sep 23, 2016 • edited Loading

kaskavalci commented Sep 23, 2016

dadgar commented Sep 26, 2016

kaskavalci commented Sep 26, 2016

kaskavalci commented Sep 27, 2016

dadgar commented Sep 27, 2016

dadgar commented Sep 27, 2016

github-actions bot commented Dec 19, 2022

kaskavalci commented Sep 20, 2016 •

edited

Loading

kaskavalci commented Sep 23, 2016 •

edited

Loading