Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: runtime error #568

Closed
joakimberglund opened this issue Jan 22, 2016 · 3 comments
Closed

panic: runtime error #568

joakimberglund opened this issue Jan 22, 2016 · 3 comments
Labels
bug unexpected problem or unintended behavior

Comments

@joakimberglund
Copy link

Got an instans of telegraf 0.10.0-1 on RHEL6.7 that crashes a couple of times a day...

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x45f387]

goroutine 24469 [running]:
github.com/influxdb/telegraf/internal/config.Filter.ShouldPass(0x0, 0x0, 0x0, 0xc820087280, 0x6, 0x8, 0x0
, 0x0, 0x0, 0x0, ...)
        /home/telebuild/go/src/github.com/influxdb/telegraf/internal/config/config.go:154 +0x10b
github.com/influxdb/telegraf/internal/config.(*RunningOutput).FilterPoints(0xc8201f03f0, 0xc82023ce80, 0x
f, 0x10, 0x0, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/internal/config/config.go:139 +0x14a
github.com/influxdb/telegraf.(*Agent).writeOutput(0xc82006a298, 0xc82023ce80, 0xf, 0x10, 0xc8201f03f0, 0x
c820061e00, 0xc82043dc50)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:233 +0xfd
created by github.com/influxdb/telegraf.(*Agent).flush
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:274 +0x11a

goroutine 1 [select]:
github.com/influxdb/telegraf.(*Agent).Run(0xc82006a298, 0xc820061e00, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:390 +0x9d5
main.main()
        /home/telebuild/go/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:213 +0x1b45

goroutine 17 [syscall, 176 minutes, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 20 [syscall, 176 minutes]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 82 [IO wait]:
net.runtime_pollWait(0x7f5aebdb49e0, 0x72, 0xc8200620a0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82000b100, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82000b100, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82000b0a0, 0xc8203aa000, 0x1000, 0x1000, 0x0, 0x7f5aebdae028, 0xc8200620a0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82006a538, 0xc8203aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f5aed1f0180, 0xc82006a538, 0xc820143c88, 0xc8203aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc82037d940, 0xc8203aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc820287a40)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820287a40, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820143c30)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78


goroutine 76 [select]:
net/http.(*persistConn).writeLoop(0xc8203a2000)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 34 [select, 176 minutes, locked to thread]:
runtime.gopark(0x11b44e8, 0xc82001ef28, 0xfbe180, 0x6, 0xc820255018, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc82001ef28, 0x0, 0x18)
        /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82001ef28)
        /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 35 [chan receive, 176 minutes]:
main.main.func1(0xc820061e60, 0xc820061e00)
        /home/telebuild/go/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:193 +0x36
created by main.main
        /home/telebuild/go/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:195 +0x142a

goroutine 36 [select]:
github.com/influxdb/telegraf.(*Agent).flusher(0xc82006a298, 0xc820061e00, 0xc820286060, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:291 +0x44f
github.com/influxdb/telegraf.(*Agent).Run.func1(0xc8201f5920, 0xc82006a298, 0xc820061e00, 0xc820286060)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:351 +0x7f
created by github.com/influxdb/telegraf.(*Agent).Run
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:355 +0x469

goroutine 83 [select]:
net/http.(*persistConn).writeLoop(0xc820143c30)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 75 [IO wait]:
net.runtime_pollWait(0x7f5aebdb4aa0, 0x72, 0xc8200620a0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82000b170, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82000b170, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82000b110, 0xc8203ac000, 0x1000, 0x1000, 0x0, 0x7f5aebdae028, 0xc8200620a0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200221c8, 0xc8203ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f5aed1f0180, 0xc8200221c8, 0xc8203a2058, 0xc8203ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc82038a6e0, 0xc8203ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8203a6060)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8203a6060, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8203a2000)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 24470 [runnable]:
strings.Count(0xc8201f4831, 0x6, 0xfb29e8, 0x1, 0x0)
        /usr/local/go/src/strings/strings.go:84
strings.genSplit(0xc8201f4831, 0x6, 0xfb29e8, 0x1, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0x0)
        /usr/local/go/src/strings/strings.go:289 +0xef
strings.Split(0xc8201f4831, 0x6, 0xfb29e8, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/strings/strings.go:331 +0x65
github.com/influxdb/telegraf/internal.Glob(0xc8201f4831, 0x6, 0xc82043dc70, 0x4, 0x699422)
        /home/telebuild/go/src/github.com/influxdb/telegraf/internal/internal.go:116 +0xf9
github.com/influxdb/telegraf/internal/config.Filter.ShouldPass(0x0, 0x0, 0x0, 0xc8201f48e0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
        /home/telebuild/go/src/github.com/influxdb/telegraf/internal/config/config.go:154 +0x14f
github.com/influxdb/telegraf/internal/config.(*RunningOutput).FilterPoints(0xc8201f0db0, 0xc82023ce80, 0xf, 0x10, 0x0, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/internal/config/config.go:139 +0x14a
github.com/influxdb/telegraf.(*Agent).writeOutput(0xc82006a298, 0xc82023ce80, 0xf, 0x10, 0xc8201f0db0, 0xc820061e00, 0xc82043dc50)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:233 +0xfd
created by github.com/influxdb/telegraf.(*Agent).flush
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:274 +0x11a
@joakimberglund
Copy link
Author

[tags]

dc = "us-east-1"

tenant = "xyz"

[agent]

Default data collection interval for all plugins

interval = "10s"

Rounds collection interval to 'interval'

ie, if interval="10s" then always collect on :00, :10, :20, etc.

round_interval = true

Default data flushing interval for all outputs. You should not set this below

interval. Maximum flush_interval will be flush_interval + flush_jitter

flush_interval = "10s"

Jitter the flush interval by a random amount. This is primarily to avoid

large write spikes for users running a large number of telegraf instances.

ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s

flush_jitter = "0s"

Run telegraf in debug mode

debug = false

Override default hostname, if empty use os.Hostname()

hostname = ""

[[outputs.influxdb]]
urls = ["http://xyz:8086"] # required

The target database for metrics (telegraf will create it if not exists)

database = "linux" # required

Precision of writes, valid values are n, u, ms, s, m, and h

note: using second precision greatly helps InfluxDB compression

precision = "s"

pass = [ "cpu", "disk", "mem", "swap", "system", "net" ]

[[inputs.cpu]]

Whether to report per-cpu stats or not

percpu = false

Whether to report total system cpu stats or not

totalcpu = true

Comment this line if you want the raw CPU time metrics

drop = ["cpu_time"]

[[inputs.disk]]

By default, telegraf gather stats for all mountpoints.

Setting mountpoints will restrict the stats to the specified mountpoints.

Mountpoints=["/"]

[inputs.disk.tagpass]
fstype = [ "ext4", "xfs", "vxfs", "tmpfs" ]

[[inputs.diskio]]

By default, telegraf will gather stats for all devices including

disk partitions.

Setting devices will restrict the stats to the specified devices.

Devices=["sda","sdb"]

Uncomment the following line if you do not need disk serial numbers.

SkipSerialNumber = true

[[inputs.mem]]

no configuration

[[inputs.swap]]

no configuration

[[inputs.system]]

no configuration

[[inputs.netstat]]

no configuration

[[inputs.net]]

no configuration

@joakimberglund
Copy link
Author

and in telegraf.d/apache.conf

[[outputs.influxdb]]
urls = ["http://xyz:8086"] # required

The target database for metrics (telegraf will create it if not exists)

database = "apache" # required

Precision of writes, valid values are n, u, ms, s, m, and h

note: using second precision greatly helps InfluxDB compression

precision = "s"

Connection timeout (for the connection with InfluxDB), formatted as a string.

If not provided, will default to 0 (no timeout)

timeout = "5s"

username = "telegraf"

password = "metricsmetricsmetricsmetrics"

Set the user agent for HTTP POSTs (can be useful for log differentiation)

user_agent = "telegraf"

Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)

udp_payload = 512

pass = [ "apache" ]

[[inputs.apache]]
urls = ["http://xyz2/server-status?auto"]

@sparrc sparrc added the bug unexpected problem or unintended behavior label Jan 22, 2016
sparrc added a commit that referenced this issue Jan 22, 2016
@sparrc
Make each output keep it's own slice of points
799b9b4 
Also moved some objects out of config.go and put them in their own
package, internal/models

fixes #568
@sparrc sparrc mentioned this issue Jan 22, 2016
Closed
sparrc added a commit that referenced this issue Jan 22, 2016
@sparrc
Make each output keep it's own slice of points
1c0968a 
Also moved some objects out of config.go and put them in their own
package, internal/models

fixes #568
closes #285
sparrc added a commit that referenced this issue Jan 22, 2016
@sparrc
Make each output keep it's own slice of points
ac86cb9 
Also moved some objects out of config.go and put them in their own
package, internal/models

fixes #568
closes #285
sparrc added a commit that referenced this issue Jan 22, 2016
@sparrc
Make each output keep it's own slice of points
9d8b4a4 
Also moved some objects out of config.go and put them in their own
package, internal/models

fixes #568
closes #285
sparrc added a commit that referenced this issue Jan 22, 2016
@sparrc
Make each output keep it's own slice of points
b350ef9 
Also moved some objects out of config.go and put them in their own
package, internal/models

fixes #568
closes #285
@sparrc
Copy link
Contributor

sparrc commented Jan 22, 2016

Thanks for the details @joakimberglund, looks like there was a race condition that could only be tripped when multiple outputs of the same type were specified.

PR #570 will fix this, and will go into version 0.10.1

@sparrc sparrc closed this as completed in 5349a3b Jan 22, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make each output keep it's own slice of points influxdata/telegraf
2 participants
@sparrc @joakimberglund