Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Couldn't parse \"/proc/stat\": bufio.Scanner: token too long #546

Closed
drdeimos opened this issue Jul 9, 2023 · 4 comments · Fixed by #547
Closed

Couldn't parse \"/proc/stat\": bufio.Scanner: token too long #546

drdeimos opened this issue Jul 9, 2023 · 4 comments · Fixed by #547

Comments

@drdeimos
Copy link

drdeimos commented Jul 9, 2023

Host operating system:

Linux k42jv 6.4.2 prometheus/node_exporter#1 SMP PREEMPT_DYNAMIC Thu Jul 6 15:29:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version:

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d69b645cb691dd3e84dc3afc88f5c006962)
  build user:       root@f9c3ed0cfbd3
  build date:       20230527-12:03:54
  go version:       go1.20.4
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9101 --collector.netdev.address-info --collector.ntp --collector.ntp.server-is-local --collector.filesystem.mount-points-exclude (^/(dev|proc|sys|run|var/lib/kubelet)($|/))|(^/var/lib/docker/) --collector.filesystem.fs-types-exclude ^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|fuse\.lxcfs|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs|squashfs)$

node_exporter log output

log 
Jul 07 21:58:49 k42jv systemd[1]: Started Prometheus exporter for hardware and OS metrics.
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.626Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.6.0, branch=HEAD, revision=ff7f9d69b645cb691dd3e84dc3afc88f5c006962)"
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.635Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.20.4, platform=linux/amd64, user=root@f9c3ed0cfbd3, date=20230527-12:03:54, tags=netgo osusergo static_build)"
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.636Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.638Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=(^/(dev|proc|sys|run|var/lib/kubelet)($|/))|(^/var/lib/docker/)
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|fuse\.lxcfs|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs|squashfs)$
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=ntp.go:83 level=warn collector=ntp msg="This collector is deprecated and will be removed in the next major version release."
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=arp
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=bcache
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=bonding
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.639Z caller=node_exporter.go:117 level=info collector=btrfs
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=conntrack
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=cpu
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=cpufreq
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=diskstats
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=dmi
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=edac
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=entropy
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=fibrechannel
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=filefd
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=filesystem
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=hwmon
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=infiniband
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=ipvs
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=loadavg
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=mdadm
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=meminfo
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=netclass
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=netdev
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=netstat
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=nfs
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=nfsd
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=ntp
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=nvme
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=os
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=powersupplyclass
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=pressure
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.640Z caller=node_exporter.go:117 level=info collector=rapl
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=schedstat
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=selinux
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=sockstat
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=softnet
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=stat
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=tapestats
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=textfile
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=thermal_zone
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=time
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=timex
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=udp_queues
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=uname
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=vmstat
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=xfs
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.646Z caller=node_exporter.go:117 level=info collector=zfs
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.660Z caller=tls_config.go:274 level=info msg="Listening on" address=127.0.0.1:9101
Jul 07 21:58:50 k42jv node_exporter[484]: ts=2023-07-07T18:58:50.660Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9101
Jul 07 21:59:09 k42jv node_exporter[484]: ts=2023-07-07T18:59:09.457Z caller=collector.go:169 level=error msg="collector failed" name=cpu duration_seconds=0.006603211 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long"
Jul 07 21:59:09 k42jv node_exporter[484]: ts=2023-07-07T18:59:09.459Z caller=collector.go:169 level=error msg="collector failed" name=stat duration_seconds=0.007868332 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long"
Jul 07 21:59:24 k42jv node_exporter[484]: ts=2023-07-07T18:59:24.443Z caller=collector.go:169 level=error msg="collector failed" name=cpu duration_seconds=0.000539895 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long"
Jul 07 21:59:24 k42jv node_exporter[484]: ts=2023-07-07T18:59:24.446Z caller=collector.go:169 level=error msg="collector failed" name=stat duration_seconds=0.001681038 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long"
Jul 07 21:59:39 k42jv node_exporter[484]: ts=2023-07-07T18:59:39.440Z caller=collector.go:169 level=error msg="collector failed" name=cpu duration_seconds=0.001306421 err="couldn't parse \"/proc/stat\": bufio.Scanner: token too long"

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Run node_exporter on my laptop with latest kernels

What did you expect to see?

I don't see some metrics like node_boot_time_seconds or node_cpu_seconds_total

What did you see instead?

Nothing. Only log above
@drdeimos
Copy link
Author

drdeimos commented Jul 9, 2023

That's content of /proc/stat from this machine
proc.stat.txt

@SuperQ
Copy link
Member

SuperQ commented Jul 10, 2023

This is a parsing issue in procfs. Moving to the correct repo.

@SuperQ SuperQ transferred this issue from prometheus/node_exporter Jul 10, 2023
@SuperQ
Copy link
Member

SuperQ commented Jul 10, 2023

Wow, there are 65962 columns in that intr line. The default Scanner can only handle 65535 columns. Easy enough to fix.

SuperQ added a commit that referenced this issue Jul 10, 2023
@SuperQ
Increase Stat() buffer size
79e072f 
Increase the /proc/stat scanner tokens size from the default of 65k to
1024k. This allows for scanning of large `intr` lines with > 65k
columns.

Fixes: #546

Signed-off-by: SuperQ <superq@gmail.com>
@SuperQ SuperQ mentioned this issue Jul 10, 2023
Merged
SuperQ added a commit that referenced this issue Jul 10, 2023
@SuperQ
Increase Stat() buffer size
dbd0bc5 
Increase the /proc/stat scanner tokens size from the default of 65k to
1024k. This allows for scanning of large `intr` lines with > 65k
columns.
* Increase the scanner buffer from the default 4kB to 8kB bytes at a time.

Fixes: #546

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this issue Jul 10, 2023
@SuperQ
Increase Stat() buffer size
fcec2ff 
Increase the /proc/stat scanner tokens size from the default of 65k to
1024k. This allows for scanning of large `intr` lines with > 65k
columns.
* Increase the scanner buffer from the default 4kB to 8kB bytes at a time.

Fixes: #546

Signed-off-by: SuperQ <superq@gmail.com>
SuperQ added a commit that referenced this issue Jul 10, 2023
@SuperQ
Increase Stat() buffer size (#547)
4b2218e 
Increase the /proc/stat scanner tokens size from the default of 65k to
1024k. This allows for scanning of large `intr` lines with > 65k
columns.
* Increase the scanner buffer from the default 4kB to 8kB bytes at a time.

Fixes: #546

Signed-off-by: SuperQ <superq@gmail.com>
@Joseph-co
Copy link

Wow, there are 65962 columns in that intr line. The default Scanner can only handle 65535 columns. Easy enough to fix.

Could you tell me how to increase those parses, "Increase the /proc/stat scanner tokens size from the default of 65k to
1024k." and "Increase the scanner buffer from the default 4kB to 8kB bytes". How to implement stuff.

jritter pushed a commit to jritter/procfs that referenced this issue Jul 15, 2024
@SuperQ @jritter
Increase Stat() buffer size (prometheus#547)
62fab86 
Increase the /proc/stat scanner tokens size from the default of 65k to
1024k. This allows for scanning of large `intr` lines with > 65k
columns.
* Increase the scanner buffer from the default 4kB to 8kB bytes at a time.

Fixes: prometheus#546

Signed-off-by: SuperQ <superq@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Increase Stat() buffer size prometheus/procfs
3 participants
@SuperQ @drdeimos @Joseph-co