Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiline string in SNMPv2-MIB:sysDescr produces garbled output #440

Closed
localpref opened this issue Jul 25, 2019 · 12 comments
Closed

multiline string in SNMPv2-MIB:sysDescr produces garbled output #440

localpref opened this issue Jul 25, 2019 · 12 comments

Comments

@localpref
Copy link

A multiline sysDescr string on an ancient Cisco 2800 produces garbled output and no valid key{tags} value line, breaking the Prometheus HTTP Metrics output format.

Host operating system: output of uname -a

Linux (hostname) 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux
(added info: this is running within an LXC container)

snmp_exporter version: output of snmp_exporter -version

snmp_exporter, version 0.15.0 (branch: HEAD, revision: 92a3da4)
build user: root@8c3a7c03d455
build date: 20190212-13:40:02
go version: go1.11.5 (same error also with go 1.2.6)

What device/snmpwalk OID are you using?

1.3.6.1.2.1.1.1 (sysDescr)

relevant excerpt from snmp.yml:

sysdescr_test:
  get:
  - 1.3.6.1.2.1.1.1.0
  metrics:
  - name: sysDescr
    oid: 1.3.6.1.2.1.1.1
    type: DisplayString
    help: A textual description of the entity - 1.3.6.1.2.1.1.1
  version: 2
  max_repetitions: 15
  timeout: 10s
  auth:
    community: public

If this is a new device, please link to the MIB(s).

It's an ancient Cisco2800, but the query goes to SNMPv2-MIB::sysDescr

What did you do that produced an error?

Query a sysDescr that is multiline

What did you expect to see?

[...]
# HELP sysDescr A textual description of the entity - 1.3.6.1.2.1.1.1
# TYPE sysDescr gauge
sysDescr{sysDescr="Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)\nTechnical Support: http://www.cisco.com/techsupport\nCopyright (c) 1986-2011 by Cisco Systems, Inc.\nCompiled Tue 06-Dec-11 16:21 by prod_rel_team\n"} 1

# snmpbulkwalk -On -v2c -c public (ip) 1.3.6.1.2.1.1.1
.1.3.6.1.2.1.1.1.0 = STRING: Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Tue 06-Dec-11 16:21 by prod_rel_team

What did you see instead?

[...]
# HELP sysDescr A textual description of the entity - 1.3.6.1.2.1.1.1
# TYPE sysDescr gauge
\nCompiled Tue 06-Dec-11 16:21 by prod_rel_team"} 1rt(C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)
@localpref
Copy link
Author

I see the same behaviour with a manually built snmp_exporter from a few days ago( HEAD/40d8840915d961a8e3d3ae88a65f5d35072a9d8b), built with go 1.2.6

@brian-brazil
Copy link
Contributor

Something very odd is going on here, invalid output like this should be impossible. Can you get the hex version of both the sysDescr and what's on the /metrics?

@localpref
Copy link
Author

(small omissions of internal data, as bulkwalk returned more objects than necessary)

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:12:04.437905 IP mgr > device:  GetBulk(27)  N=0 M=10 .1.3.6.1.2.1.1.1
	0x0000:  4500 0046 f26a 4000 4011 d386 a9fe fd0d  E..F.j@.@.......
	0x0010:  c0a8 0d01 85c4 00a1 0032 74f9 3028 0201  .........2t.0(..
	0x0020:  0104 0670 7562 6c69 63a5 1b02 046e 161c  ...public....n..
	0x0030:  8602 0100 0201 0a30 0d30 0b06 072b 0601  .......0.0...+..
	0x0040:  0201 0101 0500                           ......
15:12:04.442150 IP device > mgr:  GetResponse(478)  .1.3.6.1.2.1.1.1.0=43_69_73_63_6f_20_49_4f_53_20_53_6f_66_74_77_61_72_65_2c_20_32_38_30_30_20_53_6f_66_74_77_61_72_65_20_28_43_32_38_30_30_4e_4d_2d_41_44_56_45_4e_54_45_52_50_52_49_53_45_4b_39_2d_4d_29_2c_20_56_65_72_73_69_6f_6e_20_31_35_2e_31_28_34_29_4d_33_2c_20_52_45_4c_45_41_53_45_20_53_4f_46_54_57_41_52_45_20_28_66_63_31_29_0d_0a_54_65_63_68_6e_69_63_61_6c_20_53_75_70_70_6f_72_74_3a_20_68_74_74_70_3a_2f_2f_77_77_77_2e_63_69_73_63_6f_2e_63_6f_6d_2f_74_65_63_68_73_75_70_70_6f_72_74_0d_0a_43_6f_70_79_72_69_67_68_74_20_28_63_29_20_31_39_38_36_2d_32_30_31_31_20_62_79_20_43_69_73_63_6f_20_53_79_73_74_65_6d_73_2c_20_49_6e_63_2e_0d_0a_43_6f_6d_70_69_6c_65_64_20_54_75_65_20_30_36_2d_44_65_63_2d_31_31_20_31_36_3a_32_31_20_62_79_20_70_72_6f_64_5f_72_65_6c_5f_74_65_61_6d 
	0x0000:  4500 020d 4349 0000 fd11 03e1 c0a8 0d01  E...CI..........
	0x0010:  a9fe fd0d 00a1 85c4 01f9 37f1 3082 01ed  ..........7.0...
	0x0020:  0201 0104 0670 7562 6c69 63a2 8201 de02  .....public.....
	0x0030:  046e 161c 8602 0100 0201 0030 8201 ce30  .n.........0...0
	0x0040:  8201 0906 082b 0601 0201 0101 0004 81fc  .....+..........
	0x0050:  4369 7363 6f20 494f 5320 536f 6674 7761  Cisco.IOS.Softwa
	0x0060:  7265 2c20 3238 3030 2053 6f66 7477 6172  re,.2800.Softwar
	0x0070:  6520 2843 3238 3030 4e4d 2d41 4456 454e  e.(C2800NM-ADVEN
	0x0080:  5445 5250 5249 5345 4b39 2d4d 292c 2056  TERPRISEK9-M),.V
	0x0090:  6572 7369 6f6e 2031 352e 3128 3429 4d33  ersion.15.1(4)M3
	0x00a0:  2c20 5245 4c45 4153 4520 534f 4654 5741  ,.RELEASE.SOFTWA
	0x00b0:  5245 2028 6663 3129 0d0a 5465 6368 6e69  RE.(fc1)..Techni
	0x00c0:  6361 6c20 5375 7070 6f72 743a 2068 7474  cal.Support:.htt
	0x00d0:  703a 2f2f 7777 772e 6369 7363 6f2e 636f  p://www.cisco.co
	0x00e0:  6d2f 7465 6368 7375 7070 6f72 740d 0a43  m/techsupport..C
	0x00f0:  6f70 7972 6967 6874 2028 6329 2031 3938  opyright.(c).198
	0x0100:  362d 3230 3131 2062 7920 4369 7363 6f20  6-2011.by.Cisco.
	0x0110:  5379 7374 656d 732c 2049 6e63 2e0d 0a43  Systems,.Inc...C
	0x0120:  6f6d 7069 6c65 6420 5475 6520 3036 2d44  ompiled.Tue.06-D
	0x0130:  6563 2d31 3120 3136 3a32 3120 6279 2070  ec-11.16:21.by.p
	0x0140:  726f 645f 7265 6c5f 7465 616d 3015 0608  rod_rel_team0...
	0x0150:  2b06 0102 0101 0200 0609 2b06 0104 0109  +.........+.....
	0x0160:  0184 4130 1106 082b 0601 0201 0103 0043  ..A0...+.......C
	0x0170:  0500 91c0 4b51 3013 0608 2b06 0102 0101  ....KQ0...+.....

curl | hexdump -C

00000180  2d 2d 3a 2d 2d 20 32 31  32 38 32 0a 23 20 48 45  |--:-- 21282.# HE|
00000190  4c 50 20 73 6e 6d 70 5f  73 63 72 61 70 65 5f 64  |LP snmp_scrape_d|
000001a0  75 72 61 74 69 6f 6e 5f  73 65 63 6f 6e 64 73 20  |uration_seconds |
000001b0  54 6f 74 61 6c 20 53 4e  4d 50 20 74 69 6d 65 20  |Total SNMP time |
000001c0  73 63 72 61 70 65 20 74  6f 6f 6b 20 28 77 61 6c  |scrape took (wal|
000001d0  6b 20 61 6e 64 20 70 72  6f 63 65 73 73 69 6e 67  |k and processing|
000001e0  29 2e 0a 23 20 54 59 50  45 20 73 6e 6d 70 5f 73  |)..# TYPE snmp_s|
000001f0  63 72 61 70 65 5f 64 75  72 61 74 69 6f 6e 5f 73  |crape_duration_s|
00000200  65 63 6f 6e 64 73 20 67  61 75 67 65 0a 73 6e 6d  |econds gauge.snm|
00000210  70 5f 73 63 72 61 70 65  5f 64 75 72 61 74 69 6f  |p_scrape_duratio|
00000220  6e 5f 73 65 63 6f 6e 64  73 20 30 2e 30 30 34 34  |n_seconds 0.0044|
00000230  33 37 39 30 39 0a 23 20  48 45 4c 50 20 73 6e 6d  |37909.# HELP snm|
00000240  70 5f 73 63 72 61 70 65  5f 70 64 75 73 5f 72 65  |p_scrape_pdus_re|
00000250  74 75 72 6e 65 64 20 50  44 55 73 20 72 65 74 75  |turned PDUs retu|
00000260  72 6e 65 64 20 66 72 6f  6d 20 77 61 6c 6b 2e 0a  |rned from walk..|
00000270  23 20 54 59 50 45 20 73  6e 6d 70 5f 73 63 72 61  |# TYPE snmp_scra|
00000280  70 65 5f 70 64 75 73 5f  72 65 74 75 72 6e 65 64  |pe_pdus_returned|
00000290  20 67 61 75 67 65 0a 73  6e 6d 70 5f 73 63 72 61  | gauge.snmp_scra|
000002a0  70 65 5f 70 64 75 73 5f  72 65 74 75 72 6e 65 64  |pe_pdus_returned|
000002b0  20 31 0a 23 20 48 45 4c  50 20 73 6e 6d 70 5f 73  | 1.# HELP snmp_s|
000002c0  63 72 61 70 65 5f 77 61  6c 6b 5f 64 75 72 61 74  |crape_walk_durat|
000002d0  69 6f 6e 5f 73 65 63 6f  6e 64 73 20 54 69 6d 65  |ion_seconds Time|
000002e0  20 53 4e 4d 50 20 77 61  6c 6b 2f 62 75 6c 6b 77  | SNMP walk/bulkw|
000002f0  61 6c 6b 20 74 6f 6f 6b  2e 0a 23 20 54 59 50 45  |alk took..# TYPE|
00000300  20 73 6e 6d 70 5f 73 63  72 61 70 65 5f 77 61 6c  | snmp_scrape_wal|
00000310  6b 5f 64 75 72 61 74 69  6f 6e 5f 73 65 63 6f 6e  |k_duration_secon|
00000320  64 73 20 67 61 75 67 65  0a 73 6e 6d 70 5f 73 63  |ds gauge.snmp_sc|
00000330  72 61 70 65 5f 77 61 6c  6b 5f 64 75 72 61 74 69  |rape_walk_durati|
00000340  6f 6e 5f 73 65 63 6f 6e  64 73 20 30 2e 30 30 34  |on_seconds 0.004|
00000350  33 34 37 30 33 37 0a 23  20 48 45 4c 50 20 73 79  |347037.# HELP sy|
00000360  73 44 65 73 63 72 20 41  20 74 65 78 74 75 61 6c  |sDescr A textual|
00000370  20 64 65 73 63 72 69 70  74 69 6f 6e 20 6f 66 20  | description of |
00000380  74 68 65 20 65 6e 74 69  74 79 20 2d 20 31 2e 33  |the entity - 1.3|
00000390  2e 36 2e 31 2e 32 2e 31  2e 31 2e 31 0a 23 20 54  |.6.1.2.1.1.1.# T|
000003a0  59 50 45 20 73 79 73 44  65 73 63 72 20 67 61 75  |YPE sysDescr gau|
000003b0  67 65 0a 73 79 73 44 65  73 63 72 7b 73 79 73 44  |ge.sysDescr{sysD|
000003c0  65 73 63 72 3d 22 43 69  73 63 6f 20 49 4f 53 20  |escr="Cisco IOS |
000003d0  53 6f 66 74 77 61 72 65  2c 20 32 38 30 30 20 53  |Software, 2800 S|
000003e0  6f 66 74 77 61 72 65 20  28 43 32 38 30 30 4e 4d  |oftware (C2800NM|
000003f0  2d 41 44 56 45 4e 54 45  52 50 52 49 53 45 4b 39  |-ADVENTERPRISEK9|
00000400  2d 4d 29 2c 20 56 65 72  73 69 6f 6e 20 31 35 2e  |-M), Version 15.|
00000410  31 28 34 29 4d 33 2c 20  52 45 4c 45 41 53 45 20  |1(4)M3, RELEASE |
00000420  53 4f 46 54 57 41 52 45  20 28 66 63 31 29 0d 5c  |SOFTWARE (fc1).\|
00000430  6e 54 65 63 68 6e 69 63  61 6c 20 53 75 70 70 6f  |nTechnical Suppo|
00000440  72 74 3a 20 68 74 74 70  3a 2f 2f 77 77 77 2e 63  |rt: http://www.c|
00000450  69 73 63 6f 2e 63 6f 6d  2f 74 65 63 68 73 75 70  |isco.com/techsup|
00000460  70 6f 72 74 0d 5c 6e 43  6f 70 79 72 69 67 68 74  |port.\nCopyright|
00000470  20 28 63 29 20 31 39 38  36 2d 32 30 31 31 20 62  | (c) 1986-2011 b|
00000480  79 20 43 69 73 63 6f 20  53 79 73 74 65 6d 73 2c  |y Cisco Systems,|
00000490  20 49 6e 63 2e 0d 5c 6e  43 6f 6d 70 69 6c 65 64  | Inc..\nCompiled|
000004a0  20 54 75 65 20 30 36 2d  44 65 63 2d 31 31 20 31  | Tue 06-Dec-11 1|
000004b0  36 3a 32 31 20 62 79 20  70 72 6f 64 5f 72 65 6c  |6:21 by prod_rel|
000004c0  5f 74 65 61 6d 22 7d 20  31 0a                    |_team"} 1.|

@localpref
Copy link
Author

hmm, is that some weird \LF vs \CRLF and the shell etc overwriting itself is going on?

@localpref
Copy link
Author

the device sends the 0x0d character before every newline, which is "\CR" or "Carriage Return" but no linefeed...

@localpref
Copy link
Author 

curl "http://localhost:9117/snmp?module=sysdescr_public&target=192.168.13.1" | tr -d '\r'
[...]
# HELP sysDescr A textual description of the entity - 1.3.6.1.2.1.1.1
# TYPE sysDescr gauge
sysDescr{sysDescr="Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)\nTechnical Support: http://www.cisco.com/techsupport\nCopyright (c) 1986-2011 by Cisco Systems, Inc.\nCompiled Tue 06-Dec-11 16:21 by prod_rel_team"} 1

@localpref
Copy link
Author

I think you can close this, this smells of "user too stupid" error. I don't think theres anything wrong with the exporter.

@localpref
Copy link
Author

hmm, as you do escape \n, do you also need to escape \r in the exporter?

@localpref
Copy link
Author

TLDR: Old Cisco device sends DOS-Style newlines with \r\n. snmp_exporter escapes the \n newline, but does not escape the \r carriage return, causing the output to overwrite itself.

@brian-brazil
Copy link
Contributor

False alarm then.

@localpref
Copy link
Author

localpref commented Jul 25, 2019 •
edited
Loading

Hmmm. The device sends
0d 0a (\r \n) (see tcpdump), but the /metrics output gives 0d 5c 6e (\r \\n), i.e. an escaped newline. I think that is actually a problem - the exporter should escape both control characters or none.

tcpdump / on the wire
0x00d0:  703a 2f2f 7777 772e 6369 7363 6f2e 636f  p://www.cisco.co
0x00e0:  6d2f 7465 6368 7375 7070 6f72 740d 0a43  m/techsupport..C
                                         _____

output of exporter http /snmp
00000450  69 73 63 6f 2e 63 6f 6d  2f 74 65 63 68 73 75 70  |isco.com/techsup|
00000460  70 6f 72 74 0d 5c 6e 43  6f 70 79 72 69 67 68 74  |port.\nCopyright|
                      ________

@brian-brazil
Copy link
Contributor

That's the correct encoding for a /metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brian-brazil @localpref