Conntrack XML parsing is consuming a lot of CPU #1991

2opremio · 2016-11-07T12:47:48Z

See #1985 (comment)

2opremio · 2016-11-07T13:00:48Z

@alban Suggests replacing the conntrack command (and in turn parsing its XML output) by talking to the kernel directly through a Netlink socket.

2opremio · 2016-11-09T11:47:40Z

Found https://github.com/typetypetype/conntrack but the status is not really reassuring ...

2opremio · 2016-11-10T14:03:59Z

Using netlink is going to be a bit more work than I originally expected due to the incomplete Golang support of NETLINK_NETFILTER and my almost total ignorace about netlink until yesterday.

https://github.com/vishvananda/netlink doens't support Conntrack directly (I created Add Conntrack support (NETLINK_NETFILTER protocol) vishvananda/netlink#171)
https://github.com/typetypetype/conntrack/ is in dubious state (I tested it and seems to work though) and doesn't fully support flows either. There's Follow() which reads flows into Conn. However Conn lacks the Orig side of the flow (which we need) and there is no simple or public way to send commands to netfilter (which we may need to implement conntrack --any-nat, currently used by Scope).

My plan is to fork https://github.com/typetypetype/conntrack/ to reuse all the parsing but it may be more sensible to bite the bullet and implement vishvananda/netlink#171 (which I think would take a much longer time). @awh Thoughts?

2opremio · 2016-11-15T10:29:11Z

After an offline discussion with @awh I've confirmed that supporting a new netlink protocol is a non-trivial task and that we should instead try improving parsing first. I will proceed in the following order, reevaluating performance after each step:

Try to find a better performing XML library
Try parsing the textual output of conntrack (i.e. without -o xml)
Use netlink

2opremio · 2016-11-15T12:52:56Z

Try to find a better performing XML library

I have failed to find an alternative XML library at all. Moving on to parse the conntrack text manually.

tomwilkie · 2016-11-15T13:05:32Z

Huh, I thought codec did it, but it doesn't.

https://godoc.org/github.com/ugorji/go/codec

On Tue, 15 Nov 2016 at 12:53, Alfonso Acosta notifications@github.com
wrote:

Try to find a better performing XML library

I have failed to find an alternative XML library at all. Moving on to
parse the conntrack text manually.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1991 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAbGhQ1NpJUVmd3I1U4XyyBHu7u5WPDsks5q-aspgaJpZM4KrLIo
.

2opremio · 2016-12-15T12:35:09Z

Parsing the line-based output of conntrack is a pain in the ass because some (at least 4) fields are optional. This means I cannot simply use Scanf.

I started coding a manual parser for it, but I am not sure it's worth the effort compared to spending time on https://github.com/vishvananda/netlink

tomwilkie · 2016-12-15T12:41:16Z

Does https://github.com/lucsky/go-exml help? Do you know what bit of the xml parsing is the overhead?

…

On Thu, Dec 15, 2016 at 12:35 PM, Alfonso Acosta ***@***.***> wrote: Parsing the line-based output of conntrack is a pain in the ass because some (at least 4) fields are optional. This means I cannot simply use Scanf. I started coding a manual parser for it, but I am not sure it's worth the effort compared to spending time on https://github.com/vishvananda/netlink — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1991 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhU18Q3lWAs1rPOkDdknkxZs60PxPks5rITP-gaJpZM4KrLIo> .

rade · 2016-12-15T12:43:10Z

Do you know what bit of the xml parsing is the overhead?

IIRC string allocation/gc is the issue. Due to golang's lack of string interning / symbols, or a proper generational GC.

tomwilkie · 2016-12-15T13:11:45Z

If that is the case, you could parse the IPs straight into unit32s (or net.IPs), and the Proto, Direction and State into a enum etc

…

On Thu, Dec 15, 2016 at 12:43 PM, Matthias Radestock < ***@***.***> wrote: Do you know what bit of the xml parsing is the overhead? IIRC string allocation/gc is the issue. Due to golang's lack of interns or a proper generational GC. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1991 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhYoIbW2w4x1VwNx_0H7mJEE3Lsjaks5rITXegaJpZM4KrLIo> .

tomwilkie · 2016-12-15T13:12:20Z

Note that in previous cases of encoding / decoding overhead, its been the reflection thats the problem, not allocations (cf report structure).

…

On Thu, Dec 15, 2016 at 1:11 PM, Tom Wilkie ***@***.***> wrote: If that is the case, you could parse the IPs straight into unit32s (or net.IPs), and the Proto, Direction and State into a enum etc On Thu, Dec 15, 2016 at 12:43 PM, Matthias Radestock < ***@***.***> wrote: > Do you know what bit of the xml parsing is the overhead? > > IIRC string allocation/gc is the issue. Due to golang's lack of interns > or a proper generational GC. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#1991 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAbGhYoIbW2w4x1VwNx_0H7mJEE3Lsjaks5rITXegaJpZM4KrLIo> > . >

rade · 2016-12-15T13:16:45Z

As I understand it, for the xml parsing the string allocation/gc happens mainly due to tags and attribute keys, e.g. if you have a document containing 500 <meta> tags, that's 500 allocations, and 500 items of garbage, not one.

tomwilkie · 2016-12-15T13:30:08Z

How dumb: https://golang.org/src/encoding/xml/xml.go?s=6817:6857#L1146

…

On Thu, Dec 15, 2016 at 1:16 PM, Matthias Radestock < ***@***.***> wrote: As I understand it, for the xml parsing the string allocation/gc happens mainly due to tags and attribute keys, e.g. if you have a document containing 500 <meta> tags, that's 500 allocations, and 500 items of garbage, not one. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1991 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhf_UqOIgCYYkkyWessfFFa77aJsuks5rIT2-gaJpZM4KrLIo> .

2opremio · 2016-12-15T13:36:27Z

I could create a parser with something like https://github.com/prataprc/goparsec

Maybe I could also give https://github.com/moovweb/gokogiri a try

2opremio · 2016-12-15T13:37:49Z

Does https://github.com/lucsky/go-exml help?

It doesn't promise any performance improvements plus sits on top of a standard Go encoding/xml/Decoder so I am doubtful

2opremio added the performance Excessive resource usage and latency; usually a bug or chore label Nov 7, 2016

2opremio added this to the re:Invent milestone Nov 7, 2016

rade added the chore Related to fix/refinement/improvement of end user or new/existing developer functionality label Nov 7, 2016

2opremio mentioned this issue Nov 7, 2016

Replace conntrack by ebpf #1992

Closed

2opremio mentioned this issue Nov 7, 2016

Release 1.0.0 #1985

Merged

17 tasks

2opremio mentioned this issue Nov 10, 2016

Add Conntrack support (NETLINK_NETFILTER protocol) vishvananda/netlink#171

Open

2opremio self-assigned this Nov 10, 2016

2opremio modified the milestones: December2016, EOY 2016 Dec 13, 2016

2opremio mentioned this issue Dec 15, 2016

Disable XML in conntrack parsing #2095

Merged

2opremio closed this as completed Dec 22, 2016

2opremio mentioned this issue Jan 16, 2017

Conntrack netlink #2130

Closed

leodotcloud mentioned this issue Apr 3, 2017

ipsec/vxlan: library for native conntrack support rancher/rancher#8398

Closed

2opremio mentioned this issue May 3, 2017

Use Netlink to read conntrack flows #2495

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conntrack XML parsing is consuming a lot of CPU #1991

Conntrack XML parsing is consuming a lot of CPU #1991

2opremio commented Nov 7, 2016

2opremio commented Nov 7, 2016 •

edited

Loading

2opremio commented Nov 9, 2016

2opremio commented Nov 10, 2016 •

edited

Loading

2opremio commented Nov 15, 2016 •

edited

Loading

2opremio commented Nov 15, 2016

tomwilkie commented Nov 15, 2016

2opremio commented Dec 15, 2016

tomwilkie commented Dec 15, 2016 via email

rade commented Dec 15, 2016 •

edited

Loading

tomwilkie commented Dec 15, 2016 via email

tomwilkie commented Dec 15, 2016 via email

rade commented Dec 15, 2016

tomwilkie commented Dec 15, 2016 via email

2opremio commented Dec 15, 2016

2opremio commented Dec 15, 2016

Conntrack XML parsing is consuming a lot of CPU #1991

Conntrack XML parsing is consuming a lot of CPU #1991

Comments

2opremio commented Nov 7, 2016

2opremio commented Nov 7, 2016 • edited Loading

2opremio commented Nov 9, 2016

2opremio commented Nov 10, 2016 • edited Loading

2opremio commented Nov 15, 2016 • edited Loading

2opremio commented Nov 15, 2016

tomwilkie commented Nov 15, 2016

2opremio commented Dec 15, 2016

tomwilkie commented Dec 15, 2016 via email

rade commented Dec 15, 2016 • edited Loading

tomwilkie commented Dec 15, 2016 via email

tomwilkie commented Dec 15, 2016 via email

rade commented Dec 15, 2016

tomwilkie commented Dec 15, 2016 via email

2opremio commented Dec 15, 2016

2opremio commented Dec 15, 2016

2opremio commented Nov 7, 2016 •

edited

Loading

2opremio commented Nov 10, 2016 •

edited

Loading

2opremio commented Nov 15, 2016 •

edited

Loading

rade commented Dec 15, 2016 •

edited

Loading