-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache fields initialization #642
Conversation
scapy/contrib/gtp.uts
Outdated
@@ -11,7 +11,7 @@ gtp.dport == 2123 and gtp.teid == 2807 and len(gtp.IE_list) == 5 | |||
|
|||
= GTPCreatePDPContextRequest(), basic dissection | |||
random.seed(0x2807) | |||
str(gtp) == b"E\x00\x00O\x00\x01\x00\x00@\x11|\x9b\x7f\x00\x00\x01\x7f\x00\x00\x01\x08K\x08K\x00;{N2\x10\x00+\x00\x00\n\xf7\xd2y\x00\x00\x10\xf8>\x14\x05\x14\t\x85\x00\x04\xa6A\xd8+\x85\x00\x04z\xafnt\x87\x00\x0fxKbPaePK9oq0pb5" | |||
str(gtp) == b'E\x00\x00O\x00\x01\x00\x00@\x11|\x9b\x7f\x00\x00\x01\x7f\x00\x00\x01\x08K\x08K\x00;\x97\xbd2\x10\x00+\x00\x00\n\xf7\xd2y\x00\x00\x10\xdeM\xb8\xf5\x14\x0f\x85\x00\x04\xabyk\xc1\x85\x00\x04\xb8\xcf\x96\xfe\x87\x00\x0f9Co27Fbj65eKHyQ' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the reason of this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A RandShort() is consumed
while the Packet fields are cached. Therefore, the randomly built packet is slightly different after the patch.
While debugging the HTTP2 issue @X-Cli found this corner case which is not a smaller version of the issue: $ cat test_642.py
from scapy.all import *
print '-----'
class SmallPacket(Packet):
name = 'Small Packet'
fields_desc = [ ByteField('byte', 0) ]
class TestPacket(Packet):
name = 'TestPacket'
fields_desc = [ PacketListField('list', [], SmallPacket) ]
a = TestPacket()
a.list.append(SmallPacket('a'))
a.list.append(SmallPacket('b'))
a.show()
print '-----'
TestPacket().show()
|
@guedou I'm gonna try to get this fixed |
Thanks ! I will have a look next week. I also have a fix that I did not push yet.
|
I have been working on this new patch with @X-Cli but forgot to push it ... It fixes the issue triggered by @gpotter2 & @p-l- I have several questions:
|
Codecov Report
@@ Coverage Diff @@
## master #642 +/- ##
========================================
Coverage ? 85.3%
========================================
Files ? 179
Lines ? 42300
Branches ? 0
========================================
Hits ? 36082
Misses ? 6218
Partials ? 0
|
@guedou: Can you rebase against current master? Also, can you be more specific about the changes to |
With the patch, Before:
After:
|
Hello, When someone fetches the internal representation of such a field and there is no user data for this field, then the returned value is a reference to that mutable object. Any alteration of that object leads to the corruption/alteration of the default value of that field for that Packet. @guedou's patch makes all instances of a given Packet type to share the same reference to a unique default value object. So, when one of the Packet instance returns the reference to that shared object, and the referenced object is mutated, the default value of all Packet of the same type are affected by this modification. This is what was happening in the earlier code excerpt that @guedou posted. To work around this, I submitted a patch to @guedou that deepcopies into self.fields the default value of a field upon a Packet type first instantiation, if that field internal representation of the default value is a mutable object. Unfortunately, this messes with |
Yeah, no. Your patch is similar to one of my early ideas to fix this issue. Unfortunately, it does not work. Returning a new mutable object upon |
@X-Cli (my bad for the fix) |
@X-Cli Ok, here's another possibility:
to keep the same repr behavior ? The triggered question then is the lost of performances... I guess that as |
Here is the updated version. I don't know what to do with |
We can simply remove it as defaults are hidden automatically. |
b8b2d12
to
985af22
Compare
So. Where are we here ? |
I need to find time to port it to Python3. |
test/regression.uts
Outdated
@@ -463,6 +464,9 @@ x = IP(dst="8.8.8.8")/fuzz(UDP()/NTP(version=4)) | |||
x.show2() | |||
x = IP(raw(x)) | |||
assert NTP in x | |||
======= | |||
str(fuzz(IP()/ICMP())) == '5\xe1\x00\x1c\x9dC@\x007\x01\xb7\xba\x7f\x00\x00\x01\x7f\x00\x00\x01*\xdb\xf7,9\x8e\xa4i' | |||
>>>>>>> Cache fields initialization |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
conflicts
I have done several changes to MultipleTypeField in #1485, will check if it fixes anything |
98e2b2d
to
df65e66
Compare
test/regression.uts
Outdated
p.hide_defaults() | ||
assert(repr(p) in ["<IP frag=0 proto=icmp |<ICMP |>>", "<IP frag=0 proto=1 |<ICMP |>>"]) | ||
assert(repr(p) == "<IP ttl=42 proto=icmp |<ICMP |>>") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So what does hide_defaults()
do now ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is broken. You are correct.
584598d
to
2c7d6cb
Compare
2c7d6cb
to
e905cb8
Compare
I made changes that will make codacity happy. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Amazing PR ! All good to me
e905cb8
to
de07974
Compare
Just restarted a failed test, thanks @guedou! |
This PR attempts to enhance Scapy performance by caching the initialization of packets fields. Currently, they are initialized at each packet instantiation.
The benchmarks look quite nice and indicate that we can gain between 15% and 35% execution time depending on the use case.
Here is the script used to bench this PR:
fixes #619