Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement randomised round-trip testing #453

Open
psychon opened this issue May 27, 2020 · 2 comments
Open

Implement randomised round-trip testing #453

psychon opened this issue May 27, 2020 · 2 comments
Labels
P5 Priority Nice-to-have

Comments

@psychon
Copy link
Owner

psychon commented May 27, 2020

Something like proptest or quickcheck could be used to generate random instances of "something". This random instance could then be serialised and parsed again. If "something" implements PartialEq, one could then check that "everything" worked okay.

See #452 (comment) and #184 (which is about fuzzing, which is arguably quite close to this)
@psychon psychon added the P5 Priority Nice-to-have label May 27, 2020
This was referenced May 27, 2020
Merged
Open
@psychon
Copy link
Owner Author

psychon commented May 28, 2020

quickcheck does not support generating random arrays: BurntSushi/quickcheck#226 BurntSushi/quickcheck#103 BurntSushi/quickcheck#92

proptest does not support structs with more than 10 members (?): proptest-rs/proptest#197
(There are 31 cryptic compiler errors, I only looked at one of them in detail)

psychon added a commit that referenced this issue Jul 23, 2023
@psychon
Add GenerateRandom trait to x11rb-protocol
a04dda9 
We want to have randomised round-trip testing. An instance of a struct
is generated randomly, then serialised, parsed and the result should be
equivalent to the original structure. Existing crates for this seem not
to support our use case. See [0] for details.

Thus, this commit starts work on our own implementation of this. It
begins by adding a GenerateRandom to x11rb-protocol that allows to
create a random instance of a type based on a fastrand random number
generator.

This commit also implements this trait for primitive types. Numbers are
limited to at most 4 since they could be sizes and we have nesting. For
example, if a list contains another list which contains another list,
with a limit of 4 we can already end up with 4*4*4 = 64 items. To avoid
generating huge instances, this low limit is used.

[0]: #453

Signed-off-by: Uli Schlachter <psychon@znc.in>
@psychon
Copy link
Owner Author

psychon commented Jul 23, 2023

I gave this another try in the branch generate_random. The approach is basically the same as in #737, but limiting all numbers to < 4 so that lengths do not get too large and no overflows occurs. I implemented two tests:

  • serialize() and serialize_into() produce the same result.
  • serialize() + try_parse() returns the original object again.

The second test is given me lots of problems, but the first one also does not work.

  • Events have a length field that is not really used for anything, but its value must still be large enough to cover all the input
  • Enums can have larger value than on the wire. For example, XKB's GetNameIndicatorReply contains a xproto::ModMask (which we represent as u16 in-memory), but this field only gets 8 bits on the wire. Thus, the test promptly fails because ANY turns into 0.
  • List lengths can have side conditions. For example, xproto's GetModifierMapping has a list whose length is a field's value times 8. The factor 8 is ignored when generating instances of this struct and then serialising fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P5 Priority Nice-to-have
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@psychon