[Hinty] Add type hints to Scapy

[gpotter2]

Project "Hinty" aims at adding Type hints to Scapy. It will help discover bugs, improve the API, and make Scapy up-to-date with the high standards of Python libraries.

Implementation

We use mypy to ensure automatic testing of the work that has already been completed. PRs that fall under project Hinty will process one (or a few) files and register them into the checks. The file .config/mypy/mypy_enabled.txt lists the files in which mypy checks are enabled. This process has been added by #2162

Because we support Python 2.7, the format of the Type hints should be the following:
https://mypy.readthedocs.io/en/latest/cheat_sheet.html

Guide on how to contribute

0. 💿 Make sure you've "git clonned" the latest version of scapy and are working from within it !

1. ✏️ Pick a file you want to work on. If you're new on scapy, pick a file from within the layers/ or contrib/ folder that is small in size and complexity: at this point in time, the remaining layers in Scapy's core are often hard to type.

2. ✏️ Add that file to .config/mypy/mypy_enabled.txt. This will enable it in our test suite.

3. ➡️ pip install pyannotate cryptography tox mock (install tox, pyannotate and cryptography)

4. ➡️ Run the tests for your file: tox --sitepackages -- -x -K tcpdump -K manufdb -K wireshark -K tshark -K brotli -K zstd -K ci_only -K not_pyannotate -N
   You shall specify a -e py[38]-[linux]_non_root (fill the brackets with your OS infos) option before the --sitepackages option if you have multiple versions of python installed. To see all available tox tags (the whole py...non_root string), type tox -l. The -x option tells our test suite to use pyannotate.
   This will create test/pyannotate_results

   • ⚠️ ✏️ In many cases, pyannotate will glitch out (ParseError: Invalid type comment: (__main__:f.<locals>.A) -> None dropbox/pyannotate#92). You need to open pyannotate_results with your favorite editor and replace all occurrences of <locals> (with the <>) by blanks (or anything really). For instance on vim: :%s/<locals>//g

5. ➡️ pyannotate --type-info test/pyannotate_results -w [the file.py you are working on]

6. ✏️ The file has been automatically processed. Now, edit it to fix the mistakes, and check your work with tox -e mypy. This might take time

   • ℹ️ Remember that you can use cast(type, obj) from typing to help mypy without affecting the code.
   • ⚠️ Import any types you might need from scapy.compat instead of typing. It provides a fallback if typing isn't installed (python < 3.5). I know it's annoying.
   • ℹ️ If a class extends Field directly (very few do) it will require to be passed the internal and machine representation of the field. See this if you don't know what I'm talking about. For instance: class subField(Field[float, int]): is a field where i=float and m=int. In most cases, it should be [int, int] (a normal integer field).

7. ⚠️ If you get error: No library stub file for module ..., add an exception for it in .config/mypy/mypy.ini (follow the existing format)

8. ✅ Check that the files still pass PEP8: tox -e flake8

9. 🥇 When everything passes, submit a PR. 😄

Active hinty related issues

• Discuss strict equality in Mypy [Hinty] Strict equality #2823

Documentation for advanced typing (that I always lose the links for..)

• Generics
• Methods overloading

Hinty status

MyPy Support: 25.44%

Module	Typed code (lines)	Typed files
[core]	100.00%	33/33
arch	100.00%	13/13
layers	3.06%	2/92
contrib	14.28%	27/167
libs	23.30%	6/8
asn1	100.00%	4/4
tools	11.08%	1/9
modules	0.00%	0/8

[gpotter2]

OSX is based on BSD 👍 there is a bsd.uts

[luizbarcelos]

Wanted to start helping on this issue, it's been a while since it was last updated though. Should I just follow the tutorial @gpotter2 ? Perhaps there is a higher priority issue for beginners on this repo, please let me know.

[gpotter2]

@luizbarcelos Hi and thanks for your interest. I try to keep this page updated so feel free to help

[Davidy22]

Hello, just grabbed the repo to take a shot at this and I opened up automaton.py to start working, then opened up ansmachine.py to see some examples and I noticed the way that type hinting is being done currently, with

    def __init__(self, **kargs):
        # type: (Any) -> None

While the type hints that I was aware of look something like:

    def __init__(self, **kargs: Any) -> None:

I'm wriitng automaton.py in the style that I'm seeing in ansmachine.py right now, but is there a reason why it's written like this instead of in PEP 484 style?

[polybassa]

Hi, thanks for your help. It's written in comments to support older versions of python, aswell.

…

On October 5, 2021 3:20:17 PM GMT+02:00, David Yang ***@***.***> wrote: Hello, just grabbed the repo to take a shot at this and I opened up automaton.py to start working, then opened up ansmachine.py to see some examples and I noticed the way that type hinting is being done currently, with ``` def __init__(self, **kargs): # type: (Any) -> None ``` While the type hints that I was aware of look something like: ``` def __init__(self, **kargs: Any) -> None: ``` I'm wriitng automaton.py in the style that I'm seeing in ansmachine.py right now, but is there a reason why it's written like this? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #2158 (comment)

[guedou]

With v2.5.0 being closer, can this issue be closed?

[gpotter2]

Technically there's still arch/bpf. I was secretly thinking I could finish it before 2.5.0 but it's not blocking at all.
I'll close it once it's done

[KhazAkar]

Hi,
Since there's plan to drop python 2.x support after 2.5.0, will functions etc get signatures not in comment-style, but directly in the future?

[gpotter2]

Yes ! that's planned

[rohanvp]

I am currently working on project hinty on file scapy/contrib/carp.py . The mypy documentation says it no longer supports python2 versions. The link https://mypy.readthedocs.io/en/latest/cheat_sheet.html from the page #2158 is going to non-existent page.

What is my way forward?

Secondly, is the following a correct process to contribute to hinty?

I have picked a file that I am working on (scapy/contrib/carp.py) . I run that file using "mypy scapy/contrib/carp.py .
I get multiple errors(e.g. [name-defined] , [attr-defined] etc). I start resolving those errors one by one.
Once there are no errors I continue with step 2 mentioned in #2158 .
Please advise if this is the correct way.

[gpotter2]

Typing of the core is now fully achieved. I'd consider this 'completed'.

With Python 2.7 being dropped, this issue is also now obsolete. Closing