MyPy CI failures #1488

yoonthegoon · 2023-07-24T01:26:48Z

File: theHarvester/lib/stash.py
- Line: 197 - Error: Incompatible types in assignment (expression has type "Iterable[Row]", variable has type "list[Any]") [assignment]
- Line: 218 - Error: Value of type "Optional[Row]" is not indexable [index]
- Line: 223 - Error: Incompatible types in assignment (expression has type "Iterable[Row]", variable has type "list[Any]") [assignment]
- Line: 328 - Error: Incompatible types in assignment (expression has type "Iterable[Row]", variable has type "list[Any]") [assignment]
File: theHarvester/screenshot/screenshot.py
- Line: 56 - Error: Value of type "Sized" is not indexable [index]
File: theHarvester/lib/core.py
- Line: 112 - Error: Incompatible return value type (got "tuple[Any, Any]", expected "str") [return-value]
- Line: 290 - Error: Argument "headers" to "ClientSession" has incompatible type "Sized"; expected "Union[Mapping[Union[str, istr], str], CIMultiDict[str], CIMultiDictProxy[str], None]" [arg-type]
- Line: 303 - Error: Argument "headers" to "ClientSession" has incompatible type "Sized"; expected "Union[Mapping[Union[str, istr], str], CIMultiDict[str], CIMultiDictProxy[str], None]" [arg-type]
- Line: 316 - Error: Argument "headers" to "ClientSession" has incompatible type "Sized"; expected "Union[Mapping[Union[str, istr], str], CIMultiDict[str], CIMultiDictProxy[str], None]" [arg-type]
- Line: 325 - Error: Argument "headers" to "ClientSession" has incompatible type "Sized"; expected "Union[Mapping[Union[str, istr], str], CIMultiDict[str], CIMultiDictProxy[str], None]" [arg-type]
- Line: 445 - Error: Argument "headers" to "ClientSession" has incompatible type "Sized"; expected "Union[Mapping[Union[str, istr], str], CIMultiDict[str], CIMultiDictProxy[str], None]" [arg-type]
- Line: 465 - Error: Argument "headers" to "ClientSession" has incompatible type "Sized"; expected "Union[Mapping[Union[str, istr], str], CIMultiDict[str], CIMultiDictProxy[str], None]" [arg-type]
- Line: 488 - Error: Argument "headers" to "ClientSession" has incompatible type "Sized"; expected "Union[Mapping[Union[str, istr], str], CIMultiDict[str], CIMultiDictProxy[str], None]" [arg-type]
- Line: 496 - Error: Argument 3 to "fetch" of "AsyncFetcher" has incompatible type "Sized"; expected "str" [arg-type]
File: theHarvester/discovery/takeover.py
- Line: 15 - Error: Need type annotation for "fingerprints" (hint: "fingerprints: Dict[, ] = ...") [var-annotated]
- Line: 17 - Error: Need type annotation for "results" [var-annotated]
- Line: 99 - Error: Incompatible types in assignment (expression has type "list[Any]", variable has type "tuple[Any, ...]") [assignment]
File: theHarvester/discovery/dnsdumpster.py
- Line: 28 - Error: Incompatible types in assignment (expression has type "list[str]", variable has type "str") [assignment]
- Line: 33 - Error: Incompatible types in assignment (expression has type "list[str]", variable has type "str") [assignment]
File: theHarvester/discovery/constants.py
- Line: 92 - Error: Argument "data" to "post_fetch" of "AsyncFetcher" has incompatible type "dict[str, str]"; expected "str" [arg-type]
File: theHarvester/discovery/zoomeyesearch.py
- Line: 58 - Error: Incompatible types in assignment (expression has type "Pattern[str]", variable has type "str") [assignment]
File: theHarvester/discovery/securitytrailssearch.py
- Line: 57 - Error: Incompatible types in assignment (expression has type "Union[list[Any], tuple[Any, ...]]", variable has type "tuple[]") [assignment]
- Line: 62 - Error: Tuple index out of range [misc]
- Line: 65 - Error: Tuple index out of range [misc]
File: theHarvester/discovery/intelxsearch.py
- Line: 63 - Error: Incompatible types in assignment (expression has type "tuple[Any, ...]", variable has type "None") [assignment]
- Line: 71 - Error: Incompatible types in assignment (expression has type "tuple[Any, ...]", variable has type "tuple[]") [assignment]
- Line: 71 - Error: Argument 1 to "parse_dictionaries" of "Parser" has incompatible type "None"; expected "dict[Any, Any]" [arg-type]
File: theHarvester/discovery/githubcode.py
- Line: 111 - Error: Incompatible return value type (got "Optional[Any]", expected "int") [return-value]
File: theHarvester/discovery/bingsearch.py
- Line: 51 - Error: Incompatible types in assignment (expression has type "tuple[Any, ...]", variable has type "str") [assignment]
File: theHarvester/discovery/censysearch.py
- Line: 8 - Error: Module "censys.search" has no attribute "CensysCertificates"; maybe "CensysCerts"? [attr-defined]

yoonthegoon · 2023-07-24T01:32:55Z

I need to take a look at page_from_response later.
My brain tells me that last_page should never be None, but I'd need to dig through the logic some more.
If anyone has any input, please let me know.

I run pytest and everything is passing locally.
If there are any other tests I need to run, I didn't 🤷

yoonthegoon · 2023-07-24T01:36:21Z

Also damn, I tried to do a quiet commit for formatting. You can review with shift + click over the other commits when reviewing though.

L1ghtn1ng · 2023-07-24T02:07:20Z

I need to take a look at page_from_response later.

My brain tells me that last_page should never be None, but I'd need to dig through the logic some more.

If anyone has any input, please let me know.

I run pytest and everything is passing locally.

If there are any other tests I need to run, I didn't 🤷

As far as I understand it that is to know when to stop iterating, I could be wrong on that but would need Matt to chime in.

L1ghtn1ng · 2023-07-24T02:10:27Z

With regards to more checks to run, can you fix the flake8 ci failures from this PR please?

yoonthegoon · 2023-07-24T02:13:31Z

I think it might be appropriate to also add E203, W503 to .flake8 ignore.
They are both whitespace before ':' and line break before binary operator respectively.
Both of those make for more readable code when slicing with an expression and long binary sequences like in some conditionals.

theHarvester/lib/core.py

L1ghtn1ng · 2023-07-24T02:19:50Z

Yeah sounds good 👍 On 24 Jul 2023, at 03:13, yoon ***@***.***> wrote: I think it might be appropriate to also add E203, W503 to .flake8 ignore. They are both whitespace before ':' and line break before binary operator respectively. Both of those make for more readable code when slicing with an expression and long binary sequences like in some conditionals. — Reply to this email directly, view it on GitHub<#1488 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA3V2QRBPHQYPKE43ZSQUUDXRXK5NANCNFSM6AAAAAA2U4C6XY>. You are receiving this because you commented.Message ID: ***@***.***>

yoonthegoon · 2023-07-24T02:27:21Z

I wouldn't be too worried about running these workflows yet; I still haven't fixed theHarvester/discovery/githubcode.py.
I'll take a look at that a bit later.

yoonthegoon · 2023-07-24T02:28:51Z

theHarvester/__main__.py

@@ -848,7 +848,6 @@ async def handler(lst):
        print('\n[*] Virtual hosts:')
        print('------------------')
        for data in host_ip:
-            from theHarvester.discovery import bingsearch


from theHarvester.discovery import * already exists at the top of the file.

Some times it's needed as it does not actually work without that, if you tested it and it still works then am happy for it to be removed

I haven't tested it out myself after making this change yet, only ran pytest after. I'll give it a shot tomorrow. There's still more I've got to go through it looks like.

LMK if you got a command I can run that results in some vhosts. I tried a few stuff with keeping and removing the import, but both resulsts didn't have any vhosts to show. I can experiment with harvesting more a bit later.

theHarvester/lib/core.py

yoonthegoon · 2023-07-24T02:41:03Z

Looks like the workflow isn't pointing to .flake8.

░▒▓    ~/repos/theHarvester    MyPy-CI-failures *1 ··········· theHarvester   22:39:19  ▓▒░
❯ flake8 --config .flake8 theHarvester                    

░▒▓    ~/repos/theHarvester    MyPy-CI-failures *1 ··········· theHarvester   22:39:23  ▓▒░
❯ flake8 theHarvester                                     
theHarvester/discovery/constants.py:47:13: W503 line break before binary operator
theHarvester/discovery/constants.py:48:13: W503 line break before binary operator
theHarvester/discovery/constants.py:69:13: W503 line break before binary operator
theHarvester/discovery/constants.py:70:13: W503 line break before binary operator
theHarvester/discovery/constants.py:106:9: W503 line break before binary operator
theHarvester/discovery/constants.py:107:9: W503 line break before binary operator
theHarvester/discovery/takeover.py:34:21: W503 line break before binary operator
theHarvester/discovery/zoomeyesearch.py:171:21: W503 line break before binary operator
theHarvester/discovery/zoomeyesearch.py:172:21: W503 line break before binary operator
theHarvester/discovery/zoomeyesearch.py:173:21: W503 line break before binary operator
theHarvester/discovery/zoomeyesearch.py:174:21: W503 line break before binary operator
theHarvester/screenshot/screenshot.py:57:26: E203 whitespace before ':'

L1ghtn1ng · 2023-07-24T02:47:05Z

Looks like the workflow isn't pointing to .flake8.


░▒▓    ~/repos/theHarvester    MyPy-CI-failures *1 ··········· theHarvester   22:39:19  ▓▒░

❯ flake8 --config .flake8 theHarvester                    



░▒▓    ~/repos/theHarvester    MyPy-CI-failures *1 ··········· theHarvester   22:39:23  ▓▒░

❯ flake8 theHarvester                                     

theHarvester/discovery/constants.py:47:13: W503 line break before binary operator

theHarvester/discovery/constants.py:48:13: W503 line break before binary operator

theHarvester/discovery/constants.py:69:13: W503 line break before binary operator

theHarvester/discovery/constants.py:70:13: W503 line break before binary operator

theHarvester/discovery/constants.py:106:9: W503 line break before binary operator

theHarvester/discovery/constants.py:107:9: W503 line break before binary operator

theHarvester/discovery/takeover.py:34:21: W503 line break before binary operator

theHarvester/discovery/zoomeyesearch.py:171:21: W503 line break before binary operator

theHarvester/discovery/zoomeyesearch.py:172:21: W503 line break before binary operator

theHarvester/discovery/zoomeyesearch.py:173:21: W503 line break before binary operator

theHarvester/discovery/zoomeyesearch.py:174:21: W503 line break before binary operator

theHarvester/screenshot/screenshot.py:57:26: E203 whitespace before ':'

Ah can you update it please and give it the --config flag as that is what is missing, thanks for pointing that out

yoonthegoon · 2023-07-24T16:59:55Z

It looks like expected behavior of method next_page_or_end is to return None if next_page and last_page are both None.
Therefore, I will have to change the return type of the method, and the type of the variables that call it.

yoonthegoon · 2023-07-24T17:08:50Z

❯ flake8 --config .flake8 theHarvester

❯ mypy --pretty theHarvester/*/*.py              
Success: no issues found in 51 source files

❯ pytest                                         
====================================== test session starts ======================================
platform darwin -- Python 3.9.6, pytest-7.4.0, pluggy-1.2.0
rootdir: /Users/yunis/repos/theHarvester
configfile: pytest.ini
testpaths: tests
plugins: asyncio-0.21.1, anyio-3.7.1, typeguard-2.13.3
asyncio: mode=auto
collected 16 items                                                                              

tests/test_myparser.py .                                                                  [  6%]
tests/discovery/test_anubis.py ...                                                        [ 25%]
tests/discovery/test_certspotter.py ...                                                   [ 43%]
tests/discovery/test_githubcode.py .....                                                  [ 75%]
tests/discovery/test_otx.py ..                                                            [ 87%]
tests/discovery/test_threatminer.py ..                                                    [100%]

====================================== 16 passed in 46.83s ======================================

Hopefully all checks pass this time around.
I have just once concern about removing import shadowed by loop variable.
When I run python3 theHarvester.py -v -d google.com -b all, my output is

...
zxkitk.feedproxy.ghs.google.com:142.250.204.83
zyy53f.feedproxy.ghs.google.com

[*] Virtual hosts:
------------------

❯

both with and without the looped import.
It's great that it doesn't give me a NameError from removing the import.
However, I'd like to ensure that a source and domain that result in virtual hosts also result in those same virtual hosts with the looped import removed.
Otherwise, LGTM 🤷

L1ghtn1ng · 2023-07-24T17:13:32Z

Give apple.com and or Microsoft.com On 24 Jul 2023, at 18:09, yoon ***@***.***> wrote: ❯ flake8 --config .flake8 theHarvester ❯ mypy --pretty theHarvester/*/*.py Success: no issues found in 51 source files ❯ pytest ====================================== test session starts ====================================== platform darwin -- Python 3.9.6, pytest-7.4.0, pluggy-1.2.0 rootdir: /Users/yunis/repos/theHarvester configfile: pytest.ini testpaths: tests plugins: asyncio-0.21.1, anyio-3.7.1, typeguard-2.13.3 asyncio: mode=auto collected 16 items tests/test_myparser.py . [ 6%] tests/discovery/test_anubis.py ... [ 25%] tests/discovery/test_certspotter.py ... [ 43%] tests/discovery/test_githubcode.py ..... [ 75%] tests/discovery/test_otx.py .. [ 87%] tests/discovery/test_threatminer.py .. [100%] ====================================== 16 passed in 46.83s ====================================== Hopefully all checks pass this time around. I have just once concern about removing import shadowed by loop variable<https://github.com/laramies/theHarvester/pull/1488/commits/8f7dd9d75c09fc1a09cd21b6e69f64e5eec4e1ef>](https://github.com/laramies/theHarvester/pull/1488/files#diff-e026ff479beedd53324b4e852f868c2a3ac337d2185a49c4346168cb0f472079L851). When I run python3 theHarvester.py -v -d google.com -b all, my output is ... zxkitk.feedproxy.ghs.google.com:142.250.204.83 zyy53f.feedproxy.ghs.google.com [*] Virtual hosts: ------------------ both with and without the looped import. It's great that it doesn't give me a NameError from removing the import. However, I'd like to ensure that a source and domain that result in virtual hosts also result in those same virtual hosts with the looped import removed. Otherwise, LGTM 🤷 — Reply to this email directly, view it on GitHub<#1488 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA3V2QVYDIAZUZGZ5VVMRSLXR2T2ZANCNFSM6AAAAAA2U4C6XY>. You are receiving this because you commented.Message ID: ***@***.***>

yoonthegoon · 2023-07-24T17:35:20Z

No luck with either domain.
It should be fine as bingsearch is brought into the namespace here.
Everything is passing locally and we'll have to see how the workflows look.
I think the near future, I'll probably write an issue to clean up __main__.py and probably the rest of the repo to properly order the imports and to not use import * as that can cause confusion as to where certain modules are coming from.

yoonthegoon · 2023-07-24T17:43:29Z

I see #1341 but I cannot reproduce the reference before assignment.
I imagine this may also be something cause by having imports throughout the script in conditionals rather than all at the top of the file.
This goes along with my last comment about cleaning this up.
Practically all these modules are already being imported and brought into the namespace by import *, so a refactor to have explicit imports of all of these modules would be better and less likely to cause these issues.

L1ghtn1ng · 2023-07-24T19:35:15Z

Yeah sounds good to me ☺️ major big thanks for this very much appreciated you fixing these issues

yoonthegoon added 16 commits July 23, 2023 19:37

black and isort formatting

44c7b12

laramies#1383 fix stash.py

26a1fa6

laramies#1383 laramies#1486 fix screenshot.py

684c7c2

laramies#1383 fix censysearch.py

081ce57

laramies#1383 fix bingsearch.py

0d0a9f0

laramies#1383 fix core.py

31bacdf

laramies#1383 fix takeover.py

fef8396

laramies#1383 fix takeover.py

25fe4ab

laramies#1383 improved readability

805a502

laramies#1383 fix dnsdumpster.py

2351c88

laramies#1383 fix constants.py

7475924

laramies#1383 laramies#1487 fix zoomeyesearch.py

2579d60

laramies#1383 fix securitytrailssearch.py

5db041c

laramies#1383 fix intelxsearch.py

0706579

laramies#1383 assign self.info typing

0b371c5

laramies#1383 need figure out page_from_response method

b9af76f

github-advanced-security bot found potential problems Jul 24, 2023

View reviewed changes

theHarvester/lib/core.py Fixed Show resolved Hide resolved

add ignore E203 and W503

0ac1eda

yoonthegoon added 2 commits July 23, 2023 22:23

Clear-text logging of sensitive information

8d61aef

remove import shadowed by loop variable

8f7dd9d

yoonthegoon commented Jul 24, 2023

View reviewed changes

github-advanced-security bot found potential problems Jul 24, 2023

View reviewed changes

theHarvester/lib/core.py Fixed Show resolved Hide resolved

yoonthegoon added 3 commits July 23, 2023 22:56

add flake8 config flag

b7aba8e

laramies#1383 fix githubcode.py

8468e47

Clear-text logging of sensitive information

4d0f9a5

laramies#1383 fix __main__.py

3aa25f6

L1ghtn1ng merged commit 4f0e3bc into laramies:master Jul 25, 2023

yoonthegoon deleted the MyPy-CI-failures branch July 25, 2023 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MyPy CI failures #1488

MyPy CI failures #1488

yoonthegoon commented Jul 24, 2023 •

edited

Loading

yoonthegoon commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023 via email

yoonthegoon commented Jul 24, 2023

yoonthegoon Jul 24, 2023

L1ghtn1ng Jul 24, 2023

yoonthegoon Jul 24, 2023

yoonthegoon Jul 24, 2023

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023 •

edited

Loading

L1ghtn1ng commented Jul 24, 2023 via email

yoonthegoon commented Jul 24, 2023 •

edited

Loading

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

MyPy CI failures #1488

MyPy CI failures #1488

Conversation

yoonthegoon commented Jul 24, 2023 • edited Loading

yoonthegoon commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023 via email

yoonthegoon commented Jul 24, 2023

yoonthegoon Jul 24, 2023

Choose a reason for hiding this comment

L1ghtn1ng Jul 24, 2023

Choose a reason for hiding this comment

yoonthegoon Jul 24, 2023

Choose a reason for hiding this comment

yoonthegoon Jul 24, 2023

Choose a reason for hiding this comment

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023 • edited Loading

L1ghtn1ng commented Jul 24, 2023 via email

yoonthegoon commented Jul 24, 2023 • edited Loading

yoonthegoon commented Jul 24, 2023

L1ghtn1ng commented Jul 24, 2023

yoonthegoon commented Jul 24, 2023 •

edited

Loading

yoonthegoon commented Jul 24, 2023 •

edited

Loading

yoonthegoon commented Jul 24, 2023 •

edited

Loading