Dependency problem with pydns / pyspf

[lampyjon]

Fresh install on OS X:

$ python -V
Python 3.6.3
$ pip -V
pip 9.0.1 from /Users/user/test/lib/python3.6/site-packages (python 3.6)

$ pip install trustymail
...
Successfully built trustymail py3dns pyspf
Installing collected packages: dnspython, py3dns, pyspf, trustymail
Successfully installed dnspython-1.15.0 py3dns-3.1.0 pyspf-2.0.11 trustymail-0.3.0

$ trustymail dhs.gov
Traceback (most recent call last):
File "/Users/user/test/bin/trustymail", line 7, in
from trustymail.cli import main
File "/Users/user/test/lib/python3.6/site-packages/trustymail/cli.py", line 45, in
from trustymail import trustymail
File "/Users/user/test/lib/python3.6/site-packages/trustymail/trustymail.py", line 9, in
import spf
File "/Users/user/test/lib/python3.6/site-packages/spf.py", line 110, in
import DNS # http://pydns.sourceforge.net
ModuleNotFoundError: No module named 'DNS'

[jsf9k]

This is odd. I went through what I believe are the same steps as you on my (Linux) system without hitting the same error:

$ mkdir tmail_test
$ cd tmail_test/
$ python -m venv venv
$ source venv/bin/activate
(venv) $ python -V
Python 3.6.3
(venv) $ pip -V
pip 9.0.1 from /home/jeremy_frasier/tmail_test/venv/lib/python3.6/site-packages (python 3.6)
(venv) $ pip install trustymail
Collecting trustymail
  Using cached trustymail-0.3.0.tar.gz
Collecting requests (from trustymail)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting docopt (from trustymail)
Collecting publicsuffix (from trustymail)
Collecting dnspython (from trustymail)
  Using cached dnspython-1.15.0-py2.py3-none-any.whl
Collecting py3dns (from trustymail)
Collecting pyspf==2.0.11 (from trustymail)
Collecting idna<2.7,>=2.5 (from requests->trustymail)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests->trustymail)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->trustymail)
  Using cached certifi-2017.11.5-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->trustymail)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: idna, urllib3, certifi, chardet, requests, docopt, publicsuffix, dnspython, py3dns, pyspf, trustymail
  Running setup.py install for trustymail ... done
Successfully installed certifi-2017.11.5 chardet-3.0.4 dnspython-1.15.0 docopt-0.6.2 idna-2.6 publicsuffix-1.1.0 py3dns-3.1.0 pyspf-2.0.11 requests-2.18.4 trustymail-0.3.0 urllib3-1.22
(venv) $ trustymail dhs.gov
(venv) $ ls
results.csv  venv
(venv) $ less results.csv 
Domain,Base Domain,Live,MX Record,Mail Servers,Mail Server Ports Tested,Domain Supports SMTP,Domain Supports SMTP Results,Domain Supports STARTTLS,Domain Supports STARTTLS Results,SPF Record,Valid SPF,SPF Results,DMARC Record,Valid DMARC,DMARC Results,DMARC Record on Base Domain,Valid DMARC Record on Base Domain,DMARC Results on Base Domain,DMARC Policy,Syntax Errors,Errors
dhs.gov,dhs.gov,True,True,dhs-gov.mail.protection.outlook.com,"25, 587, 465",True,dhs-gov.mail.protection.outlook.com:25,True,dhs-gov.mail.protection.outlook.com:25,True,True,v=spf1 ip4:216.81.91.184 ip4:216.81.85.157 include:spf.protection.outlook.com -all,True,True,v=DMARC1; p=none; pct=100; rua=mailto:DMARC@hq.dhs.gov,,,,none,,"timed out, timed out"

@lampyjon, is this how you created your virtual environment? Or are you not working in a virtual environment? (It looked like you were, since I saw pip 9.0.1 from /Users/user/test/lib/python3.6/site-packages (python 3.6) in the output you shared.)

[lampyjon]

Thanks @jsf9k! I've tried as per your run through above, and still see the same issue:

Macintosh:scan user$ mkdir tmail_test
Macintosh:scan user$ cd tmail_test/
Macintosh:tmail_test user$ python3 -m venv venv
Macintosh:tmail_test user$ source venv/bin/activate
(venv) Macintosh:tmail_test user$ source venv/bin/activate
(venv) Macintosh:tmail_test user$ python -V
Python 3.6.3
(venv) Macintosh:tmail_test user$ pip -V
pip 9.0.1 from /Users/user/tmail_test/venv/lib/python3.6/site-packages (python 3.6)
(venv) Macintosh:tmail_test user$ pip install trustymail
Collecting trustymail
Collecting docopt (from trustymail)
Collecting dnspython (from trustymail)
  Using cached dnspython-1.15.0-py2.py3-none-any.whl
Collecting requests (from trustymail)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting publicsuffix (from trustymail)
Collecting pyspf==2.0.11 (from trustymail)
Collecting py3dns (from trustymail)
Collecting urllib3<1.23,>=1.21.1 (from requests->trustymail)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->trustymail)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->trustymail)
  Using cached certifi-2017.11.5-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests->trustymail)
  Using cached idna-2.6-py2.py3-none-any.whl
Installing collected packages: docopt, dnspython, urllib3, chardet, certifi, idna, requests, publicsuffix, pyspf, py3dns, trustymail
Successfully installed certifi-2017.11.5 chardet-3.0.4 dnspython-1.15.0 docopt-0.6.2 idna-2.6 publicsuffix-1.1.0 py3dns-3.1.0 pyspf-2.0.11 requests-2.18.4 trustymail-0.3.0 urllib3-1.22
(venv) Macintosh:tmail_test user$ trustymail dhs.gov
Traceback (most recent call last):
  File "/Users/user/tmail_test/venv/bin/trustymail", line 7, in <module>
    from trustymail.cli import main
  File "/Users/user/tmail_test/venv/lib/python3.6/site-packages/trustymail/cli.py", line 45, in <module>
    from trustymail import trustymail
  File "/Users/user/tmail_test/venv/lib/python3.6/site-packages/trustymail/trustymail.py", line 9, in <module>
    import spf
  File "/Users/user/tmail_test/venv/lib/python3.6/site-packages/spf.py", line 110, in <module>
    import DNS    # http://pydns.sourceforge.net
ModuleNotFoundError: No module named 'DNS'

[jsf9k]

@lampyjon, as it happens I have a brand new Mac right here. It only has Python 2.7, though. How did you install version 3.6? I want to replicate what you did, but I'm not a Mac guy.

[lampyjon]

That's a good question @jsf9k - I'm struggling to remember. I strongly suspect I installed it using brew.

[jsf9k]

@lampyjon, sorry for the delay. I've been busy with the holidays and catching up on other things.

I was able to install python3 via brew on the Mac I have, and I am able to reproduce your error now. It looks like other folks are seeing it too. The issue is that the py3dns installs to DNS and dnspython installs to dns, and on OSX the file system is case-insensitive, so these directories are the same.

I'm not sure yet how to get around this. @dav3r or @felddy, can you provide any insight?

[jsf9k]

@h-m-f-t says he is seeing this issue too.

[seanthegeek]

I ended up writing my own SPF validation code before I knew pyspf existed. I could make another PR that replaces pyspf with this code. Things to consider

Pros

• Solves the pydns/dnspython conflict on macOS by only depending on dnspython
• Provides more detailed validation errors (e.g. the exact location where as syntax error occurs)

Cons

• Python 3 is required (would loose Python 2 support)
  • Do recent releases of macOS come with Python 3?
• The code is not as tested/mature as pyspf (works with every crazy record I've found, anyway)

[h-m-f-t]

@seanthegeek, my hot take is 👍. For the cons,

• I'm ok making Python 3 mandatory (no, macOS has 2.7.x natively)
• If you're up for it, I'd be especially supportive if this is/can be something that you maintain externally that we could call, which would enable larger-than-trustymail use. Either way, this would be a welcome fix.

[seanthegeek]

@h-m-f-t It already does :)

https://domainaware.github.io/checkdmarc/

It includes most of the same DMARC code we just merged too...it was just easier to copy/paste those checks than rewrite your stuff around my module at the time. Even if checkdmarc fully replaces that too, my previous work on trustymail is not a waste, because you guys caught some upstream bugs I missed!

I'll wait until @jsf9k merges the fixes that he has mentioned earlier, then set to work on refactoring the trustymail SPF and DMARC checks to use the checkdmarc module. The PR should come pretty quickly quickly :).

[bknowles]

Folks,

Any word on the status of this work? I'm running into the same problem as reported by @lampyjon .

[seanthegeek]

@bknowles Until this gets fixed, try out my SPF and DMARC parser/validator, checkdmarc the only thing it does not do that trustymail does is check for STARTTLS on the SMTP servers listed in the MX records (I ought to add that!)

https://domainaware.github.io/checkdmarc/

@h-m-f-t @jsf9k Thoughts on using checkdmarc to do the record checking?

[egyptiankarim]

For additional reference: sdgathman/pyspf#2

[refayathaque]

Hi everyone,

Having to resuscitate this because I am trying to run trustymail scans but can't even import the module.

Very similar error to others contributing to this ticket:

Traceback (most recent call last): File "handler.py", line 1, in <module> from trustymail import trustymail File "/Users/refayathaque/Desktop/trustymail/trustymail/trustymail.py", line 11, in <module> import spf File "/Users/refayathaque/Desktop/trustymail/spf.py", line 111, in <module> if not hasattr(DNS.Type, 'SPF'): AttributeError: 'module' object has no attribute 'Type'

I am running Python 3.7.3 in a virtual environment. I built the trustymail folder with all its required libraries using CISA lambda_functions guidance. I cloned the repo, moved to the master branch, then used docker-compose to get the trustymail.zip lambda deployment package, to which I added a simple file that tries to run a trustymail scan by importing trustymail and running trustymail.scan(domain_url, TIMEOUT, SMTP_TIMEOUT, SMTP_LOCALHOST, SMTP_PORTS, SMTP_CACHE, SCAN_TYPES, DNS_HOSTNAMES).

I'm not entirely convinced about this being a MacOS issue as I tried running the same file with the trustymail package and all associated libraries in lambda, AWS Lambda uses Amazon Linux and there too I am seeing the same error.

START RequestId: 5e98848c-8ef5-47b7-9327-0736e646e1fb Version: $LATEST module initialization error: module 'DNS' has no attribute 'Type' END RequestId: 5e98848c-8ef5-47b7-9327-0736e646e1fb REPORT RequestId: 5e98848c-8ef5-47b7-9327-0736e646e1fb Duration: 419.44 ms Billed Duration: 500 ms Memory Size: 128 MB Max Memory Used: 61 MB module initialization error module 'DNS' has no attribute 'Type'

The file Type.py DOES EXIST in the DNS library that comes with the trustymail installation, so I don't understand why it thinks Type.py does not exist.

I even changed DNS to dns and Type.py to type.py, based on a conversation above about upper case being an issue, but even lowercasing was futile.

Any help on this is highly appreciated.

[refayathaque]

Tried to run it locally again and seeing the same problem :(

[jsf9k]

@refayathaque, this is on OSX, right? trustymail will not run on a default installation of OSX because of the case-insensitive file system. This is due to the issue described here and here.

I am told that it is possible to install OSX with a case-sensitive filesystem, but I don't know anyone who has done so. So that's a potential solution for running locally, but an untested one.

That said, you shouldn't see this issue in Lambda. How exactly are you running the trustymail Lambda function? Are you building the Lambda function zip file in Docker on OSX? If you try to build the Lambda zip file natively on OSX the case-insensitive file system will thwart your efforts, but building via Docker should work. If you unzip and re-zip on OSX the case-insensitive file system will again burn you.

[refayathaque]

@jsf9k hello! Yes, the screenshots above are from my macOS 10.14.5

Since what you mentioned above with respect to installing macOS with a case-sensitive file system is untested we'd prefer to continue trying to run trustymail in AWS Lambda.

These are the steps I am taking to create an AWS Lambda deployment package with trustymail.

1. git clone git@github.com:cisagov/trustymail-lambda.git
2. cd trustymail-lambda
3. docker-compose down
4. docker-compose build
5. docker-compose up
6. unzip trustymail.zip -d ~/Desktop/trustymail_deployment_package
   This is the step where I have been encountering the problems, because while unzipping I am asked if I'd like to replace some dns files
   replace /Users/refayathaque/Desktop/trustymail_deployment_package/dns/opcode.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:
replace /Users/refayathaque/Desktop/trustymail_deployment_package/dns/__pycache__/opcode.cpython-36.pyc? [y]es, [n]o, [A]ll, [N]one, [r]ename:
replace /Users/refayathaque/Desktop/trustymail_deployment_package/dns/__pycache__/__init__.cpython-36.pyc? [y]es, [n]o, [A]ll, [N]one, [r]ename:
replace /Users/refayathaque/Desktop/trustymail_deployment_package/dns/__init__.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:
   I have tried to replace all AND replace none, and in both cases, I am seeing the error
   You must be asking why were are unzipping the trustymail.zip file, it is because we need to modify the supplied lambda_handler.py file to be able to do other things with the scan data, e.g., persist the data to DynamoDB
7. zip -r ~/Desktop/trustymail_deployment_package.zip .
8. Upload to AWS Lambda console and execute with domain in the test object to be picked up by Lambda as event

So it's evident that the problem lies in the unzipping and zipping of the trustymail.zip file from what is built using docker and the trustymail-lambda git repository of yours.

If I just take the trustymail.zip file I get from running docker on the trustymail-lambda git repository (no unzipping into my local directory and then zipping up for push up to AWS Lambda) and run it on AWS Lambda it works flawlessly. I'm not sure why I did not try this before.

Now the challenge lies in somehow being able to modify the supplied lambda_handler.py file in the trustymail.zip file without unzipping and then zipping back up on my macOS.

Do you recommend I use Windows or Ubuntu to do this?

Thank you once again for all your help @jsf9k !

[jsf9k]

Yep, it's when you unzip the file on MacOS that the case-insensitive filesystem bites you. If you really want to unzip the file manually you can do that on any linux distribution. Windows used to be case-insensitive as well; I don't know offhand if that is still true in Windows 10.

As a better solution, I'd recommend the following:

1. Fork the cisagov/lambda_functions repository
2. Add your custom lambda_handler.py file to the lambda_functions/trustymail directory
3. Modifying this line in build_trustymail.sh so that it copies in your custom lambda_handler.py file (from the previous step) instead of the one from domain-scan.

This way everything you want is zipped up in Docker and you don't have to go back and manually modify the zip file afterwards.

[refayathaque]

Hi @jsf9k , hope you've had a nice weekend.

Thank you so much for the guidance, this is now what I am doing to make edits to the Lambda handler file and it's working fine. I'm really glad we were able to resolve these issues and can now begin to use trustymail in AWS Lambda! :)

Based on some preliminary trustymail tests there was one thing I was hoping to get some feedback on.

{ "Domain": "samhsa.gov", "Base Domain": "samhsa.gov", "Live": true, "MX Record": true, "Mail Servers": "smtp2.ees.hhs.gov, smtp.ees.hhs.gov", "Mail Server Ports Tested": "25, 587, 465", "Domain Supports SMTP Results": "smtp.ees.hhs.gov:25", "Domain Supports SMTP": true, "Domain Supports STARTTLS Results": "smtp.ees.hhs.gov:25", "Domain Supports STARTTLS": true, "SPF Record": true, "Valid SPF": true, "SPF Results": "v=spf1 a mx ip4:52.45.112.110 -all", "DMARC Record": true, "Valid DMARC": true, "DMARC Results": "v=DMARC1; p=reject; fo=1; ri=3600; rua=mailto:hhs@rua.agari.com,mailto:reports@dmarc.cyber.dhs.gov; ruf=mailto:hhs@ruf.agari.com", "DMARC Record on Base Domain": true, "Valid DMARC Record on Base Domain": true, "DMARC Results on Base Domain": "v=DMARC1; p=reject; fo=1; ri=3600; rua=mailto:hhs@rua.agari.com,mailto:reports@dmarc.cyber.dhs.gov; ruf=mailto:hhs@ruf.agari.com", "DMARC Policy": "reject", "DMARC Policy Percentage": 100, "DMARC Aggregate Report URIs": "mailto:hhs@rua.agari.com, mailto:reports@dmarc.cyber.dhs.gov", "DMARC Forensic Report URIs": "mailto:hhs@ruf.agari.com", "DMARC Has Aggregate Report URI": true, "DMARC Has Forensic Report URI": true, "Syntax Errors": null, "Debug Info": "[STARTTLS] In starttls_scan at /var/task/trustymail/trustymail.py:118: Connection unexpectedly closed: timed out, [STARTTLS] In starttls_scan at /var/task/trustymail/trustymail.py:118: timed out, [STARTTLS] In starttls_scan at /var/task/trustymail/trustymail.py:118: timed out, [STARTTLS] In starttls_scan at /var/task/trustymail/trustymail.py:118: timed out, [STARTTLS] In starttls_scan at /var/task/trustymail/trustymail.py:118: timed out" }

In the example above we scanned samhsa.gov an operational division here at HHS. I see several STARTTLS timeout errors in the debug info. I have noticed STARTTLS timeout errors in the return data for other domains (e.g., fda.gov). Is this something we should be concerned about? Is it something that could result in incorrect/incomplete data? I.e., will it bite us in the future if we ignore this for now?

Thank you again!

[echudow]

Hi @refayathaque, in our runs of `trustymail` this is completely normal and expected. The MX record has 2 mail servers listed (`smtp2.ees.hhs.gov` and `smtp.ees.hhs.gov`), so `trustymail` will try to connect to each of them on ports 25, 465, and 587, resulting in 6 SMTP STARTTLS attempts. In the debug info, there are 5 time outs listed, so it looks like one connection was successful while the other 5 were not. Based on the data in `Domain Supports SMTP Results` and `Domain Supports STARTTLS Results` being `smtp.ees.hhs.gov:25`, I assume that that was the one connection that was successful.

[refayathaque]

@echudow thank you for that detailed explanation. I am better understanding the Debug Info now! So trustymail will declare an email domain like samhsa.gov to be supporting STARTTLS even though it could verify that only 1/2 of the domain's mail servers have STARTTLS? Is that the logic it's using? More of a server1STARTTLS || server2STARTTLS rather than server1STARTTLS && server2STARTTLS?

[echudow]

@refayathaque, yes, the domain supports STARTTLS if all mail servers that support SMTP support STARTTLS. So in the case of samhsa.gov, the scanner could only connect for SMTP to one of the listed mail servers so the other one was ignored since as far as the scanner knows it doesn’t accept any email, and the one that did accept an SMTP connect also supports STARTTLS.

[refayathaque]

@echudow thank you, well understood! So let me ask you this: In order for me to determine full BOD 18-01 compliance I am going to need to explore 3DES, RC4, SLLv2, SSLv3, and for this I will need to use SSLyze in addition to trustymail. Taking this above example into consideration, since we only got a response on 1/2 of the servers, should I only do the SSLyze scan on that one server that responded to STARTTLS instead of both?

[echudow]

@refayathaque, if you use domain-scan and and have it use both the trustymail and then the sslyze scanners, sslyze is smart enough to use the results from trustymail to only scan the servers and ports that trustymail was already able to connect to (unfortunately it also scans 443 for all of them too, see my commit on my fork at echudow/domain-scan@e3c926d to skip the HTTPS scans when only trustymail is used and not pshtt).

[reevesjeremy]

Windows used to be case-insensitive as well; I don't know offhand if that is still true in Windows 10.

I ran into the "ModuleNotFoundError: No module named 'DNS'" on Windows yesterday. After having renamed "dns" to "DNS" while troubleshooting, the error flipped to "ModuleNotFoundError: No module named 'dns'". This was my clue that there was an issue with modules trumping each other during installation, and how I ended up here today.

Windows is case sensitive by default, however, I learned this morning that folders can be changed to be case insensitive like Linux. https://learn.microsoft.com/en-us/windows/wsl/case-sensitivity#change-the-case-sensitivity-of-files-and-directories

Personally I only changed the site-packages folder. Once I did that, and re-pip'd trustymail, I now had both DNS and dns folders, and the script ran as expected.