Check python version and enable https where it's possible #2307

tigranl · 2016-12-05T19:21:04Z

No description provided.

Freso · 2016-12-05T19:38:35Z

@sampsyo Do we have a minimum supported Python 3 version? (SNI wasn't included in Python 3.0.0 either, so the check might need to check for early versions of Py3 as well… Depending on minimum supported Py3 version.)

Freso · 2016-12-05T19:41:40Z

beets/autotag/mb.py


 VARIOUS_ARTISTS_ID = '89ad4ac3-39f7-470e-963a-56509c546377'
-BASE_URL = 'http://musicbrainz.org/'
+
+if sys.version_info > (2, 7, 9):


The comparison should be "greater than or equal to", since 2.7.9 also has SNI support.

sampsyo · 2016-12-05T20:56:17Z

My best guess from this thread is that 3.2+ supports SNI. Fortunately, 3.2 is below our minimum required Py3 version.

Would you mind defining a SNI_SUPPORTED variable in the beets.util module so we can reuse it in other places? Other places where this might be needed are revealed by @orf's PRs from a couple weeks ago: #2247 #2243 #2245 #2244 #2248

codecov-io · 2016-12-06T13:22:50Z

Current coverage is 76.85% (diff: 75.00%)

Merging #2307 into master will increase coverage by 0.01%

@@             master      #2307   diff @@
==========================================
  Files            65         65          
  Lines         13220      13223     +3   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          10159      10163     +4   
+ Misses         3061       3060     -1   
  Partials          0          0

Powered by Codecov. Last update ae5e55c...ba5fcb1

sampsyo · 2016-12-06T15:14:33Z

OMG I wish @codecov-io would stop commenting. 😠

This is looking great so far! The last thing we'll need once all the endpoint changes are in place is to update the tests. Some tests use mocking to return fake responses to URL requests; these typically use hard-coded URLs. We'll need to change the tests to return the mocked response for both HTTP and HTTPS versions of each URL.

sampsyo · 2016-12-06T15:56:29Z

I meant to suggest that SNI_SUPPORTED be a boolean, not just the current version number. Then the code in each module can look like this:

if ui.SNI_SUPPORTED:

instead of this:

if ui.SNI_SUPPORTED >= (2, 7, 9):

This variable should also probably reside in the util module, not the ui module.

tigranl · 2016-12-06T16:13:26Z

SNI_SUPPORTED is defined in the ui module. Should I move it to the util module?

sampsyo · 2016-12-06T16:15:41Z

Yes, please.

This reverts commit f60c911.

Add SNI_SUPPORTED

Add SNI_SUPPORTED Add SNI_SUPPORTED

tigranl · 2016-12-06T17:17:52Z

Okay, so, what shall we do with tests?

sampsyo · 2016-12-06T18:15:28Z

OK, let's take a look! The rough prescription is:

Look at the test failures (either locally on your machine or on Travis) to find the tests that need updating.
For each, look for the URL that's being mocked. Add a second mock call that uses HTTPS instead of HTTP so we catch both.
In the end, check that all the tests pass without network access—e.g., with the WiFi on your laptop disabled. If that works, we'll know that all the mocking does what we want it to do.

It would also be nice to ensure that all the endpoints actually do support HTTPS. That just involves running each piece of functionality to make sure it still works when contacting the server over HTTPS.

tigranl · 2016-12-08T18:08:19Z

Hi, @sampsyo. Can you give me some comments on my work? I have never written unit tests before.

sampsyo

The progress looks great; thanks for looking at the tests! I've added a couple of comments here.

To keep the conversation going, the next step (once you have a set of changes ready) should be to look at the test failures. If you see anything obvious, you can fix that stuff right away. If you see failures you can't explain, please feel free to ask about them—point me toward the test you don't understand and I'll take a closer look.

Finally, we should remember to check that these endpoints actually support SSL. It would also be good to verify which of them use SNI—for ones that don't, we can use SSL without a version check.

sampsyo · 2016-12-08T19:15:46Z

test/test_art.py

        self.source.fetch_image(self.candidate, self.extra)
+        self.source.fetch_image(self.candidate_https, self.extra)


Since this test controls the URL on "both ends," i.e. the request and the response, I'm not sure it needs updating. Was it failing before?

sampsyo · 2016-12-08T19:16:49Z

test/test_art.py

+    AAO_URL_HTTPS = 'http://www.albumart.org/index_detail.php?asin={0}' \
+                 .format(ASIN)
+    CAA_URL_HTTPS = 'http://coverartarchive.org/release/{0}/front' \
+                 .format(MBID)


Is there any way we can avoid copying and pasting here? For example, we could put most of the URLs (starting with ://) into a variable and then concatenate on the https or http part separately.

sampsyo · 2016-12-08T19:18:01Z

test/test_art.py

        album = _common.Bag(asin=self.ASIN)
        self.plugin.art_for_album(album, [self.dpath])
        self.assertEqual(len(responses.calls), 1)
        self.assertEqual(responses.calls[0].request.url, self.AMAZON_URL)
+        self.assertEqual(responses.calls[0].request.url, self.AMAZON_URL_HTTPS)


This seems to assert that the requested URL is equal to two different URLs, which probably isn't what you meant—instead, you probably want to write an assertion that says that the requested URL is either the HTTP one or the HTTPS one.

(The next couple of tests seem to have the same issue.)

tigranl · 2016-12-10T16:59:28Z

Hi, @sampsyo. Sorry for delay. I found that amazon and albumart.org don't support https, so I removed https urls for them.

sampsyo · 2016-12-10T17:27:17Z

No worries, and thank you!

It looks like Travis still has a few style/error reports: https://travis-ci.org/beetbox/beets/jobs/182870011#L848-L853

It also looks like there might be a typo in some URL, where there's "htts" instead of https? https://travis-ci.org/beetbox/beets/jobs/182870006#L913

tigranl · 2016-12-10T18:19:34Z

It's strange, because I fixed typo in one of my previous commits b65a7da

sampsyo · 2016-12-10T20:46:31Z

Got can behave in mysterious ways. 👻

Those flake8 errors shouldn't be hard to address either.

tigranl · 2016-12-10T21:53:35Z

Okay, my numerous commits look ridiculous. I should do something with it.

sampsyo · 2016-12-10T22:05:25Z

It's cool; I don't have anything against long series of commits. But if you like cleaner histories, the terms to google are [git squash].

ghost · 2016-12-10T22:09:15Z

@sampsyo : I'd definitely prefer if we use "Squash and merge" button in cases like this. @tigranl : we can take take care of that on our end by pressing a button

sampsyo · 2016-12-10T22:13:50Z

Yeah, that's cool too.

tigranl · 2016-12-10T23:11:48Z

It would be great.

Kraymer

Instead of duplicating hardcoded urls in if/else branches, why not write :

PROXY_URL = '%s://example.com' % 'https' if util.SNI_SUPPORTED else 'http'

Kraymer · 2016-12-30T23:37:49Z

beetsplug/lastimport.py

+if util.SNI_SUPPORTED:
+    API_URL = 'https://ws.audioscrobbler.com/2.0/'
+else:
+    API_URL = 'https://ws.audioscrobbler.com/2.0/'


https => http ?

I think it's less readable.

I did a little audit using the `openssl` command-line tool to find the servers that don't require SNI. Here's what I found: icbrainz.org: SNI images.weserv.nl: inconclusive, but docs say yes SNI coverartarchive.org: SNI webservice.fanart.tv: *no* SNI dbpedia.org: *no* SNI en.wikipedia.org: *no* SNI ws.audioscrobbler.com: *no* SNI api.microsofttranslator.com: *no* SNI In summary, *only* MusicBrainz and CoverArtArchive were found to require SNI. So I'm using SSL unconditionally on all the other sites.

We don't need quite so many checks now that SSL isn't conditional most of the time.

Close #2247, close #2243, close #2245, close #2244, close #2248.

sampsyo · 2017-01-03T02:03:30Z

Merged with a few changes. Thank you! ✨

Check python version and enable https where it's possible

43fcb3d

Freso reviewed Dec 5, 2016

View reviewed changes

tigranl added 2 commits December 6, 2016 16:17

Python version check for lyrics.py

6ba5099

Merge branch 'master' of github.com:beetbox/beets into https_fix

0fb19cf

tigranl force-pushed the https_fix branch from ba5fcb1 to 0fb19cf Compare December 6, 2016 13:22

Add SNI_SUPPORTED variable for https check

f60c911

tigranl force-pushed the https_fix branch from d6d69d9 to f60c911 Compare December 6, 2016 14:11

Add SNI_SUPPORTED variable for https check

73a7a4f

tigranl force-pushed the https_fix branch from 531548a to 73a7a4f Compare December 6, 2016 15:39

tigranl added 2 commits December 6, 2016 18:42

Add SNI_SUPPORTED variable for https check

d065b33

Add SNI_SUPPORTED variable for https check

5ae1376

tigranl added 3 commits December 6, 2016 19:27

Revert "Add SNI_SUPPORTED variable for https check"

25ebf89

This reverts commit f60c911.

Add SNI_SUPPORTED

91819b2

Add SNI_SUPPORTED

efa9041

Add SNI_SUPPORTED

tigranl force-pushed the https_fix branch from 13c2f4d to efa9041 Compare December 6, 2016 16:51

Add SNI_SUPPORTED

420c451

Add SNI_SUPPORTED Add SNI_SUPPORTED

tigranl force-pushed the https_fix branch from 76520ad to 420c451 Compare December 6, 2016 17:07

Add tests for https

9bba178

tigranl force-pushed the https_fix branch from 6f681ce to 9bba178 Compare December 8, 2016 15:46

tigranl added 2 commits December 8, 2016 19:09

Add SNI_SUPPORTED

21208b8

Add SNI_SUPPORTED

b65a7da

sampsyo requested changes Dec 8, 2016

View reviewed changes

Add tests for https

68b4a03

PEP8 corrections

0868299

Fix typo

471f875

tigranl added 2 commits December 11, 2016 00:25

Fix typos

5ca664e

Add ui import

dd115b1

Kraymer suggested changes Dec 30, 2016

View reviewed changes

tigranl and others added 3 commits December 31, 2016 18:46

Fix a typo

1c5c74f

Simplify test changes

33a8e81

We don't need quite so many checks now that SSL isn't conditional most of the time.

sampsyo merged commit 33a8e81 into beetbox:master Jan 3, 2017

sampsyo added a commit that referenced this pull request Jan 3, 2017

Changelog for #2307

9e76c83

Close #2247, close #2243, close #2245, close #2244, close #2248.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check python version and enable https where it's possible #2307

Check python version and enable https where it's possible #2307

tigranl commented Dec 5, 2016

Freso commented Dec 5, 2016 •

edited

Loading

Freso Dec 5, 2016 •

edited

Loading

sampsyo commented Dec 5, 2016

codecov-io commented Dec 6, 2016

sampsyo commented Dec 6, 2016

sampsyo commented Dec 6, 2016

tigranl commented Dec 6, 2016

sampsyo commented Dec 6, 2016

tigranl commented Dec 6, 2016

sampsyo commented Dec 6, 2016

tigranl commented Dec 8, 2016

sampsyo left a comment

sampsyo Dec 8, 2016

sampsyo Dec 8, 2016

sampsyo Dec 8, 2016

sampsyo Dec 8, 2016

tigranl commented Dec 10, 2016

sampsyo commented Dec 10, 2016

tigranl commented Dec 10, 2016

sampsyo commented Dec 10, 2016

tigranl commented Dec 10, 2016

sampsyo commented Dec 10, 2016 •

edited

Loading

ghost commented Dec 10, 2016

sampsyo commented Dec 10, 2016

tigranl commented Dec 10, 2016

Kraymer left a comment

Kraymer Dec 30, 2016

tigranl Dec 31, 2016

sampsyo commented Jan 3, 2017

		self.source.fetch_image(self.candidate, self.extra)
		self.source.fetch_image(self.candidate_https, self.extra)

Check python version and enable https where it's possible #2307

Check python version and enable https where it's possible #2307

Conversation

tigranl commented Dec 5, 2016

Freso commented Dec 5, 2016 • edited Loading

Freso Dec 5, 2016 • edited Loading

Choose a reason for hiding this comment

sampsyo commented Dec 5, 2016

codecov-io commented Dec 6, 2016

Current coverage is 76.85% (diff: 75.00%)

sampsyo commented Dec 6, 2016

sampsyo commented Dec 6, 2016

tigranl commented Dec 6, 2016

sampsyo commented Dec 6, 2016

tigranl commented Dec 6, 2016

sampsyo commented Dec 6, 2016

tigranl commented Dec 8, 2016

sampsyo left a comment

Choose a reason for hiding this comment

sampsyo Dec 8, 2016

Choose a reason for hiding this comment

sampsyo Dec 8, 2016

Choose a reason for hiding this comment

sampsyo Dec 8, 2016

Choose a reason for hiding this comment

sampsyo Dec 8, 2016

Choose a reason for hiding this comment

tigranl commented Dec 10, 2016

sampsyo commented Dec 10, 2016

tigranl commented Dec 10, 2016

sampsyo commented Dec 10, 2016

tigranl commented Dec 10, 2016

sampsyo commented Dec 10, 2016 • edited Loading

ghost commented Dec 10, 2016

sampsyo commented Dec 10, 2016

tigranl commented Dec 10, 2016

Kraymer left a comment

Choose a reason for hiding this comment

Kraymer Dec 30, 2016

Choose a reason for hiding this comment

tigranl Dec 31, 2016

Choose a reason for hiding this comment

sampsyo commented Jan 3, 2017

Freso commented Dec 5, 2016 •

edited

Loading

Freso Dec 5, 2016 •

edited

Loading

sampsyo commented Dec 10, 2016 •

edited

Loading