KeyError when used with an API key with a higher rate limit

[esonderegger]

This week the FEC bumped up the rate-limit on the API key I've been using with this library.

Unfortunately, it appears the openFEC API now no longer returns an X-RateLimit-Remaining header with their responses.

This causes the _throttled_request method to throw a KeyError exception that looks like this:

>>> from pyopenfec import Candidate
>>> cands = Candidate.fetch(office='H', election_year=2018, state='VA', district='10')
>>> for cand in cands:
...     print(cand.name)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/esonderegger/code/github/pyopenfec/pyopenfec/utils.py", line 106, in fetch
    initial_results = cls._make_request(resource, **kwargs)
  File "/Users/esonderegger/code/github/pyopenfec/pyopenfec/utils.py", line 89, in _make_request
    r = cls._throttled_request(url, params)
  File "/Users/esonderegger/code/github/pyopenfec/pyopenfec/utils.py", line 60, in _throttled_request
    cls.ratelimit_remaining = int(response.headers['x-ratelimit-remaining'])
  File "/Users/esonderegger/.pyenv/versions/pyopenfec/lib/python3.4/site-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'x-ratelimit-remaining'

I think this should be a pretty easy fix and will put together a PR for it later this afternoon.

[lbeaufort]

Hi @esonderegger the missing X-RateLimit-Remaining header is a configuration issue with your API key - if you still have this issue, please email apiinfo@fec.gov and we can get that fixed for you.

[esonderegger]

Hi @lbeaufort - thank you for letting me know that this can be fixed on the FEC side of things. I just checked a test request with both my normal rate limit key and my higher rate-limit key and got the following headers back:

https://api.open.fec.gov/v1/candidates/?page=1&sort_null_only=false&candidate_status=C&per_page=20&district=09&sort_hide_null=false&cycle=2020&office=H&sort_nulls_last=false&state=NC&sort=name&api_key=my-normal-rate-key

HTTP 200 No Error

Content-Type: application/json
Access-Control-Allow-Origin: *
X-Frame-Options: DENY
Age: 1
Via: https/1.1 api-umbrella (ApacheTrafficServer [cMsSfW])
Server: openresty
X-XSS-Protection: 1; mode=block
X-Vcap-Request-Id: 855cde06-6e16-42eb-7e3f-1baa7b227338
Date: Wed, 01 May 2019 20:34:53 GMT
Cache-Control: public, max-age=3600
Content-Length: 4059
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 858
Connection: keep-alive
Strict-Transport-Security: max-age=31536000, max-age=31536000; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding, Accept-Encoding
X-Cache: MISS

https://api.open.fec.gov/v1/candidates/?page=1&sort_null_only=false&candidate_status=C&per_page=20&district=09&sort_hide_null=false&cycle=2020&office=H&sort_nulls_last=false&state=NC&sort=name&api_key=my-high-rate-key

HTTP 200 No Error

Content-Type: application/json
Access-Control-Allow-Origin: *
X-Frame-Options: DENY
Age: 305
Via: http/1.1 api-umbrella (ApacheTrafficServer [cHs f ])
Server: openresty
X-XSS-Protection: 1; mode=block
X-Vcap-Request-Id: 855cde06-6e16-42eb-7e3f-1baa7b227338
Date: Wed, 01 May 2019 20:39:57 GMT
Cache-Control: public, max-age=3600
Strict-Transport-Security: max-age=31536000, max-age=31536000; preload
Content-Length: 4059
Connection: keep-alive
X-Content-Type-Options: nosniff
Vary: Accept-Encoding, Accept-Encoding
X-Cache: HIT

Which shows that it's possible to get a response without the X-RateLimit-Remaining header in it. As a result, I think #26 is still a good idea. That said, this is super-low priority for us at National Journal because we're using this fork and it's already been addressed over there.

[lbeaufort]

Thanks for letting me know! It’s a per-key configuration issue that we only figured out fairly recently. Glad you have a workaround!