Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add conditional for hiding bills with restricted views #394

Merged
merged 3 commits into from
Feb 11, 2019

Conversation

reginafcompton
Copy link
Contributor

@reginafcompton reginafcompton commented Feb 7, 2019

This PR adds a conditional to the is_visible property: it explicitly hides bills marked as having a restricted view in Legistar.

It works in cooperation with opencivicdata/scrapers-us-municipal#258, opencivicdata/python-legistar-scraper#81, and datamade/django-councilmatic#232

N.b., I'll get the tests passing once I can add the above-mentioned changes in Django Councilmatic to the testing requirements.

@reginafcompton reginafcompton changed the title Add conditional for hiding bills with restricted views [wip] Add conditional for hiding bills with restricted views Feb 7, 2019
@reginafcompton
Copy link
Contributor Author

reginafcompton commented Feb 8, 2019

Overview

With the newly provided API token, DataMade scrapers now have access to private Metro bills. But….why do we need to scrape private bills?

We need to know which bills to show and which bills to hide on Councilmatic: the MatterRestrictedViewViaWeb field tells us which bills are public (show!) and which are private (hide!). The process:

  1. scrape bill and include the value of MatterRestrictedViewViaWeb in the extras (Add token to params opencivicdata/scrapers-us-municipal#258 and Scrape private opencivicdata/python-legistar-scraper#81)
  2. import bills and indicate if they are restricted (Add field to mark bills with a restricted view, and import it datamade/django-councilmatic#232)
  3. hide or show bills based on this value. (this PR)

But wait. Why not just skip private bills? Sometimes, Metro will toggle bills between public and private, meaning that the Councilmatic system may have bills that were previously public. Scraper needs to access those now-private bills, so that Councilmatic can make sense of their new status.

Not knowing which bills had changed between public and private has caused issues in the past:
#345 (comment)
#328


N.b. We do know that toggling “Not viewable in site” changes the timestamp of bills: https://docs.google.com/document/d/1Huv0kRQmRm-I658JE4hG8rQSdmtjy-8QUSr_cwWZnHI/edit#heading=h.hrplgr8t7ccr

Hence, the scraper should catch when a bill changes between public to private.

Copy link
Collaborator

@hancush hancush left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a+

DJANGO_SETTINGS_MODULE = tests.test_config
filterwarnings =
ignore::django.utils.deprecation.RemovedInDjango20Warning
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

@reginafcompton reginafcompton changed the title [wip] Add conditional for hiding bills with restricted views Add conditional for hiding bills with restricted views Feb 8, 2019
@reginafcompton reginafcompton merged commit 4004441 into master Feb 11, 2019
@reginafcompton reginafcompton deleted the visible-filter branch February 11, 2019 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants