Trying to understand "unresolved" bills

[reginafcompton]

The Metro scraper regularly returns warnings about Bills referenced (e.g., by an event item) but not present in the OCD database or (it follows) Legistar API. The following bills raised this error in recent history:

2015-0827
2015-0828
2015-0422
2015-0446
2015-0371
2015-0423
2015-0420
2015-0375
2015-0421
2015-0459
2015-0312
2015-0829
2015-0216
2015-0280
2015-0493
2015-0831
2015-0793
2018-0313

DataMade is trying to understand the nature of these errors, specifically:
(1) are these bills private and will remain private?
(2) are they private and will be come public?
(3) are they particular in another way?

Resolving this can help us better handle errors such as this one.

@shrayshray - would you be able to give us information about some or all of the above referenced bills?

[shrayshray]

It looks like the "2015-xxxx" reports were all created the first month Metro was using Legistar (May 2015). It's possible they were created concurrently with the manual report-drafting process, as "practice" for drafting reports in Legistar.
The final agendas for May 2015 were created manually and are in the Board Archive.
The reports are private because they were never on a Legistar Agenda released to the public. They will remain private.
@camachoo can you confirm? And should the May 2015 reports and calendar dates be removed from Legistar, as Agendas for them are in the Board Archive and will not be released via Legistar? https://boardagendas.metro.net/event/ad-hoc-sustainability-committee-ef4907f09d3a/

For the 2018 file, it's set to "Not Viewable Via InSite" because it's General Public Comment. General Public Comment used to be entered on the Agenda as just a header, but it has to be entered as a Report now, so it gets an ID our digital public comment system can use. But because there is no actual content in the Report, it's set to not show on the web, even though it's on a published Agenda, just to minimize confusion.

[reginafcompton]

@shrayshray - this is tremendously helpful, and helps us better interpret these scraper warnings. Thank you for taking the time to look!

Please let us know if//when you remove the May 2015 content from Legistar.

[reginafcompton]

I did a little digging, and I noticed that some of the General Public Comment reports are, in fact, viewable on Legistar, for instance:

http://webapi.legistar.com/v1/metro/matters/5175
http://webapi.legistar.com/v1/metro/matters/5179
http://webapi.legistar.com/v1/metro/matters/5178

Is this as you expect? If so, what makes these reports different from 2018-0313?

[shrayshray]

@reginafcompton This is not as I expect, but it's human error. The reports should have been hidden, but the person who created them forgot to check the "Not Viewable Via InSite" box.
I checked the box just a bit ago ... but since then we've gotten into a discussion about whether these should NOT be hidden, since in the future we'll need them to be viewable in order to accept public comment via the web.
Sorry for the confusion, hopefully I'll have a resolution to report by this afternoon.

[shrayshray]

@reginafcompton Some clarification: From June 2018 forward, General Public Comments should be visible on the web. This was decided after some discussion today.
2018-0313, and others in May 2018, are different because this was the first month we were using the new public comment system in "beta" in meetings. We needed General Public Comments to be reports so we could use the ID's, but I guess instead of adding the GPC reports to the final published Agenda, the Board Secretary's office stuck with the practice of adding GPC as a text header. Not sure why, but at least it's an explanation for why they're different.

[reginafcompton]

thank you @shrayshray - this is truly helpful information

[shrayshray]

Because of the anticipated migration of the Board Archive into Legistar, we're not going to delete the May 2015 calendar events from Legistar. Although these event pages do not currently show a PDFs, because they were manually created outside of Legistar, they will show the PDF after the migration.

[reginafcompton]

@shrayshray – The original intent of this issue was to help DataMade better understand scraper warnings for private bills: we now scrape those bills (in part, due to conversations like this!), and so, we can close this issue.

Thanks for the useful conversation, above!