Some RSS feeds have double-encoded HTML entities

[Mr0grog]

The San Mateo source data (which is an actual RSS feed!) currently has some double-encoded HTML entities that we should see if we can clean up. That is, the source data might have code like:

 

Which is a common symptom of an HTML entity like:

 

Getting re-encoded and second time and therefore getting ruined. See an example of this happening in practice on sfbrigade/stop-covid19-sfbayarea#309 (comment)

One common way to fix this is to just repeatedly decode the HTML into plain text until there’s nothing left to decode, and then re-encode it once. e.g:

" " → " " → " " → (then re-encode once) → " "

The downside here is that this can ruin code that was intentionally double-encoded, like source code examples. That’s probably not likely in the kind of data we’re dealing with, though.

[Mr0grog]

Note: Python’s built-in html module has escape() and unescape() functions for doing the above work.