Getting data from remote sources into the spec

[olizilla]

There are a few places where we want data from remote sources to be included in the spec

• the filecoin network parameters - Show the Filecoin Network Parameters in the spec #1058
• source files from other repos
  • option to include specific ranges of code rather than entire files.
• CI and code coverage data
• security and theory audit dates
• and more... TODO: define all.

For some cases like audits we are adding yaml frontmatter to markdown docs, which is simple enough, but requires manual maintenance.

For Code files we are using hugo modules, which is not ideal as it fetches the entire external repo into the spec working tree.

For CI status we pull it at build time via hugo data helpers... this is neat, but if the remote is unavailable the build will fail. It also means the info is only as fresh as the last build, but a cron job would be sufficient to solve that.

I suspect an api that aggregates the info we want to see in the sepc would be helpful... it can hide the complexity of extracting the data we want, and maintain a cache so it can return the last good result if any endpoint fails... we can also query it at build time, and for anything data that should be as fresh as possible we can also query it from the client.

This could just be a bunch of github actions that commit data files to the spec repo when something changes, or an app you run locally, or a hosted service (a cloudflare web worker or similar). A local app would be nice and hackable / upgradeable. A hosted service would mean authors don't have to think about providing github tokens or similar auth issues to get access to data to be scraped.

The situation right now is too spread out and requires too much knowledge, and is hard to include all the transformations we want.

TODO

• Get @hugomrdias to add his thoughts and more detail to this issue.
• Define what data we want
• Decide on the mechanism for including it in the spec (local json files, and api, html post processing, etc)

[olizilla]

Here is a PR from today where CI has failed because the build did not get a response from codecov.io in time

https://github.com/filecoin-project/specs/pull/1111/checks?check_run_id=1036073403

 Building sites … ERROR 2020/08/27 10:48:15 Failed to get JSON resource "https://codecov.io/api/gh/filecoin-project/lotus": Get "https://codecov.io/api/gh/filecoin-project/lotus": dial tcp 35.199.43.247:443: i/o timeout

[olizilla]

The dream is real thanks to @hugomrdias in #1159