feat: use the new api proxy and improve embed shortcode (#1159)

This PR adds:

- Support for the new API proxy
  - The proxy will handle all the external resources, caching and transforming data to our needs
  - New repo with the Proxy will be published asap
- Code coverage and CI status is now handled by the proxy
- Code blocks from github are now support in the embed shortcode using the Proxy
- Code blocks with title or symbols have permalinks

.github/workflows/main.yml

@@ -1,12 +1,6 @@
 name: CI 
 
-on:
-  push:
-    branches:
-      - "master"
-  pull_request:
-    branches:
-      - "*"
+on: [push]
 
 jobs:
   deploy:
@@ -21,8 +15,6 @@ jobs:
       - uses: actions/setup-node@v2-beta
         with:
           node-version: '12'
-      # get lotus deps...
-      - run: sudo apt install bzr
 
       - run: npm install
       - run: npm test
@@ -31,7 +23,6 @@ jobs:
       # Pin the built site to ipfs-cluster, output the cid as `steps.ipfs.outputs.cid`
       # see: https://github.com/ipfs-shipyard/ipfs-github-action
       - uses: ipfs-shipyard/ipfs-github-action@v2.0.0
-        if: github.event.pull_request == 0 || github.event.pull_request.head.repo.full_name == github.repository
         id: ipfs
         with:
           path_to_add: public

.gitignore

@@ -3,6 +3,7 @@
 public/
 .DS_Store
 yarn.lock
+package-lock.json
 node_modules
 resources
 static/_gen

README.md

@@ -37,12 +37,11 @@ To build the spec website you need
 
 -   [`node` & `npm`](https://nodejs.org/en/download)
 -   [`go`](https://golang.org/doc/install)
--   `bzr` (required to build lotus)
 
-On macOS you can get go and bzr from Homebrew
+On macOS you can get go from Homebrew
 
 ```bash
-brew install go bzr
+brew install go
 ```
 
 Clone the repo, and use `npm install` to fetch the dependencies
@@ -110,10 +109,12 @@ Your algorithm here
 ```
 ````
 
-You can embed source code from other repos. Mount the repo as a hugo modules as descibed in [External Modules](#external-modules) then use the [`embed shorcode`](#embed) to link to a specific symbol.
+You can embed source code from local files or external other repos using the `embed` [shortcode](#embed).
 
-```go
-{{<embed src="/externals/go-data-transfer/types.go"  lang="go" symbol="Channel">}}
+```text
+{{<embed src="/path/to/local/file/types.go"  lang="go" symbol="Channel">}}
+
+{{<embed src="https://github.com/filecoin-project/lotus/blob/master/build/bootstrap.go" lang="go">}}
 ```
 
 ## Images
@@ -173,15 +174,20 @@ hugo shortcodes you can add to your markdown.
 {{<embed src="piece_store.go" lang="go">}}
 
 # src relative to content folder
-{{<embed src="/systems/piece_store.id" lang="go">}}
+{{<embed src="/systems/piece_store.go" lang="go">}}
 
 # can just embed a markdown file
 {{<embed src="section.md" markdown="true">}}
 
 # can embed symbols from Go files
 # extracts comments and symbol body
-{{<embed src="/externals/go-data-transfer/types.go"  lang="go" symbol="Channel">}}
+{{<embed src="types.go"  lang="go" symbol="Channel">}}
+
+# can embed from external sources like github
+{{<embed src="https://github.com/filecoin-project/lotus/blob/master/build/bootstrap.go" lang="go">}}
 ```
+This shortcode also supports the property `title` to add a permalink below the embed.
+
 
 ### `listing`
 
@@ -231,7 +237,7 @@ For short snippets of math text you can just use the `{{<katex>}}` shortcode, bu
 
 Parses math typesetting with [KaTeX](https://katex.org/docs/api.html)   
 
-Check this example [example](https://deploy-preview-969--fil-spec-staging.netlify.app/math-mode/)
+Check this example [example](https://spec.filecoin.io/math-mode/)
 
 > Some syntax like `\_` can't go through HUGO markdown parser and for that reason we need to wrap math text with code blocks, code fendes or the shortcode `{{<plain>}}`. See examples below.
 >
@@ -374,5 +380,4 @@ hugo mod get github.com/filecoin-project/specs-actors@v0.7.2
     -   [config](https://github.com/mermaid-js/mermaid/blob/master/docs/mermaidAPI.md#mermaidapi-configuration-defaults)
     -   [editor](https://mermaid-js.github.io/mermaid-live-editor)
 -   [Pan/Zoom for SVG](https://github.com/anvaka/panzoom)
--   [Icons](https://css.gg/)
--   [Working with submodules](https://github.blog/2016-02-01-working-with-submodules/)
+-   [Icons](https://css.gg/)

assets/_custom.scss

@@ -47,6 +47,23 @@ input.toggle {
     }
 }
 
+.markdown .permalink {
+    color: inherit;
+    text-decoration: none;
+    &:visited {
+        color:inherit;
+    }
+    &:hover {
+        text-decoration: none;
+        &::after {
+            font-size: 90%;
+            padding-left:0.3em;
+            color: $color-link;
+            content: " #"
+        }
+    }
+}
+
 
 // SVG Diagrams
 .diagrams-container {

assets/_icons.scss

@@ -1,9 +1,3 @@
-// Modifiers
-.gg-s-half {
-    --ggs: 0.5
-}
-
-
 // External Icon
 .gg-external {
     box-sizing: border-box;
@@ -40,4 +34,19 @@
     border-right: 2px solid;
     border-top: 2px solid;
     top: -4px
+}
+
+
+// Modifiers
+.gg-s-half {
+    --ggs: 0.5
+}
+
+
+.gg-inline {
+    display: inline-block;
+}
+
+.gg-middle {
+    vertical-align: middle;
 }

config.toml

@@ -1,7 +1,6 @@
 # baseURL = '/'
 title = 'Filecoin Spec'
 ignoreFiles= ["externals", "\\.dot$", "\\.mmd$"]
-# theme = 'book'
 canonifyurls = false
 # relativeURLs = true
 
@@ -33,11 +32,6 @@ enableGitInfo = true
   contentDir = 'content-pt'
   weight = 2
 
-# [languages.zh]
-#   languageName = 'Chinese'
-#   contentDir = 'content.zh'
-#   weight = 3
-
 [menu]
 # [[menu.before]]
 # [[menu.after]]
@@ -46,6 +40,7 @@ enableGitInfo = true
 #   weight = 10
 
 [params]
+  API = 'https://specs-api.hugomrdias.workers.dev'
   # (Optional, default true) Controls table of contents visibility on right side of pages.
   # Start and end levels can be controlled with markup.tableOfContents setting.
   # You can also specify this parameter per page in front matter.
@@ -101,24 +96,4 @@ enableGitInfo = true
 
 [module]
   [[module.imports]]
-    path = "github.com/alex-shpak/hugo-book"
-  [[module.imports]]
-    path = "github.com/filecoin-project/specs-actors"
-    [[module.imports.mounts]]
-    source = "."
-    target = "content/externals/specs-actors"
-  [[module.imports]]
-    path = "github.com/filecoin-project/go-fil-markets"
-    [[module.imports.mounts]]
-    source = "."
-    target = "content/externals/go-fil-markets"  
-  [[module.imports]]
-    path = "github.com/filecoin-project/lotus"
-    [[module.imports.mounts]]
-    source = "."
-    target = "content/externals/lotus"
-  [[module.imports]]
-    path = "github.com/filecoin-project/go-data-transfer"
-    [[module.imports.mounts]]
-    source = "."
-    target = "content/externals/go-data-transfer"
+    path = "github.com/alex-shpak/hugo-book"

content/algorithms/crypto/randomness.md

@@ -68,7 +68,7 @@ GetRandomness(dst, l, s):
     return H(buffer)
 ```
 
-{{<embed src="/externals/specs-actors/actors/crypto/randomness.go"  lang="go">}}
+
 {{<embed src="/systems/filecoin_blockchain/struct/chain/chain.go" lang="go">}}
 
 ## Drawing tickets from the VRF-chain for proof inclusion

content/intro/_index.md

@@ -30,6 +30,7 @@ and other contract mechanisms recorded on the chain continue to be processed
 over time, without requiring further interaction from the original parties
 (such as the clients who requested the data storage).
 
+
 ## Spec Status
 
 Each section of the spec must be stable and audited before it is considered done. The state of each section is tracked below. 

content/systems/filecoin_blockchain/storage_power_consensus/storage_power_actor.md

@@ -8,13 +8,10 @@ dashboardTests: 0
 
 # Storage Power Actor
 
-## `StoragePowerActorState` implementation
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/power/power_state.go" lang="go" title="StoragePowerActorState implementation" symbol="State">}}
 
-{{<embed src="/externals/specs-actors/actors/builtin/power/power_state.go" lang="go" >}}
 
-## `StoragePowerActor` implementation
-
-{{<embed src="/externals/specs-actors/actors/builtin/power/power_actor.go" lang="go" >}}
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/power/power_actor.go" lang="go" title="StoragePowerActor implementation" >}}
 
 ## The Power Table
 

content/systems/filecoin_files/data_transfer/_index.md

@@ -88,14 +88,11 @@ transport mechanism (including offline mechanisms) is acceptable.
 
 ## Data Structures
 
-**Data Transfer Types**
 
-{{<embed src="/externals/go-data-transfer/types.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-data-transfer/blob/master/types.go"  lang="go" title="Data Transfer Types">}}
 
-**Data Transfer Statuses**
 
-{{<embed src="/externals/go-data-transfer/statuses.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-data-transfer/blob/master/statuses.go"  lang="go" title="Data Transfer Statuses">}}
 
-**Data Transfer Manager**
 
-{{<embed src="/externals/go-data-transfer/manager.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-data-transfer/blob/master/manager.go"  lang="go" symbol="Manager" title="Data Transfer Manager">}}

content/systems/filecoin_markets/onchain_storage_market/_index.md

@@ -20,4 +20,4 @@ This section describes the storage deal data type and a technical outline for de
 
 ## Data Types
 
-{{< embed src="/externals/specs-actors/actors/builtin/market/deal.go" lang="go" >}}
+{{< embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/deal.go" lang="go" >}}

content/systems/filecoin_markets/onchain_storage_market/storage_market_actor.md

@@ -11,13 +11,9 @@ dashboardTests: 0
 
 `StorageMarketActor` is responsible for processing and managing on-chain deals. This is also the entry point of all storage deals and data into the system. It maintains a mapping of `StorageDealID` to `StorageDeal` and keeps track of locked balances of `StorageClient` and `StorageProvider`. When a deal is posted on chain through the `StorageMarketActor`, it will first check if both transacting parties have sufficient balances locked up and include the deal on chain. 
 
-## `StorageMarketActor` implementation
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/market_state.go" lang="go" symbol="State" title="Storage Market Actor State">}}
 
-The implementation of the Storage Market Actor can be found [here](https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/market_actor.go).
-
-## `StorageMarketActorState` implementation
-
-The Storage Market Actor Statuses can be found [here](https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/market_state.go).
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/market_actor.go" lang="go" title="Storage Market Actor" >}}
 
 The Storage Market Actor Balance states and mutations can be found [here](https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/market_balances.go).
 

content/systems/filecoin_markets/retrieval_market/_index.md

@@ -81,6 +81,4 @@ Trust establishment proceeds as follows:
 
 The retrieval market works based on the Payload CID. The PayloadCID is the hash that represents the root of the IPLD DAG of the UnixFS version of the file. At this stage the file is a raw system file with IPFS-style representation. In order for a client to request  for some data under the retrieval market, they have to know the PayloadCID. It is important to highlight that PayloadCIDs are not stored or registered on-chain.
 
-## Common Data Types
-
-{{<embed src="/externals/go-fil-markets/retrievalmarket/types.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-fil-markets/blob/master/retrievalmarket/types.go"  lang="go" title="Retrieval Market - Common Data Types">}}

content/systems/filecoin_markets/retrieval_market/deal_status.md

@@ -9,4 +9,4 @@ dashboardTests: 0
 
 # Retrieval Deal Status
 
-{{<embed src="/externals/go-fil-markets/retrievalmarket/dealstatus.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-fil-markets/blob/master/retrievalmarket/dealstatus.go"  lang="go">}}

content/systems/filecoin_markets/retrieval_market/retrieval_client.md

@@ -18,6 +18,4 @@ The Retrieval Client Depends On The Following Dependencies
 - **BlockStore**: Same as one used by data transfer module
 - **Data Transfer**: Module used for transferring payload. Writes to the blockstore.
 
-## API
-
-{{<embed src="/externals/go-fil-markets/retrievalmarket/client.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-fil-markets/blob/master/retrievalmarket/client.go"  lang="go" title="Retrieval Client API">}}

content/systems/filecoin_markets/retrieval_market/retrieval_provider.md

@@ -19,6 +19,4 @@ The Retrieval Provider depends on the following dependencies
 - **BlockStore**: Same as one used by data transfer module
 - **Data Transfer**: Module used for transferring payload. Reads from the blockstore.
 
-## API
-
-{{<embed src="/externals/go-fil-markets/retrievalmarket/provider.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-fil-markets/blob/master/retrievalmarket/provider.go"  lang="go" title="Retrieval Provider API">}}

content/systems/filecoin_markets/storage_market/_index.md

@@ -107,11 +107,9 @@ Data submitted to the Filecoin network go through several transformations before
 5. At this point, the Piece is included in a Sector together with data from other deals. The `StorageProvider` then calculates Merkle root for all the Pieces inside the sector. The root of this tree is _CommD_. This is the _unsealed sector CID_.
 6. The `StorageProvider` is then sealing the sector and the root of the resulting Merkle root is the _CommR_.
 
-## Data Types
-
 The following data types are unique to the Storage Market:
 
-{{<embed src="/externals/go-fil-markets/storagemarket/types.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-fil-markets/blob/master/storagemarket/types.go"  lang="go" title="Storage Market Data Types">}}
 
 
 Details about `StorageDealProposal` and `StorageDeal` (which are used in the Storage Market and elsewhere) specifically can be found in Storage Deal.
@@ -141,12 +139,12 @@ It is worth highlighting that a single participant can be a `StorageClient`, `St
 
 Because most of what a Storage Provider does is respond to actions initiated by a `StorageClient`, most of its public facing methods relate to getting current status on deals, as opposed to initiating new actions. However, a user of the `StorageProvider` module can update the current Ask for the provider.
 
-{{<embed src="/externals/go-fil-markets/storagemarket/provider.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-fil-markets/blob/master/storagemarket/provider.go"  lang="go" title="Storage Market Provider">}}
 
 ## Storage Client
 
 The `StorageClient` is a module that discovers miners, determines their asks, and proposes deals to `StorageProviders`. It also tracks deals as they move through the deal flow. Note that any address registered as a `StorageMarketParticipant` with the `StorageMarketActor` can be used with the `StorageClient`.
 
 Recall that a single participant can be a `StorageClient`, `StorageProvider`, or both at the same time.
 
-{{<embed src="/externals/go-fil-markets/storagemarket/client.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/go-fil-markets/blob/master/storagemarket/client.go"  lang="go" title="Storage Market Client">}}

content/systems/filecoin_mining/storage_mining/storage_miner_actor.md

@@ -8,10 +8,7 @@ dashboardTests: 0
 
 # Storage Miner Actor
 
-## `StorageMinerActorState` implementation
 
-{{<embed src="/externals/specs-actors/actors/builtin/miner/miner_state.go"  lang="go" >}}
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/miner/miner_state.go"  lang="go" symbol="State" title="Storage Miner Actor State">}}
 
-## `StorageMinerActorCode` implementation
-
-{{<embed src="/externals/specs-actors/actors/builtin/miner/miner_actor.go"  lang="go" >}}
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/miner/miner_actor.go"  lang="go" title="Storage Miner Actor">}}

content/systems/filecoin_token/payment_channels.md

@@ -134,15 +134,9 @@ Payment Channels are used in the Filecoin [Retrieval Market](retrieval_market) t
 In particular, given that there is no proving method provided for the act of sending data from a provider (miner) to a client, there is no trust anchor between the two. Therefore, in order to avoid mis-behaviour, Filecoin is making use of payment channels in order to realise a step-wise "data transfer <-> payment" relationship between the data provider and the client (data receiver). Clients issue requests for data that miners are responding to. The miner is entitled to ask for interim payments, the volume-oriented interval for which is agreed in the Deal phase. In order to facilitate this process, the Filecoin client is creating a payment channel once the provider has agreed on the proposed deal. The client should also lock monetary value in the payment channel equal to the one needed for retrieval of the entire block of data requested. Every time a provider is completing transfer of the pre-specified amount of data, they can request a payment. The client is responding to this payment with a voucher which the provider can redeem (immediately or later), as per the process described earlier.
 
 
-## Payment Channel Implementation
+{{<embed src="https://github.com/filecoin-project/lotus/blob/master/paychmgr/paych.go" lang="go" title="Payment Channel Implementation">}}
 
-{{<embed src="/externals/lotus/paychmgr/paych.go" lang="go" >}}
+{{<embed src="https://github.com/filecoin-project/lotus/blob/master/chain/types/voucher.go" lang="go" title="Payment Channel Voucher">}}
 
-## Payment Channel Voucher
-
-{{<embed src="/externals/lotus/chain/types/voucher.go" lang="go" >}}
-
-## Payment Channel Actor
-
-{{<embed src="/externals/specs-actors/actors/builtin/paych/paych_actor.go" lang="go" >}}
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/paych/paych_actor.go" lang="go" title="Payment Channel Actor">}}
 

content/systems/filecoin_vm/actor/_index.md

@@ -17,6 +17,6 @@ There are eleven (11) types of _builtin_ Actors in total, not all of which inter
 
 The _actor address_ is a stable address generated by hashing the sender's public key and a creation nonce. It should be stable across chain re-organizations. The _actor ID address_ on the other hand, is an auto-incrementing address that is compact but can change in case of chain re-organizations. That being said, after being created, actors should use an _actor address_.
 
-{{<embed src="/externals/specs-actors/actors/builtin/singletons.go"  lang="go">}}
+{{<embed src="https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/singletons.go"  lang="go">}}
 
 The `ActorState` structure is composed of the actor's balance, in terms of tokens held by this actor, as well as a group of state methods used to query, inspect and interact with chain state.