Publish specifications to specs.ipfs.tech

[hacdias]

⚠️ This PR must not be squashed in order to preserve history.

This PR has as a goal to clean up some of the specs and move them to a mature repository, where specs are published to https://specs.ipfs.tech.

Try it out locally:

1. Clone PR
2. npx spec-generator -c .config.json -w

Things to do:

• Editors based on GitHub history by chronological order
• Add date: to the metadata based on when the spec was last updated
• Add index.html
• Move the mature IPFS specs to the designated repository to generate the website (src / mature)
• Ensure all links are working properly between specs
• Decide on how to link the old markdown files to the new ones (link to markdown or to website?)
• Move the Fleek config here for specs.ipfs.tech - disable elsewhere.

[hacdias]

@lidel @darobin I think the most important next step is to add PR previews with Fleek, and also publish the main branch to specs.ipfs.tech.

1. I moved all gateway and IPNS specs to the src directory, which contains the specs. I added links in the old markdown files indicating that they have been moved. These now link to specs.ipfs.tech - once we publish them, the links will work properly. Updated all the links to RFCs to citations, looks nice.
2. Should I move the IPIPs and the IPIP Process spec too? If so, do we want links like specs.ipfs.tech/ipip/XXXX or with the full name? We should also update the IPIP process to indicate the new directories.
3. I looked at all the other specifications and they all say "wip". Should I move them too? @darobin added a maturity field to the frontmatter we could use for advantage.

[darobin]

@hacdias I think that it would be good to ship something good enough and then we can figure out how to improve on it.

I'd be in favour of putting all the specs on the site and using maturity to indicate which ones are good and which are not. I think that giving visibility to stuff that's less camera-ready is valuable as well.

For PR previews, I was thinking that what would be nice would be to have PR branches exposed at prXXX.specs.ipfs.tech but I haven't looked at what that involves on the Fleek side. For IPIPs, we might want to process them somewhat differently compared to the specs, no?

Either way, I think we might be better off shipping what's here (or something close to it) to main & specs.ipfs and then iterating?

[hacdias]

@darobin definitely, I would be happy to publish all and use the maturity for that.

For IPIPs, we might want to process them somewhat differently compared to the specs, no?

Probably, we can just keep them as plain markdown, or publish them with some different styles?

Either way, I think we might be better off shipping what's here (or something close to it) to main & specs.ipfs and then iterating?

I do agree with that. I think PR already has good enough stuff to publish to specs.ipfs.io - I would just prefer that @lidel took a look at it before 😃

And also someone to setup the Fleek deployment on this repository!

[darobin]

Probably, we can just keep them as plain markdown, or publish them with some different styles?

The way I think about it, the more we publish, the more we give visibility to the work, the more we make it look like "not just a thing on GitHub", the better it is for both the project and its contributors.

Just a quick thought but we could add a matchProposals config (or a frontmatter field) to identify IPIPs, and based on that give them a different style and include some different metadata in the headers (like start date and related issues).

Fleek

I used my personal Fleek account to set up the demo. Do you know if we have anything more official?

👍 on @lidel review first :)

[hacdias]

Do you know if we have anything more official?

I think so, but I'm not sure /cc @lidel

[lidel]

This is awesome, thank you so much for pushing on this ❤️

• ⚠️ 💭 I don't think many specs "meet the bar"
  • I would not publish anything more than Gateway and IPNS specs for now
    • We may also include BITSWAP one, which describes wire format, but that's probably all we have to show atm.
  • Unixfs is not ready: PR is wip, but needs a lot of editorial work.
  • I know it is frustrating, but it is not just "low quality" or "unfinished specs" – some things are outdated, invalid, or both. We need to audit existing content base-by-case, and remove it from repo, because it is just noise (I'm happy to advise on cleanup PR, if needed)
  • Adding IPIP process description and figuring out permalinks for each IPIP sgtm, but should be separate PR – let's keep scope small.
• 💭 would be cool if http-gateways/ had index.html that is the same as HTTP Gateways section on the main website (but having unstyled dir listing for now is also ok)
• 💭 moving ipns/ to ipns/record/ will probably save us a lot of headache in the future
  • This way, one can enumerate all IPNS specs by opening ipns/
    • We are missing DHT router description, we may have double-hashed records too, there will be also IPNI one in the future i think
• 🟢 I've set up Fleek
  • using the same account as docs.ipfs.tech for now
  • build is defined in Makefile
    • make website is expected to produce website in ./out directory
  • Opened https://github.com/protocol/infra/pull/1140 to update DNS config but need @darobin to remove it from his account first (details on the linked PR, but DM me if you have no access)
    • We should do this after the Makefile lands in main branch.
  • Enabled PR previews based on make website

    Example: https://bafybeihl4rvlt2mh2cjw7qwraxcweqd3qbuql6jaudfn35uie2r3gtz47y.on.fleek.co
    

• 🔴 We kept Markdown so we should be able to avoid losing git history.
  • This is avoidable if we moved files first, made a commit, and then created placeholder in the old place using separate commit (I did that for IPNS before and worked fine).
    • @hacdias it is not too late to fix it with either a rebase or manually copying files in a new PR – either works, and saves us from losing it 🙏

[hacdias]

@lidel

⚠️ 💭 I don't think many specs "meet the bar"

I agree. Let's keep the scope of this PR small and then I can work on the IPIPs in a separate PR. I would love to see this published.

💭 would be cool if http-gateways/ had index.html that is the same as HTTP Gateways section on the main website (but having unstyled dir listing for now is also ok)

I added some minimal indexes for http-gateways, ipns and meta. I tried to use the templating system to re-use some stuff. I added minimal navigation. My idea was to use the main page to have a list and then inside each index make further distinctions (such as HTTP vs. Web gateways). I can restore the headers in the main index though.

💭 moving ipns/ to ipns/record/ will probably save us a lot of headache in the future

Yes. I agree. I moved the following:

• ipns/ to ipns/ipns-record
• ipns/pubsub-router to ipns/ipns-pubsub-router
• http-gateways/redirects-file to http-gateways/gateway-redirects-file

The reason for renaming (that applies to all of them) is that we cite as :cite[filename]. If we did ipns/record, we would have to :cite[record] which is extremely unclear. The URLs don't look as nice, but I think it is a necessary evil to ensure citations are clear (applies to the remaining moves).

🟢 I've set up Fleek

🎉

🔴 We kept Markdown so we should be able to avoid losing git history.

Rebased and taken care of! It works flawlessly. Let's just not forget to not squash when merging.

[hacdias]

@darobin can we remove/disable the dark mode for now? And then add/improve it after?

[hacdias]

@lidel I see there's a lot of reference failures on the new previews. Do you know why that may be caused by? AFAIK, ipseity needs to be able to connect to the Internet during build.

[lidel]

• I've updated DNS, https://specs.ipfs.tech works.
  • The content is published from try-ipseity branch for now, I will update it to main when this PR gets merged to main branch.

I'd like us to merge this soon, but there are some things that we need to address first:

• a toggle in UI and cookie to enable/disable dark mode (or disable it for now, and focus on polishing light mode)
• canonical link is missing
  • We want to copies on gateways to contribute SEO to the canonical URL, so every spec needs to have canonical link defined, for example:

  <link rel="canonical" href="https://specs.ipfs.tech/meta/spec-for-specs/">

  I suspect we need to modify template.html or update https://www.npmjs.com/package/spec-generator to support this. @darobin thoughts/ideas? This is to avoid issues like Block internet search engines from indexing the mirror distributed-wikipedia-mirror#48
• figure out why dig +short ALL _dnslink.specs.ipfs.tech @1.1.1.1 returns no results – moved to Fix DNSLink at specs.ipfs.tech #393
  • TTL is 3600, so 1h, but maybe some aggressive caching happens on dnssimple? let's wait 24h
• add missing bibliographic references
  • I think we need to add these before merge @hacdias @darobin

[hacdias]

@lidel regarding references: everything works well when I build locally. I think there are two problems at play here:

1. npm install -g and npx behave differently, as their node_modules is located in a different place relative to the package. In spec-generator, there is a harcoded path for the webref dependency that does not work properly through npx, explaining many of the non-working references.
   • For this, I pushed a commit to use npm install -g in the Makefile. However, I would prefer to keep using npx. For now, it works. If you see the latest preview, you'll notice that the only failing citations/refs are local. See point 2.
2. References within the website itself will work once the website is published once. But it already is! So, what's wrong? The problem is that we're publishing the references and definitions to one path, and then looking for them in a different place. I opened a PR: Spec Generator Improvements spec-generator#1

/cc @darobin

ipfs/spec-generator#1 also adds support for canonical links in the specs, we may also need to do something to add it to the other pages.

[darobin]

• Dark mode removed
• PR for local lookup fix and canonical folded in
• New release pushed out (this is still published to npm from my account — happy to switch to publishing from org account, but that's not high priority)
• Regarding the problem with webref: there is an upstream issue in webref that will eventually fix this, but it's not there yet. I will look at better resolution methods in the meantime.

Getting there!

[lidel]

💭 running install on every make watch is slow, we can avoid it if its already installed in proper version

[hacdias]

How can I add such check in the Makefile? We can't use npx here for the reasons I've outlined before.

[lidel]

💭 It does not seem to bring much value in the top level dir, can we move it to src/_includes to reduce noise?

[hacdias]

Well, we probably can, but src/_includes is a special repository for the static website generator that backs spec-generator (https://www.11ty.dev/). And template.html is a file specifically required by spec-generator that works differently from the 11ty system. So I'm not sure if it's the best idea.

[lidel]

Continued in #391

[lidel]

💭 would it be possible to separate content from website, so it is easier to work with for people who refuse to deal with anything other than Markdown on Github?

@darobin @hacdias how about: keep src/_includes, src/css, but everything else move to content/ or text/?

[hacdias]

@lidel this has been discussed before. This is the format expected by Eleventy. It is possible to change it, but that requires changes to spec-generator afaik.

[lidel]

Continued in #392

[lidel]

Given:

• various places already pointing at https://specs.ipfs.tech/,
• work like Blog post on implementations and principles ipfs-blog#550 and ongoing IPIPs waiting for this to rebase, - IPFS thing being <3 weeks from now

@darobin ok merging as-is?

We can improve and address UX papercuts for editors later, but better to land this with some buffer before IPFS thing.