Gather and link existing docs and resources

[Mr0grog]

Fixes #59, #36, #35.

This makes a lot of very messy changes to the site and recasts what we have as mostly links to existing documentation and resources in various repos across the ipfs org. Resources from the existing docs site are copied instead of linked because this site is meant to replace that one (with the notable exception of the examples). Features of this site that were totally broken (e.g. search, submenus in the sidebar) are hidden or overridden just enough to get them functional.

The code and visuals here are not meant to be the focus — in fact, they’re all pretty bad and will need plenty of re-working in the future. This PR is about making sure the site is:

• Basically workable for users (not full of pages with no text except TODO; no frustratingly broken features like search)
• Covers all the major resources we already have in a clear, understandable, navigable way
• Provides sensible spaces for people to contribute new or improved documentation

I’ve organized things here into four major categories with several subcategories:

I'm hoping this lends a little more clarity and organization to things where we had a lot of major topic areas before. We still have those same sections, but now they are organized under 4 main sections:

1. Introduction (this is basically just the existing “getting started” guide, plus a link to the FAQ)
2. Guides
   • Concepts (These are all new. The docs here don’t cover nearly all the concepts we need, but it’s a start. Each is also on the minimal side; they could definitely be fleshed out more. See [NEW CONTENT] [BOUNTY] Glossary #56 for more about this.)
   • Guides (This is a little redundant. Trying to get the nav working nicely made this the most expedient way to do things, but it probably makes sense to eventually move the links here directly into the top-level “guides” section.)
   • Examples
3. Reference
   • Commands & API
     • CLI Commands (Freshly re-generated, with a little extra descriptive info at the top)
     • HTTP API (Freshly re-generated, with a little extra descriptive info at the top)
     • JS & Go Libraries (This is a link to ipfs/interface-ipfs-core, but folks have now told me the Go parts may never be filled in, so I think this needs some re-thinking.)
   • Go Implementation (All the written guides that are specific to the Go version. Also includes generated GoDoc package documentation for coreapi and go-ipfs-api. @lgierth had already written tooling to automatically pull in and generate these. Are these the right ones to start with? What should we add? Should we keep them at all?)
   • JS Implementation (A very brief high-level overview of ways to use IPFS in JS, plus links to js-ipfs and js-ipfs-api repos. This does not have package API docs like the Go section does because the JS libraries don’t have much in the way of generatable docs. What they do have is semi-structured in the READMEs — if we came up a standard way to structure it, we could include them here, but I eventually gave up on finding a way to do this that worked well and didn’t feel totally janky. This requires a broader effort. cc @diasdavid @alanshaw + other JS folks)
   • Specifications & Planning (In addition to specs, the whitepaper, roadmap, and implementation status docs are here. Not sure if there’s a better place for them.)
4. Community (To be honest, this section turned into a bit of a grab-bag.)
   • Applications (Official/semi-official applications built on top of IPFS. This is a short list; I think we should keep it to things that are really pretty solid efforts and which are at least sort of ready for serious usage. Cluster has already gone ahead and made its own site and documentation, which seems good; I’m not sure this site can do a perfect job of serving all the specific needs around Cluster and around IPFS proper, so this was a good place to link to it. Are there other things I missed that should be linked here?)
   • Community (Like guides/guides, this is kinda redundant ¯\_(ツ)_/¯ — should these also be lifted up so each link is a direct child of the top-level community section? Would love some feedback here.)
   • Get Involved (Includes a very lightly edited copy of @victorbjelkholm’s work in [WIP] Add contribute page website#237, plus links to other docs that felt really important — contribution guidelines, code of conduct, and style guides.)
   • Related Projects (Links to Libp2p, IPLD, and Multiformats projects. These are links to the home pages for those projects, not the Github orgs. Is that right? Should shipyard be here?)

In this version, nearly everything is listed directly in the sidebar. Hopefully this structure allows us to try some nicer things with navigation and layout (e.g. the top level sections could be collapsible, or maybe they could be listed in the top bar, while the sidebar just has the links for the current subsection). This version goes as far as tweaking the style to include some resources from ipfs-css, but I wanted to do the simplest thing that worked for now and leave better design and layout for a subsequent PR with more feedback :)

Additional Notes

The Reference Section

I think there are a lot of unsolved questions and concerns about various items in the reference section, though I think what’s in this PR might be acceptable for now so people can at least find all the relevant bits.

• The CLI and HTTP docs are separated from the Go and JS implementation sections. We only have one unified set of docs for these right now, so I at least think this sort of makes sense. Since the HTTP API for Go can be used from JS and vice-versa, I do think it makes sense to keep that reference at the top level (though we need to add clear info about what endpoints are supported for each and in what versions). I’m not sure if this is as true for the CLI. While JS and Go are largely aiming for parity here, my understanding is that there will probably always be a few commands that are unique to each.

• The “JS & Go Libraries” section that links to the ipfs/interface-ipfs-core docs really only covers the JS library. It appears as though it was intended to cover Go, too, but what I’ve uncovered from asking around is that many folks aren’t actually working towards that, let alone are even sure it totally makes sense. I’ve kept it in the common “commands & API” section for now because A) that’s all still unclear and B) pretty much every outside developer who’s seen it has found it to be extraordinarily useful compared to the HTTP docs because it includes clear examples of use and output. The HTTP docs only show schemas, which is not nearly as helpful for many folks (see also the research docs). However, all of this really needs some thinking and resolution:

  • The interface docs are still not a perfect replacement for the HTTP docs — they don’t discuss how arguments are set over HTTP and they don’t cover some more complicated cases like https://discuss.ipfs.io/t/use-http-to-get-a-file-why-has-extra-data/3039/3
  • If the Go bits are never really going to be documented in the interface docs, we should be clear about that and remove all the references to Go. We should also figure out if this is the best way to document that API rather than just test it. (Should we move those docs here? Should we copy them here as part of the build the way we generate the Go package docs? Should we reformat them as JSDoc comments and generate them here? (That would certainly allow for some fancier formatting.) Should we do something that allows them to feed into package/API docs for js-ipfs and js-ipfs-api [see above]?)

• The HTTP docs are generated from the Go implementation, where the documentation is very focused on the CLI (so it doesn’t explain how things work over HTTP when the HTTP version differs drastically, as in the /get discussion linked above. This also guarantees that it does a poor job covering differences between the Go implementation and the JS implementation. (Which endpoints where implemented in which versions; which aren’t implemented yet in one or the other implementation; which don’t quite work the same?) At this point, I’m feeling that the right future direction for the HTTP doc is to write and update it manually (rather than generate it), even though that may be a lot of work.

• The CLI docs are largely just the same as using ipfs --help on the CLI. How much does it make sense to keep them? What about the differences between go-ipfs and js-ipfs as noted with the HTTP API above? If we keep them, should we move them under the Go Implementation subsection?

I’m happy to make issues here in which to discuss these if folks think they warrant it. Hopefully we can keep most commentary on this PR to what is relevant to shipping it — i.e. do these docs need to move or change now?

The Old Examples

The examples from the old docs site are not quite perfectly copied over and the page for them here (http://localhost:1313/guides/examples if you’ve got it running locally) links to the old site. They are copied here in the content/guides/viewer directory, but marked as drafts so they don’t get published. I went back and forth on a few questions around these:

• The idea of an “examples viewer” based on bl.ocks.org is a neat one, but it’s yet another thing to maintain. It doesn’t even fully serve the existing examples (e.g. https://ipfs.io/docs/examples/example-viewer/example#../api/service/readme.md has a lot more stuff you can’t see in the viewer). Should we continue with the viewer idea or just recast these examples as normal pages on the site?

• They are all pretty old and some are out-of-date enough to be potentially inaccurate. Should we spend some time auditing them for correctness or would it be better to throw them out?

Other Technical Bits

• I tried to largely keep the build workflow with make the same as it is in ipfs/website and ipfs/ipfs-cluster-website, using Less, autoprefix, and clean-css. However, the latest version of everything doesn’t seem to work together perfectly (e.g. I ran into this bug in Less). As we move towards cleaning up the layout and design, maybe we want to switch things up (e.g. use PostCSS, build with Webpack or Parcel)? I’m also not 100% sure Hugo is the best fit (this nav bar code is horrifying, and while it could be better, it would still be pretty complicated because of Hugo’s lack of plugins). Again, I wanted to keep the the changes minimal and simple for this PR, but there’s definitely plenty to think about here.

• I pulled in ipfs-css, but didn’t really wind up using it directly. Instead, it copies the font files in a make step and uses another script to convert theme.json into a set of Less variables (since I’m using the colors in ways that aren’t perfectly well supported by the included Tachyons-style CSS rules). I had to make my own copy of the CSS that references the fonts, though, because it’s not separable from the Inter-UI font CSS (which we don’t use here) in the distributed package. I’m happy to post some issues on that module about making that process better supported. /cc @olizilla

• I’ve hidden the search input because it doesn’t actually work (I didn’t remove it from the code, again, because I realized I didn’t want to go too far in changing the site layout here). Even though it’s really useful, I think we should simply drop search for now. Figuring out how to make it work well when half the content isn’t actually in this site is its own crazy pile of work.

• The existing JS seems to get stuck in a pretty tight loop and can eventually start taxing your CPU, so fixing it up or just throwing it out and entirely redoing the layout (the existing theme requires the JS for proper layout) is pretty important in a subsequent PR. I think the theme we have here isn’t an actual public Hugo theme, but instead a result of @lgierth half-porting this theme for MkDocs. The assets are all pre-minified, so it’s a bit of a mess to try and actually fix.

• The menu/nav system is a little nutty and flips between an old system of enumerating all the contents of a section and using Hugo’s built-in menu system (originally needed because of the external links, but then I also made this ugly hack). At any rate, this really needs cleanup:

  • We can use the old method of enumerating the contents of each section, with hacks like the FAQ’s external frontmatter field to get external links. It’s not well supported in Hugo and results in some really complicated code, but it does keep things visible in the directory structure instead of in the config.toml file.

  • We can just wholesale use the Hugo menu system, which is well supported and more clean, if limited in functionality.

  • We can ditch both those and just encode the menus in the layout.

TODO

• Per @olizilla, switch to Less v2 because of bugs :(
• Re-generate HTTP API docs to display better type info for CIDs and multiaddrs (Bitswap wantlist API ipfs/kubo#5077)
• Signoff/changes from @akrych on fonts
• Feedback on selection of Go packages that we are generating docs for

[Mr0grog]

@victorbjelkholm is Jenkins using an up-to-date version of Hugo? I don’t get these errors locally using v0.41:

ERROR 2018/06/01 23:22:51 error processing shortcode "_internal/shortcodes/ref.html" for page "introduction/usage.md": template: _internal/shortcodes/ref.html:1:73: executing "_internal/shortcodes/ref.html" at <ref .Page (.Get 0)>: error calling ref: No page found with path or logical name "guides/examples".
ERROR 2018/06/01 23:22:51 Error while rendering "page": template: /site/themes/material/layouts/_default/single.html:52:19: executing "/site/themes/material/layouts/_default/single.html" at <partial "nav" .>: error calling partial: template: theme/partials/nav.html:42:77: executing "section-menu-standard" at <.Title>: can't evaluate field Title in type *hugolib.MenuEntry
make: *** [build] Error 255

[Mr0grog]

In the mean time, you can browse a rendered preview at: https://ipfs.io/ipfs/QmVUdHfpo9hyC8wXmgd2frRrsp83iRvuL8HWyp1LPzjsPq/

[olizilla]

This is very rad @Mr0grog!

This is a link to ipfs/interface-ipfs-core, but folks have now told me the Go parts may never be filled in

Can we fix that? Who do we need to convince @diasdavid @whyrusleeping?

JS Implementation (A very brief high-level overview of ways to use IPFS in JS, plus links to js-ipfs and js-ipfs-api repos. This does not have package API docs like the Go section does because the JS libraries don’t have much in the way of generatable docs.

@alanshaw let's JSDoc js-ipfs and js-ipfs-api. I don't see any reason not to, and am happy to start PRing things.

Related Projects (Links to Libp2p, IPLD, and Multiformats projects. These are links to the home pages for those projects, not the Github orgs. Is that right? Should shipyard be here?)

No, i don't think shipyard should be in there. it's not a coherent thing, but a staging area for many projects and ideas.

At this point, I’m feeling that the right future direction for the HTTP doc is to write and update it manually (rather than generate it), even though that may be a lot of work.

A useful http api doc would be accurate for a given implementation and version, so you know exactly what to expect. Ideally it should let you diff any differences between versions. being able to diff between implementations at given versions would be super rad. Something like: http://apidocjs.com/ (example docs: http://apidocjs.com/example/) could work for creating versioned, generated docs.

If we go the other route and create a separate, hand crafted api documentation, then it would be most useful if it was in the form of an api spec, that we could then use to show where each impl complies and drifts from the ideal. Somthing like Swagger / openAPI / RAML. I'd go for that over hand crafted html api docs; I think they'd just be different again, and probably out of date, and not programatically verifiable without a lot of bespoke work.

The CLI docs are largely just the same as using ipfs --help on the CLI. How much does it make sense to keep them?

No harm in making help text available over http. It allows you to discover commands via your search engine of choice.

I tried to largely keep the build workflow with make the same as it is in ipfs/website and ipfs/ipfs-cluster-website, using Less, autoprefix, and clean-css. However, the latest version of everything doesn’t seem to work together perfectly (e.g. I ran into this bug in Less)

Yep, i've used less@2 for all the other sites because of that, and it works fine.

As we move towards cleaning up the layout and design, maybe we want to switch things up (e.g. use PostCSS, build with Webpack or Parcel)?

I love new shiny things, but there are a bunch of sites to maintain so I'm leaning towards consistency, unless there are issues, like...

I’m also not 100% sure Hugo is the best fit (this nav bar code is horrifying, and while it could be better, it would still be pretty complicated because of Hugo’s lack of plugins)

Strongly agree. I find Hugo's template system deeply bizarre. I feel I must be missing a super strong grounding in Go templates. Anything even mildy data driven gets really complex. And the error messages are infuriatingly cryptic too. I'd gladly discuss moving to something else, the challenge is finding a static site generator that can deal with relative paths for links and resouces like css background-image urls and inline styles.

ipfs-css: pulled in ipfs-css, but didn’t really wind up using it directly. Instead, it copies the font files in a make step and uses another script to convert theme.json into a set of Less variables

Did you see: https://github.com/ipfs-shipyard/ipfs-css#less
Happy to chat about it on irc, or just ping me an issue on ipfs-css.

[Mr0grog]

@alanshaw let's JSDoc js-ipfs and js-ipfs-api. I don't see any reason not to, and am happy to start PRing things.

Sweet! I was about to experimentally start trying that myself, but if you want to jump on it, that’d be fantastic. I made a new issue to track this, and included two obvious concerns to think about: #70.

A useful http api doc would be accurate for a given implementation and version, so you know exactly what to expect…

@olizilla I wrote up some thoughts on specifically this a couple days ago in #69. Let’s move this conversation there, unless you think this should not be merged as-is :)

Yep, i've used less@2 for all the other sites because of that, and it works fine.

👍 even though having to be behind by a major version makes me sad. I’ll look into switching this to Less v2.

I love new shiny things, but there are a bunch of sites to maintain so I'm leaning towards consistency

👍 👍 to that.

I’m also not 100% sure Hugo is the best fit

Strongly agree. I find Hugo's template system deeply bizarre. I feel I must be missing a super strong grounding in Go templates.

FWIW, I think Go templates are really designed to be used from Go code, so your logic would live in your program, rather than something like Hugo, where the author of the template is not also the author of the program.

Anyway, let’s use #71 to discuss. I’m thinking the Hugo question is too open-ended of a discussion to be allowed to block this PR.

Did you see: https://github.com/ipfs-shipyard/ipfs-css#less

Yes! The problem there is that I have to include parts I don’t need (like InterUI) and that it’s not flexible enough to handle things like this or arbitrary gradients, etc. Anyway, I’ll file some issues over on ipfs-css and bug you more on IRC :)

[daviddias]

On JSDocs, check out ipfs/js-ipfs#651 and ipfs/js-ipfs#615

[daviddias]

Good resource on starting a new IPFS implementation -- https://github.com/ipfs/community/issues/177

[daviddias]

README creation, guidelines and automation - ipfs/community#45

[daviddias]

List of previous top topics that need better docs - https://github.com/ipfs/community/issues/199

[olizilla]

For the font, I'm only using Monserrat for headers text. It's not legible enough for longer blocks of text. I vote for sticking to the users system ui font stack for everything but the topbar. Legibility and familiarity are much more valuable than branding here. We'll get the brand guide updated.

Otherwise, this change is too important to leave in a PR much longer; Let's get it merged! As you say, with this we can finally direct people to update a canonical resource for whatever docs may be missing currently.

Can I vote to make @Mr0grog the lead maintainer of this repo, so he can make the call as to when this PR is ready to merge? We just went through a useful exercise in leadership delegation across the js-* modules and it's working well.

[whyrusleeping]

This is a link to ipfs/interface-ipfs-core, but folks have now told me the Go parts may never be filled in

Can we fix that? Who do we need to convince @diasdavid @whyrusleeping?

Idk, is this like coreapi stuff? Maybe ping @magik6k about it

[magik6k]

This is a link to ipfs/interface-ipfs-core, but folks have now told me the Go parts may never be filled in

Can we fix that? Who do we need to convince @diasdavid @whyrusleeping?

Idk, is this like coreapi stuff? Maybe ping @magik6k about it

Yep, coreapi is what will eventually be the 'go part' of interface-ipfs-core. Currently coreapi lives in go-ipfs repo because it's easier to develop it there. The roadmap is in ipfs/kubo#4498

[Mr0grog]

[Montserrat is] not legible enough for longer blocks of text. I vote for sticking to the users system ui font stack for everything but the topbar. Legibility and familiarity are much more valuable than branding here. We'll get the brand guide updated.

I was going by the style guide in ipfs-inactive/ipfs-ui-style-guide#3, which says Montserrat for body text, but I am totally fine with changing this (I agree that Montserrat Regular is not super legible at a regular 16px body size; it’s a little better on a high-res screen, but still not ideal). Can you comment, @akrych? I want to make sure you agree as well :)

@1x type specimen for reference:

@2x type specimen, where Montserrat Regular is a little more readable:

[Mr0grog]

Yep, coreapi is what will eventually be the 'go part' of interface-ipfs-core. Currently coreapi lives in go-ipfs repo because it's easier to develop it there.

Moving this conversation to ipfs/kubo#3176 — I wanted to surface the issue, but probably should have done so there so it doesn’t distract from this PR (I don’t believe it’s germane to getting something currently relevant shipped for this site). Sorry.

[Mr0grog]

This Go question, however, is important: @whyrusleeping or @magik6k or @Stebalien, are the generated package docs we have here now the right ones for right now? (go-ipfs/core/coreapi and go-ipfs-api) If no, which (if any) should be here?

[Mr0grog]

Can I vote to make @Mr0grog the lead maintainer of this repo, so he can make the call as to when this PR is ready to merge?

I’m going to go ahead and do that anyway :)

I need to execute on the two TODO items in this PR description:

• Switch to Less v2 because of bugs :(
• Re-generate HTTP API docs to display better type info for CIDs and multiaddrs (Bitswap wantlist API ipfs/kubo#5077)

And I need feedback in order to execute on two others (if I don’t hear back relatively soon, I’ll just make a decision and move forward):

• Signoff from @akrych on fonts
• Are the generated package docs we have here now the right ones for right now? (go-ipfs/core/coreapi and go-ipfs-api) If no, which (if any) should be here?

After that, I’m going to merge, since there have been no real concerns on the principal content of the PR.

[magik6k]

This Go question, however, is important: @whyrusleeping or @magik6k or @Stebalien, are the generated package docs we have here now the right ones for right now? (go-ipfs/core/coreapi and go-ipfs-api) If no, which (if any) should be here?

Current situation as I understand it is as follows: CoreAPI is experimantal (i.e. might change), but can be used, is quite well documented and is aiming to be /the/ interface to ipfs in Go. go-ipfs-api is in use now, but it will likely be replaced by a coreapi implementation(ipfs/go-ipfs-api#76)

The answer really depends on how quickly coreapi efforts move (most of the current work is blocked by ipfs/kubo#4672).

[Mr0grog]

CoreAPI is experimantal (i.e. might change), but can be used

Cool, so it sounds like it makes sense to keep here. Can we some how note its experimental status in the code comments? Maybe:

// Package coreapi provides direct access to the core commands in IPFS. If you
// are using an in-process IPFS node, you should use this package to read and
// write files or otherwise control it.
//
// If you are working with an out-of-process IPFS node, you should use
// `go-ipfs-api`. As we finalize the interfaces here, they will replace
// `go-ipfs-api`, but for now, they only work with in-process nodes.
//
// **NOTE: this package is experimental.** go-ipfs has mainly been developed
// as a standalone application library-style use with this package is still
// new. Interfaces here aren't yet completely stable.
package coreapi

import (

is quite well documented

Do you mean the Godoc generated from it? Otherwise, can you point me to the documentation? I feel like I’ve heard statements like this a couple times, but wasn’t really sure what docs they were referencing.

As a side note, the godoc generation script @lgierth put together here (predates me) doesn’t seem to include subdirectories/subpackages like godoc.org does. Would it make sense to manually add them for now, like this (Note separate entries in the sidebar for coreapi, coreapi/interface, coreapi/interface/options) Asking because I don’t really write a lot of Go, so I want to make sure this makes sense as a temporary fix.

go-ipfs-api is in use now, but it will likely be replaced by a coreapi implementation

OK, seems like this is still relevant now and makes sense to keep, then.

Thanks!

[akrych]

@Mr0grog

[Montserratis] not legible enough for longer blocks of text. I vote for sticking to the users system ui font stack for everything but the topbar. Legibility and familiarity are much more valuable than branding here. We'll get the brand guide updated.

I was going by the style guide in ipfs-inactive/ipfs-ui-style-guide#3, which says Montserrat for body text, but I am totally fine with changing this (I agree that Montserrat Regular is not super legible at a regular 16px body size; it’s a little better on a high-res screen, but still not ideal). Can you comment, @akrych? I want to make sure you agree as well :)

Yeah, we was already talking about it with Oli and we change font for web to "Inter UI" - just don't had time to update the Brandbook, sorry :) So you can use Montserrat to titles and Inter UI for text 👍

[Mr0grog]

OK, this should be merge-ready. Preview: https://ipfs.io/ipfs/QmNNiMkXKMD5AhhFinvGZN251mAi13bx6coThhQYVb1mgV/

I’ll merge in a few hours if nobody has additional feedback.

[whyrusleeping]

👍 👍 👍 🚢

This helped me already, I couldnt find the js-ipfs examples, so i came here and found them through this page :)

[daviddias]

I couldnt find the js-ipfs examples,

Hm.. 🤔 I'm curious why. A search for examples on https://github.com/ipfs/js-ipfs will show you a folder full of them on first hit (which I've linked to you in the past). What makes it so hard?

[whyrusleeping]

@diasdavid they arent linked to by any of our webpages, and googling for them doesnt return me great results. Theyre easy to find if i remember that they are inside the top level js-ipfs repo, but if i'm (for example) looking for an example of how to simply use ipld in javascript, my first thought isnt to look in the js-ipfs repo. Its to look in the js-ipld repos, which doesnt have any examples in it. Then i go to googling, and didn't really find much (though now, searching 'js ipld examples' gives me the right link as the second result... odd).