Content (was pages) design ideas

[rcoreilly]

This is a discussion of ways to organize markdown-based web content in a scalable, flexible way.

We currently have a repo called pages that is used for the core docs at: https://cogentcore.org/core and we want to be able to support everything from blogs to a larger-scale wiki-like collection of pages, etc. Target functionality includes things like Hugo https://gohugo.io/ and wordpress and wikipedia, etc.

The current design has a single tree panel on the left, and then the current page, with nav buttons on the page that can move you along. The docs content is organized hierarchically with numbers indicating the ordering within level, and the tree browser reflects this.

The main problem is that a hierarchical structure does not scale well. We are already starting to see this with the docs. Instead a more heterogenous, flat, tag-based organization generally scales better. The other problem is that we recently used the name "Pages" for a new widget type in #1293. I'm suggesting the name content as a good description of what this package does: manages and marshals web content.

The large-scale paradigmatic example is Wikipedia / Mediawiki. Some key design features:

• Flat namespace with disambiguation pages to resolve conflicts, and typically various template-based links at the start of a page to redirect to other pages of relevance / confusion. Some pages serve as higher-level nodes that feed into many subtopics, but everything is in the same flat namespace.
• Left tree is an index within the current article, not to other articles.
• Extensive use of templates to create various topic-related navigation boxes (navbox), along with more automated Category tags that generally live at the bottom of a page, providing links to other pages with the same categories.

Example navbox template, from https://en.wikipedia.org/wiki/Emergence :

which is included at the top of the page along with other such templates:

{{Short description|Unpredictable phenomenon in complex systems}}
{{Other uses}}
{{See also|Emergent (disambiguation)|Irreducibility (disambiguation)|Spontaneous order|Self-organization}}
...
{{Complex systems}}

Example of another organizational template at the end:

{{Cybernetics}}
{{Philosophy of biology}}

And a category tag list at the end, where the links pull up alpha-organized glossary / index-like pages that provide broad linking to related topics (perhaps too broad):

[[Category:Emergence| ]]
[[Category:Concepts in metaphysics]]
[[Category:Metaphysics of mind]]
[[Category:Pattern formation]]
[[Category:Concepts in the philosophy of mind]]
[[Category:Holism]]
[[Category:Ontology]]
[[Category:Philosophy of physics]]

The navbox is itself a template, with lots of collapsable content etc.

Hugo also makes extensive use of templates. Templates give a lot of power at a high price in terms of learning curve and functionality limitations, and general overall complexity.

Both Hugo and wikipedia are highly complex systems. Hugo requires complex css / HTML styling and templates. Hosting your own mediawiki site is a server-side affair and well beyond the scope of most people.

Content goals

The goal here is to come up with a design that covers most of what you really want to do, using relatively simple markdown + custom extensions where absolutely necessary / highly beneficial. We want to avoid templates if at all possible.

Starting principles

• Flat namespace (i.e, a big list of .md files). This is the only scalable design. Nothing is ever a clean hierarchy beyond some minimal scale.

• Direct [[Other Page]] wikilink-style links to other pages.

• Tag-driven, automatically-generated navigation mechanisms. Can we anticipate the relevant types of such mechanisms and just build them in, or have some very simple mechanism for defining these, that is much better than a template?

More detailed and / or controversial ideas to follow in subsequent comments, where they can be individually debated more directly.

[rcoreilly]

Keeping the left tree browser

One idea is that you add [[Category:XYZ|Tree]] tags at the end of a page, and those with the | Tree modifier get a top-level tree node, which you can then expand. The list of other pages is automatically organized so that if it is above some number (40?) then alpha sub-nodes are created to keep it manageable.

Clicking on any Category-level tree node brings up a page with a list of the pages, similar to wikipedia, so entries at the end without a | Tree tag just do that, like in wikipedia.

The first branch in the left tree node would be the page name itself, with nav links within the page itself.

Everything in the tree after the first page-level node would start out collapsed.

The advantage here is that it would provide convenient access to relevant other pages, without having to leave the current page.

[rcoreilly]

See also links

At the top of the page, you could have [[See also:XYZ | other...]] links where XYZ is the primary category of most closely-related pages (that also have this same See also: link -- nothing shows up if there are no others).

[rcoreilly]

References

For the Neuropedia wiki system, we want to be able to add references into a shared bibtex reference list, and have it auto-generate a reference list from that on each page. The source for the bibtex is originally on zotero -- investigate any updates on live-linking into that.

[rcoreilly]

The official pandoc reference syntax is [@CiteKey] so we can just support that.

You can also use this syntax for referencing figures and tables: [@fig:figname] -- ultimately want to support these kinds of things so the same source can generate a paper PDF through pandoc, while also providing consistent in-app formatting and live links, etc.

[rcoreilly]

Media pages

Also need a separate flat list of media that you can link into: images, etc.

[rcoreilly]

Markdown reviewer editing syntax

Would be great to have a nice markdown extension syntax for adding reviewer-style editing and comments directly into the source, with a toggle for whether to show these or not. Ideally, a tool for accepting changes etc.

All the content would be hosted on github and the automatic revision history would handle all that stuff, but perhaps some way of turning diffs into the markdow edit syntax to show them more intuitively would be awesome.

This shades into "Cogent Author" functionality, and the idea is that 1 paper (1 page) is just the start of multiple papers which ultimately becomes a whole collection of relevant work, so having this author functionality built in is good.

Also need an ability to run through pandoc and generate all the more standard representations of a page (pdf, ebook, etc).

[rcoreilly]

This can probably be managed using existing markdown extended syntax, per: https://www.markdownguide.org/extended-syntax/ with just a few mods.

For basic edits, you just need insert and delete operations, which can be done with strikethrough and superscript, which are similar to standard copyediting symbols (note that GFM doesn't support superscript, so I'm using <sup> here):

Here is some ~very~ verbose text ^that needs^ fixing.

Here is some very verbose text ^{that needs} fixing.

For comments, the footnote syntax would be good, with a specific modification:

This is really good.[^you need to be more specific here]

This is really good.¹

The current footnote syntax requires you to add two different tags, [^1] for the marker and [^1]: the note for the note -- putting all in one spot would be much better.

current github fails reasonably with that:

This is really good. [^you need to be more specific here]

The actual comment would be rendered in the right margin, instead of at the bottom, so you can actually see it in context.

Also, to emphasize a point from above: one way for this to work is you make the changes you want directly in the text, commit that, and then it optionally renders the edits. In this case, we could even just generate the html directly. But it will also probably be good to support an editing mode where you annotate and then the original author has the opportunity to review and accept / reject the changes on a case-by-case basis.

Footnotes

1. you need to be more specific here ↩

[kkoreilly]

I think we need to consider your last point further; is there really any use case in which you do this bizarre thing instead of just using Git and then rendering the differences as Git already does? We can do the case-by-case review things based on Git changes, and you don't want to directly edit the source with all of this markup, right? How do you know who left the feedback on a multi-person project? Just use Git!

[kkoreilly]

We might need to develop some more tooling, but I think it should all be Git based, not with this non-standard markdown syntax, and you haven't made any compelling points to the contrary as of yet.

[rcoreilly]

Yeah, agree about Git based, but note that the syntax actually is "standard" (advanced).

The one thing that we definitely will want is the comments -- which is nonstandard.

Again we'd have people commit these comments in git, and then have an option to display them or not, along with other edits.

[rcoreilly]

Content as the name

Here's the topic to bikeshed the name.

How you could improve on Cogent Core Content is beside me, especially given that every such package names their main directory content (including Hugo and our pages). But anyway, go ahead and try... :)

[kkoreilly]

I think that our ongoing discussion on contentSize (#1306) demonstrates that content is too generic of a name. content refers to everything inside of a GUI framework as that function demonstrates, so it is not an appropriate name for a specific package with clear functionality: managing multiple pages of static web content.

[rcoreilly]

content is indeed a very generic name, which is appropriate for the broad nature of what this package does :)

you might be tempted to call it "web content" or something more verbose like that, but then I think you can see that content is preferable.

In any case, it is incumbent upon you to suggest alternatives at this point :)

[kkoreilly]

Use of HTML for custom functionality

Instead of developing another convoluted templating system based on arbitrary characters ({%[[), I propose that we use HTML for all functionality not already present in markdown such as macros, except for wikilinks and possibly collapsed headers.

[rcoreilly]

HTML won't work for the reviewer editing syntax (no such html exists, and even if it did, it would be too verbose for this high-frequency use-case), and otherwise I'm really hoping we avoid any such templates or anything, and pack everything into [[ ]] :)

[rcoreilly]

Quick notes on decisions w Kai

• toml metadata headers with optional things including: all categories as a string slice, tree categories for what shows up in tree, in order, name if default is not good, optional title if name is not good, date, etc. this is a struct.
• parse all metadata at startup to compile lists of categories etc -- just read up to metadata separator -- all embeded so should be fast.
• file-names.md are kebab, render into Camel_Names for page names, with optional title metadata
• media subdir under content so there is one embed.
• singluar for cogent docs: Button, not Buttons
• Widgets is the category, Widget is the base widget
• category page like Widgets can have its own content, and then automatically has all the pages that list this category below. this is a replacement for filename number scheme etc -- just do it.
• tutorials likewise has curated list at top and then full list below.
• headings after first one start collapsed on mobile
• longer pages for docs in general with more subheads including lots of usage tips for different things
• links go into headings with standard # syntax.

longer term issues:

• pathway to scale up using efficient distributed databases for hosting larger amounts of content, instead of embedding -- but keep the full range supported, so you can start out with small and simple and grow easily.