Improve the appearance of crates on `crates.io` #5243

rzadp · 2024-08-05T13:34:56Z

Context

Currently, many crates have no readme files, even though they are public and available on crates.io.
Others have just a couple of words that does not look super presentable.

Even though probably nobody starts a journey with polkadot-sdk or its documentation with a readme of a random low-level crate, I think it would look more mature to have a little better readmes there.

So, in an attempt to improve the aesthetics of polkadot-sdk, I propose a set of consistent, branded, better-looking readmes for all published crates.

What's inside

~~New readme files for published crates.~~
A python script to generate new readmes, for the crates that have none.
- It will skip crates that do have a readme, and private crates.
Added a new image asset to the repo - logo with a background.
- The main readme of the repo uses a nice trick to accompany both light and dark modes - but that doesn't work on crates.io so a single logo with a background is needed.

Example

Current

Changed

kianenigma · 2024-08-05T15:35:15Z

meta: I think this type of work is great! Thanks for taking it on.

About the actual details:

With consistent format, all readmes can be updated via search&replace tools.

I wish there was a standard template file + a python script that would do it? For example, imagine we want to change the image, or add a new line to all READMEs. The manual search and replace indeed works, but I think nowadays with AI-assisted coding it is worthwhile to script it in py or js.

I imagine:

.template file that has a few fields like:

crate description (coming from cargo.toml)
crate public doc URL (hardcoded and generated based on the crate name)

Prior to each release, the script is triggered, and the new READMEs are pushed to master.

By the way, this closes #1260.

I think it is also great that you are focusing on building a system here, rather than fixing e.g. crate descriptions. End of the day, only devs can provide good one-liner description of what a crate is, and we can hope they do that in the future. But you or I are not the right people to retroactively try and change these descriptions.

As noted in the original Aesthetics issue, we should pay closer attention to the README of only a few creates, and myself and or @ggwpez @gupnik will eventually do so, if it makes sense:

polkadot-sdk
polkadot-sdk-frame
frame-system
frame-support
polkadot
polkadot-parachain

Perhaps for these we can have a special README that is not auto-generated.

kianenigma · 2024-08-05T15:36:45Z

Actually, actually, as the part of the same python script that is ran upon new release, we can also solve https://github.com/paritytech/release-engineering/issues/196, which IMO is an important issue. Thoughts @ggwpez @EgorPopelyaev?

ggwpez · 2024-08-05T15:45:01Z

cumulus/client/consensus/common/README.md

+
+# Cumulus-Specific Common Consensus Implementations
+
+This crate is part of the [Polkadot SDK](https://github.com/paritytech/polkadot-sdk/).


How where those generated? Looks good IMHO, just asking.
Eventually we should add a CI that ensures that they stay in sync with the rust files.

How where those generated? Looks good IMHO, just asking.

I did some one-off local bash scripting to generate the initial files, but then I relied on manual work to finish them, or to adapt already-existing readmes.

I guess it can be fully-automatic if:

We accept the Cargo.toml's name as a title.

For example, crate pallet-nft-fractionalization can have an automatic title like Pallet Nft Fractionalization. But not a nicer NFT Fractionalization Pallet as I have handcrafted.

The Cargo.toml's description is taken at face value, not rewritten into another sentence, etc.
Missing capitalization or a dot at the end can be auto-added.

We leave a pack of handcrafted readmes untouched by the auto-generation script, as mentioned by @kianenigma

CI that ensures that they stay in sync with the rust files.

What do you mean exactly? In my mind, those readmes were never supposed to be out-of-sync with the rust files.

For example, this crate cumulus-client-consensus-common will always have a link to docs (...)/cumulus_client_consensus_common, and it will never contain anything other than "Cumulus-specific common consensus implementations" - at least I hope not :D

I just wondered if you used something like this https://crates.io/crates/cargo-readme to generate the READMEs from the rust docs, since that can be put into CI to have up-to-date READMEs.
Maybe i misunderstood what you are trying to do here?

I was thinking the READMEs would be a small shell with a small intro, that invites the user to go to rendered rustdocs to consume the docs - not a copy of the rustdocs text converted to markdown.

I dont really mind, but i personally understood the "everything in rust docs" maxime as having the source of truth in rust docs, and then making it more pleasing to the Eye by auto generating different formats. Eg READMEs or Wiki entries.

The main issue about generating README from .rs files is that they would not be able to parse docify stuff.

Based on https://github.com/sam0x17/docify, there is a way to do this, but it would require first generating markdown files from .rs files, then tweaking the docify syntax, then running another process around it etc.

I recall talking with @sam0x17 last year about a CLI, but we never did it.

In general, it is a cool idea, but not sure if it is a good bang for our buck atm. I would be happy with an auto-generated, mostly empty .md file that points to the right rustdoc page. It is less pretty, but works as good.

ggwpez · 2024-08-05T15:46:10Z

Actually, actually, as the part of the same python script that is ran upon new release, we can also solve paritytech/release-engineering#196, which IMO is an important issue.

We would have to do these changes just before releasing. @Morganamilo i think its a good idea to append the SDK version to the README as well (additionally to the crate description).

rzadp · 2024-08-06T10:57:53Z

I wish there was a standard template file + a python script that would do it? For example, imagine we want to change the image, or add a new line to all READMEs. The manual search and replace indeed works, but I think nowadays with AI-assisted coding it is worthwhile to script it in py or js.

I was thinking that by adding those minimal readmes without a script, I could avoid adding a thing that could go wrong to the release process - even if it's a small thing. But perhaps I was overthinking.

Perhaps for these we can have a special README that is not auto-generated.

Yes, we can have that. I was operating under the impression that we don't want that, and have everything in .rs. But I guess there's always exceptions :D

I'll rewrite the PR to have an automatic py script with the following plan:

Delete readmes that only duplicate information from Cargo.toml - like this.
Add a readme template file.
For every (published) crate:

If a readme file exists, skip - meaning it's one of the handcrafted ones.
Fill the template with information based on Cargo.toml.

The script could either take SDK version as a parameter and be used just before releasing, or it could be used to update files on master and the release process would append the version separatelly.

rzadp · 2024-08-12T15:33:31Z

I have recreated the PR with a different approach.

It now only contains a python script that generates a readme for each public crate - based on the name, manifest description, license, and (optionally) SDK release version.

The script will omit crates that already have a readme - that way we can have special, handcrafted readmes.

It can be included in the release process as a non-critical step.

Example

Running ./.github/scripts/generate-readmes.py . will produce readmes looking like this.
Running ./.github/scripts/generate-readmes.py . --sdk-version 1.15.0 will produce readmes looking like this.
- Aiming to help with https://github.com/paritytech/release-engineering/issues/196.

Link to documentation

Initially, I thought it was favorable to point the user to consume rustdocs about a crate on the Polkadot SDK docs site - for example: https://paritytech.github.io/polkadot-sdk/master/sc_consensus_beefy_rpc/

I now think the opposite, because:

Polkadot SDK docs don't contain versioned docs for single crates - we can only point to master, but that can contain docs for a different version of a crate that the user is interested in.
Crates.io already contain a docs.rs link with a proper version.
Some crates don't yet have a proper documentation - it creates a false promise, generating a link that doesn't give you much when you follow it.

So, I'm not generating a link like that anymore.

Special crates

As @kianenigma you said, there are special crates, such as polkadot-sdk or polkadot, that have the most visibility and should be treated specially.

I thought about handcrafting readmes those, but I didn't have a good vision for that, so I settled for a few fixes in the one that existed.

kianenigma · 2024-08-14T15:16:26Z

.github/scripts/generate-readmes.py

+VERSION_TEMPLATE = """
+## Version
+
+This version of `{name}` is associated with Polkadot {sdk_version} release.


The sdk release names will change to the new format highlighted by Oliver in https://forum.parity.io/t/new-release-process-brainstorming-on-improvements/2296/8

But it should be fine, just so you know.

kianenigma

Your opinions in the last iteration make sense to me. Let's keep this iterative.

You can explore how hard is it for the release team to update these in the first stable release, that woudl be cool.

This makes me again try and get polkadot-sdk-docs to publish to crates.io, it will then solve a lot of our issues. The reason why have paritytech.github.io now is that we cannot publish to crates.

kianenigma · 2024-08-23T11:56:31Z

Super nice, looking forward to nice looking stuff in docs.rs and crates.io in our next release!

* master: (36 commits) Bump the ci_dependencies group across 1 directory with 2 updates (#5401) Remove deprecated calls in cumulus-parachain-system (#5439) Make the PR template a default for new PRs (#5462) Only log the propagating transactions when they are not empty (#5424) [CI] Fix SemVer check base commit (#5361) Sync status refactoring (#5450) Add build options to the srtool build step (#4956) `MaybeConsideration` extension trait for `Consideration` (#5384) Skip slot before creating inherent data providers during major sync (#5344) Add symlinks for code of conduct and contribution guidelines (#5447) pallet-collator-selection: correctly register weight in `new_session` (#5430) Derive `Clone` on `EncodableOpaqueLeaf` (#5442) Moving `Find FAIL-CI` check to GHA (#5377) Remove panic, as proof is invalid. (#5427) Reactive syncing metrics (#5410) [bridges] Prune messages from confirmation tx body, not from the on_idle (#5006) Change the chain to Rococo in the parachain template Zombienet config (#5279) Improve the appearance of crates on `crates.io` (#5243) Add initial version of `pallet_revive` (#5293) Update OpenZeppelin template documentation (#5398) ...

rzadp added R0-silent Changes should not be mentioned in any release notes T11-documentation This PR/Issue is related to documentation. labels Aug 5, 2024

rzadp requested a review from kianenigma August 5, 2024 13:34

ggwpez reviewed Aug 5, 2024

View reviewed changes

rzadp force-pushed the rzadp/crate-readmes branch from a55bc48 to 6394221 Compare August 12, 2024 13:51

rzadp changed the title ~~Improve the appearance of Substrate and Cumulus on crates.io~~ Improve the appearance of crates on crates.io Aug 12, 2024

rzadp added 2 commits August 12, 2024 16:28

Introduce script for generating readmes

2fa683e

Add imgs with background

5e0993f

rzadp force-pushed the rzadp/crate-readmes branch from 6025469 to 5e0993f Compare August 12, 2024 14:28

update ver

ac4b5df

rzadp marked this pull request as ready for review August 12, 2024 15:34

rzadp requested review from a team as code owners August 12, 2024 15:34

rzadp and others added 2 commits August 13, 2024 07:55

start path lookup from workspace dir

6ff9715

Merge branch 'master' into rzadp/crate-readmes

088ba32

kianenigma reviewed Aug 14, 2024

View reviewed changes

kianenigma approved these changes Aug 14, 2024

View reviewed changes

Merge branch 'master' into rzadp/crate-readmes

4e6770f

rzadp requested review from mutantcornholio, mordamax, EgorPopelyaev, Bullrich and Morganamilo August 16, 2024 09:19

EgorPopelyaev approved these changes Aug 22, 2024

View reviewed changes

rzadp added this pull request to the merge queue Aug 23, 2024

rzadp mentioned this pull request Aug 23, 2024

[EngAut] Aesthetics of Polkadot SDK paritytech/eng-automation#10

Open

Merged via the queue into master with commit c66f7bd Aug 23, 2024
185 of 188 checks passed

rzadp deleted the rzadp/crate-readmes branch August 23, 2024 11:26

github-actions bot mentioned this pull request Oct 8, 2024

Update polkadot-sdk from stable2407 to stable2409 moonbeam-foundation/moonbeam#2994

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the appearance of crates on `crates.io` #5243

Improve the appearance of crates on `crates.io` #5243

rzadp commented Aug 5, 2024 •

edited

Loading

kianenigma commented Aug 5, 2024 •

edited

Loading

kianenigma commented Aug 5, 2024

ggwpez Aug 5, 2024

rzadp Aug 6, 2024

rzadp Aug 6, 2024

ggwpez Aug 6, 2024

rzadp Aug 6, 2024

ggwpez Aug 6, 2024

kianenigma Aug 6, 2024

ggwpez commented Aug 5, 2024

rzadp commented Aug 6, 2024

rzadp commented Aug 12, 2024

kianenigma Aug 14, 2024

kianenigma left a comment •

edited

Loading

kianenigma commented Aug 23, 2024


		# Cumulus-Specific Common Consensus Implementations

		This crate is part of the [Polkadot SDK](https://github.com/paritytech/polkadot-sdk/).

Improve the appearance of crates on crates.io #5243

Improve the appearance of crates on crates.io #5243

Conversation

rzadp commented Aug 5, 2024 • edited Loading

Context

What's inside

Example

Current

Changed

kianenigma commented Aug 5, 2024 • edited Loading

kianenigma commented Aug 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggwpez commented Aug 5, 2024

rzadp commented Aug 6, 2024

rzadp commented Aug 12, 2024

Example

Link to documentation

Special crates

Choose a reason for hiding this comment

kianenigma left a comment • edited Loading

Choose a reason for hiding this comment

kianenigma commented Aug 23, 2024

Improve the appearance of crates on `crates.io` #5243

Improve the appearance of crates on `crates.io` #5243

rzadp commented Aug 5, 2024 •

edited

Loading

kianenigma commented Aug 5, 2024 •

edited

Loading

kianenigma left a comment •

edited

Loading