Add doc gen flags for lib, private, and all

[nobodywasishere]

• Resolves Improving crystal docs for C bindings #6721
• Resolves [RFC] Optional documentation generation for crystal version, stdlib and shards. #7904
• Resolves Include more types in crystal tool hierarchy #14711

Three new flags are added to the documentation generator:

• --include-all will include everything in the namespace (stdlib, shards, project files)
  • this is useful for locally viewing all of the methods/objects available without needing to reference several different documentation pages
  • this is also useful for tooling, as running this command will provide the entire namespace/modules/types/methods/etc

• --include-private will include all private methods
  • this is mostly intended for tooling, to allow knowing the private/protected methods of objects when doing autocomplete and the like

• --include-lib will generate documentation for lib/fun/cstruct/union/typedef/etc
  • this is useful for autocomplete with tooling
  • this is also useful for libraries meant to bind a C library, to know what methods are available without reading the documentation

[nobodywasishere]

One area this doesn't cover is includes in c structs, as TypeDeclarationVisitor hard-codes the included parts

[straight-shoota]

This looks like a good idea. However, I think this topic needs more in-depth analysis and exploration of different angles before going for a solution.
For example, this might resolve the mentioned issues, but there is a very big field of different ways to utilize the doc generator. And I'd like to make an informed decision about how to address those needs.
This evaluation should probably go into the form of an RFC at https://github.com/crystal-lang/rfcs

Of course, it's tempting to just go with this as it's already implemented (and relatively simple). So it could be easily available.
But it comes with a risk to miss out on important details and use cases, and commiting to a solution that's not ideal in the bigger picture.

[nobodywasishere]

Trying to put together an RFC, but I'm not sure there's enough here to warrant it, as this is a minor improvement upon what currently exists. I'm also not sure what further exploration there could be into this subject, or what part of this could be non-ideal in the future that also wouldn't apply to the docs generator itself as a whole.

This PR would enable more documentation for C bindings, and allow people to generate documentation for everything in their projects. It would also allow me to create a better development experience in VS Code. By having the entire top-level semantic, I can add better autocomplete/hover documentation/jump to definition, maybe even for static methods too, without the need to use an LSP.

Do you have any examples in mind of what details and use cases this misses that couldn't be fixed in future PRs? In my mind, this discussion and development belongs in #6947 (which should be turned into an RFC), independent of this PR.

[nobodywasishere]

@straight-shoota If I were to create a separate PR just adding support for generating documentation for C bindings (lib/fun/type/enum/etc), would that be accepted? That part is somewhat separate from the discussion about getting all semantic information from the compiler, and the RFC I'm trying to put together for that.

[straight-shoota]

Hm, this subset would certainly be easier to merge. However, I'd think it would be best to include the handling of lib types into the bigger discussion.

[sol-vin]

Hm, this subset would certainly be easier to merge. However, I'd think it would be best to include the handling of lib types into the bigger discussion.

If possible could you direct where that discussion should happen? I am very invested in getting this change through, it greatly helps me as a C binding author, and I would love to show this PR some support.

[straight-shoota]

As mentioned in #14816 (comment), the RFC repo https://github.com/crystal-lang/rfcs would be an ideal place for discussing a concrete proposal. Maybe before that we could collect some general ideas in a forum thread.

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/extending-api-docs-and-compiler-semantic-output/7314/1