feat: Add website layout

[Xuanwo]

This PR will add mdbook as a start for our website.

Our workflow will commit the built content to gh-pages, and asf.yml will deploy to rust.i.a.o.

close #114

Show Time

[Xuanwo]

JIRA ticket to enable the action has been submitted: https://issues.apache.org/jira/browse/INFRA-25298

[liurenjie1024]

The general layout LGTM. I will help to add more contents once the whole flow is done.

[liurenjie1024]

LGTM, thanks!

[Fokko]

There is an effort going on to centralize everything at some point: apache/iceberg#8919 We noticed in practice that many people don't find PyIceberg because it is a separate site. However there is a link to it, it does not seem intuitive to have different sites. The Java one is being merged into the repository itself, and then the idea is to also include Python and later on Rust and Go. This way we can have a single search to go through the different languages and such.

This is why I mentioned mkdocs earlier in the issue. Is there a specific reason to go with mdbook?

[Xuanwo]

This is why I mentioned mkdocs earlier in the issue. Is there a specific reason to go with mdbook?

mdbook is widely used in rust community to build a docs site. It's easy to use and deploy: one simple binary, not extra setup.

Its output can be easily embedded elsewhere: simply move the output from book to the desired location. This process is similar to integrating a site built with MkDocs.

[Fokko]

@bitsondatadev WDYT?

[bitsondatadev]

mdbook is widely used in rust community to build a docs site

While I can respect this, it is larger than just a Rust project, and we've had ongoing discussions on around efforts to make the documentation uniform across Iceberg.

Why?

1. Consistency for the Iceberg user. I understand that Rust users will come in and see a less familiar interface, but I doubt any user will find looking at mkdocs vs mdbook a large hurdle, and what's more important to them is that the content is maintained well, which brings me to the following points.
2. Consistency for the Iceberg contributor. While some Rust users who decide to join the Iceberg project will have the inconvenience of learning mkdocs if they haven't used it before, largely folks in the Iceberg community will be tasked to maintain these docs. It doesn't serve the Iceberg community well to have to learn separate docs paradigms for every sub project of Iceberg. I would posit that it will cause docs to grow outdated much faster.
3. The core documentation has many requirements around versioning, where to store versioned documentation to avoid obstruction to local search indexing in the code, and how to integrate versioned docs into the larger Iceberg site, apply the same look and feel, and also apply the same syntax, prose, and link checking across the site as a whole. Most of these same requirements extend to the sub projects outside of the Java API.
4. Not all markdown is the same. Every framework (including Hugo that we're currently using and migrating from, had a lot of special syntax specific to hugo to enable different docs features. If we use one thing, documentation ports much easier, again, enabling a faster maintenance cycle as updates happen int he Rust project, most of the markdown and explanations can be borrowed from the parent site, and may require some changes to the context in which the docs are written.

There are a few things I actually haven't mentioned but hopefully that makes enough sense to clarify that we're not pushing back for little to no reason and a lot of thought has gone into how we are structuring and pulling the documentation together. While I'm still happy to hop on and have a discussion if you all still feel really strongly about this, I'm interested to hear if this changes your perspective or if there are reasons we've missed not using mdbooks.

It's easy to use and deploy: one simple binary, not extra setup.

Mkdocs material only relies on a couple of extra python libs. It has the option to compile offline and we're already done the heavy lifting to apply Iceberg formatting to this theme and it utilizes the mkdocs web classes to give a consistent look and feel to each Iceberg site.

Its output can be easily embedded elsewhere.

As mentioned in the last answer, when created the layout for Iceberg, we do so under the assumption that all sub documentation will follow mkdocs. We will in the interim, be using this monorepo docs plugin to centralize all projects and Java api versions into one big build. Over time, mkdocs-material is building their own solution to this that is in beta right now, but we'll migrate to this to have one less external dependency. What this does is allows us to pull in multiple sub-mkdocs projects and do one big build and set of validations, a unified search index (while also enabling sub project search later on).

There was no way for you to know this without following all the conversations, which there were a lot of them and it was a lot to expect of you all as you're doing this amazing work building out this project. So hopefully this summary helps.

One thing that comes to mind is I certainly do not want to impede the writing of documentation for the project. So once we come to an agreement of framework choice. If we decide there's a compelling enough case to use mdbooks you all can carry on, if we decide to use mkdocs, then I think either just writing all docs in plaing .md files for now, will suffice, and I am happy to convert those over to the mkdocs format when we are done pulling python docs in.

Let me know your thoughts, and great work on this api team!

[bitsondatadev]

Update

After having a meeting with Fokko and him showing me the Arrow site documentation I am not totally opposed to the idea despite my long-winded blurb before.

I think the crux of the decision lies with the simple argument for docs contributor experience and lowering the bar for documentation to be contributed. The worst outcome is the one where we don't keep the documentation updated.

There's largely three models for docs that I've seen in open source where there are sub projects.

1. The docs are broken, out of sync, and out of date.
2. Docs are handled and maintained by a collective group of technical writers, developer experience engineers, and newcomers to the project that would benefit greatly from an aligned structure.
3. The core docs are handled by the collective "docs people" but sub projects are maintained by the project owners themselves.

In my experience I've seen a lot of success from the second model and I tend to lean in that direction. This is because engineers have a nasty habit of not prioritizing docs as their time is stretched thin to contribute the code, and nobody wants to block code making it in to avoid the feature. However, there are a lot of unknown/silent features even living in Iceberg today that many are unaware of due to missing documentation. Further, if documentation follows the same structure changes are the same across every docs site.

That said, I am impressed with the Arrow setup and would be interested to discuss a bit with their docs people to see how things are going from their perspective. I also would like to bring this up in a future community sync. If at the end of the day, you all maintain the rust docs well, and are happy to take on that focused role, then I'm not entirely against having us link the the Rust docs from the main Iceberg site and keep that more of a iceberg-rust contributor task as opposed to a collective Iceberg community task.

Of course anyone in the Iceberg community can also easily work with mdbooks as well, and help contribute and we can just monitor if the docs start falling behind or missing a lot of feautres, then we may revisit the discussion at another time. Happy to hop on a call or even open up a mailing list discussion to dive in further.

[bitsondatadev]

Okay, so to avoid any delays or concern here, I don't want you all to stop with writing documentation because you're not sure how this will fall inline with the other docs.

So please go ahead and use mdbooks.

If documentation stays relevant and we're not seeing rust docs fall behind, this should be fine and we lose out on a few other things, but the most important thing is that docs themselves are added/maintained. I fear if we don't use this rust framework we may have the opposite problem and block the docs from new rust contributors. So just assume that the Iceberg docs will link out to this from a navigation menu in the Iceberg site.

Sorry for the back and forth here, I'm still trying to consider the best way to do docs all around.

[Xuanwo]

Hi, @bitsondatadev, thanks a lot for your effort of maintaining docs. I can imagine that coordinating such a huge amount of work is challenging.

I'm open to switching to MkDocs or any other framework that Iceberg uses. I agree that using the same tools makes it easier to maintain the docs, especially since Iceberg has so many components.

---

Let's discuss our next steps: How can we align with our current plan?

iceberg-rust is going to make it's first release, so we want a website like rust.i.a.o for users to get started. I don't have a picture that how to integrate with i.a.o for now.

Should I just use mkdocs to setup?

[liurenjie1024]

I'm open to switching to MkDocs or any other framework that Iceberg uses. I agree that using the same tools makes it easier to maintain the docs, especially since Iceberg has so many components.

+1 for this. If we switch to MkDocs, things will be easier if the iceberg community plans to merge docs together in future.

The tradeoff is that it maybe not so rusty. But given that MkDocs is also quite popular and easy to use, I think it's worth it.

[Fokko]

Thinking more of it, and looking at other projects, both Avro and Arrow use the docs.rs website. I'm also fine with that if that's the de facto way of doing things for a language. Sorry for all the confusion here. I'll leave it up to you guys to decide.

iceberg-rust is going to make it's first release, so we want a website like rust.i.a.o for users to get started. I don't have a picture that how to integrate with i.a.o for now.

I think you're almost there. You have to publish to the gh-pages branch and add a CNAME file to let the Apache DNS know what the domain name is.

[Xuanwo]

You have to publish to the gh-pages branch and add a CNAME file to let the Apache DNS know what the domain name is.

I already configure the .asf.yml. I'm guessing it should work as expected.

[bitsondatadev]

Thank you all for being so considerate and willing to move outside of Rust. When I first replied yesterday I only saw not using our unified framework as a hindrence to update documentation. In past experiences, I have seen the unified model work where less pressure is put on the direct developers of the code to maintain documentation as the project can have generalists help out with this and not have to learn about different frameworks. Fokko pointed out the other communities doing a decentralized approach to docs and it seems to be working fine so I would like to almost run the Rust project as an experiment to see how it goes if that is okay with you all. Let's stick to your original plan.

The goal boils down to being flexible to accept whatever model will provide the most up-to-date documentation. Hopefully as the Iceberg and Rust community grows, we can establish a healthy culture that the implementors are doing a large majority of the work with documentation as they go. If that changes over time, we can discuss this again as time goes on.

Edit: I would like to see the mdbooks setup once it's going so that a future migration wouldn't be difficult in the future if we ever decide to make that move.

[Xuanwo]

Thanks a lot, @bitsondatadev!

---

@Fokko, what else needs to be done for this PR to be merged?

[Fokko]

@liurenjie1024 If there is general concensus then we can move this forward. Thanks for working on this @Xuanwo