Permit search engines to index Github wikis

[ishepherd]

There is useful content in Github wikis.

It's unfortunate that external search engines are banned in https://github.com/robots.txt
Edit: no longer banned in robots.txt but still banned

People have put time into building their wikis and it's sad that the information is almost impossible to discover - unless you think to try Github search.

[nelsonjchen]

For reference here's an old issue from the old isaacs repo: isaacs/github#1683

[nelsonjchen]

Another old discussion, which pretty much is a repeat: https://git.luolix.topmunity/t/request-github-open-up-project-wiki-pages-to-web-indexes/122096

[nelsonjchen]

FYI @EdVassie

[nelsonjchen]

It also blocks archive.org.

[asdfugil]

they are not banned, robots.txt does nothing to their ability to crawl. It is just that they should not crawl and they don't.

[rfjakob]

I'm not sure what you mean, but line 11 in https://github.com/robots.txt is literally this:

Disallow: /*/wiki*

[timothywcrane]

Disallow: //wiki is just a directive... if you build your crawler to ignore directives... there is no tech involved trying to stop your crawl provided by the robots.txt file.

[mck1117]

@timothywcrane the question is not about us writing some crawler to crawl GitHub Wikis (because you could just ignore robots.txt and crawl it anyway), but that we want Google to crawl our wikis so that our users/contributors can find the wiki. Google respects robots.txt (shocker, I know), so the only way to get a GitHub Wiki indexed by Google is to remove the line from robots.txt.

[rusefillc]

LIne 11 of https://github.com/robots.txt does not mention wiki anymore

Actually wiki is no longer mentioned at https://github.com/robots.txt?

[nelsonjchen]

Sent in a PR to remove the statement in the docs: github/docs#10836 .

[ann0see]

Great! Thank you GH and @nelsonjchen for the service you provided. We now moved to GH pages, but for sure this is a step in the right direction

[nelsonjchen]

No kidding! It's nice to see they opted to change the policy after nearly a decade.

[nelsonjchen]

Oh, um, nevermind. 😖

[nelsonjchen]

The nightmare continues.

[dipree]

Hi everyone 👋 we have intentionally excluded Disallow: /*/wiki* from the robots.txt. However, we have also introduced an x-robots-tag: none in the http response header of Wiki pages. As a result, Wikis are still not visible to search engine crawlers. Why have we done this? Abusive behavior in Wikis had a negative impact on our search engine ranking and therefore we had to exclude Wikis from getting crawled to mitigate the effects. We are now investigating options how we can open the gates again so that everyone can benefit from the great information documented in Wikis. At this point, we have not decided on whether or when we will allow Wikis to be crawled again, but we are actively reviewing this and will keep you posted with any updates.

[gwlucastrig]

I can see a counter argument for the star count criteria. It seems to me that the projects that most need to be search-enabled are those that aren't already well known. A project with a high star count probably already has links on lots of web pages. Now it is true, absolutely true, that the high-star projects ought to be indexed. But the place where being search-enabled can provide the most benefit is for those projects that serve a niche application and haven't yet been discovered by a lot of developers.

[Anouomgharbivsseagjdzzvz]

Qweqwe4844
content://media/external/file/3567

[1989shack]

Okay, I hope in the nearest future we will begin to see some changes...

[nelsonjchen]

Update: ~500 star wikis that are non-publically-editable appear to be indexable now: https://github.com/IgniteUI/igniteui-angular/wiki

No x-robots-tag: none present.

x-robots-tag: none is still present on publically-editable or wikis on repos with less than 500 stars.

[Novustat-Statistik-Beratung]

but we are actively reviewing this and will keep you posted with any updates

Please announce changes before. We are currently adding a statistics wiki to our git profile which is also found on our website. If this would become crawlable for search engines, it would cause some major SEO issues (duplicate content) for us. However, allowing users to add html tags like rel="canonical" to wiki pages would solves this issue.

[jumpjack]

Wikis are not just the problem: google simply ignores ANY page in github, even readme.md! I can never find any suitable result in google coming from github. The only way for searching in github is using github search engine. But non-developers do not know github, so why should they search there?
Try searching for "lunar lander game": there are some dozens of variants on github, no results in google!

[nelsonjchen]

Maybe you could just add github to your query if you're looking for GitHub variants. GitHub variants don't rank high because lunar lander game is something that have existed for decades with a rich history, ready to play variants (not necessarily GitHub/OSS) are what most people are looking for, and there's only so much space for results for Google to return in the first few pages.

[jumpjack]

it's just an example; github never comes up in generic google searches, although it contains thousands of programs, not just sources.

[dimpase]

perhaps things changed meanwhile, but https://www.google.com/search?q=%22lunar+lander+game%22+%22github%22+%22wiki%22
brings up GitHub wikis.

[nelsonjchen]

@dimpase https://github.com/orgs/community/discussions/4992#discussioncomment-3563126

[JackTrapper]

Fortunately there are some search engines that ignore the robot exclusion rule. They will in turn allow DuckDuckGo, Bing, and other bots to crawl their cached version of GitHub wiki pages - allowing us to find stuff on GitHub.

Its nice that someone cares more about finding content on the Internet

Now that the disallowing of robots has been rendered moot: lets do the right thing and remove any robots.txt or x-robots-tag headers and let information be available to as many people as possible - rather than caring about "our search engine rankings".

[otykier]

Not sure what the conclusion of this discussion is, but we're facing this problem:

Around a year ago, I renamed my org from github.com/Otykier to github.com/TabularEditor. Now it seems, that none of the repo issues pages are being crawled. For example, I can take a literal quote from this issue (which was created more than a year ago) and get no hits on Google. This goes for all issues - none of them seem to be crawled, making it very hard for users of our software to find solutions to problems they encounter. We can still get to the main page by searching for "github tabular editor 3", for example, but no search results from the issues pages.

Is there some org or repo setting we're missing? I am 90% sure that it used to work before we renamed the org - could that have something to do with it? Thanks!

[DF4IAH]

I am thinking to move away from GitHub with this robots.txt policy :-(

[jumpjack]

alternatives?

[DF4IAH]

I switched over to SourceForge.

[dimpase]

SF is not an alternative - they do not offer anything like GitHub actions, their GUI to work with code online is not existing, AFAIK.
The only real alternative is GitLab.

[ann0see]

We moved away from wikis to a website (hosted by GitHub Pages). I'd also consider GitLab an alternative.

[dimpase]

I am sure GitHub can easily arrange with at least one of the big search engine to be well-crawled and indexed. And this will give said engine a considerable boost in popularity.

…

On Wed, 1 Mar 2023, 09:04 jumpjack, ***@***.***> wrote: alternatives? — Reply to this email directly, view it on GitHub <#4992 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJXYHBJHJMFBPVOBZ4XUQTWZ37HBANCNFSM5B3UN4OQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DF4IAH]

GitHub is now Microsoft owned. Don't know if they want their Azure in competition to GitHub. That is the reason that I move over to an alternative.

[dimpase]

azure does not do what github does, there is no competition; github uses azure to host everything they need, bringing extra business to azure.

[maxandersen]

We are seeing github wiki pages now being indexed - has this changed permanently or is it temporarily?

[nelsonjchen]

Observationally:

• Above 500 stars
• "Editing Restricted to Repository Collaborators" turned on

[MaartenHilferink]

I asked OpenAI: https://chat.openai.com/c/308cf4b9-2759-41ca-a243-740279bbe1a7

[nelsonjchen]

You have to press "Share Conversation". Copy/pasting the URL won't work.

[ocornut]

I'm getting fairly desperate about various GitHub behaviors, but this is cherry on cake :(
It took me a long time to realize that energy spent on investing Wiki being a point documentation was hindered by the fact none of the info appears on e.g. Google, if it wasn't for a third-party creating a mirror of Wikis for that exact purpose. I have countless users searching for documentation bits that exists but aren't indexed 🤷

My repo has 54K stars and in spite of a comment above suggesting that 500+ stars repos wikis would be indexed it doesn't seem like it is. EDIT I WOULD NEED TO MAKE MY WIKI UNEDITABLE BY USERS TO DO SO ?!!!

[nelsonjchen]

It must also not permit non-collaborators to edit in addition to the star requirement. I've noticed this criteria while developing that mirror thing. You can find a bit more info on the front page of that mirroring service https://github-wiki-see.page as discussions are always a bit fragmented.

Does your repo also meet that criteria?

[ocornut]

It doesn’t, sorry i amended my post afterwards. It seems the betray the desirable purpose of my wiki to restrict edits.

Thank you for your work on this Nelson .

[ocornut]

I need to clarify this didn't really help as well as I'd hoped, in spite of adding links here and there, I don't often stumble on the right mirrored info from a search engine.

[leandromsft]

I have a wiki in my repository containing documentation and coding standards for my repository.
I would like Copilot chat (Web Interface) to index these files. It's possible?

[ocornut]

(reposting some details not as a threaded reply)

@dipree: This is really harmful and disappointing, actively discouraging people investing in Wiki. I can't comprehend why it hasn't been solved. I understand that abuses are a problem, but a single "Allow only to collaborators" options is very limiting, other mitigating options should be possible (see below).

I have 56K+ stars, a growing Wiki https://github.com/ocornut/imgui/wiki and I would like other people than me to work on it, but right now its contents is not indexed in spite of best effort to even link to https://github-wiki-see.page etc. meaning people are having difficulty finding the information they need, and keep asing us the same things, lowering the quality of their software experience and hogging support/dev resources. :(

Can't github implement some alternative options? Suggestions in order of simplicity:

• a white list of users (who are not necessarily code collaborators)
• a system for users to request approval
• a system to let collaborator approve on first edit (~queued patches: I understand it is limiting in the sense that it means non-approved users wouldn't be able to make successive edits to queue, perhaps the UI can warn them beforehand).

If even the first option was added, I would cave in and lock the wiki tomorrow and add likely contributors + infos to request.

I suspect that part of your underlying thought might be that by requiring developers to grant full access to a repository to allow people edit a wiki, it is ensuring we don't add too many people? Can't exceptions be made to well behaving projects?

[nelsonjchen]

Unfortunately, the system, for better or worse, isn't really ranking that high on search engines nowadays.

Even I find it pretty darn hard to use Google to look up openpilot wiki information which was the original impetus.

I'm thinking of maybe producing some sort of self mirroring toolkit for projects. Like a recipe for what the shellcheck project did. The existing system will still be there for general wikis though.