Big reorganisation needed

[gingerbeardman]

For those that aren't aware the tldr project was originally created for just macOS X commands. since then other platforms have been added: linux, sunOS and most recently Windows. as well as the "common" folder.

IMHO the introduction of common was a mistake and now we are all suffering because of it. whilst command names may be common even between BSD and Linux the arguments or parameters accepted by identically named commands are different.

ideals:

• common would only contain commands that are truly common and interoperable across platforms (a big ask!). there may not be many with if that criteria is upheld to the letter, especially now that Windows has been added.

• clients should look into the folder of the OS they are running on first, and then refer to common only if the OS folder does not contain the command. for me this is just the obvious way to do it, but it is not currently in the client spec

• during the migration away from common, it could contain stubs that inform users of the change, information of the pending deprecation

those are my initial thoughts on the matter.

[NargiT]

+1

[waldyrious]

What are your thoughts on #190?

[gingerbeardman]

@waldyrious my thoughts are that a flat structure is a better approach. crazy that that issue is 4 years old!

Another thought I had would be for "us" to implement updates to each client to ensure all clients are onboard with any new structure from day one.

[waldyrious]

@waldyrious my thoughts are that a flat structure is a better approach.

I was asking more in the lines of whether you see any overlap in that issue and this one.

(As for the clients, yes, that's always been our policy: we tend to make sure client authors are at least aware of upcoming changes, and give them ample time to adapt if they need so.)

[gingerbeardman]

Yes 190 is just what I think is needed.

With regards to clients, what I meant is that we make the code changes and submit PRs in the repos of the clients we use as part of this endeavor. So the client authors simply have to merge and release.

For example, I use the MacOS shell client, and the Dash docset client, both open source on github. I would be happy to make the changes needed for those clients to meet any new tldr organisation.

I think we need to roll up our sleeves here and get stuck in.

[notpeter]

Hi folks,

First, let's not hold any misconceptions, we can go to great lengths to smooth the transition (updating clients, etc) but I don't see how this can be done without impacting existing installed clients. An old client will always expect an interface (pages/common & pages/platform) which this repo will no longer provide. So at that point those old clients will cease to function. This is ok, but we should be clear this is a breaking interface change.

Personally I really like the flattened structure proposed in #190:

pages
   |__ ssh.md
   |__ ping.md
   |__ ping.windows.md
   |__ tar.md
   |__ tar.linux.md
   |__ tar.osx.md
   |__ tar.sunos.md
   |__ npm.md

If at all possible we should try and enable clients to support both the old and new style pathing without a configuration change or subsequent upgrade. For example using a template for search paths:

search_path = [
    "pages/{platform}/{cmd}.md",
    "pages/common/{cmd}.md",
    "pages/{cmd}.{platform}.md",
    "pages/{cmd}.md"]

Since language matters, I propose we switch stop using the word common altogether:

• tar.linux.md and ping.sunos.md are platform specific pages
• ping.md and tar.md are default pages (or perhaps genertic pages).

What should be in a default page is certainly up for debate, but we need realize that "common usage across all platforms" is an impossible standard, defaults instead can be considered "our best guess what might be helpful." So even though devfsadm is a solaris-only command, it should be pages/devfsadm.md not devfsadm.sunos.md. Because searching for devfsadm should return "no results" from a client just because you're running on a mismatched platform.

This simplicity also helps first time contributors. Wanna create a new page for qbasic? Use qbasic.md (not qbasic.dos.md, qbasic.windows.md nor qbasic.generic.md).

Adopting #190 also means we could adopt future platforms or other namespaces (fdisk.dos.md, tar.bsd.md, tar.busybox.md) gradually, first by creating pages and then by adding client support.

P.S. Let's make sure that this reorganization preserves git history. Not pointing fingers, but previous re-orgs failed to do this and in my opinion it's worth the extra git-effort.

[gingerbeardman]

Superb summary and proposal. I agree with almost all of it @notpeter thanks for taking the time to write it!

The only thing I am a little unsure about is how to handle devfsadm in your example. I'm not 100% convinced it should be added without the platform qualifier.

Anyway, some further thoughts...

I wonder if the client could scan all the template locations and present the correct one, but let the user know that others exist? For example, I'm on macOS and I do tldr find and it gives me find.macos.md but also gives me a hint that there's find.md.

Maybe the default find.md could contain only the description of the command (probably very wrong of me to assume commands with the same name serve the same purpose across platforms). Then the examples would be pulled in from platform specific files? I see pros and cons to that approach, but it might be worth discussion?

Finally, is there somehow a way a message could be prepended to a data update requested by an outdated client? For example saying "new format coming soon - please update your client"?

[notpeter]

I took a stab at #190 style flattening: notpeter/tldr#uncommon

cd pages/
rename -v 's/(.*)\/(.*)\.md/\2.\1.md/' {windows,osx,linux,sunos}/*
rename -v  's/common\/(.*)\.md/\1.md/' common/*
rmdir {common,linux,osx,sunos,windows}
git add .

With this no platform page gets promoted (ncat.linux.md, devfsadm.sunos.md, etc). That can be handled in its own PR.

Someone wanna take a stab at altering a client so it could consume notpeter/tldr#uncommon or a similarly re-org'd branch?

[gingerbeardman]

I'll modify the MacOS shell client today

[chamini2]

What about making symbolic links for backwards compatibility?

cd pages
for os in {windows,osx,linux,sunos}
do
  for f in $os/*
  do
    newf=$(sed -E 's/(.*)\/(.*)\.md/\2.\1.md/' <<< $f)
    mv $f $newf
    ln -s "../$newf" $f
  done
done

for f in common/*
do
  newf=$(sed -E 's/(.*)\/(.*)\.md/\2.md/' <<< $f)
  mv $f $newf
  ln -s "../$newf" $f
done

This leaves us with the new structure and the old ones pointing to the new one.

This could give chance for the clients to adapt, then we could give some time of backwards compatibility, and finally erase the old directories.

[gingerbeardman]

@chamini2 good idea, Windows support proves to be a sticking point though?

I vote to remove Windows support.

[sbrl]

Regarding symlinks, check out #787. I think a PR needs to be made against the node client to support them.

[zlatanvasovic]

This issue seems very obsolote now, years after it was opened.

First, it's pretty much clear commands common for Windows and Unix are very rare, if not non-existent. If anything, you have to type directories differently.

Second, it's normal for Windows developers to run Linux shell alongside the Windows one, especially if they are git users. Programs like Ubuntu shell for Windows also help this cause.

Overall, it seems to me that common serves its purpose as it should – it includes commands that can be run on all platforms.

[sbrl]

Overall, it seems to me that common serves its purpose as it should – it includes commands that can be run on all platforms.

Technically, it's for pages that run on more than 1 platform @zdroid

[zlatanvasovic]

Technically yes. But what I wanted to say is that with omnipresent Linux shell emulators you can run those commands pretty much anywhere, both on Unix and Windows. Since that's the case, common sounds like a pretty good name, while unix label is redundant.

[gingerbeardman]

@zdroid, the issue with common is:

whilst command names may be common even between BSD and Linux, etc, the arguments or parameters accepted by identically named commands are different.

[zlatanvasovic]

Can you point to any such example? Also you have to keep in mind we have zero BSD pages so far. That's a pretty good predictor of (almost) non-existent BSD usage here. Complicating the already over-complicated structure over really few usage cases is not worthy enough in my opinion.

Even if that's the case, I don't see how this is an issue of directory structure. A client for BSD would check BSD pages first, then the others afterwards.

[gingerbeardman]

Sure.

When comparing Mac OS X (macOS) and Linux, we're really comparing BSD and GNU.

Which kind of rules out your notion that nobody uses BSD: actually all Mac users do.

grep on macOS is the BSD style whereas on Linux it is the GNU style, ref.

date too, ref.

sort too, ref.

find is different in another small way.

du is another, for units used, compare common vs osx.

...and many more.

[zlatanvasovic]

I understood, some specifics do differ. Just another thing, are those refs up-to-date to latest version of macOS? Reading the links you've given, the differences don't seem really big. Most of those use cases aren't even documented in tldr.

You can install Linux versions of all those utilities on macOS without any problem (e.g. from coreutils package), so they are common in any case. The only real solution here is to add macOS-specific pages like it's been done for du.md.

[gingerbeardman]

Just another thing, are those refs up-to-date to latest version of macOS?

Yes

Reading the links you've given, the differences don't seem really big.

Any difference is enough to cause confusion and waste time.

Most of those use cases aren't even documented in tldr.

Shame

You can install Linux versions of all those utilities on macOS without any problem (e.g. from coreutils package), so they are common in any case.

Not the best solution to have to install and learn whole new commands.

The only real solution here is to add macOS-specific pages like it's been done for du.md.

As osx or bsd?

[zlatanvasovic]

I didn't mean to bash BSD or macOS, just wanted to clarify why this reorganization would cause more problems than it'd solve.

A difference that we cannot document doesn't really matter to 99% of the potential tldr users. An example of that is displaying current date in nanoseconds, something you normally use another program to do (mostly programming language specifics). If they see a command that works both on macOS and Linux, it's common enough for them, I guess.

What I meant about installing coreutils is that all those commands (in the provided form) are usable on basically all platforms. You can run them on macOS, Linux and Windows with Shell-emulator.

As osx I guess, since that's the more useful label. There's an open issue for making bsd as BSD-only (and not macOS) commands, but not even a single one appeared so far. :/

It's really a simple approach anyway. If you see an example in any common page that doesn't work on macOS, then you can make a macOS-specific page and document it.

[sbrl]

Actually @zdroid, we have run into these issues before - where 1 command is supported by multiple platforms, but functions slightly differently on each.

Currently, our approach is to put the 'master' copy in common, and then take a copy and put it in the directory for a given platform (e.g. windows) if it deviates from the common copy in any way.

This works because the client spec specifies the order in which clients should look pages up.

[zlatanvasovic]

I tried to say almost the exact thing, but maybe you didn't understand me. In any case, I think client spec should be responsible for solving this issue, not reorganization of pages.

[Artoria2e5]

Another helpful thing to have is to symlink GNU things to a g prefix in other systems like osx, e.g. osx/gxargs -> linux/xargs. This is how many systems like homebrew on macOS handles the co-existence of GNU tools.

[sbrl]

Yeah, symlinks are another issue we'd like to solve @Artoria2e5. I believe it's #787 that we're discussing that.

[Artoria2e5]

The other good thing about symlinks for this issue is that we can be a bit more pedantic and start referring things to the names of where they came from (like bsd, gnu, sun, posix...), so when we add say FreeBSD we can share the BSD command parts of OSX but not the XCode stuff. I mean, if 787 is good to go, it should definitely be part of the solution.

[sbrl]

I'd suggest sharing your ideas in #787 to keep things organised here @Artoria2e5 :-)

[vitorhcl]

This issue is 6 years old already, any update on this? I think the last breaking changes on client specification were too big headaches, so it's unlikely that a big change like this will be accepted

However, I really like this idea considering how the project evolved, maybe a smooth transition would make things easier. What do you think?

@kbdharun @sebastiaanspeck @waldyrious @sbrl

[Managor]

I will close this issue due to lack of interest. If you think we need symlinks or reorganization, please open a new issue so we can start fresh from the point of view of the current status of the project.