Enable merging of directories

[anibali]

If, for example, I put .config/MyApp in a castle, homesick symlink should have an option to merge with an existing .config folder in my home directory (ie ./config/MyApp is symlinked rather than .config).

[technicalpickles]

This is a pretty tough problem, and one I more or less intentionally avoided up to now :)

One thing that comes to mind is that I've typically just symlinked entire directories, rather than trying to merge them. For example, just symlinking .vim wholesale.

Definitely open to suggestions/patches/etc!

[anibali]

Perhaps if you're processing a directory and you come across a conflict, the user could opt to merge. Then you could simply skip the directory itself and process each of its entries. A 'merge all' option would also be useful in this case.

[technicalpickles]

That would work for the case you described, ie ~/.config already exists, and you want to symlink in ~/.config/MyApp into it.

It'd be a bit tricky if you have multiple castles that want to put files into the same directory though. It's less defined how that should work, since ~/.config/ is already a confict. What happens then? Does it get converted to a directory, and the existing contents moved into place? Doable, but I think it's getting pretty hairy. It'd also make things a lot more complex on the command line side of things.

I'm not opposed to this idea in general, but I'd definitely like to flesh out how things would pan out first. This might sound crazy, but might be worth doing some interface-first design of the command line output/input when dealing with a merge.

[wjbuys]

How about having some kind of merge marker file in the directory that should be merged? Homesick could then treat that folder almost like a sub-castle ("minaret"? ;) ). Something along the lines of an empty file in ~/.homesick/repos/foo/home/.config/.is_subcastle, or having a ~/.homesick/repos/foo/.subcastles file listing all folders that should be treated in this way.

[anibali]

I quite like this idea - people who don't care about subdirectories can simply ignore this feature and people who currently can't do anything have an option. I think it would probably be best to allow both methods (the list and empty file), as I can think of different instances where each is preferable.

[wjbuys]

I've been thinking through this a bit, and I've come up with another idea. Have a look at the writeup at https://github.com/wjbuys/homesick/wiki/MergingDirectories, and let me know what you think.

I'm probably going to push ahead and implement this in my fork soon, as I've been moving/upgrading computers a bit, and this is becoming a pain.

[anibali]

That solution looks pretty good to me. Handling conflicts is a bit of a headache though...

[skryl]

One way to avoid the manifest file and the edge case you mentioned is to only ever symlink regular files and just rebuild the directory structure of the home directory on the fly. By only symlinking files you end up building many more symlinks but save a ton of code complexity since your only two moves (when symlinking a castle) are:

1. Create the directory tree if it doesn't exist or is incomplete
2. Create a symlink if it doesn't exist

An added benefit of this approach is the ability to symlink multiple castles into a single environment without having to code for special merge cases. For example, imagine i have three different castles, common stuff, osx only stuff, and ubuntu only stuff. When I setup a new machine I'd want to symlink at least two castles into my home directory. If these two castles share any symlinked directories then my only choice would be to keep one or the other instead of doing the correct thing and merging the two. In order to account for the merge case you'd essentially have to do what I mentioned above anyway.

You can definitely go with a hybrid approach, symlink directories until there is a conflict and only then move to a file based approach but I feel like a lack of a manifest file alone is worth it.

This would be an awesome feature to have, hope this helps, good luck!

[nickserv]

👍

[andsens]

Omnomnom, I think I'll go right ahead and steal that one for the same issue on homeshick :-D

[edubkendo]

Did anyone ever do anything with this? I think @skryl 's approach, which it appears @andsens has chosen to go with for homeshick, referenced above , is ultimately cleanest. The only problem I see, and maybe I'm missing something that would alleviate this, is the case where a person would like to have a directory set up so that any new files added to it will automatically become part of the castle. For instance, if I have ~/.config/sublime-text-2/Packages/User/, which is where User settings are saved whenever a new plugin is added to Sublime Text 2, and I want any new settings file to automatically become part of my castle, this approach would no longer make that possible, as far as I can see. Anyone have any thoughts?

The only thing I can see that allows for both is the use of config or manifest, like @wjbuys suggested. I believe it's already possible to create a .homesickrc file in a castle, so it might be possible to simply adopt that to this, rather than creating yet another special config file.

If anyone has implemented any version of a solution to this problem, I'd like to take a look at it.

[technicalpickles]

I would vote for having a flag in .homesickrc. That file basically is evaled in the context of the Homesick instance, so could add a bang method like merge_directories! or similar.

On May 17, 2013, at 11:29 AM, edubkendo notifications@github.com wrote:

Did anyone ever do anything with this? I think @skryl 's approach, which it appears @andsens has chosen to go with for homeshick, referenced above , is ultimately cleanest. The only problem I see, and maybe I'm missing something that would alleviate this, is the case where a person would like to have a directory set up so that any new files added to it will automatically become part of the castle. For instance, if I have ~/.config/sublime-text-2/Packages/User/, which is where User settings are saved whenever a new plugin is added to Sublime Text 2, and I want any new settings file to automatically become part of my castle, this approach would no longer make that possible, as far as I can see. Anyone have any thoughts?

The only thing I can see that allows for both is the use of config or manifest, like @wjbuys suggested. I believe it's already possible to create a .homesickrc file in a castle, so it might be possible to simply adopt that to this, rather than creating yet another special config file.

If anyone has implemented any version of a solution to this problem, I'd like to take a look at it.

—
Reply to this email directly or view it on GitHub.

[andsens]

How about just recursing directories when using track?
Ultimately there could be a "track" autodetect feature which figures out what castle a directory belongs to, so you just type: homesick track .config/sublime_text and homesick automatically figures out the corresponding castle.
If it is made easy enough, I don't think you need an rc-file.

[muratayusuke]

I also think @wjbuys 's approach is better because of the same reason @edubkendo mentioned. "Never symlink directories" approach requires homesick track every time.

homesick symlink looks up all directories and files, so I think just writing down "directories I want to lookup subdirectories" in .manifest or .homesickrc is needed.

For example:

$ homesick track .config/sublime-text-2
move ~/.config/sublime-text-2 -> castle/home/.config/sublime-text-2
symlink ~/.config/sublime-text-2 -> castle/home/.config/sublime-text-2

and automatically add ".config" in manifest file.

If manifest files has .config line, homesick symlink makes link of all directories and files in home and in home/.config.

[jridgewell]

I think @skryl's solution is by far the best, and easiest to implement. Not to mention the fact that I hate adding unnecessary .files to my home directory that @technicalpickles is suggesting.

In @edubkendo case, just manually add a symlink the directory that you want things auto-added to where it needs to be in your home directory. So, once we get homesick to symlink only regular files, just delete the directory ~/.config/sublime-text-2/Packages/User/, then ln -s ~/.homesick/edubkendo/sublime/home/.config/sublime-text-2/Packages/User/ ~/.config/sublime-text-2/Packages/User/, and that will solve the problem... If my thinking is correct, homesick will identify any file in ~/.config/sublime-text-2/Packages/User/ as identical, and just continue on if he does homesick symlink edubkendo/sublime, right?

[edubkendo]

@jridgewell that works for the install from which I start tracking, but then when I get a new machine, or decide to try a new flavor of linux, or whatever, I have to manually do a bunch of stuff to get it working again, which rather defeats the point of homesick, which in my opinion, is to create repeatable installations which require as little manual intervention as possible. I realize that even heavyweight tools like chef and puppet can't do this perfectly, for every single possible use case and desired outcome, but it seems that quite a few of us have the use case I am describing.

As far as .homesickrc goes, if you read the code where it is eval'd, it actually goes into the castle directory, outside of castle home, like ~/.homesick/repos/technicalpickles/pickled-vim/.homesickrc so it really wouldn't be cluttering up your home directory (and I definitely understand the desire not to do that). Same for @wjbuys .manifest file, if you read the description he wrote up.

However, I have a new idea. It seems very, evil, but I actually like it. However, I thought I'd float it out here before I attempted an implementation since it just hit me a few minutes ago and might be a terrible idea. What we are dealing with is a loss of information, essentially, as @wjbuys correctly pointed out in his proposal. Usually, when you need to preserve information after the program's state is lost, you save it to the hard disk either in a file or a database. So, if we need to save information about the file structure of the repository we are symlinking from, our instinct is to create some kind of manifest file or config file to do it. We are, however, working with a tool that was explicitly designed for preserving lots of information about a repository: git.

My idea is this: when we track a new file or directory, Homesick examines the path, and if what we are tracking is nested in another directory, as opposed to tracking a file or directory that is a direct child of $HOME, it does this: First Homesick stashes any previous uncommitted changes. Then Homesick recreates the nested directory structure inside the castle (with something like mkdir_p), moves the file, symlinks it back, adds the file to the repo, and then makes a commit with a commit message indicating which file or directory is being tracked. In the future, when Homesick clones this repo, it can then scan the commit messages and symlink out exactly what it should, creating any needed directories as real directories. If these is any conflicting information, homesick can check time stamps and go with the most recent.We'll give it a special #hashtag or something to begin these commit messages to make them easy to pick out from any others. We could also allow a recursive merging option, viz-a-viz @skryl , etc, which could be set via command-line, .homesickrc, or commit message.

Let me know your thoughts. Am I needlessly complicating this, and should just go with a .manifest file? Going to Linux-Hell for abusing git? It seems to me that commit messages were genuinely designed for preserving information about a repository that isn't represented by the files themselves. Am I missing anything?

[andsens]

It's good you mentioned this @edubkendo, because your idea gave me an idea: What about custom attributes in .gitattributes?
This way homesick doesn't even need to parse stuff, it can read it directly with git!

[muratayusuke]

I think pull request #38 requires additional task, rm -r and ln -s for existing users to do same thing as before, as @jridgewell and @edubkendo mentioned. So basically agree with adding information somewhere.

But it's a little confusing to have the information on commit message or .gitattributes, isn't it? Commit message should be written for "castle repository", not for homesick command. And .gitattributes should manage behavior of git, not homesick.

Yes, I also don't want to have unnecessary dot files, but in this case, I prefer additional file like .homesick_subdir to tricky implementation deeply depending on git because it's simpler for me. How do you think about it?

[edubkendo]

ok, so, bearing this discussion in mind, I think the best implementation would be like @wjbuys documented https://github.com/wjbuys/homesick/wiki/MergingDirectories but to allow passing a flag that would let homesick use @skryl / @jridgewell style merging instead. And If you want homesick to continue operating as it always has, simply track the directory that is the direct child of the home directory, or delete the .homesick_subdir file. I think going with a different file is better than using .homesickrc because .homesickrc gets eval'd, and thus requires prompting the user.

I rather like the idea of using .gitattributes, and may actually play with implementing that for myself, once I get this working, but I definitely see your point about abusing git this way as being tricky. There's a reason I called my idea to use commit messages "evil". :)
cheers

[andsens]

And .gitattributes should manage behavior of git, not homesick.

Not exclusively, there's a reason custom attributes are supported

I prefer additional file like .homesick_subdir to tricky implementation deeply depending on git because it's simpler for me

folder_to_symlink     symlink=true

check with

git check-attr symlink folder_to_symlink

You do not even have to do the .gitattributes editing, you can leave that to the user.

[jridgewell]

Apologies, I didn't think it was added outside the castle's homedir. This changes my stance entirely. I'd prefer @wjbuys .manifest file over using .gitattribues, partially because I'm completely inexperienced with .gitattributes, and partially because git check-attr symlink folder_to_symlink always returns true when run inside ruby (just the stdout changes).

Off of @wjbuys:

1. Regular file linking should be the default.
2. If a directory matches a line in the .manifest, it is linked instead of it's children.

Let's not add flags for how we should do the linking, it would just complicate the codebase. We just follow the above.

I think only directories that should be linked should be added into the .manifest. Just think what would happen if files got out of sync with it: we would just default back to regular file linking anyways, so why add individual files at all?

[wjbuys]

Wow, great to see more people looking at this. I'm (obviously) very much in favour of not symlinking every single file.

Using .gitattributes to store homesick's metadata is a clever hack, but seems like it would unnecessarily tie us down to git. For example, I've got some castles that I store with no version control in Dropbox for larger files like fonts.

1. If a directory matches a line in the .manifest, it is linked instead of it's children.

Just to be clear: Today, if you have a folder in your castle, it will be linked as as single symlink. Lines in the manifest should indicate that the folder's children should be linked (and merged) instead (i.e., I meant the reverse of this bullet point).

I think only directories that should be linked should be added into the .manifest. Just think what would happen if files got out of sync with it: we would just default back to regular file linking anyways, so why add individual files at all?

The only thing that can be linked in different ways is a directory, so that should be what goes in the manifest. Again, for backwards compatibility, anything not listed in the manifest should behave exactly the way it does today.

Eagerly awaiting a pull request for this, so get cracking! 😉

[edubkendo]

First, I think we need to clarify what homesick's behavior is right now:

As of now, if you tell homesick to track something, no matter where it is, or how deeply nested, homesick moves that file or directory directly into the castle's home directory. Then it places a symlink to it into $HOME. Also, if you try to track a directory, and tab complete , so that the path you hand homesick ends in a '/', the symlink will fail with an error. Because of this, I actually thought homesick wasn't even capable of tracking a directory at first, but I digress.

Then, when you clone a castle and run symlink, it just places flat symlinks from whatever is at the top level of the castle's home, directly into $HOME. So, if inside my castle, I have a folder structure like : .config/sublime-text-2/Packages/User then what homesick will currently do is only see that .config dir, and just replace my current .config with a symlink to this one.

This means that we basically have TWO issues here. First, making homesick aware of nested directories. The second is what should homesick do with those nested structures. If the thing being tracked is a file, this is pretty easy, we just want homesick to place a symlink into the appropriate directory. If the thing being tracked is a directory, though, we have a question, do we want homesick to symlink the directory itself, or the directories contents?

@wjbuys you are contradicting what you wrote in your MergingDirectories proposal. In that proposal, the directories you wanted to track completely were bin and .config/barapp, and then wanted to just track specific files inside .config/fooapp and .config/systemwide, and bin and .config/barapp were what got written to the .manifest, with the assumption that homesick would then examine any other nested folders and symlink out whatever the last thing in those paths were (otherwise, how would it properly handle something like .config/fooapp/config1). This would mean, however, breaking backwards compatability.

However, the way you are describing now, which is, I believe, what @muratayusuke also said, is the simplest way to do it to preserve backwards compat and get what we want to work. When the user tracks something nested, whether its a file or a directory, we record the parent directory in the .manifest and then when the user later runs symlink, we symlink any of that directories children. The only real edge case i see is when the user also wants to track something that is inside a directory whose parent is already recorded in the manifest. But we can handle this case too, by examining the manifest and excluding a folder from being symlinked if it or its children is listed in the manifest. So, in the case of the examples in @wjbuys proposal here, the .manifest should have looked like this:

.config
.config/fooapp
.config/systemwide

We don't need to list bin because homesick will already do what we want here. The .config folder will not get symlinked because it's listed in the manifest, instead its children will, except of it's children, fooapp and systemwide will again be excluded because they are listed, and instead THEIR children will. .config/barapp will get symlinked out when homesick hits .config in the manifest, and everyone is happy. Meanwhile, users without a manifest file will just keep chugging along as always.

I believe a LOT of the confusion and disagreement in this thread has been the result of there being essentially two things at issue here, as well as some confusion about the current behavior of homesick as it is now. As I have gone about trying to implement this, I kept finding myself extremely mixed up. So I would re-read the comments, re-read the proposal on @wjbuys wiki, and not realize that they were in conflict about how to implement it. Anyways, going to move forward now following the above suggestion. I have most of track written, along with tests, though I will need to do some serious refactoring once my tests are done. Will be pushing up a PR soon, hopefully.

[edubkendo]

Working on this here if anyone wants to take a look: https://github.com/edubkendo/homesick/tree/nested_dirs

[muratayusuke]

And I made symlink: https://github.com/muratayusuke/homesick/tree/feature/merge_directory

I use ".homesick_subdir" instead of ".manifest" because I don't think I can remember what is .manifest about a year later :P
I'll pr this branch after implementing track.

[edubkendo]

Excuse me if I'm just missing something @muratayusuke , but I don't think your impl handles this https://github.com/edubkendo/homesick/blob/nested_dirs/spec/homesick_spec.rb#L156 edge case? I have both parts implemented now, but it needs cleaned up a little bit before doing anything with it. Also, agree with you on name of file, that's a good point.

[muratayusuke]

@edubkendo Wow, our work may be duplicating. I merged your homesick track into my above branch and made some changes including rename and path change (castle/.manifest -> castle/home/.homesick_subdir).
muratayusuke@3559d82

Should we make working branch in technicalpickles/homesick?

Anyway, I'll check your branch with the point you mentioned, and think about how to merge our work.

[edubkendo]

yes i unfortunately missed your comment before I did that, and just hadn't pushed it up yet. I wrestled with the edge cases a bit, and ended up with some pretty ugly code. Yours is much cleaner, so if you can preserve that and still handle the edge cases, it would be better.

Glad this is getting done. It's going to be very useful. If we're going to be hacking on this much more before merging into master a working branch is a good idea so I don't duplicate your efforts again.

[muratayusuke]

@edubkendo OK, thank you. I pushed here, but test failed on ruby 1.8.7...
https://github.com/technicalpickles/homesick/tree/feature/merge_directory

Now remaining tasks are:

• Fix test failure
• Add ruby 2.0.0 to travis
• Add test for your edge case
• Update README

I'll deal with them, but please commit freely if you can finish some work.

[jridgewell]

@edubkendo @muratayusuke, can you check #38? I think I've got all the edge cases covered.

[muratayusuke]

@jridgewell Thankyou! I'll follow them.

[jridgewell]

@muratayusuke, I didn't say it properly. I've put all the edge cases I can think of in #38, and gotten them all to pass in 1.8.7, 1.9.3, and 2.0. Could you check over the spec to make sure all the edge cases are covered, and if so, merge it in?

The important bits:

• homesick
  • symlink
    • links dotfiles from a castle to the home folder
    • links non-dotfiles from a castle to the home folder
    • when symlink to directory exists
      • asks if it should remove symlink
      • detects identical files
    • when same directory exists in multiple castles
      • will merge directories
    • when directory listed in .manifest
      • will symlink the directory instead of it's children
    • when forced
      • can override symlinks to files
      • when symlink to directory exists
        • does remove symlink
  • track
    • file
      • should move the file into the castle
      • should move the file in a nested folder structure into similar structure
      • .manifest
        • should not add file to manifest
    • directory
      • should move a nested directory
      • should move the directory into the castle
        • without trailing slash
        • with trailing slash
      • .manifest
        • should add directory to manifest
        • should add nested directory to manifest
        • should not add nested directory if parent is already listed
        • should remove nested directory if parent is tracked

[muratayusuke]

@jridgewell Yeah I wanted to mean that, but sorry to confuse you by my poor English . I'll check all your spec and add the necessary cases to the working branch.

[edubkendo]

@jridgewell So, this works great and the code looks good to me. You've also definitely been thorough on the edge cases.

Only problem, this seems to break backwards compatibility. Everything is linked from the deepest nest point if its not specifically tracked, which means people could find themselves extremely surprised if they have had a castle/home/bin directory and have been depending on it to be symlinked directly, and not each of its children. This would be ok with me, as I'm a recent user, but I don't know how others will feel about it.

This is why @muratayusuke and myself have been approaching this from the opposite direction, if a folder is listed in the .manifest, symlink all of its children, otherwise default to typical homesick behavior (which is just flat symlinks for everything at the top level of castle/home, with no awareness of nested structures).

I actually really like your approach, it's easier to think about logically, and it comes with the benefit of "when same directory exists in multiple castles" getting merged by default. That's something I haven't even tried to implement yet. But not my call to make.

Oh, also you forgot to prevent the .manifest from being symlinked out. That should be pretty simple.

@muratayusuke : his pull request is a complete implementation. It does everything that our work does, and more. I think he meant that if its working, we should just accept his pull request instead of continuing to work on it. BUT, it is a different approach and it seems to break backwards compatibility, I think.it's actually a cleaner approach, though,

[muratayusuke]

Oh, sorry, I misunderstood @jridgewell 's comment. As @edubkendo say, we cannot merge the pr #38 because of backwards compatibility. So I'm thinking to understand each cases in #38 , and add some cases manually if working branch lack them.

[jridgewell]

Defaulting to linking everything into just the home directory has always seemed backwards to me. And I think maintaining perfect backwards compatibility shouldn't force us to continue doing so, especially when the core idea (to symlink all the dotfiles in place with a single command) is still being preformed. I think in the worst case, it would just need to be cut as a 1.0 release, once the .manifest has been fully doc'd.

@edubkendo, all the symlinking is done from inside the castle's home directory. The globbing should never see the .manifest file.

[muratayusuke]

@jridgewell I think the biggest difference between these 2 approach is linking directories. Just thinking about merging directory function, "Never symlink directory" approach is the best way, but it will lose the benefit of just linking directories. For example, I'm linking .emacs.d and .zsh.d, and I can track new files under them automatically. It's very useful for me. If we adopt your approach, I need to add manifest file to do that. This is backwards compatibility we mean.

I agree with your opinion "I think maintaining perfect backwards compatibility shouldn't force us to continue doing so", but we can enjoy both functions of linking directories and merging directories with manifest file on both approach, one supports backwards compatibility, another doesn't. I think we should adopt the former.

Of course, this is just my viewpoint. If I miss some point, please let me know.

[muratayusuke]

I opened pr #39, so please check this implementation. Please comment if I lack some edge cases, documents or others. If there is no problem, I'll merge it.

[muratayusuke]

We finally introduced this function with #39!