Implement knife bootstrap client.d RFC #4529

jaym · 2016-02-05T01:49:18Z

This implements the RFC described in chef-boneyard/chef-rfc#183

There are some minor differences:

client.d is natively interpreted by chef-client in this implementation. It does not rely on a block at the end of the client.rb that loads the files in client.d
client.d is not found in the bootstrap directory, instead just the repo path.
used client_d_dir instead of client_d_path to keep it consistent with trusted_certs_dir

cc @chef/client-core

coderanger · 2016-02-05T05:18:44Z

How does this interact with chef-solo and local mode/knife? Do we want to make a solo.d and knife.d?

stevendanna · 2016-02-05T10:21:19Z

lib/chef/knife/core/bootstrap_context.rb

+        def client_d_content
+          content = ""
+          if @chef_config[:client_d_dir] && File.exist?(@chef_config[:client_d_dir])
+            root = Pathname(@chef_config[:client_d_dir])


Should we add a require "pathname" to this file?

yes, good catch

jaym · 2016-02-05T15:46:49Z

@coderanger good question. My intention was that this does not get loaded with knife as it shouldn't be reading client.rb, and I don't personally see it useful to have for knife.

As for solo, I'm open to suggestions. Would people who use solo find it useful? I think solo.d would probably make the most sense for a directory name.

danielsdeleo · 2016-02-05T16:57:34Z

One use case for knife.d would be for folks in the same org to share most of their knife config in a git repo and gitignore the actual knife.rb where they'd set username and other stuff.

jaym · 2016-02-05T17:22:53Z

Interesting. That seems useful.
Perhaps we add that one.

What about solo?

thommay · 2016-02-08T17:57:38Z

Just spoke to Jay about this. We agree with Dan that this would be useful for knife users, and the directory for that should be .chef/config.d.

However, my feelings are that we should not do this work for solo.

coderanger · 2016-02-08T20:05:35Z

The party line is that solo is fully supported, if we do it for client we should do it for solo.

btm · 2016-02-08T21:31:24Z

@coderanger chef-solo being supported doesn't mean guaranteed feature parity. we don't require all features we add to the client mode to also work in chef-solo.

Further, RFC031 is accepted as the path to deprecate chef-solo by ensuring local-mode has feature parity and then having chef-solo commands be an alias for local-mode.

I don't see any reason to require new feature for chef-solo, only that by supporting it we're promising to not break it.

coderanger · 2016-02-08T21:56:28Z

@btm Given how little work it would be (the impl is the same as for chef-client), I don't see any reason not to do it other than trying to convince people that they shouldn't use chef-solo and that is something we should never do.

btm · 2016-02-08T22:33:48Z

@coderanger that's a different argument. If @jaym or whoever implements it looks and thinks the code paths are similar enough that it's easy to add .d support to chef-solo, that's fine, go wild. But it's not required.

coderanger · 2016-02-08T22:38:19Z

@btm I'm saying it should be required as long as chef-solo is a supported tool/workflow. AFAIK it still is.

jaym · 2016-02-08T22:52:10Z

Added conf.d for knife things

btm · 2016-02-08T23:03:09Z

@coderanger then I'm disagreeing. There is no chef-solo based workflow in RFC019 and chef-solo is effectively deprecated except for it's existing UX by RFC031. There's no reason to add additional UX to chef-solo when all it is going to be is a backwards-compatible interface to local-mode, which does support .d configs.

coderanger · 2016-02-08T23:09:38Z

RFC 19 is both not an exhaustive list and not fully implemented and RFC 31 hasn't been implemented at all. The whole point of RFC 31 is that this would be supported by way of solo being client internally, so why not support it from the start as literally any chef-solo user (and yes, they exist) would expect. How would you justify not adding new features to solo other than "we don't think you should be using it"?

coderanger · 2016-02-08T23:10:12Z

Or alternatively "we don't want to support it", which is fine but it has to be supported until RFC 31 is fully implemented, that was the whole point of RFC 31.

thommay · 2016-02-09T00:38:00Z

I'm pretty sure we've never said nor intended that support implies complete feature parity.

coderanger · 2016-02-09T01:19:30Z

@thommay What does it mean then? Remember that solo is explicitly not deprecated. Keeping a high-quality user experience should mean that non-server-related features should go in both unless there is a compelling reason not to and no one has mentioned a reason other than "we don't want people using chef-solo anymore" which is not a reason I accept.

btm · 2016-02-09T04:04:00Z

I hate the word support. Support means we don't break something. Everything beyond that is trouble. I can't formulate how support relates to new features. We certainly haven't required resource parity across all supported platforms before adding a new resource for one or more platforms. We've added a few features to the client that didn't make it into solo until someone added some kind of compatibility later, e.g. support in solo or via an external cookbook.

There's an issue here with our differing development priorities. Let's talk on chef-dev.

luckymike · 2016-02-12T17:44:22Z

@btm @thommay Solo was actually just broken (existing functionality completely changed) in 12.7 to ensure that its flags had parity with client.

Chef Solo -r (–recipe-url) changes
https://www.chef.io/blog/2016/02/12/chef-client-12-7-released/

If breaking existing (if rarely or accidentally used) functionality was in order, why not supporting the conf directory?

Is the goal for solo to be a debasing experience that encourages switching to client?

danielsdeleo · 2016-02-12T17:57:08Z

@luckymike

Instead invokes the --recipe-url code, which had the side effect of running an immediate unprompted rm -rf * in the current working directory of the user. Due to this problem and other issues around this rm -rf * behavior it has been removed from the --recipe-url code in chef-solo. The use of -r in chef-solo to mean --recipe-url has also been deprecated.

So basically we screwed up, and merged an "improvement" that was actually a net negative for Chef Solo users, including ourselves, who were rm -rf *-ing their home directories. I guess in some pedantic sense this change violates SemVer, and it happens to only impact solo users, but tbqf I would rather delete all copies of SemVer from the internet than delete your home directory on accident.

Is the goal for solo to be a debasing experience that encourages switching to client?

Is accidentally deleting a bunch of your files "a debasing experience"? If so, wouldn't making it less likely that you do this on accident be the opposite of that?

In any case, this discussion is about adding a feature.

btm · 2016-02-12T18:00:43Z

@luckymike the issue that @danielsdeleo explained aside, the debate is about policy, i.e. by policy do we require feature parity to merge a new feature. It's less about adding the feature this PR represents for solo (I think there's more work/shed-building to figure out the right medium-term UX decisions given RFC031 than there is writing the code for solo.d support).

luckymike · 2016-02-12T18:17:26Z

I wasn't arguing for deleting home directories, I was trying to offer a clear example of rightfully favoring parity over policy (sem ver). Turning @coderanger's argument for including support in solo into a larger policy debate effectively decides this issue without anyone taking responsibility for it.

btm · 2016-02-12T18:27:43Z

@luckymike by "this issue" you mean #4529, not something related to the -r stuff, right?

This feature is part of the plan I envisioned in the ohai-config RFC discussion for getting ohai UX happy and @coderanger happy about ohai UX being happy. So I'm responsible for it [and @coderanger's happiness!]. But we're in the middle of another hairy chef-client release so it's not my top priority at the moment.

luckymike · 2016-02-12T18:29:46Z

Yes, the issue #4529, the issue we're commenting on :)

lamont-granquist · 2016-02-12T18:48:24Z

We can argue about the semantics of the word "deprecated" but I think the reality is that chef-solo is "deprecated". Time spent on chef-solo-specific codepaths is wasted time since chef-zero eventually will be replacing chef-solo via RFC 031. That also completely forces our hand when it comes to conflicting features that have been caused by chef-solo and chef-client/chef-zero having different codepaths. RFC 31 means that at some point in the future the conflicting meanings of the '-r' flag need to be resolved. Generally chef-solo is the smaller userbase and is going to lose that argument. In the case in question its just laughable since making the solo behavior the new meaning of '-r' would mean that everyone using '-r' on the client to mean '--run-list' and doing elastic scaling would have their argument changed to '--recipe-url' and get a complimentary rm -rf run in the direction they're executing chef-client in. We have to deprecate the meaning of '-r' for chef-solo users and I don't think there can be any meaningful debate about that.

So there isn't parity here. And in fact I think that "solo.d" may buy us a worse compatibility issue since how is that supposed to work when we change chef-solo to invoke chef-client local-mode? Not only is that doing work to create "solo.d" in codepaths that we'll throw away, but we need to have zero pick up solo.d for backcompat when RFC 31 is executed, which is just going to be digging ourselves a deeper hole to climb out of, it doesn't move us any closer towards executing RFC 031.

And removal of the rm_rf is entirely because when I was debugging #4368 recently, which is a chef-solo specific issue, I took my usual chef-zero command line argument:

chef-client -z -r 'recipe[test]'

And blindly converted that into a chef-solo command line:

chef-solo -r 'recipe[test]'

Which immediately wiped out all the cookbooks in my current working directory.

Which made me more or less like this:

I was actually the one that merged the rm_rf line initially:

#2418

We've also had chef-solo user requests to remove that line:

#3802

And not only that but I went and added the "--delete-entire-chef-repo" line and added the rm_rf TO chef-client from chef-solo which didn't exist there before. I just did it in a much safer way where users of "--recipe-url" who do NOT want that behavior do not get it forced down their throat and where a simple command line option like "-r" is not brutally destructive.

And I don't see how having to change a command line from:

chef-solo -r http:///whatever.com/recipies.tgz

To:

chef-solo --recipe-url http:///whatever.com/recipies.tgz --delete-entire-chef-repo

And getting exactly_the_same_behavior_as_you_had_before is "breaking functionality" or a "debasing experience"

btm · 2016-02-12T19:02:27Z

someone please file an issue about chef-solo -r if they want to keep talking about it beyond this point.

This configuration defaults to config_dir/client.d, but can be set to nil if you do not want chef-client loading any extra config files.

When knife finds a client.d/ directory, it will upload all files nested under that directory.

client.d/*.rb will be read in sorted order. All directories will be ignored.

This will behave similarly to the client.d directory. The top-level ruby files will be loaded in sorted order.

Also, do less mocking so the test will catch the error

Passing the value of `escape_glob` to Dir.glob does not always work correctly on Windows. Always use forward slashes when globbing, even on Windows. We now do this via `escape_glob_dir`, which calls Pathname.cleanpath, which does the foward slash thing.

This makes it the same as application/client.rb. This will let the mixin be used the same way in both places.

jaym · 2016-02-25T05:28:43Z

@chef/client-maintainers I added solo.d, could probably use another round of reviews

This commit refactors the spec to use shared examples as the test for client.d and solo.d is the same. knife.d is a little special so it cannot use this.

btm · 2016-02-25T18:28:17Z

chef-config/lib/chef-config/config.rb

+
+    # A directory that contains additional configuration scripts to load for
+    # the workstation config
+    default(:conf_d_dir) { PathHelper.join(config_dir, "conf.d") }


I think this is supposed to be config.d because the workstation config is config.rb.

Isn't workstation config usually knife.rb?

knife.rb is very softly deprecated. There's no real upside to forcing people to switch, but we decided to use config.rb going forward because it's used by many tools other than knife.

config.rb is supposed to be used in preference to knife.rb according to the docs

btm · 2016-02-25T18:37:12Z

👍 with the s/conf_d/config_d/ change.

smurawski · 2016-02-25T19:15:47Z

confirming my 👍 after the latest updates

Implement knife bootstrap client.d RFC

jaym force-pushed the jdm/knife-bootstrap-clientd branch from 849c82d to 4608f47 Compare February 5, 2016 03:06

stevendanna reviewed Feb 5, 2016
View reviewed changes

jaym force-pushed the jdm/knife-bootstrap-clientd branch from 1b7d6cd to 29db233 Compare February 5, 2016 17:24

jaym force-pushed the jdm/knife-bootstrap-clientd branch from 29db233 to 6e4e712 Compare February 8, 2016 22:51

danielsdeleo mentioned this pull request Feb 9, 2016

knife bootstrap named_run_list not supported #4365

Closed

jaym force-pushed the jdm/knife-bootstrap-clientd branch from 13f8b8b to b5e6119 Compare February 17, 2016 05:24

jaym added 7 commits February 24, 2016 21:24

Add client_d_dir configuration

0584f32

This configuration defaults to config_dir/client.d, but can be set to nil if you do not want chef-client loading any extra config files.

Copy client.d files during bootstrap

d0f2106

When knife finds a client.d/ directory, it will upload all files nested under that directory.

Make chef-client read client.d

9cfe82b

client.d/*.rb will be read in sorted order. All directories will be ignored.

Allow the workstation config to load a conf.d directory

352c98e

This will behave similarly to the client.d directory. The top-level ruby files will be loaded in sorted order.

Fix broken load_conf_d_file

571ad24

Also, do less mocking so the test will catch the error

Refactor workstation_config_loader's conf_d_files

05f5c67

jaym force-pushed the jdm/knife-bootstrap-clientd branch from b5e6119 to 32e7c41 Compare February 25, 2016 05:24

jaym added 4 commits February 24, 2016 21:27

Refactor dot_d loading for workstation_config_loader to use mixin

3077de7

Rename read_config to apply_config

df7683d

This makes it the same as application/client.rb. This will let the mixin be used the same way in both places.

Refactor application/client.rb to use DotD mixin for loading client.d

8741f64

Add support for solo.d

cbb0f56

jaym force-pushed the jdm/knife-bootstrap-clientd branch from 32e7c41 to 4cc7325 Compare February 25, 2016 05:27

Add test for solo.d

158178c

This commit refactors the spec to use shared examples as the test for client.d and solo.d is the same. knife.d is a little special so it cannot use this.

jaym force-pushed the jdm/knife-bootstrap-clientd branch from 4cc7325 to 158178c Compare February 25, 2016 06:18

btm reviewed Feb 25, 2016
View reviewed changes

Rename conf.d to config.d

c9b5c7a

btm added a commit that referenced this pull request Feb 25, 2016

Merge pull request #4529 from chef/jdm/knife-bootstrap-clientd

dbedc9c

Implement knife bootstrap client.d RFC

btm merged commit dbedc9c into master Feb 25, 2016

btm deleted the jdm/knife-bootstrap-clientd branch February 25, 2016 19:57

mwrock added the Enhancement label Mar 2, 2016

chef-supermarket added the Signed CLA label May 25, 2016

chef-supermarket assigned stevendanna May 25, 2016

thommay added Type: Enhancement Adds new functionality. and removed Enhancement labels Jan 25, 2017

chef locked and limited conversation to collaborators Nov 16, 2017

Implement knife bootstrap client.d RFC #4529

Implement knife bootstrap client.d RFC #4529

Conversation

jaym commented Feb 5, 2016

coderanger commented Feb 5, 2016

stevendanna Feb 5, 2016

Choose a reason for hiding this comment

jaym Feb 5, 2016

Choose a reason for hiding this comment

jaym commented Feb 5, 2016

danielsdeleo commented Feb 5, 2016

jaym commented Feb 5, 2016

thommay commented Feb 8, 2016

coderanger commented Feb 8, 2016

btm commented Feb 8, 2016

coderanger commented Feb 8, 2016

btm commented Feb 8, 2016

coderanger commented Feb 8, 2016

jaym commented Feb 8, 2016

btm commented Feb 8, 2016

coderanger commented Feb 8, 2016

coderanger commented Feb 8, 2016

thommay commented Feb 9, 2016

coderanger commented Feb 9, 2016

btm commented Feb 9, 2016

luckymike commented Feb 12, 2016

danielsdeleo commented Feb 12, 2016

btm commented Feb 12, 2016

luckymike commented Feb 12, 2016

btm commented Feb 12, 2016

luckymike commented Feb 12, 2016

lamont-granquist commented Feb 12, 2016

btm commented Feb 12, 2016

jaym commented Feb 25, 2016

btm Feb 25, 2016

Choose a reason for hiding this comment

coderanger Feb 25, 2016

Choose a reason for hiding this comment

danielsdeleo Feb 25, 2016

Choose a reason for hiding this comment

jaym Feb 25, 2016

Choose a reason for hiding this comment

coderanger Feb 25, 2016

Choose a reason for hiding this comment

btm commented Feb 25, 2016

smurawski commented Feb 25, 2016