development.md env setup is broken: vendor_gems script fails with gem not found error

[Justin-W]

Versions (please add them if you have more releases for this issue)

Dev env OS/HW: darwin_arm64
Ruby version: ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [arm64-darwin20]
Bundler version: Bundler version 2.2.32 (2021-11-23 commit 20d4957649)

CPI Version: Not applicable
BOSH Version: latest source (commit f501223)
CF Version: Not applicable
Diego Version: Not applicable

Stemcell

Not applicable

Environment

• Azure Environment: Not applicable

Issue Details

The env setup instructions in 'docs/development.md' seem to have become broken, likely due to this recent commit.

Logs (Please redact all sensitive information in your logs)

% ./vendor_gems
+ '[' -z x ']'
++ mktemp -d /var/folders/vy/ryp4d95n75q5j6kf58hc551m0000gn/T//bundler_config_XXXXXX
+ export BUNDLE_APP_CONFIG=/var/folders/vy/ryp4d95n75q5j6kf58hc551m0000gn/T//bundler_config_Ekf2yf
+ BUNDLE_APP_CONFIG=/var/folders/vy/ryp4d95n75q5j6kf58hc551m0000gn/T//bundler_config_Ekf2yf
+ export BUNDLE_CACHE_PATH=vendor/package
+ BUNDLE_CACHE_PATH=vendor/package
+ export BUNDLE_WITHOUT=development:test
+ BUNDLE_WITHOUT=development:test
+ bundle package --all-platforms
Fetching gem metadata from https://rubygems.org/.......
Installing faraday-excon 1.1.0
Installing faraday-net_http 1.0.1
Installing faraday-net_http_persistent 1.1.0
Installing multipart-post 2.1.1
Installing ruby2_keywords 0.0.4
Installing faraday 1.4.1
Installing faraday_middleware 1.0.0
Installing connection_pool 2.2.5
Installing net-http-persistent 4.0.1
Installing mini_portile2 2.5.1
Installing racc 1.5.2 with native extensions
Fetching nokogiri 1.11.4 (arm64-darwin)
Installing nokogiri 1.11.4 (arm64-darwin)
Installing azure-storage-common 2.0.2
Installing azure-storage-blob 2.0.1
Installing azure-storage-table 2.0.2
Installing little-plugger 1.1.4
Installing multi_json 1.15.0
Installing logging 1.8.2
Installing semi_semantic 1.2.0
Installing bosh_common 1.3262.24.0
Installing httpclient 2.8.3
Installing membrane 1.1.0
Installing bosh_cpi 2.5.0
Using bundler 2.1.4
Installing concurrent-ruby 1.1.8
Installing deep_merge 1.2.1
Installing jwt 2.2.2
Updating files in vendor/package
  * nokogiri-1.11.4-arm64-darwin.gem
Removing outdated .gem files from vendor/package
  * addressable-2.7.0.gem
Bundle complete! 14 Gemfile dependencies, 27 gems now installed.
Gems in the groups development and test were not installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Updating files in vendor/package
Could not find rspec-retry-0.6.2 in any of the sources

Repro steps (if necessary, delete otherwise)

1. Followed the env setup instructions in 'docs/development.md'.
2. At step 8, running ./vendor_gems produced the error (see above)

Workaround

1. Follow the Repro steps above
2. Run gem install rspec-retry -v 0.6.2 (to manually install the missing gem)
3. Re-running Step 8 now works without error.

I suspect that either the docs/development.md docs need to be updated (e.g. to explicitly install 'test' group gem dependencies?), or else the rspec-retry gem recently-added to the "test" group needs to be updated (e.g. added to a different group?).

[Justin-W]

Also, I just noticed that the script-vendored gems are actually committed as part of this repository (under src/bosh_azure_cpi/vendor/package), which raises a few additional questions:

1. Is src/bosh_azure_cpi/vendor/package/rspec-retry-0.6.2.gem supposed to be included within the repo? It is currently missing. FYI: This seems directly related to the same recent commit that I originally mentioned above.
2. The nokogiri gem is currently vendored in the reop as src/bosh_azure_cpi/vendor/package/nokogiri-1.11.4-x86_64-linux.gem, but on my dev env, the script created src/bosh_azure_cpi/vendor/package/nokogiri-1.11.4-arm64-darwin.gem. Which version(s) of platform-dependent gems should be included within the src/bosh_azure_cpi/vendor/package/ directory? FYI: This seems related to both PR Bump nokogiri from 1.11.0.rc3 to 1.11.4 in /src/bosh_azure_cpi #637 and commit 7aeab760.
3. The addressable gem is currently vendored in the repo as src/bosh_azure_cpi/vendor/package/addressable-2.7.0.gem, but on my dev env, the vendoring script deleted that file and created src/bosh_azure_cpi/vendor/package/addressable-2.8.0.gem. Under what circumstances are the gems within the src/bosh_azure_cpi/vendor/package/ directory supposed to be updated? FYI: This seems related to PR Bump addressable from 2.7.0 to 2.8.0 in /src/bosh_azure_cpi #639.

All of the above questions also seem at least partially related to PR #133, which seems to have modified several vendoring-related behaviors. But I couldn't find any documentation within the repo about the vendoring script/process.

---

Update

I submitted PR #649 to fix the above issues. I had to make some assumptions about which files do vs do not get committed (due to the limited documentation about the gem vendoring process, and conflicting info/contents in various past commits). Feel free to correct the PR as needed if any of my assumptions were wrong.

[Justin-W]

Related: This repo's dev scripts also impose specific constraints about the location at which chruby is installed (see this script code). Those constraints were previously undocumented within docs/development.md. I submitted PR #647 to document the previously-undocumented constraints (including a workaround for homebrew-based install).

[rkoster]

cc @ragaskar @julian-hj

[rkoster]

#649 got merged