Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DO NOT MERGE: Habitize the Chef Server #1416

Closed
wants to merge 99 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
c2b14a4
Add Habitat plan to bookshelf
Feb 3, 2017
78fb7f6
:construction: WIP - Run erchef in Habitat :construction:
Apr 7, 2017
087adbe
fixups
jeremymv2 Oct 25, 2017
218708c
Add rabbitmq to erchef plan (this may be wrong)
markan Oct 26, 2017
32c897b
First cut at oc_bifrost plan
markan Oct 26, 2017
769fd98
conflict resolution
Oct 27, 2017
730af82
docker compose for oc_erchef and rabbitmq
jeremymv2 Oct 27, 2017
6d7f025
Use sql for bookshelf
markan Oct 27, 2017
07d3bd5
Start wiring up erchef to talk to the other services
markan Oct 27, 2017
8f5b70a
Almost working bifrost
Oct 27, 2017
c7a2180
Working bifrost, replicating most of the magic to bookshelf and erche…
Oct 27, 2017
5285a8f
Fix erchef database migrations
Oct 27, 2017
b3eec1b
Get rid of the bad ERL_EPMD_ADDRESS environment variable that just se…
Oct 27, 2017
a147b69
Presenting: an oc_erchef service that starts
Oct 30, 2017
f5b71cf
health check for oc_erchef
jeremymv2 Oct 30, 2017
613b654
Fix erchef health check by initializating elasticsearch
Oct 30, 2017
eac0d8d
working bookshelf and moved the docker-compose.yml to top level
jeremymv2 Oct 30, 2017
7ffcd5b
working depsolver in erchef
Oct 31, 2017
11d3825
health check for oc_bifrost
jeremymv2 Nov 1, 2017
4a6eb2b
removed pkg_svc_user=root requirement for oc_bifrost
jeremymv2 Nov 1, 2017
8ec4526
removed pkg_svc_user=root for oc_erchef
jeremymv2 Nov 1, 2017
a8fe524
removed pkg_svc_user=root for bookshelf
jeremymv2 Nov 2, 2017
766f618
Fix typo
markan Nov 1, 2017
82865b5
Very tenative plan; need work for lua and the default.toml
markan Nov 2, 2017
b350060
More work for nginx config
markan Nov 2, 2017
d6f5b48
More fixes; things seem to start up, who knows if it works
markan Nov 2, 2017
dff11b6
nginx config and init fixes
Nov 3, 2017
e71cc91
wip for boostrapping the initial data
jeremymv2 Nov 3, 2017
6868303
bootstrap improvements - it is working at least but is mvp
jeremymv2 Nov 3, 2017
9fcfd3e
WIP improvements for nginx
markan Nov 3, 2017
702df54
Add the openresty-lpeg plan and integrate it into chef-server-nginx
Nov 4, 2017
a04c9ba
Finish wiring up nginx and add a health_check
Nov 5, 2017
6960c17
Nginx config cleanup
Nov 5, 2017
6f99934
chef-server-ctl container and attempting better secrets mgmt
jeremymv2 Nov 7, 2017
def0380
Fix TOML and secrets walking/updating logic so now it stabilizes afte…
Nov 7, 2017
18baab6
WIP, maybe remove nginx changes
markan Nov 3, 2017
9551c01
Get nginx lua stuff cleaned up, seems to be working ok
markan Nov 7, 2017
fb3231d
secrets mgmt gets a big boost - seeds initial values and injects back…
jeremymv2 Nov 7, 2017
768535e
chef-server-ctl pushes secrets to oc_erchef and all is well in the world
jeremymv2 Nov 8, 2017
cc9ef8d
Wire up bookshelf and bifrost to secrets the same as erchef. also mak…
Nov 8, 2017
55d5323
first stab at pedant
jeremymv2 Nov 8, 2017
ceaad0b
wip wip wip debugging nginx lua
jeremymv2 Nov 9, 2017
d858164
A few minor fixes
markan Nov 9, 2017
2291c9e
QA testing revealed a number of config problems in oc_erchef init/rec…
Nov 9, 2017
4c549f6
Fix a typo from the last commit
Nov 9, 2017
e61663f
Fixes to get us a clean pedant run
Nov 9, 2017
1c83aff
working on getting chef server data bootstrapped correctly
jeremymv2 Nov 9, 2017
b423313
Fix SQL syntax so that updates work correctly
Nov 10, 2017
215232e
cleaner oc-chef-pedant installation. no longer git cloning.
jeremymv2 Nov 13, 2017
c02448d
WIP nginx cleanup
markan Nov 10, 2017
6993da5
First hack at knife opc
markan Nov 15, 2017
d4dde6b
Rename knife-opc to knife-pivotal
markan Nov 15, 2017
cda8000
Fix bad root path
markan Nov 15, 2017
0b65c9d
build our own dbdpg and update bookshelf, bifrost and erchef to use it
Nov 15, 2017
efc7e8d
openresty is now running as hab user instead of root
thomascate Nov 15, 2017
a05ebf9
hacky fix for pedant on docker
thomascate Nov 15, 2017
9e8874f
Correct a few instances of tcate->chef-server and add a data volume f…
Nov 15, 2017
c73517a
Correct the path to the mime.types file and remove the unnecessary us…
Nov 15, 2017
d31ecc4
Update the pivotal config in the same way as the 'pedant' config
Nov 15, 2017
d735da2
fixes to get knife/pedant working within the container
thomascate Nov 16, 2017
a5dd445
WIP don't merge doesn't work
markan Nov 16, 2017
7a220f2
minor fixes to get the bootstrap script executing without error
jeremymv2 Nov 16, 2017
6631437
WIP Fix bootstrap
markan Nov 17, 2017
f92975f
Pedant mostly passes now
markan Nov 17, 2017
6a8d5dd
Remove cruft
markan Nov 17, 2017
c3eef0d
added dynamic versioning
jeremymv2 Nov 18, 2017
4aed862
Refactor nginx to use partials
markan Nov 18, 2017
ecf370d
Fixes from review comments
markan Nov 19, 2017
126363f
minor modification to plan and template so nginx builds and starts
jeremymv2 Nov 20, 2017
96257a8
Fix spelling error (pedant)
markan Nov 21, 2017
c4c62a7
adding initial plan for oc-id
jeremymv2 Nov 22, 2017
212fbe6
deleted unecessary conf file
jeremymv2 Nov 22, 2017
cc36311
oc_id no longer needs to be run under root user
jeremymv2 Nov 23, 2017
2cc03f0
docker-compose host mode networking example
Dec 1, 2017
e0d1743
Secrets for private-chef-ctl
markan Dec 1, 2017
29aa80f
Fixes for paths and keys
markan Dec 1, 2017
52c45c3
Fixes for chef-server-ctl
markan Dec 4, 2017
10e5b2c
Minor fixes for typos
markan Dec 4, 2017
ee68492
Review fixups
markan Dec 4, 2017
c0335a9
Database timezone
markan Dec 4, 2017
2c825a5
Add missing hab secrets config for chef-server-ctl
markan Dec 5, 2017
c6a1db2
Cleanup chef-server-ctl
markan Dec 5, 2017
700fdea
Explicitly specify openssl dependency to fix runtime PATH
Dec 6, 2017
a8bfceb
wire up the data_collector
jeremymv2 Dec 6, 2017
be52f28
fixed pkg_path for wrap-knife-opc.rb script
jeremymv2 Dec 7, 2017
bd797d5
make chef server api ip configurable for chef-server-ctl
jeremymv2 Dec 8, 2017
a1ed803
Refactor and fixup of erlang services so that they look similar.
markan Dec 6, 2017
e0bafc7
moved data_collector.token ownership to chef-server-ctl
jeremymv2 Dec 8, 2017
dc8ea16
adding actions_fqdn back into oc_chef_wm to fix missing config item e…
jeremymv2 Dec 8, 2017
8622082
Make port configurable for chef-server-ctl
markan Dec 9, 2017
f096749
Fix bad rendering of ports and data collector token
markan Dec 11, 2017
461d921
no need to check if uid == 0 in chef-server-ctl, removing..
jeremymv2 Dec 11, 2017
b247712
Fix bad rendering of ports and data collector token
markan Dec 11, 2017
f6c9a12
fix for data-collector location not rendering
jeremymv2 Dec 12, 2017
a54a4ff
Fix to knife command
markan Dec 13, 2017
92f13f8
Use common vendor dir for ruby, do misc cleanups, save 300MB (1.01GB-…
markan Dec 13, 2017
852c447
Fix host headers in nginx (we were dropping the port)
markan Dec 16, 2017
cef4d35
[oc-chef-pedant] Normalize chef server uri
markan Dec 19, 2017
0a869c0
chef-server-ctl binds to chef-server-nginx for dynamic host/port reso…
jeremymv2 Dec 19, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,10 @@ services:
--bind database:postgresql.default
--bind elasticsearch:elasticsearch.default
--bind chef-server-ctl:chef-server-ctl.default
environment:
HAB_OC_ERCHEF: |
[data_collector]
enabled = false

chef-server-nginx:
image: $HAB_ORIGIN/chef-server-nginx
Expand Down
27 changes: 26 additions & 1 deletion src/nginx/habitat/config/chef_http_lb_common
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,17 @@
listen {{port}};
server_name api.chef-server.dev;

{{#if bind.oc_erchef ~}}
{{#eachAlive bind.oc_erchef.members as |member| ~}}
{{#if @last ~}}
{{#if member.cfg.data_collector.enabled ~}}
set_by_lua $data_collector_token '{{member.cfg.data_collector_token}}';
{{else ~}}
set_by_lua $data_collector_token 'return os.getenv("DATA_COLLECTOR_TOKEN")';
{{/if ~}}
{{/if ~}}
{{/eachAlive ~}}
{{/if ~}}

access_log stdout opscode;
{{~#if is_ssl}}
Expand Down Expand Up @@ -56,7 +66,22 @@
proxy_pass http://opscode_erchef;
}


{{#if bind.oc_erchef ~}}
{{#eachAlive bind.oc_erchef.members as |member| ~}}
{{#if @last ~}}
{{#if member.cfg.data_collector_enabled ~}}
location ~ "^/organizations/([^/]+)/data-collector$" {
set $request_org $1;
access_by_lua_block { validator.validate("POST") }
proxy_set_header x-data-collector-token $data_collector_token;
proxy_set_header x-data-collector-auth "version=1.0";
rewrite ^ /data-collector/v0/ break;
proxy_pass https://data-collector;
}
{{/if ~}}
{{/if ~}}
{{/eachAlive ~}}
{{/if ~}}

location ~ "^/organizations/([^/]+)/required_recipe$" {
# This endpoint is unique because it is defined via nginx and is not
Expand Down
11 changes: 11 additions & 0 deletions src/nginx/habitat/config/nginx.conf
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,17 @@ http {
{{/eachAlive}}
{{/if}}
}
{{#if bind.oc_erchef ~}}
{{#eachAlive bind.oc_erchef.members as |member| ~}}
{{#if @last ~}}
{{#if member.cfg.data_collector_enabled ~}}
upstream data-collector {
server {{member.cfg.data_collector_server}}:{{member.cfg.data_collector_port}};
}
{{/if ~}}
{{/if ~}}
{{/eachAlive ~}}
{{/if ~}}

# Include upstream definitions for addons
# TODO HABITAT plugins not yet
Expand Down
4 changes: 0 additions & 4 deletions src/nginx/habitat/default.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,6 @@ server_name = "chef-server"
ssl_protocols = "TLSv1.2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:+100:

ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:!aNULL:!eNULL:!EXPORT"

# Copied from workflow-server's default.toml
[data_collector]
token = "93a49a4f2482c64126f7b6015e6b0f30284287ee4054ff8807fb63d9cbd1c506"

# Ports used for SSL traffic; used to render partials for chef_http[s]_lb_conf
[http]
port = 8080
Expand Down
2 changes: 1 addition & 1 deletion src/nginx/habitat/plan.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ pkg_exports=(
)
pkg_binds_optional=(
[bookshelf]="port"
[oc_erchef]="port"
[oc_erchef]="port data_collector_enabled data_collector_server data_collector_port data_collector_token"
[oc_bifrost]="port"
[elasticsearch]="http-port"
[oc_id]="port"
Expand Down
14 changes: 12 additions & 2 deletions src/oc_erchef/habitat/config/sys.config
Original file line number Diff line number Diff line change
Expand Up @@ -256,8 +256,18 @@
{depsolver_timeout, 20000},
{depsolver_pooler_timeout, 0}
]},


{{#if cfg.data_collector.enabled ~}}
{data_collector, [
{root_url, "https://{{cfg.data_collector.server}}:{{cfg.data_collector.port}}/data-collector/v0/"},
{timeout, 30000},
{init_count, 25},
{max_count, 100},
{cull_interval, {1, min}},
{max_age, {70, sec}},
{max_connection_duration, {70,sec}},
{ibrowse_options, [{connect_timeout, 10000}]}
]},
{{/if ~}}
{stats_hero, [
{udp_socket_pool_size, 20 },
{protocol, estatsd},
Expand Down
3 changes: 3 additions & 0 deletions src/oc_erchef/habitat/default.toml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ queue_at_capacity_affects_overall_status=""

[data_collector]
enabled=false
server="automate-server.test"
port="443"
token="93a49a4f2482c64126f7b6015e6b0f30284287ee4054ff8807fb63d9cbd1c506"

[stats_hero]

Expand Down
6 changes: 5 additions & 1 deletion src/oc_erchef/habitat/plan.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,11 @@ pkg_upstream_url="https://github.com/chef/chef-server"

pkg_exposes=(port)
pkg_exports=(
[port]=oc_chef_wm.port
[port]=oc_chef_wm.port
[data_collector_enabled]=data_collector.enabled
[data_collector_server]=data_collector.server
[data_collector_port]=data_collector.port
[data_collector_token]=data_collector.token
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to figure how to just export the top level toml table key [data_collector]=data_collector

whenever I tried it would get:

oc_erchef_1          | hab-sup(MR): Starting jeremymv2/oc_erchef
oc_erchef_1          | thread 'main' panicked at 'Struct should serialize to bytes: ValueAfterTable', /checkout/src/libcore/result.rs:906:4
oc_erchef_1          | note: Run with `RUST_BACKTRACE=1` for a backtrace.

)

pkg_binds_optional=(
Expand Down