splitting up services into modules

[joshleecreates]

Fixes: #866

• splits up services into multiple docker files
• experiment with changing service definition for checkout service when kafka is enabled

todo:

• fix checkout service to work without kafka
• fix services that error without feature flag service
• provide noop config for collector when observability backends are disabled

Changes

Please provide a brief description of the changes here.

Merge Requirements

For new features contributions please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

[joshleecreates]

Amazing, thanks @puckpuck !! This is working as expected for me now.

[puckpuck]

This one is ready for review.

It splits out all components except from what is absolutely required to run the demo with an exposed UI. A start and stop script is included to facilitate specifying the proper command line options to start the various modules.

The following modules are what are split out from the core:

• load - includes the load generator
• observability - includes jaeger, prometheus, and grafana, as well as necessary changes to the collector
• featureflags - includes the feature flag service with postrgres, as well as an environment variable flag for several services that have logic for feature flags.
• kafka-extras - includes kafka, accounting, and fraud detection services as well as an environment variable flag for the checkout service
• tests - tests included with demo

A full version of the demo can be run with a special full or all module when specified with the start script.

Starting the demo without any module, you use: ./start-demo.sh
Starting the demo with the load generator and observability components is: ./start-demo.sh load observability
Starting the full demo is: ./start-demo.sh full

Stopping the demo is: ./stop-demo.sh - which will stop all components and remove orphans if they exists.

/cc @open-telemetry/demo-approvers

[cartersocha]

Wow this is fantastic. Need to spend some more time running it but looks good so far.

Very exciting update! This opens up some really interesting extensibility options and a bit of maintainer dread 🍷

[cartersocha]

It seems to spam the starting and stopping logs for some services.

[cartersocha]

product catalog, ad, checkout, and recommendation are failing when running just the script with no arguments due to the feature flag address env variable not being set

[joshleecreates]

@cartersocha

product catalog, ad, checkout, and recommendation are failing when running just the script with no arguments due to the feature flag address env variable not being set

This sounds like the images need to be rebuilt maybe? These services work for me in all modes.

docker system prune followed by docker compose build should then let you run the script. This will be resolved for all users when the version tag gets bumped and the images based on these changes are published.

[austinlparker]

2 things -

1. Invoke scripts thru makefile
2. Disable 'docker compose up' and instead direct people to use make targets

[austinlparker]

Per discussion on call today and in thread -

• We should try to keep docker compose up behavior the same between versions

Open question, could we make docker compose take parameters? i.e., docker compose up lite or whatever.

[joshleecreates]

I can't find the original comment but I believe @julianocosta89 mentioned that docker compose up behavior should stay the same as this feature is implemented. I'm curious to hear wider thoughts on what is the most desirable dev experience that supports this enhancement.

Challenges:

• make doesn't have a built in mechanism for arguments so it cannot be as dynamically extensible as the bash script provided by @puckpuck
• the docker compose -f syntax is verbose
• Either bash or make introduce an additional dev environment dependency (right now make is only needed for creating releases)

Ideas

Option 1:

Use the bash script and add a docker-compose.override.yml to restore default docker compose up functionality to be the full demo.
Full demo: docker compose up
Modules: ./start-demo.sh module

Option 2:

Use make with an environment variable for modules and add a docker-compose.override.yml.
Full demo: docker compose up
Modules: export MODULE=module && make run

Option 3:

Use make with hard-coded targets for modules and add a docker-compose.override.yml
Unlike with options 1 or 2, the script would need to be edited any time new modules are added, which would limit flexibility for vendor extensions on forks.
Full demo: docker compose up
Modules: make run module

Option 4:

Restore docker compose up and require docker compose -f syntax for users who want to run the demo in lite mode. No additional dependencies, and the annoying syntax is applied to the less common use case. We could make a single docker-compose.lite.yaml that simplifies the amount of chaining needed to run the "lite" version with the observability backends and feature flag services.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[julianocosta89]

Hey @joshleecreates I've mentioned that in here:
#927 (comment)

But that's only my opinion, I'm open to discuss this and go with the choice most of people like.

[austinlparker]

I think we need to do Option 4. I don't want to break default behavior between releases, and ultimately we're designing around 'batteries included' experiences rather than limited ones.

[joshleecreates]

@austinlparker makes sense.

I should have just made this last commit with my comment. It enables Options 1 + 4.

I do find the bash scripts convenient and will probably keep them locally somewhere if the decision is to remove them from the official. If that's the case, then this probably just needs a documentation update showing how an example of the lite usage without the scripts.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[puckpuck]

As discussed during the June 26th SIG meeting.

docker compose up will run the full demo and preserve functionality that was always there before.
The standard docker-compose.yml file is the same as it was before, with some updates on how the depends_on sections are done to be in line with the latest docker-compose file format.
A new file called docker-compose.minimal.yml can be used to run the demo without the Feature Flag service, or anything that depends on Kafka. Load Generator, and all Observability tooling will still run in this mode, as well as the core components for the demo itself.

Technically someone could specify docker compose -f docker-compose.minimal.yml up to run the demo in minimal mode, where the -f argument appears in the command line matters, and humans make errors. So we have some new make targets to provide easy buttons to start and stop the demo in full or minimal mode.

• make start will start the full demo
• make start-minimal will start the minimal demo
• make stop will stop all things related to the demo

[joshleecreates]

This looks great, I appreciate the simplification over my original breakdown. 🚀 🎉

[julianocosta89]

Great work on this everyone!
Awesome!

I think we can have the docs updated in a new PR.