Integrate Plank into Transport as a module (#15)

* Move Plank code from personal repo

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Update package reference to vmware/transport-go/plank

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Add pipeline configs for Plank

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Fix broken tests

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Update Plank README

Signed-off-by: Josh Kim <jsk9260@gmail.com>

* Fix bug in REST bridge override & mux.Router concurrency

Current implementation of REST bridge override had some critical issues
that could lead to memory leaks and new URIs not being properly
reflected or old URIs still accessible. These issues mostly originate
from mux.Router not allowing full CRUD operations on the Router
instance. (e.g. there was no way to effecively remove an existing route
from any given router)

Signed-off-by: Josh Kim <kjosh@vmware.com>

* chore: update Dockerfile to match latest work

Signed-off-by: Josh Kim <kjosh@vmware.com>

* chore: Add copyright notices in src files

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Document broker sample code

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Minor changes to README & remove unnecessary files

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Add --rest-bridge-timeout to set REST bridge timeout

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Remove redundant bus and service registry init

Both instances are initialized way before StartServer() is called,
initialize() for example.

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Use go for build

Closes #19

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Document broker_sample package code

Signed-off-by: Josh Kim <kjosh@vmware.com>

* More documentation and gofmt and cleanup

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Break up server.go into multiple files

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Add a more complex sample

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Expose stompserver's ConnEvent and EventType

These two structs are essential in notifying services about
session disconnection hence opening the minimal structure for external
packages to consume.

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Notify certain STOMP session events through internal channels

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Use Proxyheaders to get real IPs for proxied setups

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Prohibit STOMP clients sending directly to topic/q

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Change env vars for certain flags to avoid conflicts

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Turn requestBuilder signature into its own type

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Provide a bit more detail to request timeout

Signed-off-by: Josh Kim <kjosh@vmware.com>

* Replace urfavecli with Cobra

Closes #18

Signed-off-by: Josh Kim <kjosh@vmware.com>

.github/workflows/plank-postmerge.yml

@@ -0,0 +1,38 @@
+name: Plank Post-merge pipeline
+
+on:
+  push:
+    paths:
+      - plank/**/*
+    branches:
+      - main
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up repo
+        uses: actions/checkout@v2
+      - uses: actions/setup-go@v2
+        with:
+          go-version: '^1.16'
+      - run: go get ./...
+        working-directory: plank
+      - run: |
+          go get github.com/axw/gocov/gocov
+          go get github.com/AlekSi/gocov-xml
+          go install github.com/axw/gocov/gocov
+          go install github.com/AlekSi/gocov-xml
+        working-directory: plank
+      - run: |
+          go test -v -coverprofile cover.out ./...
+          gocov convert cover.out | gocov-xml > coverage.xml
+        working-directory: plank
+#      - uses: codecov/codecov-action@v1
+#        with:
+#          token: ${{ secrets.CODECOV_TOKEN }}
+#          files: ./coverage.xml
+#          flags: unittests
+#          fail_ci_if_error: true
+#          verbose: true
+#        working-directory: plank

.github/workflows/plank-premerge.yaml

@@ -0,0 +1,28 @@
+name: Plank pre-merge pipeline
+
+on:
+  pull_request:
+    paths:
+      - plank/**/*
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up repo
+        uses: actions/checkout@v2
+      - uses: actions/setup-go@v2
+        with:
+          go-version: '^1.16'
+      - run: go get ./...
+        working-directory: plank
+      - run: |
+          go get github.com/axw/gocov/gocov
+          go get github.com/AlekSi/gocov-xml
+          go install github.com/axw/gocov/gocov
+          go install github.com/AlekSi/gocov-xml
+        working-directory: plank
+      - run: |
+          go test -v -coverprofile cover.out ./...
+          gocov convert cover.out | gocov-xml > coverage.xml
+        working-directory: plank

.github/workflows/transport-postmerge.yml

@@ -2,6 +2,8 @@ name: Transport Post-merge pipeline
 
 on:
   push:
+    paths-ignore:
+      - plank/**/*
     branches:
       - main
 

.github/workflows/transport-premerge.yml

@@ -2,6 +2,8 @@ name: Transport Pre-merge pipeline
 
 on:
   pull_request:
+    paths-ignore:
+      - plank/**/*
 
 jobs:
   test:

.gitignore

@@ -2,3 +2,6 @@
 /.idea
 /bus/bus.iml
 /bifrost
+
+**/*.log
+**/cover.out

bus/fabric_endpoint.go

@@ -14,6 +14,11 @@ import (
     "sync"
 )
 
+const (
+    TRANSPORT_INTERNAL_CHANNEL_PREFIX = "_transportInternal/"
+    STOMP_SESSION_NOTIFY_CHANNEL = TRANSPORT_INTERNAL_CHANNEL_PREFIX + "stomp-session-notify"
+)
+
 type EndpointConfig struct {
     // Prefix for public topics e.g. "/topic"
     TopicPrefix           string
@@ -55,6 +60,11 @@ type channelMapping struct {
     autoCreated bool
 }
 
+type StompSessionEvent struct {
+    Id string
+    EventType stompserver.StompSessionEventType
+}
+
 type fabricEndpoint struct {
     server stompserver.StompServer
     bus    EventBus
@@ -93,6 +103,18 @@ func newFabricEndpoint(bus EventBus,
 }
 
 func (fe *fabricEndpoint) Start() {
+    fe.server.SetConnectionEventCallback(stompserver.ConnectionStarting, func(connEvent *stompserver.ConnEvent) {
+        busInstance.SendResponseMessage(STOMP_SESSION_NOTIFY_CHANNEL, &StompSessionEvent{
+            Id: connEvent.ConnId,
+            EventType: stompserver.ConnectionStarting,
+        }, nil)
+    })
+    fe.server.SetConnectionEventCallback(stompserver.ConnectionClosed, func(connEvent *stompserver.ConnEvent) {
+        busInstance.SendResponseMessage(STOMP_SESSION_NOTIFY_CHANNEL, &StompSessionEvent{
+            Id: connEvent.ConnId,
+            EventType: stompserver.ConnectionClosed,
+        }, nil)
+    })
     fe.server.Start()
 }
 
@@ -114,6 +136,12 @@ func (fe *fabricEndpoint) addSubscription(
         return
     }
 
+    // if destination is a protected channel do not establish a subscription
+    // (we don't want any clients to be sending messages to internal channels)
+    if isProtectedDestination(channelName) {
+        return
+    }
+
     fe.chanLock.Lock()
     defer fe.chanLock.Unlock()
 
@@ -262,3 +290,10 @@ func (fe *fabricEndpoint) getChannelNameFromSubscription(destination string) (ch
     }
     return "", false
 }
+
+// isProtectedDestination checks if the destination is protected. this utility function is used to
+// prevent messages being from clients to the protected destinations. such examples would be
+// internal bus channels prefixed with _transportInternal/
+func isProtectedDestination(destination string) bool {
+    return strings.HasPrefix(destination, TRANSPORT_INTERNAL_CHANNEL_PREFIX)
+}

bus/fabric_endpoint_test.go

@@ -24,6 +24,7 @@ type MockStompServer struct {
     started bool
     sentMessages []MockStompServerMessage
     subscribeHandlerFunction stompserver.SubscribeHandlerFunction
+    connectionEventCallbacks map[stompserver.StompSessionEventType]func(event *stompserver.ConnEvent)
     unsubscribeHandlerFunction stompserver.UnsubscribeHandlerFunction
     applicationRequestHandlerFunction stompserver.ApplicationRequestHandlerFunction
     wg *sync.WaitGroup
@@ -67,10 +68,15 @@ func(s *MockStompServer) OnSubscribeEvent(callback stompserver.SubscribeHandlerF
     s.subscribeHandlerFunction = callback
 }
 
+func (s *MockStompServer) SetConnectionEventCallback(connEventType stompserver.StompSessionEventType, cb func(connEvent *stompserver.ConnEvent)) {
+    s.connectionEventCallbacks[connEventType] = cb
+    cb(&stompserver.ConnEvent{ConnId: "id"})
+}
+
 func newTestFabricEndpoint(bus EventBus, config EndpointConfig) (*fabricEndpoint, *MockStompServer) {
 
     fe := newFabricEndpoint(bus, nil, config).(*fabricEndpoint)
-    ms := &MockStompServer{}
+    ms := &MockStompServer{connectionEventCallbacks: make(map[stompserver.StompSessionEventType]func(event *stompserver.ConnEvent))}
 
     fe.server = ms
     fe.initHandlers()
@@ -111,6 +117,7 @@ func TestFabricEndpoint_StartAndStop(t *testing.T) {
 func TestFabricEndpoint_SubscribeEvent(t *testing.T) {
 
     bus := newTestEventBus()
+    bus.GetChannelManager().CreateChannel(STOMP_SESSION_NOTIFY_CHANNEL) // used for internal channel protection test
     fe, mockServer := newTestFabricEndpoint(bus,
         EndpointConfig{TopicPrefix: "/topic", UserQueuePrefix:"/user/queue"})
 
@@ -167,6 +174,11 @@ func TestFabricEndpoint_SubscribeEvent(t *testing.T) {
     assert.Equal(t, len(fe.chanMappings["test-service"].subs), 3)
     assert.Equal(t, fe.chanMappings["test-service"].subs["con1#sub3"], true)
 
+    // attempt to subscribe to a protected destination
+    mockServer.subscribeHandlerFunction("con1", "sub4", "/topic/" + STOMP_SESSION_NOTIFY_CHANNEL, nil)
+    _, chanMapCreated := fe.chanMappings[STOMP_SESSION_NOTIFY_CHANNEL]
+    assert.False(t, chanMapCreated)
+
     mockServer.wg = &sync.WaitGroup{}
     mockServer.wg.Add(1)
 

plank/.gitignore

@@ -0,0 +1,2 @@
+build/
+cert/

plank/Dockerfile

@@ -0,0 +1,6 @@
+FROM photon:latest
+COPY build/plank /usr/local/bin/
+RUN mkdir -p /opt/plank
+WORKDIR /opt/plank
+STOPSIGNAL SIGTERM
+ENTRYPOINT ["plank", "start-server"]

plank/README.md

@@ -0,0 +1,165 @@
+# Plank
+
+## What is Plank?
+Plank is just enough of a platform to build whatever you want on top. It is a small yet powerful Golang server that can serve
+static contents, single page applications, create and expose microservices over REST endpoints or WebSocket via a built-in
+STOMP broker, or even interact directly with other message brokers such as RabbitMQ. All this is done in a consistent
+and easy to follow manner powered by Transport Event Bus.
+
+Writing a service for a Plank server is in a way similar to writing a Spring Boot `Component` or `Service`, because a lot of tedious
+plumbing work is already done for you such as creating an instance of a service and wiring it up with HTTP endpoints using routers etc.
+Just by following the API you can easily stand up a service, apply any kinds of middleware your application logic calls for, and do all these
+dynamically while in runtime, meaning you can conditionally apply a filter for certain REST endpoints, stand up a new service on demand, or
+even spawn yet another whole new instance of Plank at a different endpoint.
+
+All features are cleanly exposed as public API and modules and, combined with the power of Golang's concurrency model using channels,
+the Transport Event Bus allows creating a clean application architecture, with straightforward and easy to follow logic.
+
+Detailed tutorials and examples are currently in progress and will be made public on the [Transport GitHub page](https://vmware.github.io/transport/).
+Some topics that will initially be covered are:
+
+- Writing a simple service and interacting with it over REST and WebSocket
+- Service lifecycle hooks
+- Middleware management (for REST bridged services)
+- Concept of local bus channel and galactic bus channel
+- Communicating between Plank instances using the built-in STOMP broker
+- Securing your REST and WebSocket endpoints using Auth Provider Manager
+
+## Hello world
+### How to build Plank
+First things first, you'll need the latest Golang version. Plank was first written in Golang 1.13 and it still works with it, but it's advised
+to use the latest Golang (especially 1.16 and forward) because of some nice new packages such as `embed` that we may later employ as part of Plank codebase.
+Once you have the latest Golang ready, follow the commands below:
+
+```bash
+# get all dependencies
+go get ./...
+
+# To build against your operating system
+go run build.go
+
+# or, to build against a specific operating system
+BUILD_OS=darwin|linux|windows go run build.go
+```
+
+Once successfully built, `plank` binary will be ready under `build/`.
+
+> NOTE: we acknowledge there's a lack of build script for Windows Powershell, We'll add it soon!
+
+### Generate a self signed certificate
+Plank can run in non-HTTPS mode but it's generally a good idea to always do development in a similar environment where you'll be serving your
+audience in public internet (or even intranet). Plank repository comes with a handy utility script that can generate a pair of server certificate
+and its matching private key at `scripts/create-selfsigned-cert.sh`. Simply run it in a POSIX shell like below. (requires `openssl` library
+to be available):
+
+```bash
+# generate a pair of x509 certificate and private key files
+./scripts/generate-selfsigned-cert.sh
+
+# cert/fullchain.pem is your certificate and cert/server.key its matching private key
+ls -la cert/
+```
+
+### The real Hello World part
+Now we're ready to start the application. To kick off the server using the demo app you have built above, type the following and you'll see something like this:
+
+```bash
+./build/plank start-server --config-file config.json
+
+ ______ __      ______  __   __  __  __    
+/\  == /\ \    /\  __ \/\ "-.\ \/\ \/ /    
+\ \  _-\ \ \___\ \  __ \ \ \-.  \ \  _"-.  
+ \ \_\  \ \_____\ \_\ \_\ \_\\"\_\ \_\ \_\ 
+  \/_/   \/_____/\/_/\/_/\/_/ \/_/\/_/\/_/ 
+                                           
+Host			localhost
+Port			30080
+Fabric endpoint		/ws
+SPA endpoint		/public
+SPA static assets	/assets
+Health endpoint		/health
+Prometheus endpoint	/prometheus
+...
+time="2021-08-05T21:32:50-07:00" level=info msg="Starting HTTP server at localhost:30080 with TLS" fileName=server.go goroutine=28 package=server
+```
+
+Open your browser and navigate to https://localhost:30080, accept the self-signed certificate warning and you'll be greeted with a 404!
+This is an expected behavior, as the demo app does not serve anything at root `/`, but we will consider changing the default 404 screen to
+something that looks more informational or more appealing at least.
+
+## All supported flags and usages
+
+|Long flag|Short flag|Default value|Required|Description|
+|----|----|----|----|----|
+|--hostname|-n|localhost|false|Hostname where Plank is to be served. Also reads from `$PLANK_SERVER_HOSTNAME` environment variable|
+|--port|-p|30080|false|Port where Plank is to be served. Also reads from `$PLANK_SERVER_PORT` environment variable|
+|--rootdir|-r|<current directory>|false|Root directory for the server. Also reads from `$PLANK_SERVER_ROOTDIR` environment variable|
+|--static|-s|-|false|Path to a location where static files will be served. Can be used multiple times|
+|--no-fabric-broker|-|false|false|Do not start Fabric broker|
+|--fabric-endpoint|-|/fabric|false|Fabric broker endpoint (ignored if --no-fabric-broker is present)|
+|--topic-prefix|-|/topic|false|Topic prefix for Fabric broker (ignored if --no-fabric-broker is present)|
+|--queue-prefix|-|/queue|false|Queue prefix for Fabric broker (ignored if --no-fabric-broker is present)|
+|--request-prefix|-|/pub|false|Application request prefix for Fabric broker (ignored if --no-fabric-broker is present)|
+|--request-queue-prefix|-|/pub/queue|false|Application request queue prefix for Fabric broker (ignored if --no-fabric-broker is present)|
+|--shutdown-timeout|-|5|false|Graceful server shutdown timeout in minutes|
+|--output-log|-l|stdout|false|File to output platform logs to|
+|--access-log|-l|stdout|false|File to output HTTP server access logs to|
+|--error-log|-l|stderr|false|File to output HTTP server error logs to|
+|--debug|-d|false|false|Debug mode|
+|--no-banner|-b|false|false|Do not print Plank banner at startup|
+|--prometheus|-|false|false|Enable Prometheus at /prometheus for metrics|
+|--rest-bridge-timeout|-|1|false|Time in minutes before a REST endpoint for a service request to timeout|
+
+Examples are as follows:
+```bash
+# Start a server with all options set to default values
+./plank start-server
+
+# Start a server with a custom hostname and at port 8080 without Fabric (WebSocket) broker
+./plank start-server --no-fabric-broker --hostname my-app.io --port 8080
+
+# Start a server with a 10 minute graceful shutdown timeout
+# NOTE: this is useful when you run a service that takes a significant amount of time or might even hang while shutting down 
+./plank start-server --shutdown-timeout 10
+
+# Start a server with platform server logs printing to stdout and access/error logs to their respective files
+./plank start-server --access-log server-access-$(date +%m%d%y).log --error-log server-error-$(date +%m%d%y).log
+
+# Start a server with debug outputs enabled
+./plank start-server -d
+
+# Start a server without splash banner
+./plank start-server --no-banner
+
+# Start a server with Prometheus enabled at /prometheus
+./plank start-server --prometheus
+
+# Start a server with static path served at `/static` for folder `static`
+./plank start-server --static static 
+
+# Start a server with static paths served at `/static` for folder `static` and
+# at `/public` for folder `public-contents`
+./plank start-server --static static --static public-contents:/public
+
+# Start a server with a JSON configuration file
+./plank start-server --config-file config.json
+```
+
+## Advanced topics (WIP. Coming soon)
+### OAuth2 Client
+Plank supports seamless out of the box OAuth 2.0 client that support a few OAuth flows. such as authorization
+code grant for web applications and client credentials grant for server-to-server applications.
+See below for a detailed guide for each flow.
+
+#### Authorization Code flow
+You'll choose this authentication flow when the Plank server acts as an intermediary that exchanges
+the authorization code returned from the authorization server for an access token. During this process you will be
+redirected to the identity provider's page like Google where you are asked to confirm the type of 3rd party application and
+its scope of actions it will perform on your behalf and will be taken back to the application after successful authorization.
+
+#### Client Credentials flow
+You'll choose this authentication flow when the Plank server needs to directly communicate with another backend service.
+This will not require user's consent like you would be redirected to Google's page where you confirm the type of application
+requesting your consent for the scope of actions it will perform on your behalf. not requiring any interactions from the user. You will need to
+create an OAuth 2.0 Client with `client_credentials` grant before following the steps below to implement the
+authentication flow.