Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] Inspect frames for commands #3

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions Dockerfile.build
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
FROM golang:1.12 AS builder

ENV CGO_ENABLED=0

WORKDIR /build

COPY go.mod /build/go.mod
RUN go mod download

COPY cmd /build/cmd
COPY pkg /build/pkg
RUN go build -o mc-router cmd/mc-router/main.go

FROM scratch

COPY --from=builder /build/mc-router /bin/mc-router

ENTRYPOINT ["/bin/mc-router"]
2 changes: 1 addition & 1 deletion cmd/mc-router/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"context"
"flag"
"fmt"
"github.com/itzg/mc-router/server"
"github.com/itzg/mc-router/pkg/server"
"github.com/sirupsen/logrus"
"net"
"os"
Expand Down
44 changes: 44 additions & 0 deletions molecule/default/asserts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing the molecule files I realize the motivation for shifting the source code into a pkg directory.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's a really good point, if you're writing tests, updates to the tests may end up triggering rebuilds in the container, even if none of the application logic changes. For the molecule stuff, where I'm building the image in a docker-in-docker fashion (so it's available to the k8s cluster in the container without needing to be pushed to an external registry), COPY . adds significant overhead to the testing loop since it'll basically always result in a container rebuild, and by extension a cleanup and redeploy of the router deployment.

On another note, how do you feel about the molecule stuff? I like it for end-to-end testing in CI environments, and I like using Ansible for basic API interaction with Kubernetes, but I'm also a contributor to molecule and also the maintainer of the k8s/ansible integration so I'm about as biased as possible. I can remove it from the final PR once it's working if you prefer a different testing infrastructure.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't heard of molecule before, but I'm liking the look of it. I was already a fan of Ansible, so anything that plugs into that ecosystem seems good to me.


- name: Verify
hosts: localhost
connection: local
vars:
ansible_python_interpreter: '{{ ansible_playbook_python }}'
deploy_dir: "{{ lookup('env', 'MOLECULE_PROJECT_DIRECTORY') }}/deploy"
tasks:
- block:
- name: Assert that an address has been set on the custom resource
assert:
that: custom_resource.0.status.address is defined
vars:
custom_resource: "{{ q('k8s',
api_version='games.fabianism.us/v1alpha1',
kind='Minecraft',
resource_name='example-minecraft',
namespace=namespace
)}}"

- name: Store address
set_fact:
address: "{{ q('k8s',
api_version='games.fabianism.us/v1alpha1',
kind='Minecraft',
resource_name='example-minecraft',
namespace=namespace
).0.status.address}}"
rescue:
- name: Output CR
debug: var=custom_resource
vars:
custom_resource: "{{ q('k8s',
api_version='games.fabianism.us/v1alpha1',
kind='Minecraft',
resource_name='example-minecraft',
namespace=namespace
)}}"

- fail:
msg: "Failed in asserts.yml"

- import_playbook: '{{ playbook_dir }}/../default/asserts.yml'
54 changes: 54 additions & 0 deletions molecule/default/molecule.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
enabled: False
platforms:
- name: kind-test-local
groups:
- k8s
image: bsycorp/kind:latest-1.12
privileged: True
override_command: no
exposed_ports:
- 8443/tcp
- 10080/tcp
published_ports:
- 0.0.0.0:${TEST_CLUSTER_PORT:-10443}:8443/tcp
pre_build_image: yes
volumes:
- ${MOLECULE_PROJECT_DIRECTORY}:/src/mc-router:Z
provisioner:
name: ansible
log: True
lint:
name: ansible-lint
enabled: False
inventory:
group_vars:
all:
namespace: ${TEST_NAMESPACE:-osdk-test}
env:
K8S_AUTH_KUBECONFIG: /tmp/molecule/kind-test-local/kubeconfig
KUBECONFIG: /tmp/molecule/kind-test-local/kubeconfig
KIND_PORT: '${TEST_CLUSTER_PORT:-10443}'
scenario:
name: default
test_sequence:
- lint
- destroy
- dependency
- syntax
- create
- prepare
- converge
- side_effect
- verify
- destroy
verifier:
name: testinfra
lint:
name: flake8
81 changes: 81 additions & 0 deletions molecule/default/playbook.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---

- name: Build Controller in Kubernetes docker container
hosts: k8s
vars:
image_name: games.fabianism.us/mc-router:testing
tasks:
# using command so we don't need to install any dependencies
- name: Get existing image hash
command: docker images -q {{image_name}}
register: prev_hash
changed_when: false

- name: Build Controller Image
command: docker build -f /src/mc-router/Dockerfile.build -t {{ image_name }} /src/mc-router
register: build_cmd
changed_when: not prev_hash.stdout or (prev_hash.stdout and prev_hash.stdout not in ''.join(build_cmd.stdout_lines[-2:]))

- name: Converge
hosts: localhost
connection: local
vars:
ansible_python_interpreter: '{{ ansible_playbook_python }}'
image_name: games.fabianism.us/mc-router:testing
tasks:
- block:
- name: Delete the Operator Deployment
k8s:
state: absent
namespace: '{{ namespace }}'
name: mc-router
api_version: apps/v1
kind: Deployment
register: delete_deployment
when: hostvars[groups.k8s.0].build_cmd.changed

- name: Wait 30s for Operator Deployment to terminate
k8s_facts:
api_version: 'apps/v1'
kind: 'Deployment'
namespace: '{{ namespace }}'
name: 'mc-router'
register: deployment
until: not deployment.resources
delay: 3
retries: 10
when: delete_deployment.changed

- name: Create the Operator Deployment
k8s:
namespace: '{{ namespace }}'
definition:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: mc-router
name: mc-router
spec:
selector:
matchLabels:
run: mc-router
strategy:
type: Recreate
template:
metadata:
labels:
run: mc-router
spec:
serviceAccountName: mc-router
containers:
- image: '{{ image_name }}'
name: mc-router
args: ["--api-binding", ":8080", "--in-kube-cluster"]
ports:
- name: proxy
containerPort: 25565
- name: web
containerPort: 8080

- import_playbook: verify.yml
93 changes: 93 additions & 0 deletions molecule/default/prepare.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---

- name: Prepare
hosts: k8s
gather_facts: no
vars:
kubeconfig: "{{ lookup('env', 'KUBECONFIG') }}"
tasks:
- name: delete the kubeconfig if present
file:
path: '{{ kubeconfig }}'
state: absent
delegate_to: localhost

- name: Fetch the kubeconfig
fetch:
dest: '{{ kubeconfig }}'
flat: yes
src: /root/.kube/config

- name: Change the kubeconfig port to the proper value
replace:
regexp: 8443
replace: "{{ lookup('env', 'KIND_PORT') }}"
path: '{{ kubeconfig }}'
delegate_to: localhost

- name: Wait for the Kubernetes API to become available (this could take a minute)
uri:
url: "https://localhost:8443/apis"
status_code: 200
validate_certs: no
register: result
until: (result.status|default(-1)) == 200
retries: 60
delay: 5

- name: Prepare operator resources
hosts: localhost
connection: local
vars:
ansible_python_interpreter: '{{ ansible_playbook_python }}'
deploy_dir: "{{ lookup('env', 'MOLECULE_PROJECT_DIRECTORY') }}/deploy"
tasks:
- name: Ensure specified namespace is present
k8s:
api_version: v1
kind: Namespace
name: '{{ namespace }}'

- name: Create RBAC resources and Service
k8s:
namespace: '{{ namespace }}'
definition:
- apiVersion: v1
kind: ServiceAccount
metadata:
name: mc-router
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: services-watcher
rules:
- apiGroups: [""]
resources: ["services"]
verbs: ["watch","list"]
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: mc-router-services-watcher
subjects:
- kind: ServiceAccount
name: mc-router
namespace: '{{ namespace }}'
roleRef:
kind: ClusterRole
name: services-watcher
apiGroup: rbac.authorization.k8s.io
- apiVersion: v1
kind: Service
metadata:
name: mc-router
spec:
type: NodePort
ports:
- targetPort: web
name: web
port: 8080
- targetPort: proxy
name: proxy
port: 25565
selector:
run: mc-router
48 changes: 48 additions & 0 deletions molecule/default/verify.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---

- hosts: localhost
connection: local
gather_facts: no
vars:
ansible_python_interpreter: '{{ ansible_playbook_python }}'
tasks:
- name: Wait for minecraft deployment to report ready
k8s_facts:
api_version: 'apps/v1'
kind: 'Deployment'
namespace: '{{ namespace }}'
label_selectors:
- 'app=example-minecraft'
register: deployment
until: "'MinimumReplicasAvailable' in (deployment | json_query('resources[].status.conditions[].reason'))"
delay: 10
retries: 60

- set_fact:
address: 172.17.0.2:{{ node_port }}
vars:
node_port: "{{ q('k8s',
api_version='v1',
kind='Service',
namespace=namespace,
resource_name='mc-router'
).0.spec.ports.1.nodePort }}"

- debug: var=address

- name: Wait two minutes to confirm server is reachable
command: 'mcstatus {{ address }} ping'
changed_when: false
retries: 24
delay: 5
register: result
until: not (result is failed)

- name: Fetch status of server
command: 'mcstatus {{ address }} status'
changed_when: false
register: mcstatus

- name: Output status of server
debug:
var: mcstatus.stdout_lines
9 changes: 9 additions & 0 deletions mcproto/types.go → pkg/mcproto/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"errors"
"io"
"strings"

"github.com/sirupsen/logrus"
)

type Frame struct {
Expand Down Expand Up @@ -38,6 +40,12 @@ func ReadVarInt(reader io.Reader) (int, error) {
for numRead <= 5 {
n, err := reader.Read(b)
if err != nil {
logrus.WithError(err).WithFields(logrus.Fields{
"b": b,
"numRead": numRead,
"result": result,
"n": n,
}).Infof("")
return 0, err
}
if n == 0 {
Expand Down Expand Up @@ -131,6 +139,7 @@ func ReadPacket(reader io.Reader) (*Packet, error) {

packet.PacketID, err = ReadVarInt(remainder)
if err != nil {
logrus.WithField("remainder", remainder.String()).Info("Failed to find PacketID")
return nil, err
}

Expand Down
File renamed without changes.
Loading