Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added information about Agones running on Azure Kubernetes Service #269

Merged
merged 4 commits into from
Jun 18, 2018
Merged

added information about Agones running on Azure Kubernetes Service #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7b7c842
added information about Agones running on Azure Kubernetes Service
dgkanatsios Jun 14, 2018
787e2f0
Merge branch 'master' into master
dgkanatsios Jun 14, 2018
73f2a62
Merge branch 'master' into master
dgkanatsios Jun 17, 2018
9e400f6
Merge branch 'master' into master
dgkanatsios Jun 18, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ For more details on why this project was written, read the

## Requirements
- Kubernetes cluster version 1.9+
- [Minikube](https://github.com/kubernetes/minikube) and [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) have been tested
- [Minikube](https://github.com/kubernetes/minikube), [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) and [Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service/) have been tested
- If you are creating and managing your own Kubernetes cluster, the
[MutatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19), and
[ValidatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#validatingadmissionwebhook-alpha-in-18-beta-in-19)
Expand All @@ -35,7 +35,7 @@ For more details on why this project was written, read the

## Installation

Follow [these instructions](install/README.md) to create a cluster on Google Kubernetes Engine (GKE) or Minikube, and install Agones.
Follow [these instructions](install/README.md) to create a cluster on Google Kubernetes Engine (GKE), Minikube or Azure Kubernetes Service (AKS), and install Agones.

## Usage

Expand Down
2 changes: 1 addition & 1 deletion docs/create_fleet.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The following prerequisites are required to create a GameServer:

>NOTE: Agones required Kubernetes versions 1.9+ to run. See the [cluster requirements](../README.md#requirements) for more details.

If you don't have a Kubernetes cluster you can follow [these instructions](../install/README.md) to create a cluster on Google Kubernetes Engine (GKE) or Minikube, and install Agones.
If you don't have a Kubernetes cluster you can follow [these instructions](../install/README.md) to create a cluster on Google Kubernetes Engine (GKE), Minikube or Azure Kubernetes Service (AKS), and install Agones.

For the purpose of this guide we're going to use the [simple-udp](../examples/simple-udp/) example as the GameServer container. This example is very simple UDP server written in Go. Don't hesitate to look at the code of this example for more information.

Expand Down
2 changes: 1 addition & 1 deletion docs/create_gameserver.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The following prerequisites are required to create a GameServer :

>NOTE: Agones required Kubernetes versions 1.9 with role-based access controls (RBAC) and MutatingAdmissionWebhook features activated. To check your version, enter `kubectl version`.

If you don't have a Kubernetes cluster you can follow [these instructions](../install/README.md) to create a cluster on Google Kubernetes Engine (GKE) or Minikube, and install Agones.
If you don't have a Kubernetes cluster you can follow [these instructions](../install/README.md) to create a cluster on Google Kubernetes Engine (GKE), Minikube or Azure Kubernetes Service (AKS), and install Agones.

For the purpose of this guide we're going to use the [simple-udp](../examples/simple-udp/) example as the GameServer container. This example is very simple UDP server written in Go. Don't hesitate to look at the code of this example for more information.

Expand Down
72 changes: 72 additions & 0 deletions install/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ In this quickstart, we will create a Kubernetes cluster, and populate it with th
1. [Installing Minikube](#installing-minikube)
1. [Creating an agones profile](#creating-an-agones-profile)
1. [Starting Minikube](#starting-minikube)
1. [Setting up an Azure Kubernetes Service (AKS) cluster](#setting-up-an-azure-kubernetes-service-aks-cluster)
1. [Choosing your shell](#choosing-your-shell)
1. [Creating the AKS cluster](#creating-the-aks-cluster)
1. [Allowing UDP traffic](#allowing-udp-traffic)
1. [Creating and assigning Public IPs to Nodes](#creating-and-assigning-public-ips-to-nodes)
1. [Enabling creation of RBAC resources](#enabling-creation-of-rbac-resources)
1. [Installing Agones](#installing-agones)
1. [Install with yaml](#install-with-yaml)
Expand Down Expand Up @@ -167,6 +172,73 @@ minikube start --kubernetes-version v1.9.4 --vm-driver virtualbox \
--extra-config=apiserver.Authorization.Mode=RBAC
```

# Setting up an Azure Kubernetes Service (AKS) Cluster

Follow these steps to create a cluster and install Agones directly on [Azure Kubernetes Service (AKS) ](https://docs.microsoft.com/azure/aks/).

## Choosing your shell

You can use either [Azure Cloud Shell](https://docs.microsoft.com/azure/cloud-shell/overview) or install the [Azure CLI](https://docs.microsoft.com/cli/azure/?view=azure-cli-latest) on your local shell in order to install AKS in your own Azure subscription. Cloud Shell comes preinstalled with `az` and `kubectl` utilities whereas you need to install them locally if you want to use your local shell. If you use Windows 10, you can use the [WIndows Subsystem for Windows](https://docs.microsoft.com/windows/wsl/install-win10) as well.

## Creating the AKS cluster

If you are using Azure CLI from your local shell, you need to login to your Azure account by executing the `az login` command and following the login procedure.

Here are the steps you need to follow to create a new AKS cluster (additional instructions and clarifications are listed [here](https://docs.microsoft.com/azure/aks/kubernetes-walkthrough)):

```bash
# Declare necessary variables, modify them according to your needs
AKS_RESOURCE_GROUP=akstestrg # Name of the resource group your AKS cluster will be created in
AKS_NAME=akstest # Name of your AKS cluster
AKS_LOCATION=westeurope # Azure region in which you'll deploy your AKS cluster

# Create the Resource Group where your AKS resource will be installed
az group create --name $AKS_RESOURCE_GROUP --location $AKS_LOCATION

# Create the AKS cluster - this might take some time. Type 'az aks create -h' to see all available options
# The following command will create a single Node AKS cluster. Node size is Standard A1 v1 and Kubernetes version is 1.9.6. Plus, SSH keys will be generated for you, use --ssh-key-value to provide your values
az aks create --resource-group $AKS_RESOURCE_GROUP --name $AKS_NAME --node-count 1 --generate-ssh-keys --node-vm-size Standard_A1_v2 --kubernetes-version 1.9.6 --enable-rbac

# Install kubectl
sudo az aks install-cli

# Get credentials for your new AKS cluster
az aks get-credentials --resource-group $AKS_RESOURCE_GROUP --name $AKS_NAME
```

Alternatively, you can use the [Azure Portal](https://portal.azure.com) to create a new AKS cluster [(instructions)](https://docs.microsoft.com/azure/aks/kubernetes-walkthrough-portal).

### Allowing UDP traffic

For Agones to work correctly, we need to allow UDP traffic to pass through to our AKS cluster. To achieve this, we must update the NSG (Network Security Group) with the proper rule. A simple way to do that is:

* Login to the Azure Portal
* Find the resource group where the AKS resources are kept, which should have a name like `MC_resourceGroupName_AKSName_westeurope`. Alternative, you can type `az resource show --namespace Microsoft.ContainerService --resource-type managedClusters -g $AKS_RESOURCE_GROUP -n $AKS_NAME -o json | jq .properties.nodeResourceGroup`
* Find the Network Security Group object, which should have a name like `aks-agentpool-********-nsg`
* Select **Inbound Security Rules**
* Select **Add** to create a new Rule with **UDP** as the protocol and **7000-8000** as the Destination Port Ranges. Pick a proper name and leave everything else at their default values

Alternatively, you can use the following command, after modifying the `RESOURCE_GROUP_WITH_AKS_RESOURCES` and `NSG_NAME` values:

```bash
az network nsg rule create \
--resource-group RESOURCE_GROUP_WITH_AKS_RESOURCES \
--nsg-name NSG_NAME \
--name AgonesUDP \
--access Allow \
--protocol Udp \
--direction Inbound \
--priority 520 \
--source-port-range "*" \
--destination-port-range 7000-8000
```

### Creating and assigning Public IPs to Nodes

Nodes in AKS don't get a Public IP by default. To assign a Public IP to a Node, find the Resource Group where the AKS resources are installerd on the [portal](https://portal.azure.com) (it should have a name like `MC_resourceGroupName_AKSName_westeurope`). Then, you can follow the instructions [here](https://blogs.technet.microsoft.com/srinathv/2018/02/07/how-to-add-a-public-ip-address-to-azure-vm-for-vm-failed-over-using-asr/) to create a new Public IP and assign it to the Node/VM. For more information on Public IPs for VM NICs, see [this document](https://docs.microsoft.com/azure/virtual-network/virtual-network-network-interface-addresses).

Continue to [Installing Agones](#installing-agones).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For AKS, did you need to do anything for RBAC?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! So, I can confirm that nothing else needs to be set for RBAC apart from the existing instructions/commands, just did a quick test by creating a cluster from scratch with RBAC enabled, installed Agones and quickstarting a game server, having a successful ACK. Apparently, the Agones controllers are working just fine.
Anything else I should check to make sure Agones RBAC stuff is working as intended?

Copy link
Collaborator

@cyriltovena cyriltovena Jun 17, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @dgkanatsios ,
if you created the gameserver in the default namespace and the controller lived in the agones-system you have tested RBAC, if RBAC was active it wouldn't work. Assuming pods of the gameserver was 2/2 active.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also I think @markmandel was wondering if the kubectl config fetched by az aks get-credentials get a cluster-admin role by default, I guess the answer is yes again since you were able to install agones, is this correct ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I vaguely remember some conversation about whether you needed the AKS equivalent of

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin --user `gcloud config get-value account`

Did you end up needing to do that? If not, we should probably also make that clear.

Copy link
Contributor Author

@dgkanatsios dgkanatsios Jun 18, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for your comments @Kuqd and @markmandel! To answer your questions

  • the gameserver is created in the default namespace
dgkanatsios@digkanat:~$ k get po --all-namespaces
NAMESPACE       NAME                                    READY     STATUS    RESTARTS   AGE
agones-system   agones-controller-cfccbcc6b-v9xr2       1/1       Running   0          15h
default         simple-udp-msmr9                        2/2       Running   0          15h
kube-system     azureproxy-899885bfb-5rsql              1/1       Running   2          15h
kube-system     heapster-56c6f9566f-pkbxw               2/2       Running   0          15h
kube-system     kube-dns-v20-7c556f89c5-75qzb           3/3       Running   0          15h
kube-system     kube-dns-v20-7c556f89c5-msfbc           3/3       Running   0          15h
kube-system     kube-proxy-rsjpn                        1/1       Running   0          15h
kube-system     kube-svc-redirect-gtghx                 1/1       Running   0          15h
kube-system     kubernetes-dashboard-5ffc5c5558-tdkwh   1/1       Running   4          15h
kube-system     tunnelfront-856cdb8bb7-khs86            1/1       Running   0          15h
  • Correct, az aks get-credentials fetches admin credentials. You end up with a kubeconfig with that has an embedded cert with an identity of system:masters, which has a cluster-admin ClusterRoleBinding. Agones itself as well as test game server are installed with kubectl using these credentials. Moreover, all default cluster roles are there, including cluster-admin.
  • a clusterrolebinding creation wasn't necessary, this is why I opted to skip the 'configure RBAC section'. Open to rewrite this in a better way, any suggestions?
    Some commands output as well:
dgkanatsios@digkanat:~$ kubectl get clusterrolebindings | grep -v system
NAME                                                   AGE
agones-controller-access                               15h
cluster-admin                                          15h
heapster                                               15h


dgkanatsios@digkanat:~$ k describe clusterrolebinding cluster-admin
Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind   Name            Namespace
  ----   ----            ---------
  Group  system:masters


dgkanatsios@digkanat:~$ k describe clusterrolebinding agones-controller-access
Name:         agones-controller-access
Labels:       app=agones
              chart=agones-0.2.0
              heritage=Tiller
              release=agones-manual
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"labels":{"app"
:"agones","chart":"agones-0.2.0","...
Role:
  Kind:  ClusterRole
  Name:  agones-controller
Subjects:
  Kind  Name                                                   Namespace
  ----  ----                                                   ---------
  User  system:serviceaccount:agones-system:agones-controller


dgkanatsios@digkanat:~$ kubectl get clusterroles | grep -v system
NAME                                                                   AGE
admin                                                                  15h
agones-controller                                                      15h
agones-sdk                                                             15h
cluster-admin                                                          15h
edit                                                                   15h
view                                                                   15h


dgkanatsios@digkanat:~$ kubectl describe clusterrole cluster-admin
Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  *.*        []                 []              [*]
             [*]                []              [*]


dgkanatsios@digkanat:~$ k get serviceaccounts --all-namespaces | grep -v kube-system
NAMESPACE       NAME                                 SECRETS   AGE
agones-system   agones-controller                    1         18h
agones-system   default                              1         18h
azure-system    default                              1         18h
default         agones-sdk                           1         18h
default         default                              1         18h
kube-public     default                              1         18h
  • some gameserver command output
dgkanatsios@digkanat:~$ kubectl apply -f
https://raw.githubusercontent.com/GoogleCloudPlatform/agones/master/examples/simple-udp/server/gameserver.yaml
gameserver.stable.agones.dev "simple-udp" created
dgkanatsios@digkanat:~$ k get gs
NAME         AGE
simple-udp   21s


dgkanatsios@digkanat:~$ k describe gs simple-udp
Name:         simple-udp
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"stable.agones.dev/v1alpha1","kind":"GameServer","metadata":{"annotations":{},"name":"simple-udp","name
space":"default"},"spec":{"contain...
API Version:  stable.agones.dev/v1alpha1
Kind:         GameServer
Metadata:
  Cluster Name:
  Creation Timestamp:  2018-06-18T07:07:17Z
  Finalizers:
    stable.agones.dev
  Generation:        0
  Resource Version:  64414
  Self Link:         /apis/stable.agones.dev/v1alpha1/namespaces/default/gameservers/simple-udp
  UID:               3c38fb5f-72c6-11e8-b304-beff59d82c7f
Spec:
  Container:       simple-udp
  Container Port:  7654
  Health:
    Failure Threshold:      3
    Initial Delay Seconds:  5
    Period Seconds:         5
  Host Port:                7849
  Port Policy:              dynamic
  Protocol:                 UDP
  Template:
    Metadata:
      Creation Timestamp:  <nil>
    Spec:
      Containers:
        Image:  gcr.io/agones-images/udp-server:0.1
        Name:   simple-udp
        Resources:
Status:
  Address:    137.117.171.57
  Node Name:  aks-nodepool1-26745675-0
  Port:       7849
  State:      Ready
Events:
  Type    Reason          Age   From                   Message
  ----    ------          ----  ----                   -------
  Normal  PortAllocation  2m    gameserver-controller  Port allocated
  Normal  Creating        2m    gameserver-controller  Pod simple-udp-qnmqx created
  Normal  Starting        2m    gameserver-controller  Synced
  Normal  Ready           1m    gameserver-controller  Address and Port populated


dgkanatsios@digkanat:~$ nc -u 137.117.171.57 7849
HALO
ACK: HALO
^C

thanks!

# Enabling creation of RBAC resources

To install Agones, a service account needs permission to create some special RBAC resource types.
Expand Down