Skip to content

Commit

Permalink
add hadoop-yarn chart
Browse files Browse the repository at this point in the history
Signed-off-by: zwzhang0107 <zuoweizhang@outlook.com>
  • Loading branch information
zwzhang0107 authored and eahydra committed Nov 15, 2023
1 parent 592197e commit 48851c3
Show file tree
Hide file tree
Showing 11 changed files with 850 additions and 0 deletions.
1 change: 1 addition & 0 deletions charts/hadoop-yarn
6 changes: 6 additions & 0 deletions hadoop-yarn/3.3.2-v1.0/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: v2
name: hadoop-yarn
description: A Helm chart for hadoop yarn.
type: application
version: 3.3.2
appVersion: "3.3.2"
19 changes: 19 additions & 0 deletions hadoop-yarn/3.3.2-v1.0/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
0. Check running state of yarn pods
kubectl get pod -n {{ .Values.installation.namespace }}

1. You can list the yarn nodes by running this command:
kubectl exec -n {{ .Values.installation.namespace }} -it ${yarn-rm-pod-name} -- /opt/hadoop/bin/yarn node -list

2. Create a port-forward to the yarn resource manager UI:
kubectl port-forward -n {{ .Values.installation.namespace }} service/{{ .Values.yarn.resourceManager.serviceName }} {{ .Values.yarn.resourceManager.webPort}}:{{.Values.yarn.resourceManager.webPort}}

Then open the ui in your browser:

open http://localhost:{{.Values.yarn.resourceManager.webPort}}

3. You can run included hadoop tests like this:
kubectl exec -n {{ .Values.installation.namespace }} -it ${yarn-rm-pod-name} yarn-rm -- /opt/hadoop/bin/hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-{{ .Values.hadoopVersion }}-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB -resFile /tmp/TestDFSIOwrite.txt
kubectl exec -n {{ .Values.installation.namespace }} -it ${yarn-rm-pod-name} yarn-rm -- /opt/spark/bin/spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi /opt/spark/examples/jars/spark-examples_2.12-3.3.3.jar 1000

4. You can list the mapreduce jobs like this:
kubectl exec -n {{ .Values.installation.namespace }} -it ${yarn-rm-pod-name} -- /opt/hadoop/bin/mapred job -list
51 changes: 51 additions & 0 deletions hadoop-yarn/3.3.2-v1.0/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "hadoop-yarn.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "hadoop-yarn.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "hadoop-yarn.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "hadoop-yarn.labels" -}}
helm.sh/chart: {{ include "hadoop-yarn.chart" . }}
{{ include "hadoop-yarn.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "hadoop-yarn.selectorLabels" -}}
app.kubernetes.io/name: {{ include "hadoop-yarn.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
Loading

0 comments on commit 48851c3

Please sign in to comment.