Talend Docker Example

This repository includes sample bash scripts and utilities which allow you to deploy Talend jobs to Docker containers.

The job2docker approach converts a single Talend job zip file to a container. The resulting Docker image will have a single entry point for the job. It is intended for use by developers during their build / test / debug cycle and provides desktop parity.

• Job2Docker Workflow
• Prerequisites
• Setup
• Environment
• Directory Index
• Getting Started

Job2Docker Workflow

1. A Talend job2docker_listener job is used to monitor a shared directory.
2. The developer clicks Build in Talend Studio to create Talend job zip file in the shared directory.
3. The Talend job2docker_listener triggers the job2docker script to convert the Talend zip file to a tgz ready for Docker.
4. The Talend job2docker_listener triggers the job2docker_build script.
5. The Talend job2docker_listener optionally publishes the resulting container to a Docker Registry.

Job2docker can be incorporated into a CI build environment, but it is out of scope for this cookbook. When run as part of the CI build script, job2docker will presumably run on a CI server local to Nexus and the SCM. Running job2docker in such a CI server will create a Docker image and supporting artifacts closer to the Docker registry so there will be less network overhead than transferring from a laptop.

Prerequisites

• Docker
• Git
• Oracle JRE
• Talend Studio
• A shared folder accessible from both Studio and docker machines.

Setup

A setup script is provided for linux.
The script will clone the job2docker directory and working directories under a target directory which defaults to ${HOME}/talend.

    bash <(curl https://raw.githubusercontent.com/Talend/job2docker/master/job2docker-setup)

Jobs built by Studio are dropped into a shared directory that defaults to ${HOME}/shared_jobs. This can be overridden by passing a parameter.

    bash <(curl https://raw.githubusercontent.com/Talend/job2docker/master/job2docker-setup) ${HOME}/my_shared_jobs

The files can be installed under an alternate directory by passing a second parameter to the script. Note that the shared job directory parameter must also be supplied as the first argument.

    bash <(curl https://raw.githubusercontent.com/Talend/job2docker/master/job2docker-setup) ${HOME}/shared_jobs ${HOME}/mytalend

Launch the job2docker_listener from the target directory.

    ${HOME}/talend/job2docker_listener

Manual install instructions are available if you prefer.

Environment

• All containerization work is done on Linux with Docker installed.

• Talend Studio steps can run on a separate machine if desired, it could be a Windows machine.

• A common drop point (shared directory, shared network drive, shared folder) for Jobs built from Studio to be processed by job2docker.

• The environment used to test these scripts was a Windows laptop running Talend Studio.

• The docker scripts were run on an Ubuntu 16.04.2 LTS VM running kernel 4.4.0-97-generic.

• It was also tested with a Centos 7 VM running kernel 3.10.0-862.6.3.el7.x86_64.

• VirtualBox was used for the VM hosting.

• A shared folder was created using VirtualBox so that Studio builds would be visible to the Linux VM.

Directory Index

• job2docker/job2docker-setup - install script
• job2docker/bin - scripts for creating docker images, creating containers, and deploying images to the cloud
• job2docker/docs - additional readme files
• job2docker/job2docker_build - sample Dockerfile used to create Docker image containing the Talend job
• job2docker/util - utility bash scripts
• job2docker/jobs - sample jobs
• j2d - created during setup, working directory for running the agent that monitors the build directory for Talend zip files

Getting Started

• Job2Docker HelloWorld
• Job2Docker HelloWorld Config Files