Workstation

The workstation project is machine learning job management system.

It consists of a task queue where user can create new jobs and one or more worker nodes that will pull jobs from this queue, run the algorithm and return the result when it is finished.

The jobs are submitted via a Docker image that shall be available on a public container registry.

The project is made of three repositories:

• the worker node: https://github.com/42-AI/ws-worker
• the backend server: https://github.com/42-AI/ws-backend
• the frontend: not yet implemented

How to run

This tutorial will guide you through running the whole project.

Pre requisite

• macOS or linux
• Docker installed
• Docker-compose installed
• go installed
• makefile support

Start the backend

Here we will run the entire project on your local machine from scratch, including the database. The database will be boostrapped with default users.

• Clone the backend repository: git clone https://github.com/42-AI/ws-backend
• Open the repo: cd ws-backend
• Create the .env file in the project root folder. For a dev environment running in localhost, use this:

WS_ES_HOST=http://localhost
WS_ES_PORT=9200
WS_KIBANA_PORT=5601
WS_API_HOST=localhost
WS_API_PORT=8080
WS_GRPC_HOST=localhost
WS_GRPC_PORT=8090
IS_DEV_ENV=true
TOKEN_DURATION_HOURS=24
WS_ES_USERNAME=""
WS_ES_PWD=""

• Start the elastic and kibana cluster: make elastic
• Check kibana container logs: docker logs ws-backend_kibana_1 -f
• Wait until you see:

{"type":"log","@timestamp":"2021-03-28T15:11:50+00:00","tags":["listening","info"],"pid":7,"message":"Server running at http://0:5601"}
{"type":"log","@timestamp":"2021-03-28T15:11:51+00:00","tags":["info","http","server","Kibana"],"pid":7,"message":"http server running at http://0:5601"}
{"type":"log","@timestamp":"2021-03-28T15:11:54+00:00","tags":["warning","plugins","reporting"],"pid":7,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}

• Start the GraphQL server make gql FLAG="--bootstrap"
  The bootstrap option initialise the DB by creating the required index and indexing default users
• Open a new terminal in the same repo
• Start the gRPC server: make grpc

At this point you have started the database, the graphQL server that interact with the frontend and the gRPC server that interact with the worker nodes.

Kibana

Before starting the worker node we will learn how to interact with the backend. First, lets check the database:

• Open Kibana: http://localhost:5601
• Click on the burger menu in the top left corner and go to the Dev Tools
• Copy / Paste the following in the console and run it: GET _cat/indices?v
  This list all the index existing in the DB. You should see an index called ws_task and one called ws_user. Index starting with a dot . are system index.
• Now run the following to list all the existing users:

GET ws_user/_search
{
  "query": {
    "match_all": {}
  }
}

• To get all the task, replace in the previous query ws_user by ws_task
• You can use this console for debug purpose if you need to check the content of your database. You could also create or delete task and user manually from here but it is better to use the GraphQL API.

Login

Before being able to create user and task you will need to login. As we started the GraphQL server with bootstrap option, two default users have been created in the DB.
We will login with the admin user using the GraphQL API.

• open the GraphQL playground: http://localhost:8080/playground
• you can find the doc and schema of our API thanks to the "DOCS" and "SCHEMA" tabs on the right side of the screen. This will help you later to build your own request
• copy / paste the following request to log in as the admin user:

query login {
  login (id: "admin-user@email.com"