Create kubeflow pipeline without Kubeflow UI

[ajinkya933]

I have Kubeflow installed on my local Ubuntu machine To install it I have used microk8 (https://ubuntu.com/tutorials/deploy-kubeflow-ubuntu-windows-mac#1-overview) .

I have following jupyter notebook

https://github.com/ajinkya933/streamlit-docker/blob/master/Simple%20Notebook%20Pipeline.ipynb

I am running this notebook from jupyter notebook via terminal. It gives me error

 HTTPConnectionPool(host='pipelines-api.kubeflow.svc.cluster.local', port=8888): Max retries exceeded with url: /apis/v1beta1/experiments (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa579f18610>: Failed to establish a new connection: [Errno -2] Name or service not known'))

I am not using kubeflow UI to run this notebook. How can I NOT use Kubeflow UI but still run this pipeline mentioned in my ipynb. ?

My work requires creating Kubeflow pipelines directly through terminal without using kubeflow UI. Is this possible ?

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/question	0.72
area/jupyter	0.51

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[jbottum]

@james-jwu Hey James, is there someone that might answer this question

[james-jwu]

/assign @Bobgy Yuan might know the answer

[Bobgy]

Hi @ajinkya933, you can follow guide here: https://www.kubeflow.org/docs/gke/pipelines/authentication-sdk/#how-sdk-connects-to-kubeflow-pipelines-api.

KFP sdk needs an endpoint it can use to access the api server, you can port-forward that endpoint to your localhost.

[ajinkya933]

@Bobgy In my ipynb Cell-10 I have (https://github.com/ajinkya933/streamlit-docker/blob/master/Simple%20Notebook%20Pipeline.ipynb)

host='pipelines-api.kubeflow.svc.cluster.local:8888'

Which throws error when I run the pipeline outside Kubeflow UI ( I run it through .py file on my ubuntu terminal).
I tried with multiple combinations:

host='http://localhost:3000'
host='http://10.64.140.43.xip.io:8888'

these donot work. The link https://www.kubeflow.org/docs/gke/pipelines/authentication-sdk/#how-sdk-connects-to-kubeflow-pipelines-api mentions steps to do it on Google cloud. How can I do the same on local ubuntu based installation of Kubeflow.

Please elaborate more on how to port forward and start pipeline on Ubuntu machine . As in what changes do I make to ipynb to make it work ?

[Bobgy]

You need to run

kubectl port-forward svc/ml-pipeline-ui 3000

first.

You can verify whether it works by opening localhost:3000 in your browser, it should show KFP UI

[ajinkya933]

I was able to solve it, thanks to @Bobgy for the pointers. Here is the solution https://ajinkya93.atlassian.net/l/c/008a1Moc
However I use a single node to deploy this pipeline. @Bobgy If you have any pointers on how to use Distributed computing and execute Kubeflow Pipeline that would be very helpful.

[Bobgy]

@ajinkya933 would you mind opening a new issue to make each issue focused?