unable to reach metadata_service after using docker compose

[ferras]

Description of steps by @aleade

I cloned the most recent git repo
I did docker pull netflixoss/metaflow_metadata_service
I changed the docker-compose file only to change the image and the volumes path (last line)
version: "3"
services:
metadata:
  image: "netflixoss/metaflow_metadata_service"
  container_name: "metadata_service"
  ports:
    - "${MF_METADATA_PORT:-5004}:${MF_METADATA_PORT:-5004}"
  volumes:
    - ./metadata_service:/code
I did docker-compose up -d
I did docker exec -it metadata_service /bin/bash
I did curl localhost:8080/ping

[ferras]

the latest image hasn't been deployed to netflixoss/metaflow_metadata_service on dockerhub so you would need to set the ports to 8080
either via the env variable MF_METADATA_PORT or by changing the default in the docker-compose file to 8080.

We will attempt to push the latest build to dockerhub by the end of the week.

you can also build the latest image locally via the following command:
docker build -t metadata_service .

[aleade]

Hi,

I changed my docker-compose file into:

version: "3"
services:
  metadata:
    image: "metadata_service"
    # image: "netflixoss/metaflow_metadata_service"
    container_name: "metadata_service"
    ports:
      - "${MF_METADATA_PORT:-8080}:${MF_METADATA_PORT:-8080}"
    volumes:
      - ./metadata_service:/code
    environment:
      - MF_METADATA_DB_HOST=db
      - MF_METADATA_DB_PORT=5432
      - MF_METADATA_DB_USER=postgres
      - MF_METADATA_DB_PSWD=postgres
      - MF_METADATA_DB_NAME=postgres
      - MF_METADATA_PORT=${MF_METADATA_PORT:-8080}
      - MF_METADATA_HOST=${MF_METADATA_HOST:-0.0.0.0}
    links:
      - db
  db:
    image: "postgres:11"
    restart: always
    container_name: "my_postgres"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres
    ports:
      - "5432:5432"
    volumes:
      - my_dbdata:/var/lib/postgresql/data2
volumes:
  my_dbdata:

I built the image using docker build -f Dockerfile ./ -t metadata_service`

Bu I am still unable to use metadata('http://0.0.0.0:8080/metadata_service') or metadata('http://172.17.0.1:8080/metadata_service') (after curling accordingly)

This is the error that I get:

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
~/software/anaconda2/envs/metaflow/lib/python3.7/site-packages/metaflow/metadata/service.py in compute_info(cls, val)
     30             resp = requests.get(os.path.join(v, 'ping'), headers=METADATA_SERVICE_HEADERS)
---> 31             resp.raise_for_status()
     32         except:  # noqa E722

~/software/anaconda2/envs/metaflow/lib/python3.7/site-packages/requests/models.py in raise_for_status(self)
    939         if http_error_msg:
--> 940             raise HTTPError(http_error_msg, response=self)
    941 

HTTPError: 404 Client Error: Not Found for url: http://0.0.0.0:8080/metadata_service/ping

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-10-8df3d891fbd2> in <module>
----> 1 metadata('http://0.0.0.0:8080/metadata_service')

~/software/anaconda2/envs/metaflow/lib/python3.7/site-packages/metaflow/client/core.py in metadata(ms)
     79             return get_metadata()
     80         current_metadata = res[0]
---> 81         current_metadata.INFO = ms
     82     return get_metadata()
     83 

~/software/anaconda2/envs/metaflow/lib/python3.7/site-packages/metaflow/metadata/metadata.py in _set_info(classobject, val)
     26 
     27     def _set_info(classobject, val):
---> 28         v = classobject.compute_info(val)
     29         classobject._INFO = v
     30 

~/software/anaconda2/envs/metaflow/lib/python3.7/site-packages/metaflow/metadata/service.py in compute_info(cls, val)
     31             resp.raise_for_status()
     32         except:  # noqa E722
---> 33             raise ValueError('Metaflow service [%s] unreachable.' % v)
     34         return v
     35 

ValueError: Metaflow service [http://0.0.0.0:8080/metadata_service] unreachable.

While if I do metadata('http://0.0.0.0:8080/') in my notebook I get: service@http://0.0.0.0:8080 (no error), but that doesn't seem to work in my flow: ERROR:root:Authorisation response: 404

[aleade]

This is what I get from a docker ps:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                               NAMES
4c7eaad12163        metadata_service    "/bin/sh -c 'python3…"   16 minutes ago      Up 7 seconds        0.0.0.0:8080->8080/tcp              metadata_service
ee1b7c0420bd        postgres:11         "docker-entrypoint.s…"   16 minutes ago      Up 8 seconds        0.0.0.0:5432->5432/tcp              my_postgres

or from a docker-compose ps

      Name                    Command               State           Ports         
----------------------------------------------------------------------------------
metadata_service   /bin/sh -c python3 -m meta ...   Up      0.0.0.0:8080->8080/tcp
my_postgres        docker-entrypoint.sh postgres    Up      0.0.0.0:5432->5432/tcp

[ferras]

@aleade sorry for the delayed response

I am not sure why you are trying to set the endpoint path to http://0.0.0.0:8080/metadata_service. metadata_service is not a valid endpoint.

in terms of getting it to run on your notebook. Where are you running your notebook? Instead of testing on your notebook can you try running the python repl in your terminal and testing to see if you can use the metaflow client? If your notebook is running in a separate environment then routing to localhost might not work.

[ferras]

@aleade I am assuming the issue is resolved. Please reopen if this isn't the case

[n3o-Bhushan]

@ferras I believe they are trying to hit http://0.0.0.0:8080/metadata_service endpoint cause something similar is mentioned here https://docs.metaflow.org/metaflow/client#metadata-provider

# Configure Client to use remote metadata provider globally
metadata('https://localhost:5000/mymetaflowservice')

I have tried it as well and I am facing similar issue.

Steps I have followed:

I cloned the most recent git repo
I did docker pull netflixoss/metaflow_metadata_service
I changed the docker-compose file only to change the image
I did docker-compose up -d
I did docker exec -it metadata_service /bin/bash
I did curl localhost:8080/ping

it all worked but when I try to change metadata store to service from local while using the metaflow client, I am getting the same error.

ValueError: Metaflow service [https://localhost:8080] unreachable.

I tested the API endpoints which return valid responses. But I am confused what to pass as an arg to metadata() function

[n3o-Bhushan]

I was able to resolve this issue. I was trying to hit https instead of http. So nevermind :D