Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ubuntu-latest (20.04.3) in Azure DevOps and latest azure-ml result in .NET Core 2.1 error #383

Open
sigeisler opened this issue Nov 14, 2021 · 4 comments
Open

Ubuntu-latest (20.04.3) in Azure DevOps and latest azure-ml result in .NET Core 2.1 error #383

sigeisler opened this issue Nov 14, 2021 · 4 comments

Comments

@sigeisler
Copy link

Hi,

I followed your templates to setup the CI/CD for Azure ML with Azure DevOps. However, I receive the following error. It seems to be related to the update of "Ubuntu-latest" from 20.04.3 to 20.04.2

Seems to be somehow related to Azure/MachineLearningNotebooks#1003 However, I am just wondering what's the recommended fix since I followed your templates.

Here the logs:

2021-11-13T10:04:08.0321354Z ##[section]Starting: Publish Pipeline
2021-11-13T10:04:08.0329746Z ==============================================================================
2021-11-13T10:04:08.0330086Z Task         : Azure CLI
2021-11-13T10:04:08.0330556Z Description  : Run Azure CLI commands against an Azure subscription in a Shell script when running on Linux agent or Batch script when running on Windows agent.
2021-11-13T10:04:08.0331000Z Version      : 1.164.0
2021-11-13T10:04:08.0331251Z Author       : Microsoft Corporation
2021-11-13T10:04:08.0331625Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-cli
2021-11-13T10:04:08.0332017Z ==============================================================================
2021-11-13T10:04:08.2124426Z [command]/usr/bin/az --version
2021-11-13T10:04:09.1258159Z azure-cli                         2.30.0
2021-11-13T10:04:09.1265234Z 
2021-11-13T10:04:09.1265701Z core                              2.30.0
2021-11-13T10:04:09.1379430Z telemetry                          1.0.6
2021-11-13T10:04:09.1379811Z 
2021-11-13T10:04:09.1380084Z Extensions:
2021-11-13T10:04:09.1380272Z 
2021-11-13T10:04:09.1381190Z azure-devops                      0.21.0
2021-11-13T10:04:09.1381367Z 
2021-11-13T10:04:09.1429224Z Python location '/opt/az/bin/python3'
2021-11-13T10:04:09.1429799Z Please let us know how we are doing: https://aka.ms/azureclihats
2021-11-13T10:04:09.1430591Z and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
2021-11-13T10:04:09.1431241Z Extensions directory '/opt/az/azcliextensions'
2021-11-13T10:04:09.1431440Z 
2021-11-13T10:04:09.1431895Z Python (Linux) 3.6.10 (default, Oct 29 2021, 10:11:34) 
2021-11-13T10:04:09.1432260Z [GCC 9.3.0]
2021-11-13T10:04:09.1432383Z 
2021-11-13T10:04:09.1432706Z Legal docs and information: aka.ms/AzureCliLegal
2021-11-13T10:04:09.1433359Z 
2021-11-13T10:04:09.1433509Z 
2021-11-13T10:04:09.1434045Z Your CLI is up-to-date.
2021-11-13T10:04:09.1436383Z Setting AZURE_CONFIG_DIR env variable to: /home/vsts/work/_temp/.azclitask
2021-11-13T10:04:09.1438056Z Setting active cloud to: AzureCloud
2021-11-13T10:04:09.1446154Z [command]/usr/bin/az cloud set -n AzureCloud
2021-11-13T10:04:11.1164663Z [command]/usr/bin/az login --service-principal -u *** --password=*** --tenant ***
2021-11-13T10:04:13.2033470Z [
2021-11-13T10:04:13.2033789Z   {
2021-11-13T10:04:13.2034084Z     "cloudName": "AzureCloud",
2021-11-13T10:04:13.2035924Z     "homeTenantId": "***",
2021-11-13T10:04:13.2036540Z     "id": "484d48d1-7d2c-4d72-a1e1-e598af02f1da",
2021-11-13T10:04:13.2036880Z     "isDefault": true,
2021-11-13T10:04:13.2037192Z     "managedByTenants": [],
2021-11-13T10:04:13.2037501Z     "name": "StorSimple",
2021-11-13T10:04:13.2037801Z     "state": "Enabled",
2021-11-13T10:04:13.2038727Z     "tenantId": "***",
2021-11-13T10:04:13.2039062Z     "user": {
2021-11-13T10:04:13.2039784Z       "name": "***",
2021-11-13T10:04:13.2040138Z       "type": "servicePrincipal"
2021-11-13T10:04:13.2040419Z     }
2021-11-13T10:04:13.2040697Z   }
2021-11-13T10:04:13.2040935Z ]
2021-11-13T10:04:13.2057544Z [command]/usr/bin/az account set --subscription 484d48d1-7d2c-4d72-a1e1-e598af02f1da
2021-11-13T10:04:13.5171369Z [command]/usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1636797848205.sh
2021-11-13T10:04:15.5707175Z Performing interactive authentication. Please follow the instructions on the terminal.
2021-11-13T10:04:15.5708163Z To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ARCV3S5BK to authenticate.
2021-11-13T10:05:07.0509645Z No Python version provided, defaulting to "3.6.2"
2021-11-13T10:05:11.2373497Z Traceback (most recent call last):
2021-11-13T10:05:11.2375193Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 271, in attempt_get_deps
2021-11-13T10:05:11.2376412Z     blob_deps_to_file()
2021-11-13T10:05:11.2379897Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 263, in blob_deps_to_file
2021-11-13T10:05:11.2381015Z     blob = request.urlopen(deps_url, context=ssl_context)
2021-11-13T10:05:11.2381509Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/urllib/request.py", line 222, in urlopen
2021-11-13T10:05:11.2381977Z     return opener.open(url, data, timeout)
2021-11-13T10:05:11.2382437Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/urllib/request.py", line 531, in open
2021-11-13T10:05:11.2382882Z     response = meth(req, response)
2021-11-13T10:05:11.2383549Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/urllib/request.py", line 640, in http_response
2021-11-13T10:05:11.2384758Z     response = self.parent.error(
2021-11-13T10:05:11.2385222Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/urllib/request.py", line 569, in error
2021-11-13T10:05:11.2385613Z     return self._call_chain(*args)
2021-11-13T10:05:11.2385992Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/urllib/request.py", line 502, in _call_chain
2021-11-13T10:05:11.2386389Z     result = func(*args)
2021-11-13T10:05:11.2386797Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/urllib/request.py", line 649, in http_error_default
2021-11-13T10:05:11.2387234Z     raise HTTPError(req.full_url, code, msg, hdrs, fp)
2021-11-13T10:05:11.2387580Z urllib.error.HTTPError: HTTP Error 404: Not Found
2021-11-13T10:05:11.2387774Z 
2021-11-13T10:05:11.2388098Z During handling of the above exception, another exception occurred:
2021-11-13T10:05:11.2388307Z 
2021-11-13T10:05:11.2388688Z Traceback (most recent call last):
2021-11-13T10:05:11.2389139Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-11-13T10:05:11.2389611Z     return _run_code(code, main_globals, None,
2021-11-13T10:05:11.2390066Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/runpy.py", line 87, in _run_code
2021-11-13T10:05:11.2390489Z     exec(code, run_globals)
2021-11-13T10:05:11.2390906Z   File "/home/vsts/work/1/s/ml_service/pipelines/build_train_pipeline.py", line 163, in <module>
2021-11-13T10:05:11.2391342Z     main(parser.parse_args())
2021-11-13T10:05:11.2392025Z   File "/home/vsts/work/1/s/ml_service/pipelines/build_train_pipeline.py", line 77, in main
2021-11-13T10:05:11.2392397Z     input_dataset = get_dataset(
2021-11-13T10:05:11.2392738Z   File "/home/vsts/work/1/s/ml_service/util/dataset.py", line 41, in get_dataset
2021-11-13T10:05:11.2393125Z     dataset = Dataset.Tabular.from_delimited_files(
2021-11-13T10:05:11.2393936Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/data/_loggerfactory.py", line 129, in wrapper
2021-11-13T10:05:11.2394382Z     return func(*args, **kwargs)
2021-11-13T10:05:11.2395047Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/data/dataset_factory.py", line 352, in from_delimited_files
2021-11-13T10:05:11.2395517Z     dataflow = dataprep().read_csv(path,
2021-11-13T10:05:11.2396186Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/readers.py", line 100, in read_csv
2021-11-13T10:05:11.2396926Z     df = Dataflow._path_to_get_files_block(path, archive_options)
2021-11-13T10:05:11.2397696Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/dataflow.py", line 2426, in _path_to_get_files_block
2021-11-13T10:05:11.2398178Z     return datastore_to_dataflow(path)
2021-11-13T10:05:11.2399059Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/_datastore_helper.py", line 27, in datastore_to_dataflow
2021-11-13T10:05:11.2399640Z     datastore, datastore_value = get_datastore_value(source)
2021-11-13T10:05:11.2400468Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/_datastore_helper.py", line 80, in get_datastore_value
2021-11-13T10:05:11.2400971Z     _set_auth_type(workspace)
2021-11-13T10:05:11.2402050Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/_datastore_helper.py", line 143, in _set_auth_type
2021-11-13T10:05:11.2402795Z     get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(AuthType.DERIVED, json.dumps(auth)))
2021-11-13T10:05:11.2403595Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/api.py", line 19, in get_engine_api
2021-11-13T10:05:11.2404026Z     _engine_api = EngineAPI()
2021-11-13T10:05:11.2404693Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/api.py", line 68, in __init__
2021-11-13T10:05:11.2405156Z     self._message_channel = launch_engine()
2021-11-13T10:05:11.2405848Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/azureml/dataprep/api/engineapi/engine.py", line 333, in launch_engine
2021-11-13T10:05:11.2406335Z     dependencies_path = runtime.ensure_dependencies()
2021-11-13T10:05:11.2406993Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 285, in ensure_dependencies
2021-11-13T10:05:11.2407455Z     if not attempt_get_deps():
2021-11-13T10:05:11.2408109Z   File "/usr/share/miniconda/envs/scikit_env_ci/lib/python3.8/site-packages/dotnetcore2/runtime.py", line 279, in attempt_get_deps
2021-11-13T10:05:11.2409153Z     raise NotImplementedError(err_msg + '\n' + _unsupported_help_msg)
2021-11-13T10:05:11.2409631Z NotImplementedError: Linux distribution ubuntu 20.04 does not have automatic support.
2021-11-13T10:05:11.2410158Z .NET Core 2.1 can still be used via dotnetcore2 if the required dependencies are installed.
2021-11-13T10:05:11.2410919Z Visit https://aka.ms/dotnet-install-linux for Linux distro specific .NET Core install instructions.
2021-11-13T10:05:11.2411914Z Follow your distro specific instructions to install `dotnet-runtime-*` and replace `*` with `2.1`.
2021-11-13T10:05:11.2412145Z 
2021-11-13T10:05:11.2430036Z You have logged in. Now let us find all the subscriptions to which you have access...
2021-11-13T10:05:11.2430610Z Interactive authentication successfully completed.
2021-11-13T10:05:11.2430942Z get_workspace:
2021-11-13T10:05:11.2432138Z Workspace.create(name='xxaml-machinelearning', subscription_id='484d48d1-7d2c-4d72-a1e1-e598af02f1da', resource_group='XX_Machine_Learning')
2021-11-13T10:05:11.2433121Z Found existing compute target cpu-cluster-ds3 so using it.
2021-11-13T10:05:11.2433716Z aml_compute:
2021-11-13T10:05:11.2435323Z AmlCompute(workspace=Workspace.create(name=xxaml-machinelearning', subscription_id='XX', resource_group='XX_Machine_Learning'), name=cpu-cluster-ds3, id=/subscriptions/XX/resourceGroups/XX_Machine_Learning/providers/Microsoft.MachineLearningServices/workspaces/xxaml-machinelearning/computes/cpu-cluster-ds3, type=AmlCompute, provisioning_state=Succeeded, location=westus, tags=None)
2021-11-13T10:05:11.2436299Z Environment(Name: scikit_env,
2021-11-13T10:05:11.2436622Z Version: None)
2021-11-13T10:05:11.3988749Z ##[error]Script failed with error: Error: The process '/usr/bin/bash' failed with exit code 1
2021-11-13T10:05:11.4028907Z [command]/usr/bin/az account clear
2021-11-13T10:05:11.8318676Z ##[section]Finishing: Publish Pipeline

Here the condo environment:

2021-11-13T10:03:48.5432958Z # packages in environment at /usr/share/miniconda/envs/scikit_env_ci:
2021-11-13T10:03:48.5433894Z #
2021-11-13T10:03:48.5434462Z # Name                    Version                   Build  Channel
2021-11-13T10:03:48.5435083Z _libgcc_mutex             0.1                        main  
2021-11-13T10:03:48.5436213Z _openmp_mutex             4.5                       1_gnu  
2021-11-13T10:03:48.5438094Z adal                      1.2.7                    pypi_0    pypi
2021-11-13T10:03:48.5438816Z applicationinsights       0.11.10                  pypi_0    pypi
2021-11-13T10:03:48.5439447Z attrs                     21.2.0                   pypi_0    pypi
2021-11-13T10:03:48.5441001Z azure-common              1.1.27                   pypi_0    pypi
2021-11-13T10:03:48.5441945Z azure-core                1.20.1                   pypi_0    pypi
2021-11-13T10:03:48.5442857Z azure-graphrbac           0.61.1                   pypi_0    pypi
2021-11-13T10:03:48.5443699Z azure-identity            1.4.1                    pypi_0    pypi
2021-11-13T10:03:48.5445701Z azure-mgmt-authorization  0.61.0                   pypi_0    pypi
2021-11-13T10:03:48.5446689Z azure-mgmt-containerregistry 8.2.0                    pypi_0    pypi
2021-11-13T10:03:48.5447585Z azure-mgmt-core           1.3.0                    pypi_0    pypi
2021-11-13T10:03:48.5448452Z azure-mgmt-keyvault       2.2.0                    pypi_0    pypi
2021-11-13T10:03:48.5449321Z azure-mgmt-resource       13.0.0                   pypi_0    pypi
2021-11-13T10:03:48.5450389Z azure-mgmt-storage        11.2.0                   pypi_0    pypi
2021-11-13T10:03:48.5451256Z azureml-automl-core       1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5452006Z azureml-core              1.26.0.post1             pypi_0    pypi
2021-11-13T10:03:48.5452750Z azureml-dataprep          2.13.2                   pypi_0    pypi
2021-11-13T10:03:48.5453908Z azureml-dataprep-native   32.0.0                   pypi_0    pypi
2021-11-13T10:03:48.5454724Z azureml-dataprep-rslex    1.11.2                   pypi_0    pypi
2021-11-13T10:03:48.5455520Z azureml-dataset-runtime   1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5456379Z azureml-pipeline          1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5457424Z azureml-pipeline-core     1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5458246Z azureml-pipeline-steps    1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5459108Z azureml-sdk               1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5460088Z azureml-telemetry         1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5461136Z azureml-train             1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5462084Z azureml-train-automl-client 1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5462910Z azureml-train-core        1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5463781Z azureml-train-restclients-hyperdrive 1.26.0                   pypi_0    pypi
2021-11-13T10:03:48.5464701Z backports-tempfile        1.0                      pypi_0    pypi
2021-11-13T10:03:48.5465526Z backports-weakref         1.0.post1                pypi_0    pypi
2021-11-13T10:03:48.5469243Z blas                      1.0                         mkl  
2021-11-13T10:03:48.5469668Z brotli                    1.0.9                he6710b0_2  
2021-11-13T10:03:48.5470598Z ca-certificates           2021.10.26           h06a4308_2  
2021-11-13T10:03:48.5470994Z certifi                   2021.10.8        py38h06a4308_0  
2021-11-13T10:03:48.5471362Z cffi                      1.15.0                   pypi_0    pypi
2021-11-13T10:03:48.5471857Z chardet                   3.0.4                    pypi_0    pypi
2021-11-13T10:03:48.5472256Z cloudpickle               1.6.0                    pypi_0    pypi
2021-11-13T10:03:48.5472933Z contextlib2               21.6.0                   pypi_0    pypi
2021-11-13T10:03:48.5473338Z coverage                  6.1.2                    pypi_0    pypi
2021-11-13T10:03:48.5473724Z cryptography              3.4.8                    pypi_0    pypi
2021-11-13T10:03:48.5474114Z cycler                    0.10.0                   py38_0  
2021-11-13T10:03:48.5474501Z dbus                      1.13.18              hb2f20db_0  
2021-11-13T10:03:48.5474895Z distro                    1.6.0                    pypi_0    pypi
2021-11-13T10:03:48.5475280Z docker                    5.0.3                    pypi_0    pypi
2021-11-13T10:03:48.5475916Z dotnetcore2               2.1.21                   pypi_0    pypi
2021-11-13T10:03:48.5476461Z entrypoints               0.3                      pypi_0    pypi
2021-11-13T10:03:48.5476881Z expat                     2.4.1                h2531618_2  
2021-11-13T10:03:48.5477285Z flake8                    3.7.9                    pypi_0    pypi
2021-11-13T10:03:48.5477984Z flake8-formatter-junit-xml 0.0.6                    pypi_0    pypi
2021-11-13T10:03:48.5478425Z fontconfig                2.13.1               h6c09931_0  
2021-11-13T10:03:48.5478839Z fonttools                 4.25.0             pyhd3eb1b0_0  
2021-11-13T10:03:48.5479236Z freetype                  2.11.0               h70c0345_0  
2021-11-13T10:03:48.5479657Z fusepy                    3.0.1                    pypi_0    pypi
2021-11-13T10:03:48.5480078Z giflib                    5.2.1                h7b6447c_0  
2021-11-13T10:03:48.5480488Z glib                      2.69.1               h5202010_0  
2021-11-13T10:03:48.5481090Z gst-plugins-base          1.14.0               h8213a91_2  
2021-11-13T10:03:48.5481520Z gstreamer                 1.14.0               h28cd5cc_2  
2021-11-13T10:03:48.5481933Z icu                       58.2                 he6710b0_3  
2021-11-13T10:03:48.5482565Z idna                      2.10                     pypi_0    pypi
2021-11-13T10:03:48.5483678Z inference-schema          1.1.0                    pypi_0    pypi
2021-11-13T10:03:48.5484542Z intel-openmp              2021.4.0          h06a4308_3561  
2021-11-13T10:03:48.5484974Z isodate                   0.6.0                    pypi_0    pypi
2021-11-13T10:03:48.5487294Z jeepney                   0.7.1                    pypi_0    pypi
2021-11-13T10:03:48.5487736Z jmespath                  0.10.0                   pypi_0    pypi
2021-11-13T10:03:48.5488167Z joblib                    0.17.0                   pypi_0    pypi
2021-11-13T10:03:48.5488590Z jpeg                      9d                   h7f8727e_0  
2021-11-13T10:03:48.5489025Z jsonpickle                2.0.0                    pypi_0    pypi
2021-11-13T10:03:48.5489749Z junit-xml                 1.9                      pypi_0    pypi
2021-11-13T10:03:48.5490194Z kiwisolver                1.3.1            py38h2531618_0  
2021-11-13T10:03:48.5490609Z lcms2                     2.12                 h3be6417_0  
2021-11-13T10:03:48.5491336Z ld_impl_linux-64          2.35.1               h7274673_9  
2021-11-13T10:03:48.5491744Z libffi                    3.3                  he6710b0_2  
2021-11-13T10:03:48.5492429Z libgcc-ng                 9.3.0               h5101ec6_17  
2021-11-13T10:03:48.5493965Z libgfortran-ng            7.5.0               ha8ba4b0_17  
2021-11-13T10:03:48.5494402Z libgfortran4              7.5.0               ha8ba4b0_17  
2021-11-13T10:03:48.5494772Z libgomp                   9.3.0               h5101ec6_17  
2021-11-13T10:03:48.5495157Z libpng                    1.6.37               hbc83047_0  
2021-11-13T10:03:48.5495853Z libstdcxx-ng              9.3.0               hd4cf53a_17  
2021-11-13T10:03:48.5496267Z libtiff                   4.2.0                h85742a9_0  
2021-11-13T10:03:48.5496770Z libuuid                   1.0.3                h7f8727e_2  
2021-11-13T10:03:48.5497183Z libwebp                   1.2.0                h89dd481_0  
2021-11-13T10:03:48.5497778Z libwebp-base              1.2.0                h27cfd23_0  
2021-11-13T10:03:48.5498455Z libxcb                    1.14                 h7b6447c_0  
2021-11-13T10:03:48.5498855Z libxml2                   2.9.12               h03d6c58_0  
2021-11-13T10:03:48.5499275Z llvmlite                  0.37.0                   pypi_0    pypi
2021-11-13T10:03:48.5499929Z lz4-c                     1.9.3                h295c915_1  
2021-11-13T10:03:48.5500360Z matplotlib                3.4.3            py38h06a4308_0  
2021-11-13T10:03:48.5500939Z matplotlib-base           3.4.3            py38hbbc1b5f_0  
2021-11-13T10:03:48.5501377Z mccabe                    0.6.1                    pypi_0    pypi
2021-11-13T10:03:48.5502083Z mkl                       2021.4.0           h06a4308_640  
2021-11-13T10:03:48.5502794Z mkl-service               2.4.0            py38h7f8727e_0  
2021-11-13T10:03:48.5503224Z mkl_fft                   1.3.1            py38hd3c417c_0  
2021-11-13T10:03:48.5503634Z mkl_random                1.2.2            py38h51133e4_0  
2021-11-13T10:03:48.5504237Z more-itertools            8.11.0                   pypi_0    pypi
2021-11-13T10:03:48.5504675Z msal                      1.16.0                   pypi_0    pypi
2021-11-13T10:03:48.5505295Z msal-extensions           0.2.2                    pypi_0    pypi
2021-11-13T10:03:48.5505845Z msrest                    0.6.21                   pypi_0    pypi
2021-11-13T10:03:48.5506262Z msrestazure               0.6.4                    pypi_0    pypi
2021-11-13T10:03:48.5506787Z munkres                   1.1.4                      py_0  
2021-11-13T10:03:48.5507166Z ncurses                   6.3                  h7f8727e_2  
2021-11-13T10:03:48.5507766Z ndg-httpsclient           0.5.1                    pypi_0    pypi
2021-11-13T10:03:48.5508213Z numba                     0.54.1                   pypi_0    pypi
2021-11-13T10:03:48.5508644Z numpy                     1.20.3                   pypi_0    pypi
2021-11-13T10:03:48.5509057Z oauthlib                  3.1.1                    pypi_0    pypi
2021-11-13T10:03:48.5509475Z olefile                   0.46               pyhd3eb1b0_0  
2021-11-13T10:03:48.5509894Z openssl                   1.1.1l               h7f8727e_0  
2021-11-13T10:03:48.5510299Z packaging                 21.2                     pypi_0    pypi
2021-11-13T10:03:48.5510720Z pandas                    1.1.5            py38ha9443f7_0  
2021-11-13T10:03:48.5511138Z pathspec                  0.9.0                    pypi_0    pypi
2021-11-13T10:03:48.5511559Z pcre                      8.45                 h295c915_0  
2021-11-13T10:03:48.5511952Z pillow                    8.4.0            py38h5aabda8_0  
2021-11-13T10:03:48.5512373Z pip                       20.3.3           py38h06a4308_0  
2021-11-13T10:03:48.5512791Z pluggy                    0.13.1                   pypi_0    pypi
2021-11-13T10:03:48.5513219Z portalocker               1.7.1                    pypi_0    pypi
2021-11-13T10:03:48.5513632Z py                        1.11.0                   pypi_0    pypi
2021-11-13T10:03:48.5514181Z pyarrow                   1.0.1                    pypi_0    pypi
2021-11-13T10:03:48.5514713Z pyasn1                    0.4.8                    pypi_0    pypi
2021-11-13T10:03:48.5515137Z pycodestyle               2.5.0                    pypi_0    pypi
2021-11-13T10:03:48.5515565Z pycparser                 2.21                     pypi_0    pypi
2021-11-13T10:03:48.5515975Z pyflakes                  2.1.1                    pypi_0    pypi
2021-11-13T10:03:48.5516400Z pyjwt                     2.3.0                    pypi_0    pypi
2021-11-13T10:03:48.5516823Z pyopenssl                 20.0.1                   pypi_0    pypi
2021-11-13T10:03:48.5517255Z pyparsing                 2.4.7                    pypi_0    pypi
2021-11-13T10:03:48.5517657Z pyqt                      5.9.2            py38h05f1152_4  
2021-11-13T10:03:48.5518076Z pytest                    5.4.3                    pypi_0    pypi
2021-11-13T10:03:48.5518708Z pytest-cov                2.8.1                    pypi_0    pypi
2021-11-13T10:03:48.5519144Z python                    3.8.12               h12debd9_0  
2021-11-13T10:03:48.5519898Z python-dateutil           2.8.2              pyhd3eb1b0_0  
2021-11-13T10:03:48.5520525Z python-dotenv             0.12.0                   pypi_0    pypi
2021-11-13T10:03:48.5520960Z pytz                      2021.3             pyhd3eb1b0_0  
2021-11-13T10:03:48.5521379Z pyyaml                    5.3                      pypi_0    pypi
2021-11-13T10:03:48.5521782Z qt                        5.9.7                h5867ecd_1  
2021-11-13T10:03:48.5522191Z readline                  8.1                  h27cfd23_0  
2021-11-13T10:03:48.5522749Z requests                  2.23.0                   pypi_0    pypi
2021-11-13T10:03:48.5523400Z requests-oauthlib         1.3.0                    pypi_0    pypi
2021-11-13T10:03:48.5524008Z ruamel-yaml               0.17.4                   pypi_0    pypi
2021-11-13T10:03:48.5524625Z ruamel-yaml-clib          0.2.6                    pypi_0    pypi
2021-11-13T10:03:48.5525353Z scikit-learn              0.23.2           py38h0573a6f_0  
2021-11-13T10:03:48.5525773Z scipy                     1.7.1            py38h292c36d_2  
2021-11-13T10:03:48.5526166Z secretstorage             3.3.1                    pypi_0    pypi
2021-11-13T10:03:48.5526697Z setuptools                58.0.4           py38h06a4308_0  
2021-11-13T10:03:48.5527117Z shap                      0.37.0                   pypi_0    pypi
2021-11-13T10:03:48.5527533Z sip                       4.19.13          py38he6710b0_0  
2021-11-13T10:03:48.5527929Z six                       1.16.0             pyhd3eb1b0_0  
2021-11-13T10:03:48.5528355Z slicer                    0.0.3                    pypi_0    pypi
2021-11-13T10:03:48.5528773Z sqlite                    3.36.0               hc218d9a_0  
2021-11-13T10:03:48.5529180Z threadpoolctl             2.2.0              pyh0d69192_0  
2021-11-13T10:03:48.5529701Z tk                        8.6.11               h1ccaba5_0  
2021-11-13T10:03:48.5530098Z tornado                   6.1              py38h27cfd23_0  
2021-11-13T10:03:48.5530509Z tqdm                      4.62.3                   pypi_0    pypi
2021-11-13T10:03:48.5530922Z urllib3                   1.25.11                  pypi_0    pypi
2021-11-13T10:03:48.5531321Z wcwidth                   0.2.5                    pypi_0    pypi
2021-11-13T10:03:48.5531923Z websocket-client          1.2.1                    pypi_0    pypi
2021-11-13T10:03:48.5532346Z wheel                     0.37.0             pyhd3eb1b0_1  
2021-11-13T10:03:48.5532997Z wrapt                     1.11.1                   pypi_0    pypi
2021-11-13T10:03:48.5533401Z xz                        5.2.5                h7b6447c_0  
2021-11-13T10:03:48.5533785Z zlib                      1.2.11               h7b6447c_3  
2021-11-13T10:03:48.5534286Z zstd                      1.4.9                haebb681_0  
2021-11-13T10:03:48.9625266Z Package                              Version
2021-11-13T10:03:48.9626464Z ------------------------------------ ------------
2021-11-13T10:03:48.9626879Z adal                                 1.2.7
2021-11-13T10:03:48.9627228Z applicationinsights                  0.11.10
2021-11-13T10:03:48.9627556Z attrs                                21.2.0
2021-11-13T10:03:48.9628093Z azure-common                         1.1.27
2021-11-13T10:03:48.9628615Z azure-core                           1.20.1
2021-11-13T10:03:48.9629220Z azure-graphrbac                      0.61.1
2021-11-13T10:03:48.9629676Z azure-identity                       1.4.1
2021-11-13T10:03:48.9630149Z azure-mgmt-authorization             0.61.0
2021-11-13T10:03:48.9630634Z azure-mgmt-containerregistry         8.2.0
2021-11-13T10:03:48.9631100Z azure-mgmt-core                      1.3.0
2021-11-13T10:03:48.9631550Z azure-mgmt-keyvault                  2.2.0
2021-11-13T10:03:48.9632154Z azure-mgmt-resource                  13.0.0
2021-11-13T10:03:48.9632782Z azure-mgmt-storage                   11.2.0
2021-11-13T10:03:48.9633432Z azureml-automl-core                  1.26.0
2021-11-13T10:03:48.9634384Z azureml-core                         1.26.0.post1
2021-11-13T10:03:48.9634951Z azureml-dataprep                     2.13.2
2021-11-13T10:03:48.9635490Z azureml-dataprep-native              32.0.0
2021-11-13T10:03:48.9636378Z azureml-dataprep-rslex               1.11.2
2021-11-13T10:03:48.9636835Z azureml-dataset-runtime              1.26.0
2021-11-13T10:03:48.9637306Z azureml-pipeline                     1.26.0
2021-11-13T10:03:48.9637778Z azureml-pipeline-core                1.26.0
2021-11-13T10:03:48.9638232Z azureml-pipeline-steps               1.26.0
2021-11-13T10:03:48.9638923Z azureml-sdk                          1.26.0
2021-11-13T10:03:48.9639400Z azureml-telemetry                    1.26.0
2021-11-13T10:03:48.9639994Z azureml-train                        1.26.0
2021-11-13T10:03:48.9640478Z azureml-train-automl-client          1.26.0
2021-11-13T10:03:48.9640978Z azureml-train-core                   1.26.0
2021-11-13T10:03:48.9641481Z azureml-train-restclients-hyperdrive 1.26.0
2021-11-13T10:03:48.9641842Z backports.tempfile                   1.0
2021-11-13T10:03:48.9642172Z backports.weakref                    1.0.post1
2021-11-13T10:03:48.9642645Z certifi                              2021.10.8
2021-11-13T10:03:48.9643126Z cffi                                 1.15.0
2021-11-13T10:03:48.9643486Z chardet                              3.0.4
2021-11-13T10:03:48.9643829Z cloudpickle                          1.6.0
2021-11-13T10:03:48.9644187Z contextlib2                          21.6.0
2021-11-13T10:03:48.9644545Z coverage                             6.1.2
2021-11-13T10:03:48.9644894Z cryptography                         3.4.8
2021-11-13T10:03:48.9645255Z cycler                               0.10.0
2021-11-13T10:03:48.9645614Z distro                               1.6.0
2021-11-13T10:03:48.9646087Z docker                               5.0.3
2021-11-13T10:03:48.9646533Z dotnetcore2                          2.1.21
2021-11-13T10:03:48.9646870Z entrypoints                          0.3
2021-11-13T10:03:48.9647207Z flake8                               3.7.9
2021-11-13T10:03:48.9647716Z flake8-formatter-junit-xml           0.0.6
2021-11-13T10:03:48.9648054Z fonttools                            4.25.0
2021-11-13T10:03:48.9648392Z fusepy                               3.0.1
2021-11-13T10:03:48.9648729Z idna                                 2.10
2021-11-13T10:03:48.9649218Z inference-schema                     1.1.0
2021-11-13T10:03:48.9649551Z isodate                              0.6.0
2021-11-13T10:03:48.9649888Z jeepney                              0.7.1
2021-11-13T10:03:48.9650226Z jmespath                             0.10.0
2021-11-13T10:03:48.9650571Z joblib                               0.17.0
2021-11-13T10:03:48.9650891Z jsonpickle                           2.0.0
2021-11-13T10:03:48.9651378Z junit-xml                            1.9
2021-11-13T10:03:48.9651727Z kiwisolver                           1.3.1
2021-11-13T10:03:48.9652050Z llvmlite                             0.37.0
2021-11-13T10:03:48.9652386Z matplotlib                           3.4.3
2021-11-13T10:03:48.9653264Z mccabe                               0.6.1
2021-11-13T10:03:48.9653874Z mkl-fft                              1.3.1
2021-11-13T10:03:48.9654397Z mkl-random                           1.2.2
2021-11-13T10:03:48.9655041Z mkl-service                          2.4.0
2021-11-13T10:03:48.9655560Z more-itertools                       8.11.0
2021-11-13T10:03:48.9656046Z msal                                 1.16.0
2021-11-13T10:03:48.9656658Z msal-extensions                      0.2.2
2021-11-13T10:03:48.9656994Z msrest                               0.6.21
2021-11-13T10:03:48.9657322Z msrestazure                          0.6.4
2021-11-13T10:03:48.9657638Z munkres                              1.1.4
2021-11-13T10:03:48.9658086Z ndg-httpsclient                      0.5.1
2021-11-13T10:03:48.9658417Z numba                                0.54.1
2021-11-13T10:03:48.9658736Z numpy                                1.20.3
2021-11-13T10:03:48.9659035Z oauthlib                             3.1.1
2021-11-13T10:03:48.9659525Z olefile                              0.46
2021-11-13T10:03:48.9659843Z packaging                            21.2
2021-11-13T10:03:48.9660158Z pandas                               1.1.5
2021-11-13T10:03:48.9660457Z pathspec                             0.9.0
2021-11-13T10:03:48.9660773Z Pillow                               8.4.0
2021-11-13T10:03:48.9661089Z pip                                  20.3.3
2021-11-13T10:03:48.9661408Z pluggy                               0.13.1
2021-11-13T10:03:48.9661707Z portalocker                          1.7.1
2021-11-13T10:03:48.9662265Z py                                   1.11.0
2021-11-13T10:03:48.9662722Z pyarrow                              1.0.1
2021-11-13T10:03:48.9663193Z pyasn1                               0.4.8
2021-11-13T10:03:48.9663535Z pycodestyle                          2.5.0
2021-11-13T10:03:48.9663893Z pycparser                            2.21
2021-11-13T10:03:48.9664251Z pyflakes                             2.1.1
2021-11-13T10:03:48.9664613Z PyJWT                                2.3.0
2021-11-13T10:03:48.9665078Z pyOpenSSL                            20.0.1
2021-11-13T10:03:48.9665421Z pyparsing                            2.4.7
2021-11-13T10:03:48.9665764Z pytest                               5.4.3
2021-11-13T10:03:48.9666511Z pytest-cov                           2.8.1
2021-11-13T10:03:48.9666988Z python-dateutil                      2.8.2
2021-11-13T10:03:48.9667457Z python-dotenv                        0.12.0
2021-11-13T10:03:48.9667786Z pytz                                 2021.3
2021-11-13T10:03:48.9668094Z PyYAML                               5.3
2021-11-13T10:03:48.9668410Z requests                             2.23.0
2021-11-13T10:03:48.9668871Z requests-oauthlib                    1.3.0
2021-11-13T10:03:48.9669199Z ruamel.yaml                          0.17.4
2021-11-13T10:03:48.9669500Z ruamel.yaml.clib                     0.2.6
2021-11-13T10:03:48.9669960Z scikit-learn                         0.23.2
2021-11-13T10:03:48.9670295Z scipy                                1.7.1
2021-11-13T10:03:48.9670609Z SecretStorage                        3.3.1
2021-11-13T10:03:48.9670910Z setuptools                           58.0.4
2021-11-13T10:03:48.9671226Z shap                                 0.37.0
2021-11-13T10:03:48.9671545Z sip                                  4.19.13
2021-11-13T10:03:48.9671984Z six                                  1.16.0
2021-11-13T10:03:48.9672306Z slicer                               0.0.3
2021-11-13T10:03:48.9672761Z threadpoolctl                        2.2.0
2021-11-13T10:03:48.9673234Z tornado                              6.1
2021-11-13T10:03:48.9673575Z tqdm                                 4.62.3
2021-11-13T10:03:48.9673938Z urllib3                              1.25.11
2021-11-13T10:03:48.9674299Z wcwidth                              0.2.5
2021-11-13T10:03:48.9674834Z websocket-client                     1.2.1
2021-11-13T10:03:48.9675197Z wheel                                0.37.0
2021-11-13T10:03:48.9675681Z wrapt                                1.11.1
2021-11-13T10:03:49.1234031Z ##[section]Finishing: Create Anaconda environment
@lokijota
Copy link
Contributor

lokijota commented Nov 15, 2021 •
edited
Loading

What's the pipeline, could you link to it?

PS: the shared link does seem to possibly be the same issue. Have you tried the solution there w/ dotnet core 2.1?

@sigeisler
Copy link
Author

sigeisler commented Nov 15, 2021 •
edited
Loading

Hmm unfortunately I am
Not able to share the pipeline :/

I also tried to install it via

  • script: |
    source activate scikit_env_ci
    from dotnetcore2 import runtime
    bin_folder = runtime._get_bin_folder()
    missing_packages = runtime._gather_dependencies(bin_folder)
    print(missing_packages)
    displayName: Debug DotNet 2.1

or what is the recommended way?

@lokijota
Copy link
Contributor

I can try to reproduce it, but need some more info. Could you just give me more info on where you are on the process? I assume you've done the bootstrapping, is that correct? what's the name of the yml file where the error is happening?

@sigeisler
Copy link
Author

It fails in this code: https://github.com/microsoft/MLOpsPython/blob/master/ml_service/pipelines/diabetes_regression_build_train_pipeline.py

As far as I understand this should be related to the azure ml dataset API. So probably https://github.com/microsoft/MLOpsPython/blob/master/ml_service/pipelines/diabetes_regression_build_train_pipeline.py#L77

The yaml is https://github.com/microsoft/MLOpsPython/blob/master/.pipelines/diabetes_regression-ci.yml

Perhaps it suffices to run the CI pipeline of this repo (if the issue has not been resolved by now): https://aidemos.visualstudio.com/MLOps/_build/results?buildId=5721&view=results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lokijota @sigeisler