CLI v2.62.0 breaks confcom extension

[Nautigsam]

Describe the bug

When using confcom extension, I get this error:

The docker process was not found. Please start Docker.

This message hides the real error raised by docker package:

Not supported URL scheme http+docker

It seems to be related to this issue. I suspect there is a conflict in the versions of docker or requests.

Related command

az confcom acipolicygen --debug --template-file "***.json"

Errors

The docker process was not found. Please start Docker.

Issue script & Debug output

cli.knack.cli: Command arguments: ['confcom', 'acipolicygen', '--debug', '--template-file', '***.json']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7d3fbbff2d40>, <function OutputProducer.on_global_arguments at 0x7d3fbbda1f80>, <function CLIQuery.on_global_arguments at 0x7d3fbbddf560>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'confcom': ['azext_confcom']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: confcom                   0.001         1         2  /home/***/.azure/cliextensions/confcom
cli.azure.cli.core: Total (1)                 0.001         1         2
cli.azure.cli.core: Loaded 1 groups, 2 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : confcom acipolicygen
cli.azure.cli.core: Command table: confcom acipolicygen
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7d3fbaf53ec0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/***/.azure/commands/2024-07-16.15-13-42.confcom_acipolicygen.334190.log'.
az_command_data_logger: command args: confcom acipolicygen --debug --template-file {}
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7d3fbaf99d00>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7d3fbafc4040>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7d3fbafc4180>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7d3fbbda2020>, <function CLIQuery.handle_query_parameter at 0x7d3fbbddf600>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7d3fbafc40e0>]
az_command_data_logger: extension name: confcom
az_command_data_logger: extension version: 0.3.5
docker.utils.config: Trying paths: ['/home/***/.docker/config.json', '/home/***/.dockercfg']
docker.utils.config: Found file at path: /home/***/.docker/config.json
docker.auth: Found 'auths' section
docker.auth: Found an IdentityToken entry for registry ***
docker.auth: Found entry (registry='***', username='***')
cli.azext_confcom.custom: The docker process was not found. Please start Docker.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7d3fbaf54180>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 0.244 seconds (init: 0.092, invoke: 0.153)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3691 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/azure-cli/bin/python /opt/azure-cli/lib/python3.12/site-packages/azure/cli/telemetry/__init__.py /home/***/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

The command should execute successfully as it did in version 2.61.0.

Environment Summary

azure-cli 2.62.0

core 2.62.0
telemetry 1.1.0

Extensions:
confcom 0.3.5

Dependencies:
msal 1.28.1
azure-mgmt-resource 23.1.1

Python location '/opt/azure-cli/bin/python'
Extensions directory '/home/***/.azure/cliextensions'

Python (Linux) 3.12.4 (main, Jun 7 2024, 06:33:07) [GCC 14.1.1 20240522]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

I use Arch and I installed the azure-cli package.

[yonzhan]

Thank you for opening this issue, we will look into it.

[anrdesai]

Any updates on this? Our customers are hitting this as well. We have multiple reports of this. FYI @hprabh @anantshankar17

[SethHollandsworth]

Hi, sorry for the inconvenience. You're right @Nautigsam, there is an incompatibility with the docker SDK and the requests package that ships with az cli version 2.62.0. A PR to fix this was merged with #29391 and will be in the next release. For the time being, downgrading the az cli to 2.61.0 will work

[cYKatherine]

Hello @SethHollandsworth , after downgrading to 2.61.0 I seem to still have the issue. Also the . dockercfg file seems not have been found:

+ az confcom acipolicygen --debug --approve-wildcards -a /home/runner/work/uid2-operator/uid2-operator/deployment-artifacts/operator.json --print-policy
DEBUG: cli.knack.cli: Command arguments: ['confcom', 'acipolicygen', '--debug', '--approve-wildcards', '-a', '/home/runner/work/uid2-operator/uid2-operator/deployment-artifacts/operator.json', '--print-policy']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fc73fc3f100>, <function OutputProducer.on_global_arguments at 0x7fc73fb32840>, <function CLIQuery.on_global_arguments at 0x7fc73f8d0360>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Command index version or cloud profile is invalid or doesn't match the current command.
DEBUG: cli.azure.cli.core: Command index has been invalidated.
DEBUG: cli.azure.cli.core: No module found from index for '['confcom', 'acipolicygen', '--debug', '--approve-wildcards', '-a', '/home/runner/work/uid2-operator/uid2-operator/deployment-artifacts/operator.json', '--print-policy']'
DEBUG: cli.azure.cli.core: Loading all modules and extensions
DEBUG: cli.azure.cli.core: Discovered command modules: ['acr', 'acs', 'advisor', 'ams', 'apim', 'appconfig', 'appservice', 'aro', 'backup', 'batch', 'batchai', 'billing', 'botservice', 'cdn', 'cloud', 'cognitiveservices', 'compute_recommender', 'config', 'configure', 'consumption', 'container', 'containerapp', 'cosmosdb', 'databoxedge', 'dla', 'dls', 'dms', 'eventgrid', 'eventhubs', 'extension', 'feedback', 'find', 'hdinsight', 'identity', 'interactive', 'iot', 'keyvault', 'kusto', 'lab', 'managedservices', 'maps', 'marketplaceordering', 'monitor', 'mysql', 'netappfiles', 'network', 'policyinsights', 'privatedns', 'profile', 'rdbms', 'redis', 'relay', 'resource', 'role', 'search', 'security', 'servicebus', 'serviceconnector', 'servicefabric', 'signalr', 'sql', 'sqlvm', 'storage', 'synapse', 'util', 'vm']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
DEBUG: cli.azure.cli.core: acr                       0.106        36       149
DEBUG: cli.azure.cli.core: acs                       0.014        14        76
DEBUG: cli.azure.cli.core: advisor                   0.001         3         6
DEBUG: cli.azure.cli.core: ams                       0.004        22       100
DEBUG: cli.azure.cli.core: apim                      0.004        14        69
DEBUG: cli.azure.cli.core: appconfig                 0.002         9        47
DEBUG: cli.azure.cli.core: appservice                0.037        79       270
DEBUG: cli.azure.cli.core: aro                       0.007         1        10
DEBUG: cli.azure.cli.core: backup                    0.003        16        60
DEBUG: cli.azure.cli.core: batch                     0.019        34       102
DEBUG: cli.azure.cli.core: batchai                   0.002        10        30
DEBUG: cli.azure.cli.core: billing                   0.007        19        52
DEBUG: cli.azure.cli.core: botservice                0.002        12        42
DEBUG: cli.azure.cli.core: cdn                       0.082         8        49
DEBUG: cli.azure.cli.core: cloud                     0.001         1         7
DEBUG: cli.azure.cli.core: cognitiveservices         0.002        10        33
DEBUG: cli.azure.cli.core: compute_recommender       0.002         1         1
DEBUG: cli.azure.cli.core: config                    0.001         2         7
DEBUG: cli.azure.cli.core: configure                 0.002         2         5
DEBUG: cli.azure.cli.core: consumption               0.010         8         9
DEBUG: cli.azure.cli.core: container                 0.006         1        11
DEBUG: cli.azure.cli.core: containerapp              0.103        36       115
DEBUG: cli.azure.cli.core: cosmosdb                  0.009        58       199
DEBUG: cli.azure.cli.core: databoxedge               0.005         5        28
DEBUG: cli.azure.cli.core: dla                       0.002        23        62
DEBUG: cli.azure.cli.core: dls                       0.002         7        41
DEBUG: cli.azure.cli.core: dms                       0.001         3        22
DEBUG: cli.azure.cli.core: eventgrid                 0.003        25        96
DEBUG: cli.azure.cli.core: eventhubs                 0.007        13        19
DEBUG: cli.azure.cli.core: extension                 0.001         1         7
DEBUG: cli.azure.cli.core: feedback                  0.000         1         2
DEBUG: cli.azure.cli.core: find                      0.001         1         1
DEBUG: cli.azure.cli.core: hdinsight                 0.005         8        39
DEBUG: cli.azure.cli.core: identity                  0.001         2        11
DEBUG: cli.azure.cli.core: interactive               0.000         1         1
DEBUG: cli.azure.cli.core: iot                       0.055        19        82
DEBUG: cli.azure.cli.core: keyvault                  0.006        20       113
DEBUG: cli.azure.cli.core: kusto                     0.001         3        14
DEBUG: cli.azure.cli.core: lab                       0.002        11        34
DEBUG: cli.azure.cli.core: managedservices           0.001         3         8
DEBUG: cli.azure.cli.core: maps                      0.001         5        13
DEBUG: cli.azure.cli.core: marketplaceordering       0.002         1         2
DEBUG: cli.azure.cli.core: monitor                   0.327        18        60
DEBUG: cli.azure.cli.core: mysql                     0.069        15        51
DEBUG: cli.azure.cli.core: netappfiles               0.025         8        17
DEBUG: cli.azure.cli.core: network                   0.151       103       337
DEBUG: cli.azure.cli.core: policyinsights            0.007         9        17
DEBUG: cli.azure.cli.core: privatedns                0.012        14        60
DEBUG: cli.azure.cli.core: profile                   0.001         2         8
DEBUG: cli.azure.cli.core: rdbms                     0.014        49       202
DEBUG: cli.azure.cli.core: redis                     0.002         7        38
DEBUG: cli.azure.cli.core: relay                     0.013         7         8
DEBUG: cli.azure.cli.core: resource                  0.009        51       231
DEBUG: cli.azure.cli.core: role                      0.002        17        61
DEBUG: cli.azure.cli.core: search                    0.005         7        19
DEBUG: cli.azure.cli.core: security                  0.006        47        98
DEBUG: cli.azure.cli.core: servicebus                0.005        12        15
DEBUG: cli.azure.cli.core: serviceconnector          0.027        20       307
DEBUG: cli.azure.cli.core: servicefabric             0.010        27        80
DEBUG: cli.azure.cli.core: signalr                   0.002         9        34
DEBUG: cli.azure.cli.core: sql                       0.010        56       215
DEBUG: cli.azure.cli.core: sqlvm                     0.017         4        20
DEBUG: cli.azure.cli.core: storage                   0.034        59       273
DEBUG: cli.azure.cli.core: synapse                   0.009        54       246
DEBUG: cli.azure.cli.core: util                      0.001         3         7
DEBUG: cli.azure.cli.core: vm                        0.036        58       233
DEBUG: cli.azure.cli.core: Total (66)                1.317      1204      4681
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
DEBUG: cli.azure.cli.core: confcom                   0.001         1         2  /opt/az/azcliextensions/confcom
DEBUG: cli.azure.cli.core: Total (1)                 0.001         1         2  
DEBUG: cli.azure.cli.core: Loaded 1191 groups, 4683 commands.
DEBUG: cli.azure.cli.core: Updated command index in 0.003 seconds.
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fc73dee0ea0>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-07-25.01-21-33.confcom_acipolicygen.49.log'.
INFO: az_command_data_logger: command args: confcom acipolicygen --debug --approve-wildcards -a {} --print-policy
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7fc73df26340>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7fc73df35580>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7fc73df356c0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [<function _documentdb_deprecate at 0x7fc73b47[144](https://github.com/IABTechLab/uid2-operator/actions/runs/10086443167/job/27888999705#step:24:145)0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fc73fb328e0>, <function CLIQuery.handle_query_parameter at 0x7fc73f8d0400>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7fc73df35620>]
INFO: az_command_data_logger: extension name: confcom
INFO: az_command_data_logger: extension version: 0.3.5
DEBUG: docker.utils.config: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
DEBUG: docker.utils.config: No config file found
DEBUG: docker.utils.config: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
DEBUG: docker.utils.config: No config file found
WARNING: cli.azext_confcom.custom: The docker process was not found. Please start Docker.
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fc73dee1120>]
INFO: az_command_data_logger: exit code: 1
INFO: cli.__main__: Command ran in 1.805 seconds (init: 0.252, invoke: 1.553)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3654 in cache
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/usr/local/bin/python /usr/local/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /root/.azure"
INFO: telemetry.process: Return from creating process
INFO: telemetry.main: Finish creating telemetry upload process.

Do you have any insight what might have gone wrong?

[SethHollandsworth]

Hey @cYKatherine! for .dockercfg not being found, I've seen a couple different possibilities: the file doesn't exist because it's not needed (not authenticating to a registry, etc.) or the owner of the file is root and we don't have access

For the Docker process not found issue, I'd double check that the docker service is started with systemctl | grep running. If that looks fine, the involved way to debug would be:

start interactive python

python3

open a client and see if it errors out

import docker
client = docker.from_env()
client.images.list()

the version of docker-py that comes with confcom v0.3.5 is 7.1.0, not sure if that's important. If all that checks out, lmk and we can dig further

[cYKatherine]

Hello @SethHollandsworth , thank you so much.

Hi, sorry for the inconvenience. You're right @Nautigsam, there is an incompatibility with the docker SDK and the requests package that ships with az cli version 2.62.0. A PR to fix this was merged with #29391 and will be in the next release. For the time being, downgrading the az cli to 2.61.0 will work

I ran az confcom acipolicygen --debug --approve-wildcards -a xxx.json --print-policy in a github workflow. Originally it worked with the latest version in a simple run script, but because of the bug mentioned above, I have to downgrade to 2.61.0.

To downgrade it to 2.61.0, I have to use the azure/cli@v2 action to specify the 2.61.0 version. However, it seems that this action doesn't allow calling docker cli: Azure/cli#152, Azure/cli#156

I'm just wondering if my understanding is correct. If so do you have any insight how to bypass this issue? Or can I expect an estimate on the new release that fixes the confcom extension bug?

Thank you so much!

[SethHollandsworth]

Hi @cYKatherine, I didn't realize it was in a Github action. To get around needing docker, you could use the --tar option in confcom which takes in a path to a tar file to hash images locally. But then there would need to be a way to put the tar file into the action's filesystem. In my opinion, this would be more pain than it's worth.

The AZ CLI looks like it makes a release ~monthly, so it'll hopefully only be a week or two until the next version with the requests package patched. Hope this helps, best of luck!

[cYKatherine]

Thank you so much! @SethHollandsworth

[shiswara]

@SethHollandsworth / @yonzhan : Do we have an update on the release date for 2.63?

[SethHollandsworth]

Hi all @cYKatherine @shiswara @Nautigsam @anrdesai , v2.63.0 was released a few days ago and I confirmed it works with confcom!
To upgrade please use: az upgrade

https://github.com/Azure/azure-cli/releases/tag/azure-cli-2.63.0