Azure IOT Edge build module images task fails to build after being automatically updated to 2.0.0

[tristanbarcelon]

• iotedgedev Version: 2.0.0
• Python Version: 2.7
• Pip Version: 8.1.1 or 19.1.1
• Development machine OS Version: Hosted Ubuntu 1604 in Azure Devops
• IoT Edge device OS Version:

Steps to Reproduce:

1. create a build definition in azure devops and enable continuous integration on master or any branch you need to build
2. add an Azure Iot Edge task with action "Build module images", default platform of amd64 and deployment.template.json as template json file

I get an error

ERROR: No module.json file found. module.json file is required in the root of your module folder Error: The process '/usr/local/bin/iotedgedev' failed with exit code 1

with versions 2.2.0 and 2.0.0 of Azure IoT Edge task and iotedgedev, respectively.

When iotedgedev and Azure IoT Edge task versions were 1.1.0 and 2.0.7 (as of 11 days ago), builds were successful.

[blackchoey]

@tristanbarcelon Thanks for reporting this issue. Could you share the file structure of your solution? This could help us troubleshot the issue you met.

[tristanbarcelon]

@blackchoey , sure thing. Here's how the repo root looks like. I have replaced actual module names with placeholders module_A, module_B, etc.

.vscode (folder)
json_module_snippets (folder)
modules (folder)
    |-- <module_A> (folder)
    |         |-- module.json
    |         |-- Dockerfile.amd64
    |         |-- module_A.csproj
    |         |-- several .cs files
    |-- <module_B> (folder)
              |-- module.json
              |-- Dockerfile.amd64
              |-- module_B.csproj
              |-- several .cs files
.gitignore
deployment.template.json

Iotedgedev is called as follows according to the log:

sr/local/bin/iotedgedev build --file /home/vsts/work/1/s/deployment.template.json --platform amd64

[blackchoey]

@tristanbarcelon thank you very much, I will continue investigating this and keep you updated

[blackchoey]

@tristanbarcelon I created a solution with above folder structure but didn't repro this issue.
There might be some special cases in the solution structure that we didn't take into consideration. I just created a private repo and invited you as collaborator: https://github.com/blackchoey/IoTEdgeTestSolution. Could you help update the project structure so we can repro the issue? You can use fake names in it and I will delete it after investigation.

[tristanbarcelon]

@blackchoey , would it be possible to set up a teams session with you to show you our Azure Devops build log and repo? I think this might be easier to do than trying to repro it. The particular solution has at least 11 modules of varying complexity and it might be too difficult try and anonymize.

[blackchoey]

@tristanbarcelon It would be great if we can chat on teams. Are you a MS employee? If yes please reach me by searching my alias chyuan so we can start talking. Otherwise I will share a meeting link with you.

[tristanbarcelon]

@blackchoey , I searched for you in teams by switching to the Microsoft guest organization but could not find your alias. Could you share a meeting link to tristan_barcelon@jabil.com please?

[jackt-moran]

I am getting the same error message with versions 2.2.0 and 2.0.0 of Azure IoT Edge task and iotedgedev, respectively. My project structure is similar to @tristanbarcelon 's.

[tristanbarcelon]

@jackt-moran , i was provided a workaround until they release a new version of iotedgedev or fix the iotedge task. In your build definition, introduce a new pipeline variable called: IOTEDGEDEV_VERSION and set its value to 1.1.0. Requeue a build and see if it fixes your issue. Mine works after that.

[jackt-moran]

@tristanbarcelon thank you for the advice. After adding that variable, my build now fails with the following error message: "ERROR: Error: 'NoneType' object has no attribute 'lower'" and exits with exit code 1. Are you still using version 2.2.0 of AzureIoTEdge?

[tristanbarcelon]

@jackt-moran Yeah, we are still using version 2.2.0 of the iotedge task. Previous to the update, we were using 2.0.7 of the task. At that time, the task was still in preview mode. This error probably has more to do with code than the task. Could you search for the string NoneType in all files in the repo to narrow down the source of the error? It might also help for you to set debug = true when you queue a build, similar to what I'm showing below.

You can look at the step in your build which fails and see where the error comes from. If it's from dockerfile lines that build your code, then the issue is definitely with code.

[jackt-moran]

@tristanbarcelon thanks again for the helpful response.

Searching through this repository I found that the "language" attribute is a required attribute of a module's module.json file. Some of our modules are just wrappers around public images, such as redis, so I had left "language" out of those module.json files. I now assign a value of "" to those modules' "language" attribute and the task runs successfully.

It may be worth considering making the "language" attribute optional. For reference, here is the line that throws what appears to be an AttributeError, which is not caught by any of the except statements:

iotedgedev/iotedgedev/module.py

Line 24 in b7742a1

self.module_language = self.file_json_content.get("language").lower()

[tristanbarcelon]

Hi @jackt-moran , we have some containers like that used in testing scenarios and the folders for such modules do not contain a module.json file. They are present in deployment.template.json and do build along with the other modules. I believe this is what broke version 2.0.0 of iotedgedev in addition to module folder name. The new version of iotedgedev expected every module folder to contain module.json and when it didn't, it threw that error you were seeing.

[tristanbarcelon]

@blackchoey , I followed this manual dev machine setup link you shared for setting up on-premise build agents that can build iot edge projects. Specifically:

• ran only steps 1, 2, 3, 4, 7, and 8 from the dev machine setup doc
• I installed moby instead of docker to match the runtime requirements for iotedge. It doesn't seem to make sense to use docker for building/development and then require moby for runtime. Could the documentation for manual dev machine setup be updated to use moby instead and align it with the runtime requirements for iotedge at least for linux?
• I used pip3 to install docker-compose and iotedgedev since step 2 indicates a preference for Python 3.6
• i got a coredump error when running step 8 as is. I needed to prefix it with sudo. I'm not sure why.

With this build agent, I've had mixed success so far. For a new iot edge solution that only has 2 modules it works. If I try it on the solution that triggers the ticket, the workaround which relies on IOTEDGEDEV_VERSION pipeline variable being set to 1.1.0 does not seem to work. Task still uses iotedgedev version 2.0.0. Is there something else that the task expects to allow it to swap out the version of iotedgedev version to 1.1.0 at runtime?

I've moved the folder with dots out of the modules subfolder and I still have issues so more than likely, iotedgedev 2.0.0 expects a module.json files to be present for every module subfolder underneath modules and that is the bug.

[blackchoey]

Hi @tristanbarcelon @jackt-moran Sorry for the late response, we're working on a fix to the errors you met and will release the changes soon. In the future if a project under modules folder doesn't contain a module.json file, the tool will ignore that project.

[blackchoey]

@jackt-moran The modules.json is designed for IoT Edge modules. If one project is not IoT Edge module, you can remove the modules.json from the project.

[blackchoey]

@tristanbarcelon Regarding the issue that you couldn't set iotedgedev version via the IOTEDGEDEV_VERSION variable. We didn't have other requirement to the the iotedgedev version. Could you please:

1. Create a simple bash task to echo the value of IOTEDGEDEV_VERSION to ensure the variable is passed to the pipline
2. Check the debug log of Azure IoT Edge task to see the version read by the task? The message looks like The specified iotedgedev version is: xxx

And for your question about moby and docker, Docker CE/EE container images are compatible with the Moby runtime (see document). So both moby and docker CE/EE can be used to build the images. We will update the document according to your feedback.

[blackchoey]

@tristanbarcelon @jackt-moran I have released the new version of iotedgedev, you should be able to build solutions with non-edge project with Linux Azure Pipelines agents now. The Windows Azure Pipelines agents need a few more time to apply the new version.

[tristanbarcelon]

Hi @blackchoey ,
I followed your suggestion and echoed the IOTEDGEDEV_VERSION pipeline variable and it prints 1.1.0

Starting: Bash Script
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.151.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
Script contents:
echo $IOTEDGEDEV_VERSION
========================== Starting Command Output ===========================
in/bash --noprofile --norc /srv/agent/work/_temp/9de6d97c-98f2-4d8b-bc88-095d23432868.sh
1.1.0
Finishing: Bash Script

Unfortunately, the Azure Iotedge task for building modules does not seem to honor the value of this variable but instead just utilizes the existing version 2.0.0 of iotedgedev installed via pip3 from step 8. Should I uninstall iotedgedev pip module instead and rely on the task to download it?

==============================================================================
Task         : Azure IoT Edge
Description  : Build and deploy an Azure IoT Edge image
Version      : 2.2.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
==============================================================================
Building module images...
/srv/agent/work/21/s /srv/agent/work/21/s
iotedgedev already installed with version: version 2.0.0

sr/local/bin/iotedgedev build --file /srv/agent/work/21/s/deployment.template.json --platform amd64


=======================================

======== ENVIRONMENT VARIABLES ========
=======================================

.env file not found on disk. Without a file on disk, you must specify all Environment Variables at the system level. (/srv/agent/work/21/s/.env)



==================================
======== BUILDING MODULES ========
==================================


ERROR: No module.json file found. module.json file is required in the root of your module folder

##[error]Error: The process '/usr/local/bin/iotedgedev' failed with exit code 1

After realizing that the IotEdge task does not switch versions, I upgraded iotedgedev sudo pip3 install -U iotedgedev and it updated to 2.0.1. What do I need to do to support pinning iotedgedev to a specific version in case we encounter a breaking change with iotedgedev in the future? Re-running the build with 2.0.1 works as shown below.

Starting: Azure IoT Edge - Build module images - QA
==============================================================================
Task         : Azure IoT Edge
Description  : Build and deploy an Azure IoT Edge image
Version      : 2.2.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/azure-iot-edge
==============================================================================
Building module images...
/srv/agent/work/21/s /srv/agent/work/21/s
iotedgedev already installed with version: version 2.0.1

sr/local/bin/iotedgedev build --file /srv/agent/work/21/s/deployment.template.json --platform amd64


=======================================

======== ENVIRONMENT VARIABLES ========
=======================================

.env file not found on disk. Without a file on disk, you must specify all Environment Variables at the system level. (/srv/agent/work/21/s/.env)


==================================

======== BUILDING MODULES ========
==================================


PROCESSING DOCKERFILE: /srv/agent/work/21/s/modules/.......

Thanks.