Add support for Pluggable Discoveries

[silvanocerza]

Please check if the PR fulfills these requirements

• The PR has no duplicates (please search among the Pull Requests
  before creating one)
• The PR follows
  our contributing guidelines
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• UPGRADING.md has been updated with a migration guide (for breaking changes)

• What kind of change does this PR introduce?

This PR introduces quite a bit of changes to add support for Pluggable Discoveries as specified by the related RFC.

• What is the current behavior?

Currently the Arduino CLI uses a bundled serial-discovery tool to search for supported boards connected to serial ports, there is no support for boards connected to the network or 3rd party boards that require specific tools to be discovered.

There is currently no way for 3rd party platforms to specify a custom tool to run to discover boards.

• What is the new behavior?

After this PR is merged any platform will be able to specify custom tools to run at discovery time to find supported boards, for example when calling arduino-cli board list.

This change is fully retro compatible with platform that don't specify a discovery tool, the bundled serial-discovery tool will be used as a fallback.

Discovery tools will be internally loaded automatically when the Arduino CLI is initialized and run when necessary.

• Does this PR introduce a breaking change, and is
  titled accordingly?

Yes, they are documented in the docs/UPGRADING.md file.

• Other information:

None.

---

See how to contribute

[per1234]

I get a panic now if I try to upload without specifying a port:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: 5ab1ecde Date: 2021-07-22T16:58:19Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0xd4c6e9]

goroutine 1 [running]:
github.com/arduino/arduino-cli/cli/compile.run(0xc000755600, 0xc00011d380, 0x1, 0x4)
        C:/Users/per/go/src/github.com/arduino/arduino-cli/cli/compile/compile.go:206 +0x969
github.com/spf13/cobra.(*Command).execute(0xc000755600, 0xc00011d340, 0x4, 0x4, 0xc000755600, 0xc00011d340)
        C:/Users/per/go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc00014fb80, 0x0, 0xc000210c60, 0x0)
        C:/Users/per/go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        C:/Users/per/go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:887
main.main()
        C:/Users/per/go/src/github.com/arduino/arduino-cli/main.go:31 +0x8f

Previously there was a friendly error message:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: fc367b77 Date: 2021-07-22T16:55:44Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
Error during Upload: uploading error: no upload port provided

[silvanocerza]

I get a panic now if I try to upload without specifying a port:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: 5ab1ecde Date: 2021-07-22T16:58:19Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0xd4c6e9]

goroutine 1 [running]:
github.com/arduino/arduino-cli/cli/compile.run(0xc000755600, 0xc00011d380, 0x1, 0x4)
        C:/Users/per/go/src/github.com/arduino/arduino-cli/cli/compile/compile.go:206 +0x969
github.com/spf13/cobra.(*Command).execute(0xc000755600, 0xc00011d340, 0x4, 0x4, 0xc000755600, 0xc00011d340)
        C:/Users/per/go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc00014fb80, 0x0, 0xc000210c60, 0x0)
        C:/Users/per/go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        C:/Users/per/go/pkg/mod/github.com/spf13/cobra@v1.0.1-0.20200710201246-675ae5f5a98c/command.go:887
main.main()
        C:/Users/per/go/src/github.com/arduino/arduino-cli/main.go:31 +0x8f

Previously there was a friendly error message:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: fc367b77 Date: 2021-07-22T16:55:44Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
Error during Upload: uploading error: no upload port provided

That's no good. It should now be possible to upload without specifying a port for boards that use tools that automatically detected the correct port, but it certainly must not crash like this for those that don't.

Thanks for the great review anyway. 🙏

[silvanocerza]

🎉

[rsora]

[matthijskooijman]

I noticed a regression that I expect is caused by this PR. When using arduino-cli from git now, uploads using (our slightly customized version of) the STM32 core fail with:

Error during Upload: Property 'upload.tool.serial' is undefined

The board in question indeed does not define upload.tool.serial, but still uses the pre-pluggable syntax of defining upload.tool, which should be translated to upload.tool.default by convertUploadToolsToPluggableDiscovery(). However, this core does not define upload.tool on the board directly, but defines it in a board menu: https://github.com/stm32duino/Arduino_Core_STM32/blob/974b35690cbf4680ce5746c61c69fb20935bae2f/boards.txt#L522

To reproduce:

$ arduino-cli compile --fqbn STMicroelectronics:stm32:Nucleo_32:pnum=NUCLEO_F031K6,upload_method=serialMethod --upload --port /dev/ttyACM0 
Sketch uses 7612 bytes (23%) of program storage space. Maximum is 32768 bytes.
Global variables use 828 bytes (20%) of dynamic memory, leaving 3268 bytes for local variables. Maximum is 4096 bytes.
Error during Upload: Property 'upload.tool.serial' is undefined

I guess this is fairly easy to fix in the core, though (just define both upload.tool.default (or upload.tool.<protocol>) in addition to the legacy upload.tool, but it might be useful to fix compatibility here too. An obvious fix would be to implement the fallback when actually doing the upload, rather than when loading the boards).

@fpistm, you might find this interesting too :-)

[cmaglie]

@matthijskooijman you're my nightmare :-)

An obvious fix would be to implement the fallback when actually doing the upload, rather than when loading the boards).

yes I think it's the thing to do, I'll move your comment in a new issue, so we can keep track of it.

[fpistm]

@matthijskooijman you're my nightmare :-)

@cmaglie Mine too 😜

[matthijskooijman]

Lol, thanks for the compliments :-p