#637, #906: Update package index on 3rd party URLs change. #1132
Conversation
per1234
added
topic: code
There was a problem hiding this comment.
UPDATE: Fixed!
🐛 On any Boards Manager operation (install, uninstall, update), the only feedback of progress is a persistent "Processing ___ notification. The user has no way to know if or when the process has completed.
To reproduce
- Open the "Boards Manager" view.
- Scroll down through the list until you find a platform that is not already installed (as indicated by the lack of an "INSTALLED" tag on the platform entry.
- Hover the mouse pointer over the platform entry.
An INSTALL button will appear. - Click the INSTALL button.
🐛 A "Processing ___ notification appears, but there is no further indication of the progress or completion of the operation (e.g., via notification updates or closure, Boards Manager entry state, Output panel text):
Expected behavior
The status of Boards Manager operations is communicated to the user.
Arduino IDE version
2.0.0-rc8-snapshot-b4ffd7e (tester build for 99723a9)
Operating system
Windows, Linux
Operating system version
Windows 10, Ubuntu 20.04
Additional context
From looking at the data directory, I can see that the operation did have an effect.
The issue does not occur when using the latest build from the main branch (a13a877).
|
Thank you, Per. Interesting defect :) I added a workaround; it should work now. |
Let's put the PR on hold. There are other strange issues (arduino/arduino-cli#1790), and I do not know yet if there are connections. Let's investigate. |
kittaakos
force-pushed
the
#637-signed
branch
2 times, most recently
from
2e4719a to
fd4625c
Compare
|
I have fixed the platform install/uninstall bug. The PR is ready for review. |
There was a problem hiding this comment.
UPDATE: Fixed!
Describe the problem
🐛 Boards platforms or platform releases of newly downloaded package indexes are not listed in Boards Manager.
To reproduce
- If present, delete the following:
<directories.data>/package_esp32_index.json<directories.data>/packages/esp32/hardware/
ⓘ You can also uninstall "esp32 by Espressif Systems" via Boards Manager if you prefer, but since you are already in<directories.data>to delete the package index, this is probably most convenient.
- Start the Arduino IDE.
- Select File > Preferences from the Arduino IDE menus.
- Add the following URL if not already present:
(The issue is not specific to this package index. I picked an arbitrary one for the sake of the demo.)
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json - Click the OK button.
- Wait for the package index download to finish, as indicated by the appearance and disappearance of the "Downloading index: package_esp32_index.json" notification.
- Open the "Boards Manager" view.
🐛 The "esp32 by Espressif Systems" boards platform from the package index is not listed. - Select File > Quit from the Arduino IDE menus.
- Start the Arduino IDE.
🙂 The "esp32 by Espressif Systems" boards platform from the package index is listed in the "Boards Manager" view.
Expected behavior
Boards Manager shows the platforms of all package indexes present the "Additional Boards Manager URLs" preference.
Arduino IDE version
2.0.0-rc8-snapshot-e618efc
Operating system
Windows, Linux
Operating system version
Windows 10, Ubuntu 20.04
Additional context
The demo establishes the following conditions:
- Package index file is not present in
<directories.data>. - The platform is not installed.
However, these conditions were chosen only to make the issue more visible. They are not required conditions:
Even if the package index is already present, Boards Manager will not reflect any changes present only in the newly downloaded version, such as a new platform or platform release.
Boards Manager lists all installed platforms regardless of whether it is listed in a package index. However, it will only list other versions available for upgrade that were present in the processed package index. So if you did not uninstall the "esp32 by Espressif Systems" boards platform before performing the demo, you would see it listed in Boards Manager, but the expected version menu would be missing.
The issue occurs regardless of whether the package index is downloaded on IDE startup or on preferences update.
The issue does not occur when using the build from the main branch (2.0.0-rc8-snapshot-5499c25). Boards Manager shows all platforms and platform releases from the package index downloads on startup.
kittaakos
force-pushed
the
#637-signed
branch
2 times, most recently
from
acea4e2 to
a2bec12
Compare
There was a problem hiding this comment.
UPDATE: now tracked at #1156
I notice that the Library Manager index is updated along with the package indexes after change to "Additional Boards Manager URL" preference.
It seems a bit arbitrary to me since there is no reason to think a change to this Boards Manager-specific preference is an indication that the user is planning to use Library Manager. I think that, for most users, just having a single library index update on startup will be sufficient. That might be problematic if a user leaves the IDE open permanently, but I think that would be very a very rare usage pattern. Perhaps a periodic update of all indexes could be implemented at some point as part of the updatable platform and library notification system.
To reproduce
- Select File > Preferences from the Arduino IDE menus.
- Change the the "Additional Boards Manager URL" preference by either adding or removing URLs.
- Click the OK button.
- Watch the index download notifications. They include "Downloading index: library_index.json.gz".
From the gRPC logs, it appears that a cc.arduino.cli.commands.v1.ArduinoCoreService/UpdateLibrariesIndex request is being sent in addition to the expected UpdateIndex for the package indexes.
daemon INFO 25 CALLED: /cc.arduino.cli.commands.v1.ArduinoCoreService/UpdateIndex STREAM_RESP
26 CALLED: /cc.arduino.cli.commands.v1.ArduinoCoreService/UpdateLibrariesIndex STREAM_RESP
Expected behavior
Only package indexes are updated after changing the "Additional Boards Manager URL" preference.
Arduino IDE version
2.0.0-rc8-snapshot-24cc9f1 (tester build for a2bec12)
Operating system
Windows, Linux
Operating system version
Windows 10, Ubuntu 20.04
I am more than happy to change the behavior, but I propose doing it in the context of a separate issue/PR. I know that the package index update has been bogus since Sep 30, 2021, but IDE2 always runs both. I prefer not to change too many things if not necessary. What do you think, @per1234? Update: the change seems trivial, but I think it does not belong to this PR. Update2: if you think having the fix here speeds up the verification, I can do it. |
|
That is fine. It looked like a regression to me, since it does not do this in the current build from the |
There was a problem hiding this comment.
UPDATE: Fixed!
Describe the problem
🐛 Although they are downloaded after "Additional Boards Manager URLs" preference change, additional package indexes are not downloaded on IDE startup.
Package index URLs must be downloaded by the IDE on two distinct classes of event:
IDE startup
The primary purpose is to update the existing package index files so that Boards Manager will reflect any changes made by the package maintainer (new platforms or new releases of existing platforms).
A less common, but important purpose is to restore the package index files if the user has deleted them. That might seem unlikely for a normal user (as opposed to beta testers and IDE developers), but clearing some or all of the contents of the data directory is actually regularly recommended both by Arduino support as well as the community as a way to recover from the contents getting into a bad state.
"Additional Boards Manager URLs" preference change
This is needed so that Boards Manager will reflect the changes to the preference without having to restart the IDE.
To reproduce
- Select File > Preferences from the Arduino IDE menus.
- Add the following URL if not already present:
(The issue is not specific to this package index. I picked an arbitrary one for the sake of the demo.)
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json - Click the OK button.
🙂 If the preference changed, the package indexes are downloaded as expected. - Select File > Quit from the Arduino IDE menus.
- Delete the following file:
<directories.data>/package_esp32_index.json - Start the Arduino IDE.
🐛 The expected "Downloading index: package_esp32_index.json" notification is not shown. - Check for presence of the following file:
🐛 It is missing
<directories.data>/package_esp32_index.json
When using the build from the main branch, I see this in the logs on startup:
daemon INFO {"level":"info","msg":"URL: https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json","time":"2022-07-05T05:39:32-07:00"}
{"level":"info","msg":"Updating index","time":"2022-07-05T05:39:32-07:00","url":{"Scheme":"https","Opaque":"","User":null,"Host":"raw.githubusercontent.com","Path":"/espressif/arduino-esp32/gh-pages/package_esp32_index.json","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
root INFO Downloading index: package_esp32_index.json completed.
I do not see that with the build for this PR.
Expected behavior
Package indexes are downloaded on IDE startup.
Arduino IDE version
2.0.0-rc8-snapshot-24cc9f1 (tester build for a2bec12)
Operating system
Windows, Linux
Operating system version
Windows 10, Ubuntu 20.04
Additional context
I instructed the deletion of the package index in the demo instructions because it makes the issue more obvious. However, even when the file is present, it is still not downloaded on startup, which means the users would only get updates to the package indexes on the rare occasion that they modified the preference.
|
I can confirm. I see the following error: So the problem here is if any package index is missing the index update must run before the core client initialization. The problem with this it will slow down the IDE2 startup to #876. @cmaglie is there a way to work around it? |
|
From here:
|
|
Please review. Thank you! |
kittaakos
force-pushed
the
#637-signed
branch
2 times, most recently
from
6010ec8 to
ee03b6d
Compare
There was a problem hiding this comment.
UPDATE: Fixed!
Describe the problem
Under the following conditions:
- The "Boards Manager" or "Library Manager" view has not been opened during the current session.
- The Boards/Library Manager view was opened soon after the completion of a package index update (either the update on startup or the update after changing the preference).
🐛 The "Boards Manager" or "Library Manager" view is empty.
To reproduce
- "Boards Manager" or "Library Manager" view in the sidebar if it is open.
- Select File > Quit from the Arduino IDE menus.
- Start the Arduino IDE.
- Select File > Preferences from the Arduino IDE menus.
- Change the the "Additional Boards Manager URL" preference by either adding or removing URLs.
- Click the OK button.
- Watch the index download notifications. As soon as the notification closes, open the Boards Manager" in the sidebar.
🐛 The view is empty and never populates.
This error appears in the logs when the view is opened:
root ERROR Request search failed with error: Channel has been shut down
There is this error in the Developer Tools console:
/E:/downloads/review…8pdx/sketch_jul5a:1 Uncaught (in promise) Error: Request 'searchBoards' failed
at Proxy.<anonymous> (bundle.js:2)
at b.searchBoards (bundle.js:2)
at d.queryBoards (bundle.js:2)
at d.updateBoards (bundle.js:2)
at bundle.js:2
at bundle.js:2
at r.invoke (bundle.js:2)
at s.fire (bundle.js:2)
at d.notifyIndexDidUpdate (bundle.js:2)
at s.onNotification (bundle.js:2)
/E:/downloads/review…8pdx/sketch_jul5a:1 Uncaught (in promise) Error: Request 'search' failed
at Proxy.<anonymous> (bundle.js:2)
at d.search (bundle.js:2)
at v (bundle.js:2)
at w (bundle.js:2)
at C (bundle.js:2)
Expected behavior
The manager views are populated when opened.
Arduino IDE version
2.0.0-rc8-snapshot-ab1929a (tester build for 6010ec8)
Operating system
Windows 10
Additional context
My hypothesis is that it is resulting from this sequence:
- UpdateIndex call
- Init call
- Some other gRPC call triggered by the opening of the Boards or Library Manager view
Step (3) is happening while the Init is still in progress, which causes the error.
As for the "soon after" condition, the time window seems to be dependent on how many platforms you have installed. It also seems to depend on the number of boards the platform defines because I find that having the ESP8266 platform (which has 37 boards) installed significantly increases the window.
I have not been able to reproduce the issue with Linux. But on both of my Windows machines I can reproduce it.
On my lower performance Windows machine, I can easily reproduce it with my current collection of only the "Arduino AVR Boards", "Arduino SAMD Boards (32-bits ARM Cortex-M0+)", and "Teensy" platforms installed. On the higher performance Windows machine I found that the window became very small with that number of platforms, meaning I had to open the view immediately after the notification closed.
I think the conditions could be produced by a common user sequence when doing a platform installation:
- Add a package index URL to the "Preferences" dialog.
- Either wait for the process to finish or else just have some chance delay (e.g., going back to the installation instructions to read the next step) of the necessary length after the closure of the "Preferences" dialog.
- Open "Boards Manager" view to install the platform.
How common depends on how large the time window is to trigger the fault. I haven't determined that yet, but I triggered it by chance on my very first use of this build and with my full collection of platforms I am able to trigger it 100% reliably without having to be too careful about watching the notification, even with my higher performance machine. So it must be on the order of seconds.
I have not identified any possible reason why I can't reproduce it on Linux.
I believe a fix for this is already in progress via https://github.com/arduino/arduino-ide/compare/6010ec81aeaaf455dc2011a7b82c47c539f52a6c..ee03b6dd0c391e297a1dbb0e738023a83ed39951, but am submitting the report publicly here for the sake of documenting the issue discovered with the build from the previous commit so that we may most effectively verify that it was resolved in the final version of the PR.
|
Thank you so much for your help, Per 🙏 Although I could not reproduce the broken Boards Manager widget issue from here, I could reproduce the I tested all cases reported as per review remarks; all should work. I hope I did not overlook something. Please verify What this PR does not solve:
|
There was a problem hiding this comment.
UPDATE: Fixed!
I found an issue with the latest build. As with my the issue from my previous report, it is related to the number of boards platforms that are installed. However, unlike the previous issue, which occurred even with a small number of platforms though at a lesser severity, in this case the issue only occurs when there are a large number of platforms.
So it might be considered just too much of an edge case to be worth spending time on. But since I encountered it during testing I thought I should at least submit a report.
Describe the problem
Under the following conditions:
- A large number (e.g, 42) of Arduino boards platforms are installed
- The "Library Manager" view is open
- Indexes are updated
🐛 The library listings disappear from the "Library Manager" view when the indexes update finishes:
To reproduce
- Open the "Library Manager" view in the sidebar if it is not already open.
- Select File > Quit from the Arduino IDE menus.
- Start the Arduino IDE.
🙂 The "Library Manager" view is populated as expected - Wait for the index update notification to appear and then close on completion of the updates.
🐛 The library listings disappeared from the "Library Manager" view.
Expected behavior
"Library Manager" view remains populated after an index update.
Arduino IDE version
2.0.0-rc8-snapshot-4e50e33
Operating system
Windows 10
Additional context
If I enter a keyword in the "Filter your search..." field, the library listings appear.
Through bisection, I determined that it is not dependent on one specific boards platform.
If I reduce the number of installed platforms significantly (e.g., removing half of them), the issue no longer occurs, regardless of which half is removed.
The issue does not occur with the "Boards Manager" view.
The issue does not occur when using the latest build from the main branch (2.0.0-rc8-snapshot-7f2b849).
I do not see any relevant errors or other interesting output in the logs, logs with Arduino CLI daemon output enabled, or Developer Tools console.
The issue occurs with either the index update triggered on startup or the index update triggered on "Additional Boards Manager URLs" preference change.
The issue does not persist into a new IDE window, but it will occur in that window as well once the required conditions are produced.
The issue occurs even with nothing in the "Additional Boards Manager URLs" preference.
My currently installed platforms:
ID Installed Latest Name
adafruit:samd 1.7.10 1.7.10 Adafruit SAMD Boards
adafruit:wiced 0.6.6 0.6.6 Adafruit WICED
arduboy-homemade:avr 1.3.4 1.3.4 Arduboy homemade package
arduino:avr 1.8.99 1.8.99 Arduino AVR Boards
arduino:mbed_edge 3.1.1 3.1.1 Arduino Mbed OS Edge Boards
arduino:mbed_nano 3.1.1 3.1.1 Arduino Mbed OS Nano Boards
arduino:mbed_nicla 3.1.1 3.1.1 Arduino Mbed OS Nicla Boards
arduino:mbed_portenta 3.1.1 3.1.1 Arduino Mbed OS Portenta Boards
arduino:mbed_rp2040 3.1.1 3.1.1 Arduino Mbed OS RP2040 Boards
arduino:megaavr 1.8.102 1.8.102 Arduino megaAVR Boards
arduino:nrf52 1.0.2 1.0.2 Arduino nRF52 Boards
arduino:sam 1.6.12 1.6.12 Arduino SAM Boards (32-bits ARM Cortex-M3)
arduino:samd 1.8.13 1.8.13 Arduino SAMD Boards (32-bits ARM Cortex-M0+)
Arrow:samd 2.1.0 2.1.0 Arrow Boards
atmel-avr-xminis:avr 0.6.0 0.6.0 Atmel AVR Xplained-minis
attiny:avr 1.0.2 1.0.2 attiny
ATTinyCore:avr 1.5.2 1.5.2 ATTinyCore
Balanduino:avr 1.2.0 1.2.0 Balanduino
bitDuino10-aTPI:avr 1.0.0 1.0.0 bitDuino10-arduinoTPI
digistump:avr 1.7.5 1.7.5 Digistump AVR Boards
DxCore:megaavr 1.4.7 1.4.7 DxCore
electroniccats:mbed 1.0.5 1.0.5 Electronic Cats Mbed OS Boards
electroniccats:samd 2.0.7 2.0.7 Electronic Cats SAMD(L)(C) Core for Arduino
emoro:avr 3.2.2 3.2.2 EMORO 2560
esp32:esp32 2.0.3 2.0.3 esp32
esp8266:esp8266 3.0.2 3.0.2 esp8266
industruino:samd 1.0.1 1.0.1 Industruino SAMD Boards (32-bits ARM Cortex-M0+)
Intel:arc32 2.0.5 2.0.5 Intel Curie Boards
Intel:i686 1.6.7+1.0 1.6.7+1.0 Intel i686 Boards
LGT8fx Boards:avr 1.0.6 1.0.6 LGT8fx Boards
littleBits:avr 1.0.0 1.0.0 littleBits Arduino AVR Modules
m5stack:esp32 2.0.2 2.0.2 M5Stack
Maixduino:k210 0.3.11 0.3.11 Maixduino(k210)
MattairTech_Arduino:samd 1.6.18-beta-b1 1.6.18-beta-b1 MattairTech SAM D|L|C core for Arduino - Beta build
MegaCore:avr 2.1.3 2.1.3 MegaCore
megaTinyCore:megaavr 2.5.10 2.5.10 megaTinyCore
microUPDIcore:avr 1.0.2 1.0.2 microUPDI Firmware Uploader
MightyCore:avr 2.1.3 2.1.3 MightyCore
MiniCore:avr 2.1.3 2.1.3 MiniCore
OpenCM904:OpenCM904 1.5.0 1.5.0 OpenCM9.04
opentracker:sam 1.0.5 1.0.5 OpenTracker ARM (32-bits) Boards
realtek:AmebaD 3.1.1 3.1.1 Realtek Ameba Boards (32-bits ARM Cortex-M33 @200MHz)
|
I have fixed the most recent review issue. It should work now.🤞 Please verify. Thank you! |
|
Thank you so much for your help, Per! Your review is gold. |
Motivation
Update package indexes without IDE2 stop/start if 3rd party URLs change.
Change description
Changes in-action:
3rd_party_urls_set.mp4
3rd_party_urls_unset.mp4
3rd_party_urls_invalid.mp4
3rd_party_urls_invalid_startup.mp4
Other information
3rd party URLs to test with:
Closes #637
Closes #906
Partial #1084
Reviewer checklist