Document Serial Monitor changes

[tigoe]

The 2.0 IDE and the Pro IDE introduced a major change to the UX of the Serial Monitor. Instead of being its own window, it now is attached to a sketch window. This is a change that is likely to cause confusion in many classrooms, and requires better thought to the UX and more documentation so as not to lose users.

Example:

1. Open a serial sketch (say AnalogReadSerial, from the Basics menu)
2. Pick a board and port
3. upload code
4. open serial monitor
5. open a new sketch window, write a new sketch
6. attempt to upload

The result will be that the second sketch will not upload and you;'ll get a "port busy" error. But you can't see the Serial Monitor, because it's in the sketch window behind the current one, so you don't know where to look.

This will result in beginning users wasting hours trying to solve the problem. It could be addressed by a better error message, e.g. "error: serial port is in use. Check to see if you have the Serial Monitor open in another sketch window."

it could also be solved by moving the Serial Monitor to the in-focus window, however, this is a problem if you have more than one board attached at time.

---

Example 2:

1. Attach a board to USB port 1
2. Attach another to USB port 2
3. In one window, select board 1 on port 1
4. Upload a sketch to board 1
5. Open serial port 1
6. In a second window, select board 2 on port 2
7. Upload a sketch to board 2
8. Attempt to open port 2

This is a nice change, it allows you to have two serial monitors open at a time. But it doesn't work by default. When you click the Serial Monitor icon in window 2, it attempts to open serial port 1 rather than the port it just uploaded to! You have to explicitly pick the second port AFTER you upload to open both ports. That's confusing. Whatever board and port you select for a given window should be the default for all the serial operations associated with that window.

---

There are a series of pop-up toasts that pop up when you upload a changed sketch while the serial monitor is open warning that the port's not available, etc. These are distracting, and not useful if you're handling the closing of the port, the upload, and the re-opening in the background. They should be kept in the error log, not in pop-up toasts.

---

These problems will become more confusing as more cases pop up. If the new model is a strong connection between window, board, and port, and the Serial Monitor (and later, the Serial Plotter) is no longer independent, that needs to be made very clear in announcements, documentations, and tutorials. It is a potential advantage, but it is also a potential learning hazard, as teachers are used to the old way and will trip over the new way if they're not informed. I did.

[ubidefeo]

hi @tigoe

I see your point.
I have sent this to Content so they can use your feedback as a source for amending the official tutorial.
Would you have time to go through it and let us know if there's something you'd change?

Documentation on IDE 2.0 is in progress, and we add more content as we go, but on such a feature your feedback is precious
https://www.arduino.cc/en/Tutorial/getting-started-with-ide-v2/ide-v2-serial-monitor

thanks
ubi

[karlsoderby]

Hi @tigoe and @ubidefeo I will revise the Serial Monitor tutorial for IDE 2.0, and see if I can highlight this better, perhaps adding a comparison section between the old Serial Monitor window and the new one!

[tigoe]

Happy to look at notes, @ubidefeo and @karlsoderby .

First thing I'd do is make sure that when you select a serial port for a given window, it stays with that window. See Example 2 above, which does not conform to what you're describing in the documentation page in progress that you're citing.

On the documentation page:
I'd avoid adverbs like "simply" and "easily" like this: "You can easily download the editor from...". While it may be an attempt to reassure the reader, for those who don't get what you're describing to work, they then feel stupid because you told them it should be easy. Be matter-of-fact about it: "you can download the editor from..."

I also think it's better to write instructions in the second person, so the reader knows what they should do, rather than the first person plural. "You" are doing this, not "we". I have a style guide I try to use that may be of use.

In this draft, you're jumping straight to the exciting (and advanced) thing without explaining the new UX concept or the potential problem it raises with with a more basic activity. I think I'd mention explicitly that the 2.0 IDE, unlike the 1.0, can keep track of multiple boards plugged into multiple ports on your computer. To you, the user, this means that you choose a board and port from the Board/port selection option menu in the sketch window, and that sketch window gets control over that port and board. When you switch from one sketch window to another, the serial port that you opened in the first window is still open in that window. If you try to upload code to the same board and port in the second window, you'll get an error. It's a change of mindset from the 1.0 IDE, where the serial monitor was an independent window, and the IDE kept track of only one port.

Then I'd go on to explain how this is an advantage as you did in this tutorial.

[tigoe]

Related to this: in IDE 1.0 and before:

1. Click upload
2. Click "Serial Monitor"
3. Serial monitor opens
4. When upload begins, serial monitor greys out
5. When upload completes serial monitor re-activates.

In IDE2.0:

1. Click upload
2. Click "Serial Monitor"
3. Console message appears, "Compilation complete. Cannot perform port reset: opening port at 1200bps: Serial port busy"
4. No indication that upload is in progress, no indication of how or when it's okay to open Serial Monitor again.
5. Several seconds later, console message appears: "No device found on XXXX. Upload error: Error: 2 UNKNOWN: uploading error: uploading error: exit status 1"

This is much less clear or useful behavior than IDE 1.0

[per1234]

Thanks so much for your valuable feedback @tigoe!

This issue ended up containing multiple distinct defect and enhancement reports. In order to facilitate tracking the progress of the work on them, I have created a dedicated issue for each.

If I have misinterpreted your reports, please let me know. Feel free to comment on those dedicated issues with any additional information.

---

4. open serial monitor
5. open a new sketch window, write a new sketch
6. attempt to upload

The result will be that the second sketch will not upload and you;'ll get a "port busy" error. But you can't see the Serial Monitor, because it's in the sketch window behind the current one, so you don't know where to look.

Moved to: #586

---

Example 2:

1. Attach a board to USB port 1
2. Attach another to USB port 2
3. In one window, select board 1 on port 1
4. Upload a sketch to board 1
5. Open serial port 1
6. In a second window, select board 2 on port 2
7. Upload a sketch to board 2
8. Attempt to open port 2

This is a nice change, it allows you to have two serial monitors open at a time. But it doesn't work by default. When you click the Serial Monitor icon in window 2, it attempts to open serial port 1 rather than the port it just > uploaded to!

I haven't been able to reproduce this bug with the current versions of Arduino IDE from the main branch, but the same problem will be introduced by a regression that comes with the pending Serial Plotter, so we can use that issue: #613

---

There are a series of pop-up toasts that pop up when you upload a changed sketch while the serial monitor is open warning that the port's not available, etc.

I'm not certain it is exactly the same, but I did observe something similar, which I reported at #620

---

1. Click upload
2. Click "Serial Monitor"
3. Console message appears, "Compilation complete. Cannot perform port reset: opening port at 1200bps: Serial port busy"
4. No indication that upload is in progress, no indication of how or when it's okay to open Serial Monitor again.
5. Several seconds later, console message appears: "No device found on XXXX. Upload error: Error: 2 UNKNOWN: uploading error: uploading error: exit status 1"

Moved to #581

[per1234]

The final very important feedback was summarized here:

These problems will become more confusing as more cases pop up. If the new model is a strong connection between window, board, and port, and the Serial Monitor (and later, the Serial Plotter) is no longer independent, that needs to be made very clear in announcements, documentations, and tutorials. It is a potential advantage, but it is also a potential learning hazard, as teachers are used to the old way and will trip over the new way if they're not informed. I did.

Since the discussion here centered around that topic, I would like to dedicate this issue exclusively to the subject of covering any intentional changes or unavoidable limitations between Arduino IDE 2.x and the classic IDE Serial Monitor in the documentation.

@karlsoderby when you get a chance, please give us a quick update in regards to the previous discussion here.

Related

• Serial Monitor is missing an autoscroll control #806

[ulihuber]

I can't see any real advantage of sharing of serial ports over Arduino 2.0 instances. If you want to develop communication between two controllers you have two active instances and you want to keep them separately and consistent. Even (or especially) if you reset a controller. This was handled ideally in the old IDE where you always knew what ressources were claimed by what instance. There was never the problem of 'loosing' the port to another instance and you were even free to switch to external terminals if special terminal features were needed.
Why is it impossible to free the port after compiling/upload? Why is it impossible to close the terminal window without closing the whole IDE.
Really not a good approach and might force me back to the old stuff where so many nice things are missing. :-(

[per1234]

Hi @ulihuber. I think we must be careful to differentiate between unintended defects and intentional/unavoidable changes.

This issue is now restricted exclusively to tracking the need to fully document the latter.

We have other dedicated issues tracking all the known defects in the Serial Monitor:

https://github.com/arduino/arduino-ide/issues?q=is%3Aopen+is%3Aissue+label%3A%22topic%3A+serial+monitor%22+label%3A%22type%3A+imperfection%22

If you encounter a defect which is not currently tracked, please open a new issue about it. If you have additional information to share about one of the defects which is already tracked, you are welcome to comment on that specific issue.

[tigoe]

@per1234 I'm not sure if you want this noted in this issue or the other one, but it may be of some help.

Though I understand the new model, namely "a strong connection between window, board, and port, and the Serial Monitor (and later, the Serial Plotter) is no longer independent" I constantly trip over the problem every time I try to use the 2.0 IDE. Typical use case:

1. open one sketch,
2. upload it,
3. open serial monitor.
4. Open a second sketch that I want to try as an alternative to the first,
5. hit upload.
6. get the error that the monitor's already open.

Even though I know what the model is, I still get it wrong. Nothing in the new IDE encourages me to get it right other than being slapped with an error message that doesn't say something helpful like "you've got the port open in another window"

However: I started experimenting with WebSerial recently in Chrome and Edge on Windows and in Chrome on MacOS. And I noticed that its control over the serial port was non-rivalrous! This would be a good thing to implement between sketch windows. It could preserve the best of the old model while including the best of the new.

For what it's worth here is how I tested it:

1. upload a serial sketch to the board (using 1.8.x IDE; will try 2.0 as soon as I can)
2. open WebSerial sketch
3. open serial port, data comes in
4. go back to IDE, make changes to the sketch, hit upload. And it uploaded!

I don't know what's going on in WebSerial that makes it not compete with the IDE, but it's the first time I've seen it outside of node-serialport. The latter also seems to not compete with the Serial Monitor or the uploader, but I haven't run across any other software that does. This suggests to me that there's been a change in how the major OSes are managing serial access that maybe I am not aware of, and that it might be possible to have more flexible serial management?

[ulihuber]

It's not only an issue with the serial monitor. The new IDE grabs and blocks any port it can find. You plug in a second device, and the open IDE instantly blocks the com port in addition to the one that is used for the actual sketch.
This makes it almost impossible to use other Software like specialized monitors.

[tigoe]

Just downloaded the latest 2.0 IDE (2.0.0 rc6) and tried the steps I outlined above. Got the following error:

C:\Users\tomig\AppData\Local\Temp\arduino-sketch-6E8E491760EA9A59D0523CF1F90870D2/..\arduino-core-cache\core_arduino_samd_nano_33_iot_695dd54ea5ff7848a353d80627971bc0.a(cortex_handlers.c.o): In function `Reset_Handler':
C:\Users\tomig\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.8\cores\arduino/cortex_handlers.c:161: undefined reference to `main'
C:\Users\tomig\AppData\Local\Temp\arduino-sketch-6E8E491760EA9A59D0523CF1F90870D2\core\variant.cpp.o: In function `SERCOM5_Handler':
C:\Users\tomig\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.8\variants\nano_33_iot/variant.cpp:192: undefined reference to `Uart::IrqHandler()'
C:\Users\tomig\AppData\Local\Temp\arduino-sketch-6E8E491760EA9A59D0523CF1F90870D2\core\variant.cpp.o: In function `SERCOM3_Handler':
C:\Users\tomig\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.8\variants\nano_33_iot/variant.cpp:199: undefined reference to `Uart::IrqHandler()'
C:\Users\tomig\AppData\Local\Temp\arduino-sketch-6E8E491760EA9A59D0523CF1F90870D2\core\variant.cpp.o: In function `SERCOM2_Handler':
C:\Users\tomig\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.8\variants\nano_33_iot/variant.cpp:206: undefined reference to `Uart::IrqHandler()'
C:\Users\tomig\AppData\Local\Temp\arduino-sketch-6E8E491760EA9A59D0523CF1F90870D2\core\variant.cpp.o: In function `__static_initialization_and_destruction_0':
C:\Users\tomig\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.8\variants\nano_33_iot/variant.cpp:188: undefined reference to `Uart::Uart(SERCOM*, unsigned char, unsigned char, SercomRXPad, SercomUartTXPad)'
C:\Users\tomig\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.8\variants\nano_33_iot/variant.cpp:195: undefined reference to `Uart::Uart(SERCOM*, unsigned char, unsigned char, SercomRXPad, SercomUartTXPad)'
C:\Users\tomig\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.8\variants\nano_33_iot/variant.cpp:202: undefined reference to `Uart::Uart(SERCOM*, unsigned char, unsigned char, SercomRXPad, SercomUartTXPad, unsigned char, unsigned char)'
collect2.exe: error: ld returned 1 exit status

Compilation error: exit status 1

[ubidefeo]

@tigoe
can you try uninstalling the samd package and installing it again?

[per1234]

Hi @tigoe

I understand the new model, namely "a strong connection between window, board, and port, and the Serial Monitor (and later, the Serial Plotter) is no longer independent" I constantly trip over the problem every time I try to use the 2.0 IDE.

I'm very happy to tell you that issue (#586) has now been fixed by #982

If you would like to try it, the tester build is currently available from the links under the "Artifacts" section of this page:

https://github.com/arduino/arduino-ide/actions/runs/2462555111#artifacts

You must be signed in to your GitHub account in order for those specific download links to be active (a limitation imposed by GitHub).

It will also be in tomorrow's nightly build and in the next release.

---

I don't know what's going on in WebSerial that makes it not compete with the IDE, but it's the first time I've seen it outside of node-serialport. The latter also seems to not compete with the Serial Monitor or the uploader, but I haven't run across any other software that does. This suggests to me that there's been a change in how the major OSes are managing serial access that maybe I am not aware of, and that it might be possible to have more flexible serial management?

Thanks for sharing this information. That is very interesting. I am accustomed to the usual situation where a serial port can only be open by one application at a time, with either the user being required to manually close and open the port, or else some sophisticated automagical juggling of control as is done by the Arduino IDE when uploading to the port while it is open in Serial Monitor. But I have not had the chance to play around with WebSerial yet.

[per1234]

Hi @ulihuber

The new IDE grabs and blocks any port it can find. You plug in a second device, and the open IDE instantly blocks the com port in addition to the one that is used for the actual sketch.

Please try again with the latest nightly build. The download links are listed here:

https://github.com/arduino/arduino-ide#nightly-builds

If it still occurs, please open a dedicated issue for it.

Make sure to provide lots of information in that issue, including detailed steps we can follow to reproduce it. None of the team working on this project have ever experienced something like this, which makes the investigation quite challenging.

[tigoe]

@per1234, that is a sweet upgrade, nice work!

There is still a mystery going on, but it might be on my end, looking into it.

Here's the test I gave it:

1. downloaded the artifact build
2. Opened File _> Examples -> Basics -> ArduinoSerialRead
3. after successful upload notification, opened serial monitor
4. Got serial data stream
5. Opened another window, opened serial monitor
6. Got serial data stream
7. Did a happy dance
8. Uploaded a variation on AnalogReadSerial from second window (delay(100) instead of delay(1))
9. Uploaded
10. Opened serial monitor, got serial stream
11. Opened WebSerial sketch referenced above. Left it open
12. Got serial stream
13. Went back to IDE, changed delay again, uploaded
14. Uploaded successfully, and WebSerial sketch restarted when upload was complete

I do not recommend this behavior, because there are still occasions where one serial listener or another misses a byte. But overall, it's hugely more robust than the previous version, makes it easy to switch between windows like in the 1.8.x IDE, and plays nice with at least one external serial API (namely WebSerial). I think this is a great step forward.

Now I'll go try to break that formatting issue :)

Thank you!

[tigoe]

Oh, and for the record, the most I had going was three Arduino windows and WebSerial, all receiving serial simultaneously. I didn't go further than that. So it may warrant a fuller load test.

[ubidefeo]

Thank you for testing this out @tigoe
It's been a long journey, but we also feel that finally the Pluggable Architecture is solid and will only get better.
Now it's not just "Serial" Monitor, it is "Monitor" because in theory you could write your own MIDI implementation and push MIDI data to the monitor view :)

[tigoe]

Hi folks, an update on this:

When you upload to a board with the Serial Monitor open, the "Done Uploading" popup notification (in UX-ese, the "toast") appears before the upload notifications in the output console. To repeat this:

1. Write some code for a Nano 33 IoT (only thing I have to test with at the moment)
2. Open Serial Monitor
3. Click Upload Button

Shortly thereafter, "Done Uploading appears". Then if you try to go back to the Serial Monitor, the Output Console forces its way to the front, and you see the usual upload messages, like so:

--------------------------
Compilation complete.
Atmel SMART device 0x10010005 found
Device       : ATSAMD21G18A
Chip ID      : 10010005
Version      : v2.0 [Arduino:XYZ] Apr 19 2019 14:38:48
Address      : 8192
Pages        : 3968
Page Size    : 64 bytes
Total Size   : 248KB
Planes       : 1
Lock Regions : 16
Locked       : none
Security     : false
Boot Flash   : true
BOD          : true
BOR          : true
Arduino      : FAST_CHIP_ERASE
Arduino      : FAST_MULTI_PAGE_WRITE
Arduino      : CAN_CHECKSUM_MEMORY_BUFFER
Erase flash
done in 1.084 seconds

Write 11896 bytes to flash (186 pages)

[==========                    ] 34% (64/186 pages)
[====================          ] 68% (128/186 pages)
[==============================] 100% (186/186 pages)
done in 0.080 seconds

Verify 11896 bytes of flash with checksum.
Verify successful
done in 0.010 seconds
CPU reset.

--------------------------
upload complete.

Once that clears, you can go back to the Serial Monitor and everything is fine. But the toast popping up before upload is confusing. I'm not sure whether it's because I'm using a USB-native board or not, but it's worth testing on both USB-native and non-USB-native.

[per1234]

Hi @tigoe. I believe you are describing this bug: #68

That was recently fixed: #1058

The printing of the output from the upload process should now keep up with the actual process, and so you should find that the output has finished printing at the same time as the notification indicates that the actual process completed.

That fix was made after the time of the 2.0.0-rc7 release, so it is only available from the nightly build. The links to download the nightly build are listed here:

https://www.arduino.cc/en/software#nightly-builds

Please give it a try.

[tigoe]

Just tried it in the nightly, with both a Nano and an Uno, and you are correct. Thanks for the update.

[per1234]

Evaluation

I have reviewed the valuable discussion from this issue and found that all the Serial Monitor changes mentioned here as requiring documentation coverage in the end turned out to be defects or deficiencies which were since resolved in the Arduino IDE code base and so no longer require documentation.

Some other defects were reported which have not yet been resolved, but dedicated issues have been created to effectively track those.

So I am going to consider this issue resolved as far as the Arduino IDE is concerned.

Additional IDE 2.x documentation tasks

There are a couple of significant remaining changes to the Serial Monitor (which surprisingly escaped previous mention in this thread):

• serial monitor: missing send button  #348
• Keyboard control for Serial Monitor send was changed #572

I have now documented those: arduino/docs-content#287.

Additional documentation tasks

I see there was some additional valuable feedback which may well still be unresolved (there was no indication here of action taken in response):

#271 (comment)

I also think it's better to write instructions in the second person, so the reader knows what they should do, rather than the first person plural. "You" are doing this, not "we". I have a style guide I try to use that may be of use.

However, it will not be effective to track that task here in the Arduino IDE code base repository. I would recommend it to be tracked in the relevant repository here:

https://github.com/arduino/docs-content

Maybe @karlsoderby or @tigoe could create an issue there for it.

Conclusion

Thanks so much for all the excellent feedback @tigoe. Please let me know if I missed anything IDE-related from this thread which is neither resolved nor tracked elsewhere.