Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update micronucleus #465

Closed
sanbeg opened this issue Oct 13, 2020 · 86 comments
Closed

update micronucleus #465

sanbeg opened this issue Oct 13, 2020 · 86 comments

Comments

@sanbeg
Copy link

sanbeg commented Oct 13, 2020

The micronucleus bootloader is bundled with a command line utility. Although this command has backwards compatibility to allow the command to work with older bootloaders, it doesn't have forwards compatibility, and thus will refuse to work with newer bootloaders.

It appears that ATTinyCore bundles an old alpha of this command from 2015, which is no longer listed in the release page https://github.com/micronucleus/micronucleus/tags (IIRC, the same version that was used by the long since unmaintained Digistump core).

When trying to flash a Digispark clone with an updated bootloader, I get this error:

Warning: device with unknown new version of Micronucleus detected.
This tool doesn't know how to upload to this new device. Updates may be available.
Device reports version as: 2.4

Although may cheap clones still use very outdated bootloaders which will work with this old tool, an updated tool would work with both old and new bootloaders.

@pcfreak1201

This comment has been minimized.

@SpenceKonde
Copy link
Owner

I took the version from what appeared to be the most recent official digispark package - I would love to get a newer version, are there any prebuilt packages to use, or do I need to get a build environment set up to compile it?

@SpenceKonde
Copy link
Owner

To expand on that - I've seen that repo, but I am not familiar with the process of compiling for desktop computers, only microcontrollers. That's more or less what we got to last time it came up - well, great, there's a new version, but it's source code, and we need binaries for multiple platforms, and I don't know how to get there from here.

@pcfreak1201
Copy link

You are write, I was on the MC side. On Linux (and may be other OS?) there is no problem to rebuild the commandline-tools with the sources on https://github.com/micronucleus/micronucleus/tree/master/commandline as desribed in #442
You only need the binutils and libusb-dev to recompile. Now I tried to CrossCompile for Windows x86, and there was a problem, because I don't know how to change the environment to MinGW. So I tried to generate a batch file xcompile.sh and it seems to work (finally no erros or warnings). But I can't test it. I've uploaded them to https://github.com/pcfreak1201/micronucleus_commandline
It is really quick and dirty!
Gladly Tim (@cpldcpu I think ?!) has written some interesting infos in the German Forum mikrocontroller.net, so I started my testing - but may be he is the better person to cooperate with, because of his experieces?

@sanbeg
Copy link
Author

sanbeg commented Oct 13, 2020

That would explain it. I don't think there are binary releases, I've just been building from source as needed. I have the binaries for the Linux command line and the default bootloader if that helps; I could probably get the other bootloaders & Mac binary without too much difficulty also.

@cpldcpu
Copy link

cpldcpu commented Oct 14, 2020

Hm... not sure what to say about this. This is the kind of clusterf* you get into when people start making rogue forks.

AFAIR the commandline tool needs MINGW32 to compile under windows due to a dependency on an old version of libusb. That is something that could be fixed, but it takes time. What would actually be preferrable is to clean up the micronucleus reposititory and add CI/CD so there are precompiled releases.

interesting infos in the German Forum mikrocontroller.net,

Which did you mean exactly? Maybe I have answered the same questions some years ago :)

@pcfreak1201
Copy link

Year, therfore I didn't make a fork, only put some code together, to make it more clear... I hope ;-)

I think, Spence wants to have a repository, which could generate a new build for each release, automated.
So fixing would be much more reliable on each OS/platform.

On Linux I needn't to fix or use an old version of libusb. With installing libusb-dev and make, the new build was ready.

mikrocontroller.net is always my preferred source for solutions - and at the end there was clearly descibed, what to do to compile on (for) Windows. And Google found it because of the stuff, you put there, years ago ;-)

@SpenceKonde
Copy link
Owner

I did not know until just now that there was any work going on in the micronucleus official repo! I looked in master - it was a ghost town (I'm used to the (commit to master, branch releases off of that) rather than doing work in branches that you won't immediately see,

I'm sad to see that we've made different decisions on the MCUSR handling. Honestly, I've turned against preserving MCUSR entirely; i'm going to be lobbying hard to clear it whenever we go to the app, stashing it's contents into GPIOR0 and clearly and vigorously documenting that, because otherwise, you really can't guarantee that you honor the requested entry conditions.... and relying on people in Arduino land to check MCUSR? You're just deluding yourself if you think even 1% of them will.

IMO one of ideals of a good bootloader is that - barring self programming by the app to overwrite the bootloader on parts where it's not restricted to boot section, of course - the only thing that the application shuold ever be able to do that has anything to do with the bootloader is confirm that the bootloader is there (by reading the flash) and enter the bootloader, either from it's start, by resetting, or (in the case of that thing to allow write from app on parts that don't allow that), via a designated entry point. A poorly written app should not be able to cause the bootloader to not honor it's entry conditions.

@SpenceKonde
Copy link
Owner

SpenceKonde commented Oct 14, 2020

The reason that none of my work was ever contributed back to @ArminJo (who I thought was the closest thing to someone who had taken over development of what I thought was the abandoned project, hence why I started from his code) was that after making hash of the build directory (this was after the hdd failure, so it's backed up in multiple places, I then got pulled away onto a few other more pressing matters, and I need to get DxCore and megaTinyCore to a state where people aren't screaming quite so loudly about their deficiencies - objectively, those are probably a more important thing for me to work on, even though micronucleus is fun). I've actually got two new pieces of hardware for IMO the most obvious candidates for micronucleusing among the Classic tinyAVR parts, because I felt like they deserved to have micronucleus... (though I'm getting like another 12+ other new designs in the same batch, so these won't be processed immediately,) plus gimped tiny841 micronucleus boards (it's like the Wattuino one, electrically... but built around the SOIC-14 package, so it cant compete on size, as I didn't want to steal anyone's thunder... but I couldn't seem to DIY one that worked, was like hrrrumph, it's just gotta be the lousy geometry of the wires, a custome PCB will work for sure.... And as I had already drawn something that was sssoooOOOOooo close, I figured I might as well do it.

Not sure what my approach going forward is going to be regarding the bootloader itself (I definitely intend to generate a large number of versions of the bootloader for each part, to cover any plausibly useful entry condition, on every part. Azzy cores tend to try to expose every conceivably useful option

@SpenceKonde
Copy link
Owner

And finally - as I've alluded to wanting to do, one day I hope to say a prayer to the lord of the AVR architecture, and armed with a sturdy laptop, clear screen, pure heart, and a copy of the bible ( http://ww1.microchip.com/downloads/en/DeviceDoc/AVR-Instruction-Set-Manual-DS40002198A.pdf ), sequester myself in my room for a some days, and port VUSB to AVRxt. From there, making the rest of the micronuclues bootloader work on the new parts would be a cake walk ((hell then it could also be a "real" bootloader, without having to fiddle with vector tables! Not that I don't think that's part of the fun...). Plus we could run them all at 20 MHz... a ATtiny3217 USB would be pretty sweet... Further beyond, once the code worked on tinyAVR, you'd have it done for the Dx-series as well (no differences in timing), just would need workaround to allow the larger flash (they can go faster than 20, but don't have to) - could use the MVIO o get rid of the need for the zener diodes (Dx series is full speed range over full voltage range - I think core is running at same voltage no matter what the input voltage is, which I don't think was how older AVRs worked). Lot of really awesome things could be made possible.... It's an exciting thing to dream of, but I have more pressing things to attend to...

@SpenceKonde
Copy link
Owner

SpenceKonde commented Oct 14, 2020

But no matter what. somehow we need to get binary packages of a current version of micronucleus for all the relevant OS's... we don't need anything fancy like continuous delivery, but we need something reasonably recent that works with the current version of the firmware, since Arduino board packages (whcih is how like 99% of people install "board packages" now) is needed to make that experience work out... If that means someone like @pcfreak1201 building it when something big happens, directing me to the files, and me swapping out the micronucleus exe with the new one, so be it... heck, it might only need to be done every time a board that broke compatibility suddenly became supported or something.... We don't need perfectly up to date micronucleus tool.... just something recent enough to be usable without problems...

@pcfreak1201
Copy link

Well, I lost the track to Windows since XP, so it is more likely @sanbeg or @cpldcpu who will build EXEs ;-)
And I think https://github.com/ArminJo/DigistumpArduino would be an nice start, to set up the sources for these 5 years old binaries. As long as the protocoll is the same as in 2.04a we don't come in a hurry!?
Even for me "software-usb" is only fun, because the price for a STM32 or ESP32Sx is not soo high and I get USB in hardware. My personal favorite is still the t16x4 with CH340. Possible to solder by my own, 16k and for most ideas enough pins and interfaces.

@sanbeg
Copy link
Author

sanbeg commented Oct 18, 2020

Well, I lost the track to Windows since XP, so it is more likely @sanbeg or @cpldcpu who will build EXEs ;-)
And I think https://github.com/ArminJo/DigistumpArduino would be an nice start, to set up the sources for these 5 years old binaries. As long as the protocoll is the same as in 2.04a we don't come in a hurry!?

We may be thinking about the opposite thing. I don't think it's necessary to set up sources for the 5 year old binaries; but rather, a way to automatically set up binaries for the current sources. For the current bootloader, the protocol isn't the same as the 2.04a version. So if you buy one of these boards now and decide to update your bootloader to a recent release, you also need to overwrite the command line tool from the core with the recent one.

I took a quick look through the core, and I wasn't quite able to track down where the core is configured to use 2.04a; so they only way I was able to make it work was by copying in the tool, then replacing packages/digistump/tools/micronucleus/2.0a4/micronucleus with a symlink to the packages/ATTinyCore/tools/micronucleus/2.04/micronucleus which I added.

@per1234
Copy link
Contributor

per1234 commented Oct 18, 2020

I took a quick look through the core, and I wasn't quite able to track down where the core is configured to use 2.04a

It's defined in the package index's packages[].platforms[].toolsDependencies[].

Here:
https://github.com/SpenceKonde/ReleaseScripts/blob/b53dc05fb66b14ee1007ffd3795e654f8fe5d85a/package_drazzy.com_index.json#L814-L818
and here:
https://github.com/ArminJo/DigistumpArduino/blob/aa2d5529c83ec1eb866ec3e6e8d3672eaeffbaaf/package_digistump_index.json#L152-L156

@SpenceKonde
Copy link
Owner

And - the reason I use that version was that that was the only version of binaries that I was able to find! I was unaware that there even WAS an updated version of those tools... and the people I was talking to and who were driving this project forward were ALSO using those old versions of the tools, because that's the version that was there...

@SpenceKonde
Copy link
Owner

I was not at all suggesting that we start some new system for hosting ancient binaries, but rather - if there's a new version, and people who know how to build it, can we please get some builds made and start using them?

Like, a way to automatically build them is lovely, but let's not let the perfect be the enemy of the good... just building a recent snapshot that we can pull in with an Arduino core and hosting them somewhere (github?) would be better than anyone has done since, well, since 2.04a!!!

@pcfreak1201
Copy link

pcfreak1201 commented Oct 18, 2020

And - if I remember correctly, all these USB implementations are based on V-USB or am I wrong? And this exists since 2005 - so old is relative ;-)

@SpenceKonde
Copy link
Owner

SpenceKonde commented Oct 18, 2020

Yes, VUSB is one of several AVR bitbang USB implementations. Was developed quite a while ago. My understanding was that it's the best out of the lot, and seems to be what all the AVR-based USB devices using AVR's that don't have "real" USB are built on. The company that does it is still around, though I don't think they're actively developing it. I was considering buying a commercial license so I could keep my cards close to my vest when I finally dig out of enough other projects to get to try to port it to the new AVRs (I did look through the assembly - it's all hand-tuned assembly - and it definitely looked doable, and I love working that close to the metal...

In any event. the whole reason this came up, though is that the micronucleus upload tool is clearly too old, as there are lots of bootloaders living on boards in the wild that these upload tool binaries won't upload to. Arduino cores live in a world of board manager, so "just build micronucleus yourself" does not fly, heh... and I don't know how to program devices with more than 128k of program memory (I was born without an EIND register you see....)

(actually, that's not true - a few years ago, we needed a dual booting bootloader for an arduino mega2560, but it had to still work like a normal mega too, but also support another protocol on another serial port under certain conditions.... Luckily the Arduino bootloader was grossly bloated with some "monitor" thing nobody ever used. Without that, I had acres of space after the end of the bootloader, and stuck this other bootloader back there. Actually though, I think my poor understanding at the time of how that was handled contributed to some difficulties getting to it (though we worked around that easy enough...

@pcfreak1201
Copy link

Yes, the tools (binaries) are too old, but not the protocol. So we have to clean up the source "HAL" to get rid of unsupported/obsolete libraries as cpldcu mentioned:

AFAIR the commandline tool needs MINGW32 to compile under windows due to a dependency on an old version of libusb. That is something that could be fixed, but it takes time.

But as I said, Linux doesn't look like the problem ;-)

@sanbeg
Copy link
Author

sanbeg commented Oct 18, 2020

So not sure how I missed this until last night, but it looks like micronucleus does include a prebuilt windows binary mixed in with the sources. So with that I have mac, windows & linux 64; the only one I don't have yet is 32 bit linux.
Since I still haven't found sources for the digispark launcher I reimplemented that in bash, since that seems better than distributing mystery binaries, and I think it's be portable across all the system we're talking about.

I've gathered up my work in progess @ https://github.com/sanbeg/MicronucleusArduino So far I've only tested on Linux, but I can do some more testing then clean and package that.

@SpenceKonde
Copy link
Owner

SpenceKonde commented Oct 19, 2020

Oh! That's wonderful...

And wait, the whole purpose of launcher is just to rewrite the arguments passed to micronucleus executable?

Couldn't we just do that in platform.txt and do away with it entirely?

Something like...
tools.micronucleus.upload.pattern="{cmd.path}" --type intel-hex --timeout 60 {build.path}/{build.project_name}.hex

(with the path adjusted to point to micronucleus instead of launcher?)
I guess that wouldn't get us the message telling the user to plug in their device (which is wrong anyway, should be "plug in or reset" or something like that, since almost every device I've found in the wild had a working reset button that could be used to upload, and plan for next release is versions of the bootloader built to support every plausible reset entry mode).... I wonder if there's a way to execute two commands, either in a single command invocation, or two two lines in platform.txt. One thing I noticed and found intensely annoying is that the output didn't seem to get forwarded to the IDE during the upload, or only got passed in big chunks, so you don't see it as it's happening. I'm wondering if it's just because of the redirection that that happened....

@pcfreak1201
Copy link

pcfreak1201 commented Oct 19, 2020

Cool! Any yes, removing the launcher would be nice. But the Digispark hasn't a reset button!
Yesterday, I found a notebook with dual-boot and Windows 7 on it. So I tried to use my built micronucleus.exe but it doesn't do anything (may be a problem with detection) - only starts. (I will repeat it with sandbegs binary.)

I also had a quick search for getting the actual libusb environment running for windows at my linux. And nope, I won't go down this rabbit hole. To start with quite well supported souces I think https://github.com/msys2 (only for LibUSB support on Windows!) looks very "promising"...
Actually there seems to be support for Windows 7 to 10, x64 and x86 (and Mac, Android, Linux) but it seems to vary how to work with it. I'm a freak, not a specialist ;-)
Ah, and during search, I also found that avrdude and the avr toolchain for windows also bases on MinGW - and avrdude needs LibUSB - may be another point to start?

@SpenceKonde
Copy link
Owner

The official digispark? Yeah that one may not have a reset button.

99% of the clones do, and don't come with reset disabled... so if we do need to keep the script for launcher, the text printed should reflect the fact that the correct course of action might be to reset it, it might be to plug it in, it might be to plug it in with a specific pin held low, and so on.....

And, somehow looking at it now, i see that his launcher script does indeed say plug in or reset.... how did I not see that last time, lol

@sanbeg
Copy link
Author

sanbeg commented Oct 20, 2020

Oh! That's wonderful...

And wait, the whole purpose of launcher is just to rewrite the arguments passed to micronucleus executable?

Couldn't we just do that in platform.txt and do away with it entirely?

Yeah, I think digistump designed it to pass through to avrdude, or to translate if you speci -c digispark. There's definitely some historical baggage there that could be cleaned up.

Something like...
tools.micronucleus.upload.pattern="{cmd.path}" --type intel-hex --timeout 60 {build.path}/{build.project_name}.hex

That should work. --type intel-hex is the default, so we could omit that. we do want the --no-ansi --run options, though.

(with the path adjusted to point to micronucleus instead of launcher?)
I guess that wouldn't get us the message telling the user to plug in their device (which is wrong anyway, should be "plug in or reset" or something like that, since almost every device I've found in the wild had a working reset button that could be used to upload, and plan for next release is versions of the bootloader built to support every plausible reset entry mode).... I wonder if there's a way to execute two commands, either in a single command invocation, or two two lines in platform.txt. One thing I noticed and found intensely annoying is that the output didn't seem to get forwarded to the IDE during the upload, or only got passed in big chunks, so you don't see it as it's happening. I'm wondering if it's just because of the redirection that that happened....

I think the message in Digistump's binary is specific for their digisparks, which I believe had the reset pin disabled. I reworded that in my script because I usually breadboard a reset button onto the clones I've been using.

Ideally, micronucleus could emit the message at the right time, but if it doesn't I don't think the launcher script is a terrible approach. We could also simplify it if we don't need to translate avrdude options and either pass through its options directly, or just hard code most options, and just pass in the filename.

I actually hadn't noticed the issue with the data coming in chunks. If that's due to the binary buffering the output incorrectly, it should go away if we move to a script.

@stonehippo
Copy link

@sanbeg If you’re getting all of these built, any chance you can do arm, too? If so, it’s fix the issue I opened (#477).

I’m down to help get them built, too.

@stonehippo
Copy link

stonehippo commented Nov 17, 2020

Well, that was dumb. After going back up the thread here, it turns out building micronucleus 2.04 on ARMv6/ARMv7 was trivial. I’ve put the it up at https://github.com/stonehippo/micronucleus-commandline-arm-builds if that will help move things along.

@SpenceKonde
Copy link
Owner

@stonehippo Would you mind specifying:
What source tree you built from?
What command was run to build it?
Is this arm-linux-gnueabihf or aarch64-linux-gnu ? (gee, such clear names, huh?)

@SpenceKonde
Copy link
Owner

And, I would disagree that it was "dumb" - if people either suffered in silence, or built it themselves, we'd never end up with an automatically installable binary for board manager installation!

@stonehippo
Copy link

I've got the tool compiling under Msys2/MinGW64, but it won't compile statically due to a lib incompatibility. Can't see why, yet, but will keep at it.

@ArminJo
Copy link
Contributor

ArminJo commented Feb 4, 2021

@SpenceKonde
I just packaged a new version with the new micronucleus binaries for all platforms - not requiring launcher any more, see platform.txt 😀- and new bootloaders.
Now it's your turn...

@SpenceKonde
Copy link
Owner

Siiiiiiick!

Need to finish DxCore 1.3.1 and get that released which I hope can happen today/tomorrow (test that changing TCA PORTMUX correctly moves analogWrite pins and that the TCD PWM pins act like the PWM pins on the ATtiny x7 of ATTinyCore now, and clean up variants for the TCA change, propagate flash-write changes from DA to DB and make at least one AVR128DB board I have run and test Opamp.h on it).and then can finally come back to ATTinyCore

@SpenceKonde
Copy link
Owner

You just had to delete the old file didn't you?

Good job - now nobody can install ATTinyCore

@SpenceKonde
Copy link
Owner

SpenceKonde/ReleaseScripts@19300de#diff-dba3a4c259e313a72ba4b704f3db5319f26fac51bde77dfa545d5acd993b7754 2.0a4 windows (same version) rehosted to restore functionality.

@ArminJo
Copy link
Contributor

ArminJo commented Feb 5, 2021

Good Morning!
Oh sorry sorry!!!
I changed my own json to point to the original digispark win zip for the 2.04 version, since this is more transparent.
I was not aware that you linked to the file. I restored it now.
Tell me when it is save to remove.

Sorry
Armin

@sanbeg
Copy link
Author

sanbeg commented Feb 6, 2021

@SpenceKonde
I just packaged a new version with the new micronucleus binaries for all platforms - not requiring launcher any more, see platform.txt - and new bootloaders.
Now it's your turn...

I just noticed that the PR to fix the mirconucleus issue with the "please plug in" message being buffered was rejected. So it looks like we'll still need a launcher script to emit that message. micronucleus/micronucleus#193

@SpenceKonde
Copy link
Owner

SpenceKonde commented Feb 6, 2021

Oh, I found a clean copy - but I only had one copy of that one. Like 4 copies of the one from the official digispark repo.

wait, you were using the digispark version? But it's not the same! one of the files - I think the .inf - is slightly different. I was under the impression that yours fixed something, which is why I used yours for windows instead of the old digispark one. But I think it was relating to the digispark USB libraries, I think the CDC one, but that one never worked for me no matter what I did! (tried both installs many times. )

Were you ever able to make the libraries work? I never was able to make any USB libraries work, which is precisely why I don't include those libraries with ATTinyCore. I got some help from someone on FB - but then life got crazy.

What I was going to do is 1) geta known working hex file and from there, I should be able to work on the driver issue until I have it working on my test systems. Then, I could work on fixing the libraries so I could build my own working ones (i'd have the advantage of being able to examine the hex files of a working and non-working build, too. Maybe I'd find that if I went back enough compiler versions, finally it would work, and I'd see what the difference was and be able to prevent it from happening. (and I could dump a .lst of both versions too)..

@ArminJo
Copy link
Contributor

ArminJo commented Feb 6, 2021

@SpenceKonde I never did a fresh install on windows for the micronucleus USB driver in the last year. Maybe I should test it again.
And I can check if the V-USB examples are still running, I never used them personally, only for issue fixing.
After this you will get feedback.

@SpenceKonde
Copy link
Owner

@SpenceKonde I never did a fresh install on windows for the micronucleus USB driver in the last year. Maybe I should test it again.

Please let me know how to set it up if you manage this - I have always really wanted to be able to make thee into USB devices! It would open so many doors!

@nerdralph
Copy link

@SpenceKonde I never did a fresh install on windows for the micronucleus USB driver in the last year. Maybe I should test it again.

Please let me know how to set it up if you manage this - I have always really wanted to be able to make thee into USB devices! It would open so many doors!

And it will have you pulling out your hair. V-USB is by no means robust. Even micronucleus isn't as robust as it could be, and that's a lot easier to do when you have 100% control of the CPU and interrupts. For example I found a bug in Tim's OSCCAL tuning code. It seems the USB hubs he tested on were more like a USB switch and didn't broadcast traffic to every port on the hub after they were enumerated. On the PC I was using, with a common Intel chipset USB EHCI, the interrupt IN tokens for my low-speed mouse get broadcast to every port on the same hub, which was sometimes throwing off the tuning.
This is one of the things I worked on fixing for MN:
https://github.com/nerdralph/micronucleus-firmware/blob/asm/firmware/osccal.S

Then on top of the software hacks, you have the common hardware hack of running the AVR at 5V, and clipping the D+/D- signals with zeners. Even if you get the "right" zeners, you've reduced your signal margins since the host outputs 3V3 (> 2.7V from what I recall of the USB spec), and the AVR thinks 5V is high. VIH for most AVRs is between 2.6V and 2.8V with 5V power. If you run the AVR at 3V3, VIH is between 1.7 and 1.9V, making things much easier.

If you want to make a USB project that other people can build and use on Windows, Linux, MACs on their various hardware platforms, it's WAY easier with a MCU that has a hardware USB SIE. For around $1 you can get an ARM Cortex-M MCU with a full-speed USB peripheral, or for less than $3 you can get a tube of 10 8051-based CH551G MCUs.

@SpenceKonde
Copy link
Owner

SpenceKonde commented Feb 26, 2021

BLOODY HELL! There are at least three distinct versions of each of the micronucleus tool files, with the same name, which differ by 20-50 bytes in size and have a different SHA hash.

Sorting that fucked up mess out is what's between here and 1.5.0. Should be ready tonight... and they are all getting repacked, WITH A UNIQUE NAME

I just went through generating SHA-256 hashes and file sizes for the 7 archives. because they didn't match what was in the archive. Then I looked at what I had on the webserver.... they're all different too! It looks like every place that the 7 packages are located, they were compressed in a slightly different way, yielding an ever-so-slightly different archive, and because the size is almost identical and the names are the same,

@SpenceKonde
Copy link
Owner

SpenceKonde commented Feb 26, 2021

And the digiusb drivers for windows are missing from the archives too - we only have the digiserial (CDC) drivers, which it has been clearly established do not work on post-XP windows or recent versions of the linux kernel... instead of the DigiUSB drivers they were meant to replace, but which still work whereas the CDC ones do not.... I don't know why we have the drivers that are known to not work, and don't have the drivers that do - I generally try for the other way around....

@SpenceKonde
Copy link
Owner

SpenceKonde commented Feb 27, 2021

Okay, 1.5.0 is getting:
For all linux: Repacked micronucleus binaries (just renamed and with a readmneto make these distinct from the other near identical archives that exist...

For all windows: Repacked micronucleus binaries, with the files related to DigiUSB and DigiX, and a post_install.bat file. We know that Arduino will not run that file, but it will print a warning mentioning that - specifically, mentioning the file path. So an enterprising user could track it down and run it to install the drivers.

My interpretation of the doom gloom and begging for salvation on other forums about the serial drivers is that:
digiCDC is dead. It relied on a hackjob that stopped working almost as soon as it was implemented.
digiCDC is deader than a doornail in the bottom of dracula's coffin. The hackjob probably never should have worked, even on XP.
forbidden eldritch magic would be needed to return it to the realm of the living. (anyone have experience writing windows drivers? crickets chirp)
All who wield such magic lost interest in digispark when the crowdfunding buzz passed, and haven't thought about it in years, unless they suffer from PTSD from their time working on it, which (based on comments about what it's like working on a bitbang USB implementation in one of the other topics) would not be entirely surprising. In any event, we should expect no help from anyone of that caliber, so it falls to mediocre minds like mine to try to chart a course to an approximation of serial, as that's the most plausible, flexible, and familiar interface for two-way communication of arbitrary data with an application on the host system.

However, all is not lost, as there are widespread reports of DigiUSB continuing to work fine. Provided that can be confirmed, this leaves us in a worse place than if we had CDC, by far - but compared to having neither of them, it is greatly preferable; while it requires custom software, such software can be written in python and examples appear to be available.

My interpretation of the body of the files is that DigiX is related to DigiCDC.
Thus, IF observations bear out my interpretation of available information above with 1.5.0, I will remove DigiX and DigiCDC drivers from the 2.0.0 release, include the DigiUSB library and not the non-functional CDC one. If evidence contradicts my interpretation above, plans will be adjusted accordingly. Goal is for known working libraries to be included, and known non-working libraries to not be included :-)
1.5.0 is an interim release, and therefor will include all the drivers so we can bang on this shit and see what works and what doesn't, so by 2.0.0, we will know what does and does not work.

@SpenceKonde
Copy link
Owner

SpenceKonde commented Feb 27, 2021

Goddamn, they have made the errors it throws over post_install.bat much nastier, can't release it like that. Ugh - why do they do that? It doesn't provide any added security! when you install a core you'd better trust the author. Like any piece of software you install on a windows PC! Nevermind, the awful error doesn't reproduce, and we get the behavior I wanted: the message:

Warning: non trusted contribution, skipping script execution (C:\arduino-1.8.13-portable\portable\packages\ATTinyCore\tools\micronucleus\2.5\micronucleus-2.5\post_install.bat) - 

gives users a path to follow to trace down the drivers. Maybe in future version I could even split the drivers off into an extra dependancy, so it could look like:

Warning: non trusted contribution, skipping script execution (C:\arduino-1.8.13-portable\portable\packages\ATTinyCore\tools\micronucleus_drivers\1.0\run_post_install_bat\post_install.bat)

@SpenceKonde
Copy link
Owner

Raarrrrrgghhh! Okay! it's done!

@stonehippo
Copy link

Oh man, thanks for all of this @SpenceKonde! Looking forward to trying out 1.5.0 from a RPi (upgrade from 1.4 worked smoothly). Sorry I couldn't get the damn binaries all the way there for you.

@per1234
Copy link
Contributor

per1234 commented Feb 27, 2021

@SpenceKonde while adding post_install.{bat,sh} support to Arduino CLI, the point was raised by Matthijs that this is really a pointless restriction, since the user is just about to run arbitrary executables from the platform anyway. So post install script now works for 3rd party boards platforms when the user is using Arduino CLI or Arduino Pro IDE.

This will also be applied to Arduino IDE as soon as the Boards Manager business code in the IDE code base is removed and replaced with a call to Arduino CLI. That process has gradually been happening with all the non-GUI code in the IDE.

@SpenceKonde
Copy link
Owner

SpenceKonde commented Feb 27, 2021 via email

@nerdralph
Copy link

It looks like those MN builds are still using Tim's osccal tuning code.
https://github.com/ArminJo/micronucleus-firmware/blob/master/firmware/osccalASM.S

As I mentioned above, it has a subtle bug that will sometimes choose a bad OSCCAL value if there's another low-speed device such as a keyboard or mouse on the same root hub. It is likely to be quite frustrating to users who encounter it, as it will seem almost random since it is only a problem when one of the first couple frames is non-empty. Tim helped review and optimize my new tuning code, which in addition to fixing the bug is smaller than Tim's original asm version.

AFAIK the version of MN used by Eric on the DigiSpark didn't have this bug. I think that version used Christian's tuning code, which doesn't have this bug. I even think Christian's osctune.h code may be a better choice for a bootloader than my asm OSCCAL tuner. For a USB library, my code is better because it doesn't tie up timer0.
https://github.com/obdev/v-usb/tree/master/libs-device

@cpldcpu
Copy link

cpldcpu commented Feb 27, 2021

Hi Ralph,

indeed, we should include your excellent code. Can you submit a pull request to the MN repository?

Btw, the issue with the hub was fixed at some time, but I had to remove the code again, because of some timing issues with windows 10 afair. So far, there wwas not really a bug report regarding USB hubs that forward all traffic. It may be that they are not that widely spread.

@nerdralph
Copy link

indeed, we should include your excellent code. Can you submit a pull request to the MN repository?

I just opened a discussion about it.
micronucleus/micronucleus#216

Btw, the issue with the hub was fixed at some time, but I had to remove the code again, because of some timing issues with windows 10 afair. So far, there wwas not really a bug report regarding USB hubs that forward all traffic. It may be that they are not that widely spread.

I think you are underestimating the extent of the bug. If you look back our email communications from May of 2014, I was having problems with OSCCAL tuning back then, but wasn't able to nail down the specifics of the bug. It's only last year after getting a better understanding of the protocol and having a good logic analyzer and scope that I was able to confirm the bug. To identify the bug I had to:

  1. capture several USB enumeration sequences in order to identify the IN token for my USB low-speed mouse.
  2. capture a few failed enumerations with a logic analyzer and a high-speed scope. Since the enumeration failures are intermittent (and dependent upon the interrupt interval of the other low-speed device), this took dozens of captures.
  3. once I had a capture on my scope of a failed enumeration, I was able to measure the bit times transmitted by the AVR to see that they were way off 666.7ns/bit.
  4. I then ran the OSCCAL tuning code on an AVR with an external xtal, with a high-speed UART debug log function added to output the attempted OSCCAL value each time through the loop.

Since the bug is intermittent (around 10-20% of the time from my experiments), many people that encountered it may have been OK with the simple unplug and re-connect "solution".

@SpenceKonde
Copy link
Owner

What I never understood is why MN needs to re-tune every time, rather than saving the correct tuning value... could even achieve that with one of the "updater" type sketches. You've got all the flash you need to write a really careful funer, cause you're gonna delete it, then it rewrites the last page of the flash (where the bootloader resides), adding the tuning value it determined, and you upload another sketch over it, and every time it starts, MN uses that saved OSCCAL value... As long as it was being used indoors from a USB port that was outputting ~5v,, there's like a 0.3v and 10 degree C window it's gotta work in... I don;t think the dependence on T or V is that strong, is it? why would we have to re-tune it? having to re-tune every start makes the whole thing way more demanding, because it means the tuner needs to be built for size and speed since it is eating up space that the sketch could be occupying and needs to be able to tune off of USB and right away be able to communicate over USB.

@nerdralph
Copy link

nerdralph commented Feb 27, 2021 via email

@cpldcpu
Copy link

cpldcpu commented Feb 28, 2021

Have you checked if this fixes the bug?

https://github.com/micronucleus/micronucleus/blob/29317ea0c5b55b704a42ad318c52945cf3e2359e/firmware/osccalASM.S#L150

As stated there, it was in the codebase for a long time, but caused MN to fail when Win 10 came out. That's why I removed it, because few people actually use USB 1.1 hubs and my expectation was that this would decline further over time.

Your new solution is much better because it does not extend the osccal duration so it should still work with Windows 10. So if we can include that everything should be fine.

@SpenceKonde Well, you can activate saving of oscall in the source.

The oscillator frequency changes with temperature and voltage (there can be quite some different in actual USB voltage) and you never know how marginal you are if you only save the value once. To do this properly you would have to find a way to define when this perfact osccal value is found and only save it then. Surely doable but it's quite some work to do this properly, it may add complexity to the usability of micronucleus and we may encounter new bugs along the way.

@nerdralph
Copy link

Have you checked if this fixes the bug?

https://github.com/micronucleus/micronucleus/blob/29317ea0c5b55b704a42ad318c52945cf3e2359e/firmware/osccalASM.S#L150

As stated there, it was in the codebase for a long time, but caused MN to fail when Win 10 came out. That's why I removed it, because few people actually use USB 1.1 hubs and my expectation was that this would decline further over time.

The issue is not limited to USB 1.1 hubs. I think I mentioned in one of the MN issues that I debugged the issue using a PC with an Intel chipset and USB 2.0 high-speed (i.e. 480Mbps) EHCI hubs. It only happens when there is traffic for a 2nd low-speed device on the same hub. In my case it was when I had my mouse plugged into the same hub. Since some ports are routed to the back of the PC case and some are routed to the front, it took some time to realize that the intermittent enumeration problems were primarily happening with certain ports. Years ago I was using genuine DigiSpark devices (with RSTDSBL programmed), and therefore had to unplug and re-connect the digispark to force a re-enumeration. I now know there was a 50/50 chance I'd plug it into a port on the same hub as my mouse. So this was a very tricky bug to track down, taking much more work than your typical MN and/or V-USB user is going to bother with.

USB hubs isolate traffic for different speed devices, so if a device running MN is plugged into the same port as a high-speed flash drive, the AVR will never see any of that traffic. In fact, if you could count on there being no other USB traffic on the port, you could eliminate the code in V-USB for handling the 7-bit USB device address.

As for Win 10, someone else will have to test that. I've tested on Win7 & Linux. I had Win10 installed on a machine a few years ago, but the motherboard died. When Win10 refused to run on the replacement motherboard (something to do with the license key), I just wiped the disk and installed Ubuntu.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants