Add experimental support for Pi 2/3 emulation

[JamesReynolds]

Support for extracting the kernel and dtb files from an SD card image to test more of an image's boot process.

1. Added 'fatcat' to extract from the boot filesystem from a standard RPI image
2. Added qemu-system-aarch64 to support raspi3 machines
3. Extracted images to search for kernel7/8.img and correct dtb files to boot kernel
4. Disabled nic for raspi2/3 as USB is not supported (yet)

Running a VM now has an optional machine target flag:

Old behaviour still supported, defaults to versatilepb with a NIC:
docker run -it -v.../buster-lite.img:/sdcard/filesystem.img lukechilds/dockerpi:vm

New behaviour, raspi2 and raspi3 do not have a NIC (its not yet supported):
docker run -it -v.../buster-lite.img:/sdcard/filesystem.img lukechilds/dockerpi:vm versatilepb
docker run -it -v.../buster-lite.img:/sdcard/filesystem.img lukechilds/dockerpi:vm raspi2
docker run -it -v.../buster-lite.img:/sdcard/filesystem.img lukechilds/dockerpi:vm raspi3

I've not updated README.md, if you are interested in this PR I can add some documentation.

Tested on raspian-lite, happy to test on others.

[aaronpropst]

Cool! I fired it up and got a pi 3.

pi@raspberrypi:~$ cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 125.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
BogoMIPS	: 125.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
BogoMIPS	: 125.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
BogoMIPS	: 125.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

Hardware	: BCM2835
Model		: Raspberry Pi 3 Model B+

Thanks for figuring that out @JamesReynolds

[lukechilds]

Thanks @JamesReynolds and thanks @aaronpropst for testing!

I've got family and friends visiting at the moment and I'm about to move after they leave so I'm a bit busy at the moment.

Really excited to take a look at this and review it when I'm available though.

[lukechilds]

Ok, I got some time to play around with this, really nice work @JamesReynolds!

Something interesting I noticed, Pi 1 has better single threaded perf than Pi 3. But then once you bench multiple threads simultaneously the Pi 3 is significantly faster as expected.

I guess maybe single threaded perf is better on the Pi 1 because the CPU is less complex so more efficiently virtualised?

Some random python benchmark I threw together quickly:

	1 Thread	4 Threads
Pi 1	0m42.746s	2m41.986s
Pi 3	0m46.143s	0m59.837s

I have a few questions @JamesReynolds.

Do you know how much work it is to get networking working on Pi 2/3?

Disabled nic for raspi2/3 as USB is not supported (yet)

Is this something that's planned and already being worked on currently in QEMU?

If not I might just merge this as-is and say in the readme that experimental Pi 2/3 support is merged and link to this PR for more details.

But if it's possible to get it working then I'd love to have Pi 3 as the default vm and have the users opt in to Pi 1/2.

Can we support Pi 4 or at least a Pi 4 like system?

I think the main difference between Pi 3 and Pi4 is Cortex A53/A72 CPU and 1GB/4GB RAM.

According to qemu-system-aarch64 -machine raspi3 --cpu help Cortex A72 virtualisation is supported.

I also tried booting the raspi3 machine with 4GB RAM but just got an error saying memory size was too large for that machine.

Would it be simple to modify the QEMU raspi3 machine to allow Cortex A72 and 4GB of RAM so we can get a Pi 4 like system?

Thanks!

[JamesReynolds]

@lukechilds Thanks!

I think you could be right about the single-threaded performance, perhaps having extra cores - even if only one is used - adds significant emulation overhead?

Do you know how much work it is to get networking working on Pi 2/3?

Yes... quite a bit of work unfortunately. I've had a fair bash at this as the latest patch sets for QEMU don't provide networking support yet. I've found a potential solution that would create a tunneling adapter over the second serial port, but unfortunately getting both serial ports working simultaneously is proving hard as well. I think there is active work here, but it sounds like the full datasheets for the SOC are hard to come by and there is a lot of guess work involved. The serial port route seems doable but a bit hacky and full networking is going to require some serious legwork by someone who knows QEMU inside-out.

Can we support Pi 4 or at least a Pi 4 like system?

This might be possible soon, as patches are in the works:
https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg00681.html

We could probably pull in that patch now to have a reasonably functional raspi4 system before they fully land in QEMU. The raspi3 definition is locked to under 1GB.

[aaronpropst]

If I have a physical usb network dongle could I instruct a qemu pi to use it? When you say usb is not supported yet, does that mean I couldn't attach any devices to the pi at all?

[lukechilds]

@JamesReynolds sorry for the delayed response.

That's a shame about networking. Ok, I'll just merge this in it's current state and then link people to this PR for docs mentioning that networking isn't yet working.

I'd like to make a few (minor) changes but I don't seem to have permission to push to your PR branch. Do you have the allow edits from maintainers checkbox checked in the PR?

I'm happy to make the changes if you can enable that. Or if not, I've highlighted the changes I'd like to make before merging in this review.

Also, one more thing I noticed. sudo poweroff correctly shuts down the machine and ends the process on versatilepb. However with the raspi* machines sudo poweroff ends up trying (and failing) to reboot and just hangs the QEMU process so the Docker container never finishes running. I have to manually kill it with docker kill. Did you notice this? Any way around it?

Thanks!

[JamesReynolds]

@lukechilds shall I make these changes tomorrow and submit a new (squashed?) PR?

[lukechilds]

@JamesReynolds if you could make the changes in this PR branch that would be great, don't worry about noisy commits.

I'll squash everything into a single commit when merging. Then I can link to a single PR in the commit message for all the context.

Thanks!

[JamesReynolds]

@lukechilds I've made the changes, and also noticed I didn't reply to your poweroff query. I've noticed this too - but I've not yet found a solution. I think there is a solution in the latest patch set, but I'm not sure. My current focus is using this for testing my image creation tools - once I'm done there I'll see where the QEMU guys have got to.

[lukechilds]

Thanks so much for this @JamesReynolds!

Got this merged in and made a few little tweaks: 7a3c875

There were a few people on Twitter asking about modern Pi support so I was gonna tweet about this. Do you have a Twitter handle so I can give you credit?

I think there is a solution in the latest patch set, but I'm not sure.

No worries, I'm not too worried about the shutdown hanging issue as I've clearly labelled this as experimental support for now.

[JamesReynolds]

@lukechilds Woooo! Amazingly, I'm not actually a twitter user... I'll put creating an account on my ever growing list of stuff!

[AFDudley]

Has anyone gotten networking functional yet?

[lukechilds]

@AFDudley Not that I'm aware of. We need to wait for it to be supported upstream in QEMU.

@JamesReynolds do you know if an issue/PR/mailing list thread exists that we can follow for progress on this?

[hardillb]

Potential good news on the USB support https://bugs.launchpad.net/qemu/+bug/1772165/comments/19

[david-mohr]

Looks like networking is now active for raspi3!

https://stackoverflow.com/a/64420363/3463019

[hardillb]

@david-mohr look at pull request #21

[lukechilds]

Yeah apologies @hardillb on the delay for review. I've been very busy recently but it's top of my list when I have some free time.