Image-builder produces stable 2.0 image

[RespawnDespair]

Fix all the known problems in the image builder so we have a stable and usable 1.6 image.

https://github.com/RespawnDespair/wifibroadcast-image-builder/projects/1

[pilotnbr1]

@RespawnDespair it’s looking good! Only a few more todo items left! How is it going? I can probably help do some testing soon.

I was just curious, and I am not second guessing you, why the choice of Raspian for the buildroot? Was it just to make it easier to get buildroot working? I believe Rodizio was hoping for some flavor of Linux. I am just trying to understand the pros and cons.

[RespawnDespair]

It's looking good now, as i said on my repository, i had some difficulties with the OSD. These seem to be fixed now. I'll put a new image up sometime today or tomorrow.

[bortek]

@pilotnbr1 about buildroot, I guess it must be easiest to build on the same platform where you going to use the image thus raspbian. Building on Ubuntu or other Linux probably needs all the to make the build working.

[RespawnDespair]

@pilotnbr1 @bortek
I try to avoid the term 'buildroot', this is a separate system which i tried first. 😄

I use Raspbian LITE, and then add all the components we need for the system to work. This leaves some extra's we don't need, but less than when using normal Raspbian.

I choose this route because the entire system depends heavily on some things raspbian-specific.

[pilotnbr1]

That makes sense. Are you using yocto? Just curious.. ;)

And all this is an image that does not support the pi3b+ yet?

[RespawnDespair]

No yocto, that would mean first recreating raspbian and then adding on top. Maybe in the future 😄

And no, i need to create the kernel8 image in the buildscript to support the 64 bit Pi3B+. Which i will do once all bugs are sorted. Also i don't have a 3B+ for testing so i would have to rely on others to do that for me.

[pilotnbr1]

@RespawnDespair Jelle thanks for the effort! Also thanks for making it easy to follow your progress on your repo, very organized!

[Yes21]

@pilotnbr1 : I have already build many times the image using @RespawnDespair script. It works perfectly with Ubuntu 18.04 following the instructions of his github.
For me it took 26 minutes, but I have a big pc (Core I7-8700K / 32 Gb ram)

[dposea]

I had one issue with the raspbian lite image. Also, it's hard to restart the build if it fails. That will just take some tweaking. If I can offer a suggestion write a last step completed message to a file, and use a restart flag. If restart, get last step and start at the next one.

[dposea]

I think I need a new image. I am having many problems with the scripts. These may already be fixed so I'm waiting on a new image to test and then I can create issues.

[dposea]

If we don't already have one I can build a Mavlink 2.0 library.

[Yes21]

@ALL

@RespawnDespair is making hard work with his script to make the new 1.6 working.
We need help to test/debug the last images.

Many things had been fixed now, but I can't have video/telemetry with CSL300 (rt5572) on the airPi. Could anyone confirm that ? Could you try with other boards ?

[htcohio]

@Yes21 I was unable to get the ground side to boot with respawndespairs latest highly experimental image with pi3b+.

[Yes21]

@ALL
@RespawnDespair did make big progresses :

• mavlink2
• rtl8812au
• RPI 3 B+

Also read the new Readme.

He will release a new test image a.s.a.p.

I am building myself the new image ...

[careyer]

Awesome work! Can't wait to test :-) Von meinem iPhone gesendet

…

Am 30.10.2018 um 18:51 schrieb Yes21 ***@***.***>: @ALL @RespawnDespair did make big progresses : mavlink2 rtl8812au RPI 3 B+ He will release a new test image a.s.a.p. I am building myself the new image ... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[RespawnDespair]

@Yes21 Please use the staged branch of the image-builder. Havent tested it on a Pi yet. Baking a new image now for testing.

[Yes21]

@RespawnDespair
Of course I used :

git clone -b staged https://github.com/RespawnDespair/wifibroadcast-image-builder.git

[RespawnDespair]

Okay, very good, please let me know the results!

[Yes21]

Ok, will tell you.
My racing Ubuntu PC is working hard ;-)

[RespawnDespair]

It takes nearly two hours for a full run on my Hyper-V (don't have a dedicated Ubuntu machine yet).
If you read the readme you'll know that a full rebuild will hopefully be something of the past 😄

[pilotnbr1]

Nice work! Wish I could try.. I have a brand new 3b+ at home but i am travelling next 3 weeks :/

[RespawnDespair]

I'm getting a Pi3B+ myself so i'm not relying on others to test.
By the time you get back we'll be working on 1.7 😄

[Yes21]

Yes, I understand now why you did "cut" the script into small feet.
Great idea !! Bravo ! smile

[Yes21]

@RespawnDespair : the work/build.log is also a great thing !!!

[RespawnDespair]

It does how the amount of time each step took nicely.
I'll probably have enough to fix in the coming days, but i'm really happy with the way it turned out.
Hopefully it will allow me to make fixes easily so we can get to 1.6 in a short while.
I'm also working on a test 'document' that will specify all things we did and didn't test with the release of a new image, but that's another discussion. It's still compiling the kernel here, so it'll be a while yet before i can test the image...

[RespawnDespair]

@Yes21 Be aware that i haven't implemented the last step. It does not copy the image to the deploy folder.
When it's done you can take the IMAGE.img from the work/04-Wifibroadcast folder.

[Yes21]

I just had a blue screen for "wireshark". I answered "Yes", and then an error message :

Setting capabilities for dumpcap failed
The attempt to use Linux capabilities to grant packet-capturing
privileges to the dumpcap binary failed. Instead, it has had the
set-user-id bit set.

Can't select "Ok". I'm blocked !

I will try to restart from "stages/03-Packages/00-run-chroot.sh" which is the last step from the log.

[RespawnDespair]

And yes, the interrupt problem should be fixed, but that might be a little outside our possibilities.
I don't know where to start looking so unless somebody else has suggestions...

[careyer]

Alright Jelle, I will be happy to send you the code this evening. Currently I am at work and don't have access to it. Can I mail it somewhere? Cheers!

[RespawnDespair]

Just attach it to a post here (drag the file), that way i will have reference.

[rodizio1]

just noticed there is a new Raspbian version 2018-11-13

http://downloads.raspberrypi.org/raspbian/release_notes.txt
https://downloads.raspberrypi.org/raspbian_lite_latest

[careyer]

@RespawnDespair : Jelle, here you go...
rctx.zip

Don't laugh at me: It is very simple. After all I check the analogue channels (Ch0-5) for invalid values (PWM==1000 which cannot be reached with the sticks and results from reading the empty joystick registers when the interrupt conflict occurs. This results in a raw value of =-32767 read from the joystick which is then mapped to PWM 1000 by the joytsick routine).
Jitter in the On/Off channels (Buttons) is mitigated by judging a value valid only if it is three times the same in a row when reading the joystick values.

It is a very crude mitigation but it works just nicely.

Cheers!

[DaddyK00l]

Hello,

It would be nice that builder use latest raspbian stretch image, so I can test with new Pi 3A+.

[Yes21]

It would be nice that builder use latest raspbian stretch image, so I can test with new Pi 3A+.

Doesn't it work with raspbian_lite-2018-10-11 ?

[DaddyK00l]

Yes you're right, I thought it could be a problem but it's booting anyway.

[Yes21]

The new Pi3A+ should work like the Pi3B+ ...

[DaddyK00l]

Yes that's what I thought too but for some reason I couldn't get it to boot with an image which was working on 3B+ so I need to make some tests again to understand what's going on.

[DaddyK00l]

Image from builder is booting and working on Pi 3A+. Strange thing is image from Lelik (russian forum) which is a modified 1.6 RC6 is working with Pi 3B+, and not working with Pi 3A+ (no boot).

[careyer]

I digged a bit deeper regarding the strange USB problems (Interrupt?) with Raspberry Pi. There are not many people arround using several high bandwidth USB devices all in parallel on the RPi. But the problem is very well known among bitcoin miners. Those people tend to plug a high number of ASIC miners via USB to their mining plattform and soon found that the Broadcom SOC has some serious USB problems when it comes to handling multiple USB devices:
https://bitcointalk.org/index.php?topic=118669.0 (preferably read the 2nd half of that page)

Problems are said to get very obvious with about 8 parallel USB devices:

1. Ethernet (internally connected to USB)
2. WiFi Adapter 1
3. WiFi Adapter 2
4. WiFi Adapter 3
5. USB-Joystick
6. Touch Screen (Offical 7" RPi Screen)
7. USB Hub
8. ...(not sure how BT and internal WiFi are connected)

Seems like we are pretty much working on the limit of what the RPi platform is capable of in terms of USB.

[rodizio1]

Yes, the Pi USB system has it's fair share of issues. Guess why I always said to not upgrade the kernel or firmware or change kernel or cmdline.txt settings without a good reason ...

Problem is, it's not a "full" USB implementation, as the SoC the Pi is using was once intended for some mobile phone which don't need several parallel devices etc. Some things are implemented in software, both inside the rpi firmware and the linux kernel.

Back in 2012 (the time of the above linked bitcoin thread) it was way worse though, since then, a lot of stuff has been completely re-written and improved (there are lots of threads and posts about usb issues on the Raspberry forum and Github issues pages).

The other problem (only affecting Atheros cards though) is that endpoint problem described here (which results in the 3 card limitation and the higher jitter the packets experience on their way from card to userspace): https://github.com/qca/open-ath9k-htc-firmware/wiki/usb-related-issues.

Regarding the Pi3A+: It's not strange that an image that supports the 3B+ doesn't support the 3A+, it's a different model. The 3A+ is supported since Raspbian 2018-10-09, see the changelog here: https://downloads.raspberrypi.org/raspbian/release_notes.txt

I have also tested the Pi3A+ with my own image (based on 1.6.RC6) with Kernel 4.14.79 and firmware from Raspbian 2018-11-13. Doesn't work correctly, after maybe 5-10 minutes it starts producing badblocks, the OSD freezes, the image freezes, sometimes the screen turns completely black comes back, totally erratic behaviour. Not sure what the cause is, did not connect via Ethernet yet to look at kernel logs etc.

[Yes21]

@DaddyK00l

It would be nice that builder use latest raspbian stretch image, so I can test with new Pi 3A+.

If you want to build an image with the latest kernel, you only need to modify the "config" file :

# Output image name
IMG_NAME=EZWFB

# Download location base image
BASE_IMAGE_URL="http://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-11-15"

# File name base image
BASE_IMAGE="2018-11-13-raspbian-stretch-lite"

# Amount of cores to use for compilation on host machine
J_CORES=12

# Git SHA-1 hash for specific kernel
GIT_KERNEL_SHA1="86e1b50e956098ab75042898b7c36c7be8d5c5cb"

[DaddyK00l]

Hello, yes I found that too, thanks. I'm building a test image with latest kernel to test.

[pilotnbr1]

So the wife is out of town for over a week and I have time off. What would be most helpful to you guys? Testing of the new image builder? Analysis of bad/lost blocks? Implement MSPv2?

I’ll attempt just about anything (and probably fail) so put me to work!

Have a new pi 3b+ and 3a+ ... only some 722s and 36nha . Rpi touchscreen

@RespawnDespair @Yes21 @rodizio1

[DaddyK00l]

Hello,

A good thing to work on should be fixing the broken atheros driver, I think. You'll easily see this testing Awus036NHA. But of course there are many other things to do.

Thanks and enjoy the time off :)

[Yes21]

@pilotnbr1 Many thanks for your proposal :-))

I would say that an interesting thing would be to make RC work with more chipsets than only AR9271. Perhaps with CSL300 (RT5572), AWUS036AC/H (RTL8812au) and AWUS1900 (RTL8814au).

I know you haven't any of these cards, but I could help you getting one of them, ... if you are interested in that subject ?

But if you are "free" for the next week, take also time to walk in the town, or in forests ...

[pilotnbr1]

Yes maybe I’ll order one or a couple of those cards..

[bortek]

If you won't find any of those activities attractive you could spend some time on testing new image builder. :) I wonder how is it going with @RespawnDespair

[dposea]

Has anyone had a problem with rtl88xx debug mb messages in the syslog (dmesg). There are some messages enabled I think by accident. I found at least one place with only:

#ifdef RTW_CONFIG_DEBUG

Y or N doesnt matter. This is only in 5.3.4. It took a while to get rid of the messages, so if it's an issue I can make respawn a patch to fix the driver source if air crack does t.

[pilotnbr1]

@careyer did good work noting the correlation between usb devices and dropped rc packets.

Glancing at @careyer rc code modification I have concerns with how it will behave in failsafe. Will it just continue to detect “corrupt” signals and thus repeat the last known good signal (3 inputs in agreement)? I would just feel better about it with some failsafe testing.. I know this is probably low on the list but just wanted to throw this out there.

[careyer]

Hi Luke, This is my very last message before WiFi runs out. Failsafe is tested with this code. It depends on either of the two things: a) joystick disconnect - will be detected immediately regardless of the mitigation b) no RC packets arrive at FC for a certain time - this is also independent from the mitigation fix because it will allways send packets: either regular or mitigated frames. If you run out off range packets get lost in the air and hence the FC will trigger failsafe. Regarding b) first I just discarded invalid frames at the transmitter which could lead to a situation where too many got discarded and failsafe would be triggered because FC had to wait for new packets to arrive. Hence I decided to introduce the mitigation strategy. 1) Send last good analogue stick position (IRQ problem) 2) make sure BUTTONS stay at least consistent over 3 cycles (Jitter+ IRQ) Have a good time. Von meinem iPhone gesendet

…

Am 25.11.2018 um 01:12 schrieb Luke ***@***.***>: @careyer did good work noting the correlation between usb devices and dropped rc packets. Glancing at @careyer rc code modification I have concerns with how it will behave in failsafe. Will it just continue to detect “corrupt” signals and thus repeat the last known good signal (3 inputs in agreement)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[dposea]

After reading more about the issues with USB and too many devices it doesn't seem like there will be a fix any time soon If you really need that many devices what about a PCI adapter for the pi? I'm 95% sure there is such a thing available and should be faster than USB as well. My laptop has a half size ath9k pci card that seems to work well in monitor mode. An adapter might be expensive though. It is worth thinking about.

[rodizio1]

RespawnDespair wrote:

Yes we should, also, we should determine the goal again since we are now 'separate' from the main EZWFB project.

New issue has been opened here: RespawnDespair/wifibroadcast-image-builder#93

Thus closing this one.

[maniranjanojha]

Hi Fellas,

I am looking for a stable Ez-wifibroascast 2.0 image.
I have already tested 1.6 RC6 but was facing an issue in route mavlink data over WFB because of some mavlink version issue (probably). I have read somewhere about WFB 2.0 beta.

Any help would be appreciated.