RPi3 CONFIG_VMSPLIT_3G

[peyo-hd]

For some applications, we need user/kernel address split of 3G/1G - rather than default 2G/2G.

Until RPi2, modifying kernel source's (github.com/raspberrypi/linux) bcm2709_defconfig
from CONFIG_VMSPLIT_2G to CONFIG_VMSPLIT_3G, and recompiling the kernel - works fine.

On RPi3, modifying it blocks kernel booting. With VMSPLIT_2G, kernel & OS are working fine.
But with VMSPLIT_3G, kernel shows nothing on HDMI output neither on serial console.

I have tried on a few branches - rpi-4.1.y / rpi-4.2.y / rpi-4.5.y - and results were same,
and used device tree files were bcm2709-rpi-2-b.dtb & bcm2710-rpi-3-b.dtb for matched RPi boards.

[pelwell]

With a 3G/1G split, the kernel virtual addresses can only cover 768MB (lowmem) because address space is needed for other areas such as the 240MB vmalloc area:

[    0.000000] Memory: 760276K/786432K available (6478K kernel code, 436K rwdata, 1732K rodata, 476K init, 775K bss, 17964K reserved, 8192K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc080cd64   (8212 kB)
[    0.000000]       .init : 0xc080d000 - 0xc0884000   ( 476 kB)
[    0.000000]       .data : 0xc0884000 - 0xc08f10e8   ( 437 kB)
[    0.000000]        .bss : 0xc08f4000 - 0xc09b5c24   ( 776 kB)

On Pi2 and Pi3 the top 16MB of RAM is not addressable by the ARM because the peripherals are mapped there, leaving 240MB of RAM that is physically addressable but can't be used without temporarily mapping it into the vmalloc or other area; this is what enabling highmem does. You could shrink the vmalloc area (and others), but the problem will remain to some extent.

The reason the kernel doesn't boot with a 3G/1G split is down to the placement of the initial Device Tree blob. It used to be located at 0x100, but this limits it to (0x4000 - 0x100 = 0x3f00) bytes (about 16KB), and some users have already exceeded this limit with sizeable (but reasonable) overlays. One of the first things the kernel does is "unflatten" the DTB, building internal data structures that are more convenient to work with. After this point, the source memory is free to be used for something else. I changed the firmware logic to place the DTB at the top of memory (just below the space reserved for the GPU and the initrd/initramfs), which works well with the 2G/2G split but can be too high for 3G/1G - the kernel faults trying to copy it.

The next (or next but one) firmware release will place a 768MB cap on the initial DTB (and initrd/initramfs) placement, avoiding this booting problem. Until then you achieve the same result by setting gpu_mem=240 (or higher), although this reduces the total memory usable by the ARM.

[pelwell]

By the way, the Pi2 is no different to the Pi3 in this respect, except that you will see "Uncompressing Linux... done, booting the kernel." on the UART before it crashes silently. Do you have a larger gpu_mem on your Pi2 card?

[peyo-hd]

On my configuration I used gpu_mem=256. This might have enabled RPi2 operation.
I am using same gpu_mem=256 on RPi3, but it does not boot up. So your temporal workaround seems not working on RPi3.

[pelwell]

It works for me on a Pi3.

My config.txt:

gpu_mem=240
enable_uart=1

My boot log start:

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.4.6-v7+ (phil@buildbot) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03) ) #324 SMP Wed Apr 6 09:44:28 BST 2016
[    0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine model: Raspberry Pi 3 Model B Rev 1.2
[    0.000000] cma: Reserved 8 MiB at 0x2f400000
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] On node 0 totalpages: 196608
[    0.000000] free_area_init_node: node 0, pgdat c08e7dc0, node_mem_map eed40000
[    0.000000]   Normal zone: 1728 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 196608 pages, LIFO batch:31
[    0.000000] [bcm2709_smp_init_cpus] enter (9520->f3003010)
[    0.000000] [bcm2709_smp_init_cpus] ncores=4
[    0.000000] PERCPU: Embedded 13 pages/cpu @effa8000 s22528 r8192 d22528 u53248
[    0.000000] pcpu-alloc: s22528 r8192 d22528 u53248 alloc=13*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 194880
[    0.000000] Kernel command line: 8250.nr_uarts=1 dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2709.boardrev=0xa02082 bcm2709.serial=0x8f6bc316 smsc95xx.macaddr=B8:27:EB:6B:C3:16 bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000  dwc_otg.lpm_enable=0 console=ttyS0,115200 kgdboc=ttyS0,115200 console=tty1 ip=dhcp root=/dev/nfs nfsroot=10.3.14.8:/home/phil/pi/root,udp,v3 rw rootfstype=nfs elevator=deadline rootwait loglevel=8 earlyprintk
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 760272K/786432K available (6478K kernel code, 436K rwdata, 1732K rodata, 476K init, 775K bss, 17968K reserved, 8192K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc080cd64   (8212 kB)
[    0.000000]       .init : 0xc080d000 - 0xc0884000   ( 476 kB)
[    0.000000]       .data : 0xc0884000 - 0xc08f10e8   ( 437 kB)
[    0.000000]        .bss : 0xc08f4000 - 0xc09b5c24   ( 776 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 32.
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] Architected cp15 timer(s) running at 19.20MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[    0.000008] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[    0.000025] Switching to timer-based delay loop, resolution 52ns
[    0.000319] Console: colour dummy device 80x30
[    0.001593] console [tty1] enabled
[    0.001647] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000)
[    0.001717] pid_max: default: 32768 minimum: 301
[    0.002045] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.002089] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.003054] Disabling cpuset control group subsystem
[    0.003114] Initializing cgroup subsys io
[    0.003166] Initializing cgroup subsys memory
[    0.003227] Initializing cgroup subsys devices
[    0.003271] Initializing cgroup subsys freezer
[    0.003314] Initializing cgroup subsys net_cls
[    0.003384] CPU: Testing write buffer coherency: ok
[    0.003473] ftrace: allocating 21691 entries in 64 pages
[    0.053963] CPU0: update cpu_capacity 1024
[    0.054027] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.054061] [bcm2709_smp_prepare_cpus] enter
[    0.054205] Setting up static identity map for 0x8240 - 0x8274
[    0.055921] [bcm2709_boot_secondary] cpu:1 started (0) 18
[    0.056099] [bcm2709_secondary_init] enter cpu:1
[    0.056143] CPU1: update cpu_capacity 1024
[    0.056150] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.056527] [bcm2709_boot_secondary] cpu:2 started (0) 17
[    0.056667] [bcm2709_secondary_init] enter cpu:2
[    0.056689] CPU2: update cpu_capacity 1024
[    0.056695] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.057057] [bcm2709_boot_secondary] cpu:3 started (0) 18
[    0.057186] [bcm2709_secondary_init] enter cpu:3
[    0.057207] CPU3: update cpu_capacity 1024
[    0.057214] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.057275] Brought up 4 CPUs
[    0.057374] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[    0.057404] CPU: All CPU(s) started in HYP mode.
[    0.057430] CPU: Virtualization extensions available.
[    0.058076] devtmpfs: initialized
[    0.069421] VFP support v0.3: implementor 41 architecture 3 part 40 variant 3 rev 4
[    0.069670] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.070423] pinctrl core: initialized pinctrl subsystem
[    0.070992] NET: Registered protocol family 16
[    0.076232] DMA: preallocated 4096 KiB pool for atomic coherent allocations
[    0.081793] bcm2709: Mini UART enabled
[    0.081856] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[    0.081903] hw-breakpoint: maximum watchpoint size is 8 bytes.
[    0.082071] Serial: AMBA PL011 UART driver
[    0.082224] uart-pl011 3f201000.uart: could not find pctldev for node /soc/gpio@7e200000/uart0_pins, deferring probe
[    0.082422] bcm2835-mbox 3f00b880.mailbox: mailbox enabled
[    0.144587] bcm2835-dma 3f007000.dma: DMA legacy API manager at f3007000, dmachans=0x1
[    0.145970] SCSI subsystem initialized
[    0.146176] usbcore: registered new interface driver usbfs
[    0.146287] usbcore: registered new interface driver hub
[    0.146405] usbcore: registered new device driver usb
...

You could try device_tree_address=0x10000000 instead.

[peyo-hd]

There was a mistake that gpu_mem=256 was not correctly applied to config.txt in my RPi3 boot partition.
After getting your last comment with kernel log, I found this mistake.

So, I cross checked that RPi3 VMSPLIT_3G boot-up failure could be avoided by larger (>=240) gpu_mem value in config.txt.

Thanks.

[pelwell]

Let's keep this issue open until the firmware takes care of it automatically.

[popcornmix]

Latest rpi-update firmware has a potential fix for this.

[peyo-hd]

On my Pi3 configuration, new firmware on following commit does not fix the issue. (made it worse)
raspberrypi/firmware@e968a4e
On both cases of adding or removing "gpu_mem=240" to config.txt - kernel failed to boot.

With previous version firmware,
raspberrypi/firmware@0a96f63
Adding "gpu_mem=240" enabled kernel boot-up.

[pelwell]

There is new rpi-update firmware that should fix this problem.

[peyo-hd]

Tried firmware on following commit, but shows same problems as before :
On both cases of adding or removing "gpu_mem=240" to config.txt - kernel failed to boot.
raspberrypi/firmware@5a1c159

I keep staying on following older version :
raspberrypi/firmware@0a96f63

[gonzoua]

I believe e968a4 has broken device_tree_address attribute. I have this in my config.txt:

disable_commandline_tags=0
device_tree_address=0x100
device_tree=rpi2.dtb
kernel=u-boot.bin
gpu_mem=128

It works fine with u-boot on 0a96f63 but on e968a4e I get
libfdt fdt_check_header(): FDT_ERR_BADMAGIC

Memory at 0x100 contains ATAG sequence. So it seems device_tree_address is ignored

[pelwell]

@gonzoua I suspect the firmware thinks the 4.4 DTB is too large. Some part of the Linux kernel, perhaps the built-in decompressor, is using the memory from 0x4000-0x8000, so if you start at 0x100 the firmware only has about 16KB to play with.

The DT compiler in the 4.4 tree generates larger output files than before. I should be able to get the firmware to strip out some unnecessary extras, but I haven't done this yet so you are probably hitting the 0x3f00-byte limit. If you are using U-boot then the 0x4000 limit probably doesn't apply; you can tell the firmware this by setting device_tree_end=0x8000 (or whatever your upper bound is). N.B. The end address is exclusive, so you are actually telling it the first address not to use.

[SylvainGarrigues]

@gonzoua I had the same problem. The only way I got to boot FreeBSD with this new firmware and with the latest u-boot was to remove the disable_commandline_tags line from config.txt and tag the u-boot.bin binary :-( with:

perl mkknlimg --dtok /Volumes/BOOT/U-BOOT.BIN.UNTAGGED /Volumes/BOOT/U-BOOT.BIN

Now another Linux dependency in our workflow. I tried all other combination disable_commandline_tags=[0,1,2] with tagged and untagged, that is the only one which works.

I guess we have to update our u-boot RPI 1 & 2 ports. In such case, a recent u-boot commit (don't know which one but less than 10 days old) made reading from the MMC 6x faster! I was reading ubldr.bin and kernel @ 700 kB/s and now u-boot reading speed is showing 5 MB/s. Let's try to use in our ports a recent u-boot GIT commit then, which takes this improvement into account :-)

[SylvainGarrigues]

@pelwell Would you make the firmware trust the "kernel" it loads is DTB compliant by default so that we don't need to tag it anymore? I believe you already have done the same thing when in 64bits mode.

[pelwell]

Yes, it's going to happen.

[pelwell]

As of yesterday's firmware release (f063c24a8307ae57040eda58f4751a97efdf7ab8), the kernel is assumed to be DT-capable by default:

001681.193: Loading 'kernel7.img' to 0x8000 size 0x3fb558
001684.838: No kernel trailer (run mkknlimg to fix) - assuming DT-capable
001688.999: Loading 'bcm2710-rpi-3-b.dtb' to 0x403558 size 0x3afe

[SylvainGarrigues]

@pelwell You're right, I no longer need to tag u-boot to make it use the DT and pass it to the FreeBSD kernel. No more disable_commandline_tags is necessary in config.txt now.

[SylvainGarrigues]

I just don't manage to boot our kernel directly however, I still need u-boot. But it may be FreeBSD-kernel related.

[dankcushions]

I hope this is ok to ask here, but is it (theoretically) possible that a change could be made so that CONFIG_VMSPLIT setting could be placed controlled via config.txt (or similar), rather than requiring a kernel recompile?

It is useful for certain emulation purposes (eg notaz/pcsx_rearmed#71) but would prefer to stick to regular raspian if possible.

[popcornmix]

No, it requires a rebuilt kernel. You could build a second kernel, call it kernel7_3g.img and choose which kernel to use in config.txt (kernel=kernel7_3g.img).
But, no it's not possible to use a single kernel for this.

[popcornmix]

Closing as original issue is resolved.