i2c_driver panics on fixed map overflow

[bcantrill]

In updating the ROM on a Gimlet Rev B, the i2c_driver was seen to have panicked. (Due to a to-be-filed issue, this caused the thermal loop to be unable to assert control of the system, which in turn caused the fans to watchdog.)

Here is the i2c_driver dump as taken by Jefe. We died because of a stack overflow:

$ humility tasks -v
ID TASK                       GEN PRI STATE    
 4 i2c_driver                   0   3 FAULT: stack overflow; sp=0x24044fa0 (was: ready)
   |
   +-----------> 0x24000730 Task {
                    save: SavedState {
                        r4: 0x1,
                        r5: 0x0,
                        r6: 0x0,
                        r7: 0x24045038,
                        r8: 0x80aafcc,
                        r9: 0x0,
                        r10: 0x0,
                        r11: 0x240450b0,
                        psp: 0x24044fa0,
                        exc_return: 0xffffffed,
                        s16: 0x0,
                        s17: 0x0,
                        s18: 0x0,
                        s19: 0x0,
                        s20: 0x0,
                        s21: 0x0,
                        s22: 0x0,
                        s23: 0x0,
                        s24: 0x0,
                        s25: 0x0,
                        s26: 0x0,
                        s27: 0x0,
                        s28: 0x0,
                        s29: 0x0,
                        s30: 0x0,
                        s31: 0x0
                    },
                    priority: Priority(0x3),
                    state: Faulted {
                        fault: StackOverflow {
                            address: 0x24044fa0
                        },
                        original_state: Runnable
                    },
                    timer: TimerState {
                        deadline: None,
                        to_post: NotificationSet(0x0)
                    },
                    generation: 0x0,
                    notifications: 0x7,
                    descriptor: 0x80055c8 (&kern::descs::TaskDesc)
                }

Looking at the stack, we overdflowed while panicking:

$ humility readmem -s 0x24045000 4k 
humility: attached to dump
0x24045000 | 0xbaddcafe
0x24045004 | 0xbaddcafe
0x24045008 | 0x080aad44
0x2404500c | 0x00000001
0x24045010 | 0x00000000
0x24045014 | 0x00000000
0x24045018 | 0x080aafcc
0x2404501c | 0x00000000
0x24045020 | 0x24045048
0x24045024 | 0x00000001
0x24045028 | 0x240450b0
0x2404502c | 0x00000008
0x24045030 | 0x080aae90
0x24045034 | 0x2404509c
0x24045038 | 0x24045088
0x2404503c | 0x080a9de7 <- i2c_driver:write+0x13d
0x24045040 | 0xbaddcafe
0x24045044 | 0xbaddcafe
0x24045048 | 0x00000000
0x2404504c | 0x00000020
0x24045050 | 0x00000000
0x24045054 | 0xbaddcafe
0x24045058 | 0x00000000
0x2404505c | 0x00000001
0x24045060 | 0x24045134
0x24045064 | 0x080ab308
0x24045068 | 0x24045103
0x2404506c | 0x240451a4
0x24045070 | 0x00000001
0x24045074 | 0x00000000
0x24045078 | 0x24045138
0x2404507c | 0x24045134
0x24045080 | 0x080ab308
0x24045084 | 0x00000001
0x24045088 | 0x240450d8
0x2404508c | 0x080aa6f5 <- i2c_driver:fmt<core::panic::panic_info::PanicInfo>+0x6d
0x24045090 | 0xbaddcafe
0x24045094 | 0x240451c0
0x24045098 | 0x24045094
0x2404509c | 0x080a9e2b <- i2c_driver:fmt<core::fmt::Arguments>+0x1
0x240450a0 | 0xbaddcafe
0x240450a4 | 0xbaddcafe
0x240450a8 | 0xbaddcafe
0x240450ac | 0xbaddcafe
0x240450b0 | 0x080aae8c
0x240450b4 | 0x00000002
0x240450b8 | 0x00000000
0x240450bc | 0xbaddcafe
0x240450c0 | 0x24045098
0x240450c4 | 0x00000001
0x240450c8 | 0x240450e8
0x240450cc | 0x00000008
0x240450d0 | 0x080ab3c8
0x240450d4 | 0x24045194
0x240450d8 | 0x24045128
0x240450dc | 0x080a9de7 <- i2c_driver:write+0x13d
0x240450e0 | 0xbaddcafe
0x240450e4 | 0xbaddcafe
0x240450e8 | 0x00000000
0x240450ec | 0x00000020
0x240450f0 | 0x00000000
0x240450f4 | 0x00000005
0x240450f8 | 0x00000000
0x240450fc | 0x00000206
0x24045100 | 0x24045134
0x24045104 | 0x080ab308
0x24045108 | 0x080aa903 <- i2c_driver:sys_borrow_write_stub+0x13
0x2404510c | 0x00000040
0x24045110 | 0x240452ec
0x24045114 | 0x00000000
0x24045118 | 0x24045202
0x2404511c | 0x00000001
0x24045120 | 0x00000000
0x24045124 | 0x00000003
0x24045128 | 0x24045160
0x2404512c | 0x080aa781 <- i2c_driver:write_fmt<userlib::panic::PrefixWrite>+0x21
0x24045130 | 0x00000000
0x24045134 | 0x24045170
0x24045138 | 0x080ab3c4
0x2404513c | 0x00000001
0x24045140 | 0x00000000
0x24045144 | 0x00000003
0x24045148 | 0x24045190
0x2404514c | 0x00000001
0x24045150 | 0x24045202
0x24045154 | 0x00000003
0x24045158 | 0x00000008
0x2404515c | 0x2404533c
0x24045160 | 0x24045198
0x24045164 | 0x080aaa41 <- i2c_driver:panic+0x31
0x24045168 | 0x00000000
0x2404516c | 0x240451a4
0x24045170 | 0x24045ec0 <- PANIC_BUFFER+0x0
0x24045174 | 0x0000000d
0x24045178 | 0x080ab3c4
0x2404517c | 0x00000001
0x24045180 | 0x00000000
0x24045184 | 0x00000003
0x24045188 | 0x24045190
0x2404518c | 0x00000001
0x24045190 | 0x2404516c
0x24045194 | 0x080aa689 <- i2c_driver:fmt<core::panic::panic_info::PanicInfo>+0x1
0x24045198 | 0x240451b8
0x2404519c | 0x080a93ab <- i2c_driver:panic_fmt+0x2b
0x240451a0 | 0x00000000
0x240451a4 | 0x080aafcc
0x240451a8 | 0x080aae9c
0x240451ac | 0x240451c0
0x240451b0 | 0x080aad4c
0x240451b4 | 0x00000001
0x240451b8 | 0x240451e8
0x240451bc | 0x080a826d <- i2c_driver:insert<(drv_i2c_api::Controller, drv_i2c_api::PortIndex), drv_stm32xx_i2c_server::MuxState, 2>+0x65
0x240451c0 | 0x080aad44
0x240451c4 | 0x00000001
0x240451c8 | 0x00000000
0x240451cc | 0x00000000
0x240451d0 | 0x080aafcc
0x240451d4 | 0x00000000
0x240451d8 | 0x24045202
0x240451dc | 0x00000000
0x240451e0 | 0x2404531c
0x240451e4 | 0x24045204
0x240451e8 | 0x24045228
0x240451ec | 0x080a8391 <- i2c_driver:reset+0xe5
0x240451f0 | 0x00000000
0x240451f4 | 0x00000000
0x240451f8 | 0x24045338
0x240451fc | 0x00000003
0x24045200 | 0x0002ffff
0x24045204 | 0x00000008
0x24045208 | 0x001000d6
0x2404520c | 0x24045380 <- __RINGBUF+0x0
0x24045210 | 0x24045344
0x24045214 | 0x00000012
0x24045218 | 0x00000007
0x2404521c | 0x00000000
0x24045220 | 0x0000ffff
0x24045224 | 0x00000001
0x24045228 | 0x24045378
0x2404522c | 0x080a919f <- i2c_driver:main+0xddf
0x24045230 | 0x24045338
0x24045234 | 0x00000004
0x24045238 | 0xbaddcafe
0x2404523c | 0x24045339
0x24045240 | 0x00000001
0x24045244 | 0x00000000
0x24045248 | 0x00000001
0x2404524c | 0x00000002
0x24045250 | 0x00000000
0x24045254 | 0x00000000
0x24045258 | 0x00000032
0x2404525c | 0x00000001
0x24045260 | 0x00000001
0x24045264 | 0x080a8091 <- i2c_driver:call_once<drv_stm32xx_i2c_server::main::{closure_env#1}, (u32)>+0x1
0x24045268 | 0x080a8089 <- i2c_driver:call_once<drv_stm32xx_i2c_server::main::{closure_env#0}, (u32)>+0x1
0x2404526c | 0x00000002
0x24045270 | 0x40005c00
0x24045274 | 0x00000000
0x24045278 | 0x00000019
0x2404527c | 0x00000000
0x24045280 | 0x2404529c
0x24045284 | 0x00000000
0x24045288 | 0x00000002
0x2404528c | 0x00000096
0x24045290 | 0x00000001
0x24045294 | 0x40005800
0x24045298 | 0xbaddca02
0x2404529c | 0x00000097
0x240452a0 | 0x00000002
0x240452a4 | 0x40005c00
0x240452a8 | 0xbaddca03
0x240452ac | 0x00000107
0x240452b0 | 0x00000004
0x240452b4 | 0x58001c00
0x240452b8 | 0xbaddca04
0x240452bc | 0xba010400
0x240452c0 | 0xba010800
0x240452c4 | 0xba040002
0x240452c8 | 0xba050002
0x240452cc | 0xba050001
0x240452d0 | 0xba040102
0x240452d4 | 0xba070080
0x240452d8 | 0xba070100
0x240452dc | 0xba040003
0x240452e0 | 0xba054000
0x240452e4 | 0xba058000
0x240452e8 | 0xba040004
0x240452ec | 0x080aafcc
0x240452f0 | 0x080aae04
0x240452f4 | 0xba0bcafe
0x240452f8 | 0x73010002
0x240452fc | 0x080aafcc
0x24045300 | 0x080aae04
0x24045304 | 0xba0bcafe
0x24045308 | 0x70010102
0x2404530c | 0x080aafcc
0x24045310 | 0x080aae04
0x24045314 | 0xba0bcafe
0x24045318 | 0x71020102
0x2404531c | 0x080aafcc
0x24045320 | 0x080aae04
0x24045324 | 0x000b0000
0x24045328 | 0x72030102
0x2404532c | 0x01010201
0x24045330 | 0x04010003
0x24045334 | 0x01000000
0x24045338 | 0x04010002
0x2404533c | 0x01000102
0x24045340 | 0x00000000
0x24045344 | 0x00000319
0x24045348 | 0x080a8089 <- i2c_driver:call_once<drv_stm32xx_i2c_server::main::{closure_env#0}, (u32)>+0x1
0x2404534c | 0x080a8091 <- i2c_driver:call_once<drv_stm32xx_i2c_server::main::{closure_env#1}, (u32)>+0x1
0x24045350 | 0x00020000
0x24045354 | 0x000000c2
0x24045358 | 0x00000003
0x2404535c | 0x00001065
0x24045360 | 0x00000000
0x24045364 | 0x24045354
0x24045368 | 0x00000001
0x2404536c | 0x080aafcc
0x24045370 | 0x00000000
0x24045374 | 0x00000000
0x24045378 | 0x00000000
0x2404537c | 0x080a804f <- i2c_driver:_start+0x4f

Our panic buffer is (unsurprisingly) incomplete:

$ humility readvar --as-c-string PANIC_BUFFER
humility: attached to dump
userlib::panic::PANIC_BUFFER (0x24045ec0) = "panicked at '"

But we are at the call to panic in fixedmap::FixedMap<K,V,_>::insert:

$ humility extract elf/task/i2c_driver > o ; arm-none-eabi-objdump -Cd o
...
080a81a0 <fixedmap::FixedMap<K,V,_>::insert>:
 80a81a0:       b5f0            push    {r4, r5, r6, r7, lr}
 80a81a2:       af03            add     r7, sp, #12
 80a81a4:       f84d bd04       str.w   fp, [sp, #-4]!
 80a81a8:       b086            sub     sp, #24
 80a81aa:       f06f 0408       mvn.w   r4, #8
 80a81ae:       b1b4            cbz     r4, 80a81de <fixedmap::FixedMap<K,V,_>::insert+0x3e>
 80a81b0:       1903            adds    r3, r0, r4
 80a81b2:       7a5d            ldrb    r5, [r3, #9]
 80a81b4:       b14d            cbz     r5, 80a81ca <fixedmap::FixedMap<K,V,_>::insert+0x2a>
 80a81b6:       7a9d            ldrb    r5, [r3, #10]
 80a81b8:       3403            adds    r4, #3
 80a81ba:       b2ce            uxtb    r6, r1
 80a81bc:       42b5            cmp     r5, r6
 80a81be:       d1f6            bne.n   80a81ae <fixedmap::FixedMap<K,V,_>::insert+0xe>
 80a81c0:       4420            add     r0, r4
 80a81c2:       2401            movs    r4, #1
 80a81c4:       7184            strb    r4, [r0, #6]
 80a81c6:       3007            adds    r0, #7
 80a81c8:       e003            b.n     80a81d2 <fixedmap::FixedMap<K,V,_>::insert+0x32>
 80a81ca:       2001            movs    r0, #1
 80a81cc:       7258            strb    r0, [r3, #9]
 80a81ce:       f103 000a       add.w   r0, r3, #10
 80a81d2:       7001            strb    r1, [r0, #0]
 80a81d4:       72da            strb    r2, [r3, #11]
 80a81d6:       b006            add     sp, #24
 80a81d8:       f85d bb04       ldr.w   fp, [sp], #4
 80a81dc:       bdf0            pop     {r4, r5, r6, r7, pc}
 80a81de:       2000            movs    r0, #0
 80a81e0:       4906            ldr     r1, [pc, #24]   ; (80a81fc <fixedmap::FixedMap<K,V,_>::insert+0x5c>)
 80a81e2:       9104            str     r1, [sp, #16]
 80a81e4:       9005            str     r0, [sp, #20]
 80a81e6:       9002            str     r0, [sp, #8]
 80a81e8:       2001            movs    r0, #1
 80a81ea:       4906            ldr     r1, [pc, #24]   ; (80a8204 <fixedmap::FixedMap<K,V,_>::insert+0x64>)
 80a81ec:       9001            str     r0, [sp, #4]
 80a81ee:       4804            ldr     r0, [pc, #16]   ; (80a8200 <fixedmap::FixedMap<K,V,_>::insert+0x60>)
 80a81f0:       9000            str     r0, [sp, #0]
 80a81f2:       4668            mov     r0, sp
 80a81f4:       f001 f8c4       bl      80a9380 <core::panicking::panic_fmt>
 80a81f8:       defe            udf     #254    ; 0xfe
 80a81fa:       bf00            nop
 80a81fc:       080aafcc        .word   0x080aafcc
 80a8200:       080aad44        .word   0x080aad44
 80a8204:       080aad4c        .word   0x080aad4c

Unfortunately, due to another Humility bug (also to-be-filed!), we neglected to get 0x080aafcc, 0x080aad44, and 0x080aad4c in the dump -- but there is only one panic! in FixedMap::insert:

    ///
    /// Inserts the `value` into the map for the specified `key`.  If the
    /// specified key already exists in the map, its value will be overwritten
    /// with the specified value.  It is up to the caller to assure that there
    /// is room in the map; if the map is full, this code will panic.
    ///
    pub fn insert(&mut self, key: K, value: V) {
        for i in 0..self.contents.len() {
            match self.contents[i] {
                None => {
                    self.contents[i] = Some((key, value));
                    return;
                }

                Some((k, _)) => {
                    if k == key {
                        self.contents[i] = Some((key, value));
                        return;
                    }
                }
            }
        }

        panic!("FixedMap overflow");
    }

Knowing that this is (likely) a FixedMap overflow, the cause becomes clear when looking at the ring buffer:

humility: ring buffer drv_stm32xx_i2c_server::__RINGBUF in i2c_driver:
 NDX LINE      GEN    COUNT PAYLOAD
   0  644        1        4 Wiggles(0x0)
   1  465        1        1 Error(0x67, BusLocked)
   2  252        1        1 Reset((I2C4, PortIndex(0x0)))
   3  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
   4  465        1        1 Error(0x20, BusLocked)
   5  252        1        1 Reset((I2C4, PortIndex(0x0)))
   6  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
   7  465        1        1 Error(0x4a, BusLocked)
   8  252        1        1 Reset((I2C2, PortIndex(0x1)))
   9  262        1        1 ResetMux(0x70)
  10  262        1        1 ResetMux(0x71)
  11  262        1        1 ResetMux(0x72)
  12  190        1        1 MuxUnknownRecover((I2C2, PortIndex(0x1)))
  13  465        1        1 Error(0x20, BusLocked)
  14  252        1        1 Reset((I2C4, PortIndex(0x0)))
  15  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  16  465        1        1 Error(0x48, BusLocked)
  17  252        1        1 Reset((I2C4, PortIndex(0x0)))
  18  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  19  465        1        1 Error(0x18, BusLocked)
  20  252        1        1 Reset((I2C3, PortIndex(0x0)))
  21  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
  22  465        1        1 Error(0x20, BusLocked)
  23  252        1        1 Reset((I2C4, PortIndex(0x0)))
  24  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  25  465        1        1 Error(0x20, BusLocked)
  26  252        1        1 Reset((I2C4, PortIndex(0x0)))
  27  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  28  465        1        1 Error(0x4a, BusLocked)
  29  252        1        1 Reset((I2C2, PortIndex(0x1)))
  30  262        1        1 ResetMux(0x70)
  31  262        1        1 ResetMux(0x71)
  32  262        1        1 ResetMux(0x72)
  33  190        1        1 MuxUnknownRecover((I2C2, PortIndex(0x1)))
  34  465        1        1 Error(0x18, BusLocked)
  35  252        1        1 Reset((I2C4, PortIndex(0x0)))
  36  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  37  465        1        1 Error(0x20, BusLocked)
  38  252        1        1 Reset((I2C4, PortIndex(0x0)))
  39  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  40  465        1        1 Error(0x20, BusLocked)
  41  252        1        1 Reset((I2C4, PortIndex(0x0)))
  42  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  43  465        1        1 Error(0x67, BusLocked)
  44  252        1        1 Reset((I2C4, PortIndex(0x0)))
  45  465        1        1 Error(0x27, BusLocked)
  46  252        1        1 Reset((I2C3, PortIndex(0x0)))
  47  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
  48  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  49  465        1        1 Error(0x48, BusLocked)
  50  252        1        1 Reset((I2C4, PortIndex(0x0)))
  51  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  52  465        1        1 Error(0x67, BusLocked)
  53  252        1        1 Reset((I2C4, PortIndex(0x0)))
  54  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  55  465        1        1 Error(0x67, BusLocked)
  56  252        1        1 Reset((I2C4, PortIndex(0x0)))
  57  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  58  465        1        1 Error(0x20, BusLocked)
  59  252        1        1 Reset((I2C4, PortIndex(0x0)))
  60  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  61  465        1        1 Error(0x4a, BusLocked)
  62  252        1        1 Reset((I2C2, PortIndex(0x1)))
  63  262        1        1 ResetMux(0x70)
  64  262        1        1 ResetMux(0x71)
  65  262        1        1 ResetMux(0x72)
  66  392        1        1 MuxError(BusLockedMux)
  67  252        1        1 Reset((I2C2, PortIndex(0x1)))
  68  262        1        1 ResetMux(0x70)
  69  262        1        1 ResetMux(0x71)
  70  262        1        1 ResetMux(0x72)
  71  465        1        1 Error(0x1a, BusLocked)
  72  252        1        1 Reset((I2C3, PortIndex(0x0)))
  73  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
  74  465        1        1 Error(0x67, BusLocked)
  75  252        1        1 Reset((I2C4, PortIndex(0x0)))
  76  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
  77  190        1        1 MuxUnknownRecover((I2C2, PortIndex(0x1)))
  78  465        1        1 Error(0x24, BusLocked)
  79  252        1        1 Reset((I2C3, PortIndex(0x0)))
  80  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
  81  465        1        1 Error(0x24, BusLocked)
  82  252        1        1 Reset((I2C3, PortIndex(0x0)))
  83  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
  84  465        1        1 Error(0x4a, BusLocked)
  85  252        1        1 Reset((I2C2, PortIndex(0x1)))
  86  262        1        1 ResetMux(0x70)
  87  262        1        1 ResetMux(0x71)
  88  262        1        1 ResetMux(0x72)
  89  190        1        1 MuxUnknownRecover((I2C2, PortIndex(0x1)))
  90  465        1        1 Error(0x18, BusLocked)
  91  252        1        1 Reset((I2C3, PortIndex(0x0)))
  92  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
  93  465        1        1 Error(0x1a, BusLocked)
  94  252        1        1 Reset((I2C3, PortIndex(0x0)))
  95  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
  96  465        1        1 Error(0x4a, BusLocked)
  97  252        1        1 Reset((I2C2, PortIndex(0x1)))
  98  262        1        1 ResetMux(0x70)
  99  262        1        1 ResetMux(0x71)
 100  262        1        1 ResetMux(0x72)
 101  392        1        1 MuxError(BusLockedMux)
 102  252        1        1 Reset((I2C2, PortIndex(0x1)))
 103  262        1        1 ResetMux(0x70)
 104  262        1        1 ResetMux(0x71)
 105  262        1        1 ResetMux(0x72)
 106  465        1        1 Error(0x1d, BusLocked)
 107  252        1        1 Reset((I2C3, PortIndex(0x0)))
 108  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
 109  465        1        1 Error(0x19, BusLocked)
 110  252        1        1 Reset((I2C4, PortIndex(0x0)))
 111  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 112  190        1        1 MuxUnknownRecover((I2C2, PortIndex(0x1)))
 113  465        1        1 Error(0x24, BusLocked)
 114  252        1        1 Reset((I2C3, PortIndex(0x0)))
 115  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
 116  465        1        1 Error(0x67, BusLocked)
 117  252        1        1 Reset((I2C4, PortIndex(0x0)))
 118  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 119  465        1        1 Error(0x48, BusLocked)
 120  252        1        1 Reset((I2C4, PortIndex(0x0)))
 121  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 122  465        1        1 Error(0x67, BusLocked)
 123  252        1        1 Reset((I2C4, PortIndex(0x0)))
 124  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 125  465        1        1 Error(0x20, BusLocked)
 126  252        1        1 Reset((I2C4, PortIndex(0x0)))
 127  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 128  465        1        1 Error(0x20, BusLocked)
 129  252        1        1 Reset((I2C4, PortIndex(0x0)))
 130  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 131  465        1        1 Error(0x18, BusLocked)
 132  252        1        1 Reset((I2C3, PortIndex(0x0)))
 133  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
 134  465        1        1 Error(0x1c, BusLocked)
 135  252        1        1 Reset((I2C3, PortIndex(0x0)))
 136  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
 137  465        1        1 Error(0x20, BusLocked)
 138  252        1        1 Reset((I2C4, PortIndex(0x0)))
 139  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 140  465        1        1 Error(0x20, BusLocked)
 141  252        1        1 Reset((I2C4, PortIndex(0x0)))
 142  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 143  465        1        1 Error(0x1f, BusLocked)
 144  252        1        1 Reset((I2C3, PortIndex(0x0)))
 145  465        1        1 Error(0x1b, BusLocked)
 146  252        1        1 Reset((I2C4, PortIndex(0x0)))
 147  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 148  190        1        1 MuxUnknownRecover((I2C3, PortIndex(0x0)))
 149  465        1        1 Error(0x1e, BusLocked)
 150  252        1        1 Reset((I2C4, PortIndex(0x0)))
 151  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 152  465        1        1 Error(0x20, BusLocked)
 153  252        1        1 Reset((I2C4, PortIndex(0x0)))
 154  190        1        1 MuxUnknownRecover((I2C4, PortIndex(0x0)))
 155  465        1        1 Error(0x49, BusLocked)
 156  252        1        1 Reset((I2C2, PortIndex(0x1)))
 157  262        1        1 ResetMux(0x70)
 158  262        1        1 ResetMux(0x71)
 159  262        1        1 ResetMux(0x72)
 160  465        1        1 Error(0x19, BusLocked)
 161  252        1        1 Reset((I2C3, PortIndex(0x0)))

Here are the busses:

$ humility manifest
...
   i2c buses => 4 controllers, 5 buses
                C PORT MODE NAME          DESCRIPTION
                1 B    trgt spd           SPD proxy
                2 B    init m2            M.2 bus
                2 F    init front         Front bus
                3 H    init mid           Mid bus
                4 F    init rear          Rear bus
...

So we have seen resets to two different busses (front bus and mid bus) -- almost certainly because we were preempted by the SPI driver blasting the hostflash image. The problem is not the bus resets -- it is that we are always setting a bus to be in the MuxState::Unknown in the reset path, but in fact the map only has room for the busses that actually have muxes. In this case, because we have been to A0, the m2 bus has a mux set (that is, it has mux state); when we add front bus, the map is full -- and when we (errantly) add mid bus, we overflow. The fix is to only set the mux state to be MuxState::Unknown if a bus, in fact, has a mux to begin with.