So I've been using the device in this state for a while, and it basically works.
Some things can be further tweaked of course.
- The software sometimes hangs or crashes on boot. Happens more often when a PC is connected via USB, I think. A bit annoying, but not a real problem because of the watchdog.
- The case design is still missing the thumb stick and some more tweaking.