Tasklist

[mariusandra]

If you'd like to give a helping hand, pick any of the unfinished items and submit a PR. Don't ask for permission, everything is available and there are no issues to assign.

Old archived tasks

July to November 2023. Python proof of concept

• Backend + frontend for control panel running
• SSH connection to frames
• Deploy and launch client software
• Launch as a daemon (via tmux for now)
• API connection back
• Streaming SSH logs to client
• UI to add frames
• Get live images back
• Reload button in FCP UI
• Show image as it's updating
• Sync frame updating state to control panel
• Sync button presses to control panel
• Add migrations
• Send logs sequentially
• Configure image via UI
• Kiosk mode URL
• Create new frame installation instructions (from rpi)
• Add apps
• Draw stuff on images
• OpenAI DallE2
• Rotate images
• Scaling mode as part of the frame, not app
• App for scaling with cover modes
• Edit via a diagram UI
• Require user/pass to see interface
• Close a tab, fix bugs with tabs
• Handle saving modified apps
• Virtualize logs (list gets too big)
• Settings page with credential storage (HA, OpenAI, Github)
• Save scenes (one or many) as templates
• Template .zip export
• Template .zip import
• Github template repositories
• Support making screenshots of web pages (on 64bit devices only)
• Apps/devices must preselect the settings/secrets they are using, only those are copied over
• SSH keys
• Python and JSON errors inline in editor (red if file is obviously wrong)
• Editing text in app config jumps to end
• Document the coding environment and the available apps
• Get CPU, temp and other metrics from the frames
• Add Sentry support
• Better login screen
• If/else branches in diagram editor
• Horizontal/vertical split (grid mode)

December to January 2024. Nim rewrite. Blog post. HN thread

• Explore migrating from PIL to pixie-python make sense? We'd get great font rendering for free, minus emoji support sadly. --> Might be great, but putting energy into the next point instead.
• Experiment if it makes any sense to rewrite the FrameOS on-device part in a compiled language. Conclusion: yes it does. We're doing real time graphics on low power devices. A compiled language brings 10x improvements in speed and/or performance. Migrating some parts to a compiled language is inevitable. After looking around a lot, nim seems like the best choice. People praise it. It's friendlier than rust/zig/go ("If you imagine native-compiled type-annotated Python where nearly 100% of your code is business logic with no cruft, you're getting close to the Nim experience."), it's easy to pick up from examples, fast, it's used for games (it's fast), and comes with an amazing image handling library and other goodies. None of the other languages hit this sweet spot, so let's go.
• Rewrite FrameOS in nim.
  • Proof of concept in the nim branch
  • Deploy the nim code onto a raspberry with on-device compilation
  • Caching for on device compilation
  • Support for apps and the execution context
  • Support numbers, color and other types of fields
  • Support template strings in config -> inline code nodes for vars
  • Support edited custom apps
  • Convert to async server with schedules, etc
  • Scheduler
  • Inky impression display support? No C bindings? Only python available. Will call python code for now.
  • Rotated display support
  • System of hardware specific drivers, and devices that group them
  • Port over the main apps
    • Unsplash
    • Text
    • Code
    • Clock
    • Download URL
    • FrameOS Gallery
    • Color
    • Gradient
    • HA Sensor
    • If
    • Split
    • Resize
    • Rotate
    • OpenAI
    • Break if rendering
  • Log batching in a separate thread
  • Rendering in a separate thread
  • HDMI support (double buffering, frame info, different color modes)
  • Sanitize app fields
  • Display on/off controls
  • Skip most logs in high fps mode
  • Metrics logger
  • Waveshare eink support (using C libraries!)
  • Touch and click handlers
  • Ignore render when rendering, run all events after
• 60fps mode (max 2fps with Python now) -> got 60fps+ (more like 100) on a rpi5 with a 720p display (pikvm). So it's possible, even with simple "software rendering". Once we have the basics covered, I can investigate something like boxy to increase performance even more.
• Stability and hours of use (been running an inky impression + rpi zero w 1 and a hyperpixel round + zero w 2 at 24fps for days now without downtime)
• Show errors when editing nim apps in the interface
• Update docs Update for the nim rewrite frameos-docs#1
• Update the Dockerfile
• Merge the PR: Rewrite in nim #5
• Add B/W and B/W/R waveshare drivers
• Full screen mode in web server
• Investigate timeouts hanging the logger
• Delete old build folders on deploy
• Any way to lock/secure dependencies
• Global last image lock
• Cache deps in docker
• Show frame metric charts
• Memory hole?? Memory is not freed when requesting the image over http.
• Add dithering for 2-color waveshare devices
• Add dithering for the rest
• Add waveshare drivers
  • Black/White
  • 4-Grayscale
  • 7-color
  • 4-color Black/White/Red/Yellow
  • Black/White/Yellow
  • Black/White/Red
• Compile lgpio from scratch if no package
• Swap from string node ID-s in compiled scenes to integers (should speed it up a bit and reduce memory usage)
• GPIO button handlers
• Outgoing event nodes
• Disable swap (saves SD card from death if we have a memory leak) --> revert? messes up compilation on W2
• Faster compilation (was: local buildchains. Decided against it for now)
• Inky impressions native driver. Went with faster python redeploys instead. The driver itself works fine as is, so not replacing today.
• Logger crashes again, and takes its thread to 100%. Let's try again...
• Debug toggle
• Double check all the gcsafe hacks
• General polish

February to April 2024. Multiple scenes, apps, regression tests and other completed tasks.

• App: QR code
• App: GitHub star count
• App: GPT text
• Scene state
• Scene state updates and persistence
• Scene state updates with built in web server
• Auth for frame API access
• Don't drag from within textfields
• Sometimes 404 error when opening frame and clicking back only during rebuilds in dev
• Multiple scenes
• Move parts of frame config to scene config/state
• Revamped scenes and templates
• New default repository, easier sharing of scenes via repositories
• Visual regression tests for apps and scenes
• Tests for scene generation
• Update docs and screenshots
• Update docs about building apps
• Easier split tool
• Initial scene wizard when setting up a new frame
• Improve path dragging/removing in the editor

May to July 2024. Data nodes, state nodes, universal caching, agenda app.

• Persist last open scene for restarts
• Schedule a restart
• Faster frontend builds
• Code nodes with arguments
• Universal caching controls
• Async data nodes (download url)
• Images as data (?)
• New nodes by dragging to the diagram
• Drag to empty space -> new node
• Data apps, universal caching, refactors all old apps, context menus
• Cursor jumps to the end of the text field
• Unsplash app
• Update all templates with data apps
• Control render/sleep durations programmatically
• Clean up nim app code
• Fix inputs cache type
• Logger leaks memory (stopped buffering, got better, added workaround: nightly restarts)
• iCal support
• Simple timezone support
• Save OpenAI images to disk
• Edit apps with multiple files
• State nodes
• Document building scenes
• Build rich text app
• Build agenda app
• Set system timezone
• Assets browser
• XKCD scene
• Push edge when adding node

Recently finished

• Show scene.nim errors
• Video intro (Nimconf 2024)
• Flask to FastAPI
• Package as a home assistant or Hacs addon

TODO next

• Adding variable/argument to code node is flakey
• Wrap up 12.48" project (upload stl files, record video, etc)
• Undo/redo support
• Fix logger bugs
• Upload images (better asset browser)
• Control URL: don't render immediately when swapping scenes
• Store diagram/edit state in URL
• Drag to select / select multiple
• Copy/paste nodes
• Autocomplete for nim code
• Monaco for code nodes
• Clean up edges that are not connected

Apps to build next

• Calendar
• Graphs and charts
• Google Photos
• RTSP video stream
• Image transitions / screensavers (e.g time based smooth zoom)

New features

• Cross compilation
• Act as wifi hotspot for initial setup
• Custom event types
• Apps and versioning
• Scenes in scenes & special "Layout" scene
• Visual, grid, layout, placement, etc editor
• Errors and exceptions in the diagram
• Scheduler nodes

All the other ideas

Here's a list of things that have come to mind. Not in order of priority.

• Release 1.0 (or 24.07 or such)
• Automatically update timezones
• Upload/sync assets
• Tests for server
• Tests for metrics
• Tests for drivers
• Community forums and device guides
• CI for all supported raspberry OS versions
• Tests for scene runners
• Backups
• Config for drivers (choose which pins are buttons, etc)
• Kindle support
• ESP32 support
• JS target for live editing
• Font with emoji support
• Better sharing/import/export of apps
• Verify support for all other inky frames
• Unified background data fetching to not slow down real time mode
• Throttle if CPU is hot
• Slow down or turn off rendering if display is off
• HTTPS access between frames and controllers
• 404 on frame that is not there
• SSH terminal
• Multiple SSH keys
• Events (e.g. GPIO buttons) as home assistant events
• Boot splash screen for hdmi devices
• UI polish, design system work, etc
• Hyperpixel native driver (screen turn on/off is via python now)
• Investigate GPU rendering via boxy
• Pallette support for type="color" node config on dithered displays
• New image data app: % option
• Resize image data app % option
• Resize image data app: center properly if source is smaller than target

Cases and devices

• Create a clean case for the 12.48" frame
• Create a modular, easy, and semi-compatible 3d printed case system for all these different frames and raspberries.
• Update the device guides with a lot more info and images
• Clean system for adding frames/drivers/capabilities

[andiohn]

Hey! I LOVE THIS PROJECT SO MUCH!

Good work man. I am very inspired by this.

Ideas:

1. Simple http served frame for crappy devices (kobo aura HD etc) with really old browsers.
2. "touch targets" for those crappy devices (so you can pretend to interact with Home Assistant toggles.

Thank you for this. I'm going to try to use it and try to contribute.