Open AI + Home Assistant Dual Assistants

[DanPatten]

I just got the OpenAI Conversation working with Home Assistant's built in assist. It's great they can work separately but has anyone figured out a dual version since OpenAI cannot control the smart home. It would be great if commands were routed to one assistant or the other depending on the incoming command.

Ex:
Turn on the kitchen lights >> Home Assistant

What should I make for dinner? >> OpenAI

[kristiankielhofner]

Our current plan for this is what we're calling the Willow Application Server (WAS) and the WAS protocol. One of the main project goals is to not be linked to Home Assistant for any functionality for universal compatibility with other supported endpoints (openHAB, Home Assistant, generic REST, etc).

WAS will live standalone (like WIS) and take over handling of much of the Home Assistant interaction (connection to HA will be configured there). We will have applications (like OpenAI) that can chain together functionality such as this (or LLM via WIS, API calls to arbitrary services, etc) to enable functionality such as:

Willow (wake) -> WIS ASR -> OpenAI/LLM/API/etc -> HA Action (optionally) -> TTS output of results

We would also love to see the community create a Willow component that integrates WAS protocol support in Home Assistant so that Home Assistant can essentially "fake" a WAS instance and provide the same integration and chaining to the ability that Home Assistant and the component are fundamentally able to.