The future of Heimcontrol: structural changes, new features (please discuss!)

[tjanson]

So I've been thinking about the state of Heimcontrol and possible new features, so I figured I'd share.

Application Structure

• Heimcontrol's client side is a perfect example for a real-time single-page application
• the bulk of HC's code is in routes/index.js; the app could use some structure
• the API is nice and fast, but needs to be implemented by each plugin; an API consumer must know and hardcode these endpoints
• HC could use better logging, authentication
• the web interface load times are very slow for no immediately-obvious reason

This to me sounds very much like it's time for a rewrite, in particular using a stronger framework [that is, something more than Express]. It should help get all the basics right, with little effort: Rest API endpoints, websockets, data store via ORM, authentication, logging, and most of all structure are all staples of current frameworks, and that's exactly what HC needs.

Do we need a framework? No, but I think it's an exciting opportunity and could solve a lot of problems (see below). And it's a learning experience, of course! :)

Features (all just ideas / a wishlist, of course)

• more clients (e.g., CLI), and making it easier to develop clients (e.g., wouldn't it be cool if the API was designed in such a way that the Android app does not need to be modified when there's a new plugin, but automatically discovers all endpoints?)
• speed/efficiency: a Pi should not be under constant load due to running HC, even when clients are actively connected
• automation rule engine — after all it's home automation, right? (simple if..then, e.g., "if dark, turn on light")
• move from MongoDB to something that supports ARM/comes with Raspbian
• different interfaces (rearrange/hide switches etc.) for different users
• keep history/log of sensor values and actions
• [your ideas?]

Here again I see the benefit of a rewrite: keeping our feature requests in mind while choosing the framework and restructuring the existing code should make the implementation a lot easier.

I'll write some more tomorrow, but for now: What do you think? Please don't hesitate to share your thoughts, feature requests, anything. :) (As always especially @ni-c, but everyone else as well!)

[tjanson]

As promised, some more details on a possible implementation:

I think it would be good to run Heimcontrol’s server side as an API server, i.e., to step away from template rendering, and handle that in the client instead. This could presumably be done with something like Angular (or Ember, or Backbone, or …) on the client, and Sails (or SocketStream, or LoopBack, or Actionhero … Derby, Meteor – the choices are overwhelming).

I don’t have experience with any of these, and they probably all have their ups and downs. For example, on first look:

• it seems LoopBack’s APIs are great, with a really handy Explorer; its docs are comprehensive; comes with Angular, Android, iOS SDKs
• SocketStream, on the other hand, seems perfectly suited for our websocket needs; but its docs are lacking
• Sails seems to be a balanced package overall with no obvious drawbacks

Again, the potential advantages would be speed, stability, structure & maintainability, and a better API. (Whether it works out that nicely remains to be seen. Maybe the frameworks will turn out to be bloated overkill with none of the benefits I hoped for, who knows.)
I’ll look into this further, maybe try some of them.

[@ni-c If you find all of this entirely repulsive, please complain ;-).]

[ni-c]

A complete rewrite will take a lot of time and at the moment i'm a little bit busy. But i like the idea and we can give it a try. I have some experience with Angular and Swagger, i will take a look at the other frameworks when i got some time.

[optikfluffel]

Maybe you could include Node-RED :)

[tjanson]

@ni-c Don’t worry, I’m not in any hurry – and if the experiment doesn’t work out well, we can always stay with the current setup. But the more I learn about those frameworks, the more I want to try them.

In case any one is interested, I’ve found (yet another) really interesting framework: Autobahn/WAMP + Crossbar can set up websockets to enable not just soft-real-time messaging/events, but remote procedure calls – not just server–client, but inter-client, too!

I’ve also played around with Angular some more, and I’m now pretty convinced that serving up a JSON API only (plus the static Angular HTML/JS files) is the way to go. Angular handles the rest with ease, and the Pi will be under less load.

[gubagu]

I think it could very nice if heimcontrol.js could be used in master slave setting where RPI is the master of several distributed remotely controlled Arduino modules. Modules could communicate with RPI through NRF24l01, RFM12B or some other low power-low - cost wireless module. The modules could be used to measure environment variables as temperature, motion, light intensity and turn on/off high voltage appliances. The master RPI could request measurements from the slaves and the slaves could report on certain time intervals their measurements to the master. The data could be stored in mongo collections for later use/reporting.
Visual interface could be adjusted in a way that you can also define rooms/places to which you can connect control/reporting modules. Every place/room can have its own dashboard and all dashboards could be aggregated in a main/home dashboard.
On the main screen/dashboard you can control user defined modules and visualize data of interest.

Cheerz,
Guby

[tjanson]

[@gubagu: That’s actually exactly what I have in mind for my own home automation: A bunch of Arduino Pro Minis w/ sensors (temp, brightness, door state) and actuators (lights and power sockets) communicating via cheap wireless. (I’m giving 433 MHz a try: it’s dirt cheap, I already use it for my power sockets, and I don’t need a whole lot of reliability – if a sensor doesn’t report I don’t care, if an actuator doesn’t respond, I’ll resend.) The goal being a unit cost of not much more than $5.]

But that inter-Arduino communication is definitely beyond the scope of Heimcontrol. Striving for modularity is clearly good though. Here’s what I’ve been thinking about:

• User interface: web interface and possibly others like mobile apps
• Sensor/Actuator: the Raspberry Pi, with duino bridging the gap to an Arduino (but any networked device could register sensors and actuators as well)
• Crossbar server: probably run on the Pi, but could just as well be run on another box

On the technical side, the sensors’ readings and other state information are distributed via websocket, the actuators are triggered as remote procedure calls:

… Well, that’s the theory. I really like it, though. I’ll try to write a minimal proof-of-concept soon.

[gubagu]

Nice to find alike thinking ppl
Just received these:
http://www.ebay.com/itm/230795578198 - ProMini clone
http://www.ebay.com/itm/170819064718 - NRF24L01 + 2.4GHz Wireless Transceiver
Some DallasTemperature ICs, ldrs, motion detectors (PIR) and mechanical relays.

I don't think the price with everything attached to one module will go above $5 per piece (work hours not included) :DDD
I hope this week I could find some time to try that http://tmrh20.github.io/RF24Mesh/index.html
I like your idea about Crossbar.io and wamp router. I'd love to see your proof of concept
Cheerz

[Sitebase]

Hi Guys,

To introduce myself, I started working on my own home automation controller only to find out later that there are already a bunch of these open source projects so after some comparing I decided to go for Heimcontrol. After all it's much more fun to work together on a project that already has some traction.

I've already seen some good ideas in the above comments.
One thing that I would like is to drop the focus on the Raspberry Pi. I think Heimcontrol just needs to be a NodeJS app that runs on every platforms that can run NodeJS. Don't get me wrong, I think the Raspberry Pi is a great device and I'm myself a proud owner of one. But with a broader focus you'll reach a bigger community instead of just a few people that play with it and then stop using it after a while because they want to try something new with their Pi.

This pull request should already help a bit with that vision. Do you guys always test the code on an actual Raspberry Pi when developing?

@tjanson I really like the suggestion you make of using Crossbar.io. It seems like a really solid and extendable setup. It's kind of on the same line with what I implemented for my home automation. See the image below for a basic graphical view of the implementation.

So all events that are triggered were sent to the PubSub system in the NodeJS app and this distributed the event again to all other protocols.
To give an example: if I would click on a button on the front-end an event is fired of websockets and received in the pubsub system in NodeJS. NodeJS would then redistrubute this event to the plugins that are subscribed and over MQTT to subscribed arduino devices.

@tjanson @ni-c are you guys still thinking about doing a rewrite because I really want to help.

[tjanson]

Hi @Sitebase,

@tjanson @ni-c are you guys still thinking about doing a rewrite because I really want to help.

Not at the moment, no. It’s definitely something I still want to do at some point, but I’m too busy at the moment. But it’s good to hear you’re interested too. :-)

I had a tiny Hello World for WAMP pub/sub/RPC using only Node.js (rather than crossbar/Python, which didn’t run well on my Pi). I remember trying to implement a way for the clients to auto-discover each other’s methods (cf. Wamp “reflection”) and being a bit frustrated that it wasn’t built in.

The following was my “dream architecture” (just brain-storming):

All user interfaces, actuators, and sensors to be derived from a very simple common Wamp client (which is where I wanted to go with the example linked above). The entire system would be decentralized in the sense that the components could be completely separate and run anywhere on the network (e.g., a temperature sensor might be a 10 line NodeJS programm running on a Pi). And dynamic in the sense that they could be added/removed from the network at any time (through the (so far non-existent) discoverability) – the interface would simply show the sensors currently available (or their last reading).

… Again, I’m unfortunately busy with school right now, but I’m still very curious to see what you come up with; and maybe we can work on it together in the future.

– TJ

[Gagnon06]

Hi, I am currently working on a new version of the Arduino plugin. This version include a way to communicate to remote Arduino with 433 MHz RF modules and the VirtualWire library. It is simple, the main Arduino is connected directly (USB) to the RPi and it communicates to the others wirelessly. To do that I fork duino project from ni-c and add some functions and a new code for the main Arduino and the remotes Arduino. For the moment, only DigitalWrite/Read and AnalogWrite/Read are implemented. I also fork the Heimcontrol.js project and tweak the Arduino plugin, but I have not test it at the moment.

Have a look at my work:
https://github.com/Gagnon06/duino/tree/custom-rf
https://github.com/Gagnon06/heimcontrol.js/tree/custom-rf

Some ideas :

• Arrange items by room (small section for each room with every buttons and sensors)
• Compatibility with Z-Wave and/or bluetooth
• Multiple motion webcam (different ip)
• Connect to ip cam

Gagnon06

[micha1983]

HI @tjanson,

I'm also working on a home visualisation with nodejs.

My Idea is to read the actual values from my plc at home (Wago controller with codesys).
I can read the values with Modbus TCP. It's working quite well.

some of the values (temperature, etc.) i want to save in a DB (Mongo, etc.) for a temperature graph.
Other values like values for switching actors, etc. i don't want to save in the DB.

The connection to the frontend i want to do with sockets for realtime visualisation.
So the target of my project is nearly the same than yours.

At the moment i'm fighting how i can implement the function that every changed value of temperature is saved in the DB and also send over the active socket connections.

At sails.js the framework will do everything for me. but i have no idea where i should implement my function for constantly reading my values with modbus TCP.

How is your status?
Maybe you can give me hint?

thanks.

greetings

Michael

[tjanson]

Hi @micha1983; I’m afraid I’m currently not working on home automation at all, nor have I tried Sails.js – so I can’t help you with that, sorry … Best of luck though. :)

[micha1983]

HI @tjanson ,

thanks for the fast feedback.

greetings from Würzburg.

Michael

[fgarci03]

Hi, where are things headed?

I think the rewrite is an awesome idea and I'd like to contribute. I'm working with AngularJS right now, so I feel somewhat comfortable to help with the frontend.

Also, today I have implemented a simple project using the RPi w/ heimcontrol & arduino and am using the RPi itself to flash the arduino, so from the moment I got everything wired up, it's 100% standalone (except electronics you need to assemble of course). Something like this could be used as a functionality like uploading sketches to the app and have it compile and flash the duino on the fly.

What do you think?