↓Skip to main content

Home Automation Without Home Assistant

·9 mins
Table of Contents

My home automation setup is based on ESPHome and MQTT, with Node-RED for automation glue, Prometheus and Grafana for making pretty graphs, and Apple HomeKit for a human-friendly UI. There’s no Home Assistant in the mix. I’ll explain how I got here and how it works.

My Requirements #

I can (retroactively) list the following requirements for my home automation setup:

  • Reliable control of lights and devices on iPhones and voice assistant for me and other members of the house.
  • Sensor data logging and pretty graphing. I like temperature graphs more than I should.
  • Local operation, no cloud dependencies. Because it’s silly to send data out and then back in.
  • Use mostly off-the-shelf hardware, but with the ability to hack together custom devices when I have time and interest.
  • Stable once configured. I want to choose to work on this project and not be interrupted with issues at other times.

Finally, I want working on the system to be fun because, let’s be honest, none of this is necessary. That means tools that:

  • Let me inspect and understand the system.
  • Allow me to version control my configuration, so I’m not scared of screwing things up.
  • Let me get the most done in the least amount of time.
  • Are familiar to me or are a learning experience I can transfer to other projects.
  • Are well-documented and have a good community around them.

Entirely Personal Frustrations with Home Assistant #

Lots of people love Home Assistant. I don’t want to tear it apart but luckily, we have a list of requirements, so it’s easy to say: it didn’t meet my requirements.

From what I can tell, originally, Home Assistant was mostly configured through YAML files. At some point, the developers decided to make it a GUI focused project. Now configuration and automation rules are created through the GUI and stored in a database. I’m sure this makes Home Assistant more approachable to non-coders, but I found myself clicking through a complex GUI I couldn’t figure out, not understanding what was happening under the hood, and struggling to fix issues.

It’s made worse because Home Assistant works best if you go with the whole Hass.io stack, which is Home Assistant packaged as an OS image with various dependencies in Docker.

So you’re clicking around in this GUI that installs and configures Docker containers in the background. When things don’t work, you’re debugging a huge complex system full of unknown layers. Then you start searching for solutions and end up burning your whole weekend trying to decipher forum posts where everyone seems confused.

After trying something, I’d often just start over with a fresh OS image to get back to a known good state. And then you need to do a bunch of GUI clicks to get back to where you were.

It wasn’t satisfying, success felt fragile, it wasn’t transferable learning, and it wasn’t fun.

That’s for me. If you love Home Assistant, ignore me. Home Assistant is great for many people. There’s clearly a lot of good engineering in there.

My New Stack #

Through those experiences with Home Assistant, I did learn some important things. I uncovered my requirements. And I learned about ESPHome, MQTT, and Node-RED. My new stack is based on those technologies, with Home Assistant removed.

Here’s the architecture:

block-beta columns 4 %% Nodes %% Row EsphomeStack("ESPHome Devices") space:2 ZigbeeDevices("Zigbee Devices") %% Row space:4 %% Row space:3 USBZigbeeAdapter("USB Zigbee Adapter") %% Row space:4 %% Row space:3 Zigbee2MQTT("Zigbee2MQTT") %% Row space:4 %% Row MQTTBroker("Mosquitto MQTT Broker"):4 %% Row space:4 %% Row MQTT2Prometheus("mqtt2prometheus") space NodeRED("Node-RED"):2 %% Row space:4 %% Row Prometheus("Prometheus") space HueBridge("Philips Hue Devices") iPhone("iPhone HomeKit") %% Row space:4 %% Row Grafana("Grafana") %% Edge connections between nodes EsphomeStack --- MQTTBroker ZigbeeDevices --- USBZigbeeAdapter USBZigbeeAdapter --- Zigbee2MQTT Zigbee2MQTT --- MQTTBroker MQTTBroker --- NodeRED MQTTBroker --- MQTT2Prometheus MQTT2Prometheus --- Prometheus Prometheus --- Grafana NodeRED --- iPhone NodeRED --- HueBridge

(The direction of those arrows is meaningless. I can’t figure out how to make Mermaid omit them in a block diagram.)

Let’s talk about each component.

MQTT Broker: Mosquitto #

MQTT is a pub-sub protocol for message queuing. It’s an application layer protocol (so you’d make a choice to communicate over MQTT versus HTTP or IMAP) and it runs over TCP/IP, which in practice means you’re typically running it over WiFi and Ethernet networks withing your home or over the internet. MQTT clients connect to an MQTT broker, where they can publish messages to a topic or subscribe to a topic. Topics are organized in a hierarchy with slashes separating levels. Message payloads are strings, and it’s common to send JSON encoded structured data as messages.

I use the Mosquitto MQTT broker. I didn’t research brokers extensively – Mosquitto seems to be popular and simple. My client count is low and message volume is low, so I think basically any broker would work.

MQTT has a couple of interesting that are worth understanding.

  • Retained messages: When a client subscribes to a topic, it can get the last message that was published to that topic. This is useful for things like battery-powered sensors that publish their value infrequently. The broker essentially holds the state of the device.
  • Last will and testament: When a client connects to the broker, it can specify a topic and message that the broker will publish when the client disconnects. This is useful to create a reliable online/offline status for devices. When the device comes online, it publishes an “online” message to a topic and sets the last will to publish an “offline” message to that topic. When the broker sees the device disconnect, it publishes the “offline” message. The ESPHome MQTT Client component can be configured to do this automatically. Read more
  • Quality of service: MQTT has three levels of quality of service (QoS) levels that provide various message delivery guarantees. QoS 0 is best-effort fire-and-forget, QoS 1 is at-least-once, and QoS 2 is exactly-once. QoS 0 is the fastest, generates the least network traffic, and uses the least power on battery-powered devices. QoS 1 is slower, more traffic, and more power. QoS 2 is the slowest, most traffic, and most power. Read more

These features are part of what makes MQTT specially designed for IoT and a good choice for home automation. Plus, MQTT is tightly integrated with ESPHome and Node-RED.

Various Devices Running ESPHome #

I have various ESP8266 and ESP32 devices I’ve put together as temperature sensors, relay controllers, displays, and other things. They all run ESPHome firmware.

ESPHome is awesome.

ESPHome is a firmware for ESP8266 and ESP32 microcontrollers that allows you to configure them with YAML files. There are out-of-the-box components for pretty much every sensor and actuator you want. You get a ton of stuff for free: OTA updates, WiFi setup, an NTP client, an MQTT client, code to control displays, logging, and a web interface for configuring devices.

I’ve written firmware of microprocessors in C, in Micropython, and in Arduino, but when I have a weekend project, I love ESPHome. When the goal is to get a thing working before Monday, the higher level of abstraction is fantastic, even if it means you’ve become a YAML engineer.

ESPHome is beautifully integrated with Home Assistant, but it also has great integration with plain MQTT, so you can use it very effectively without Home Assistant.

Zigbee #

I have a few Zigbee devices, like Aqara buttons and temperature sensors, that I bought on AliExpress. I use Zigbee2MQTT to connect them to my MQTT broker.

Zigbee2MQTT is a software bridge that lets you turn a computer into a Zigbee gateway by plugging in a supported Zigbee adapter (USB dongle). It has a simple, YAML configuration file where you specify the adapter you’re using, the MQTT broker, and the devices you want to pair.

It’s solid, simple, has great logging, and excellent documentation.

Node-RED #

Node-RED is a visual programming tool. It’s built on Node.js and provides a browser-based editor where you create “flows” by dragging nodes onto a canvas and connecting them. Messages flow between nodes, and you can use JavaScript to transform messages, make decisions, and interact with external services.

A home automation stack needs an automation engine (“if this than that” type of stuff) and a way to glue things together (“connect to this service and massage these HTTP calls into this other shape”).

I don’t think the visual programming is that compelling, but Node-RED has a good ecosystem of integrations. When you ditch Home Assistant, one of the things you lose is the incredible ecosystem of pre-built integrations for every service and device you can imagine. From what I can tell, Node-RED has the next best ecosystem of these integrations.

I recommend ignoring the flow control nodes in Node-RED and just using the function node to write JavaScript.

Prometheus and Grafana #

This is pretty standard stuff. I like having a time-series database and a dashboarding tool to make pretty graphs of sensor data.

I use Prometheus because it’s simple and incredibly space-efficient.

I use Grafana because I’ve used it before, if you spend enough time you can get it to do anything, and it looks fine.

mqtt2prometheus is a simple Go app that subscribes to MQTT topics and publishes them to Prometheus.

Apple HomeKit #

I use Apple HomeKit as the human-friendly UI. Making the iPhone Home App the primary interface to the home automation system has kept the other members of my house happy. It also gives you Siri control for free.

I use the node-red-contrib-homekit-bridged Node-RED flow to expose devices to HomeKit (NRCHKB to the cool kids). This library is a wrapper around the HAP-NodeJS library, which is a TypeScript implementation of the HomeKit Accessory Protocol.

NRCHKB lets you simulate HomeKit devices in Node-RED, so you can create switches, lights, and sensors that show up in the iPhone Home app but everything about those devices is controlled by Node-RED. For some of my devices, Node-RED is just a pass-through between HomeKit and MQTT. For others, I have more complex logic in Node-RED.

Philips Hue #

I have a Philips Hue bridge, quite a few Hue lights, and several Hue switches. Hue operates on Zigbee and it’s possible to unpair the Hue lights from the Hue bridge and pair them with Zigbee2MQTT, but it’s time-consuming and makes things more manual. Instead, I’m running two Zigbee networks in my house: one for Hue and one for Zigbee2MQTT.

I use the node-red-contrib-huemagic-fork Node-RED flow to control the Hue lights and switches. In one of my flows, I have two Aqara buttons on the Zigbee2MQTT network that control the Hue lights on the Hue network, with some custom behavior implemented in Node-RED.

Conclusion #

I’m happy with this setup. All the software runs on my home server in Docker containers.

It’s text and code heavy, so everything is in Git. The layers are easy to inspect and the architecture is fairly similar to web/cloud systems I’ve worked on. That makes it easy for me to refresh my memory when I come back to it after a few months.

I’ve been running this stack for two years, and it’s been remarkably stable. Recommended.

↑