Building Good Home Automation – Part II, The Protocols

In my last post, I mentioned that a “Good” home automation system requires the right protocols and infrastructure. What do I mean by this?

Most DIY home automation uses one of several wireless systems. While wired systems exist (KNX is supposed to be very good), they require pulling physical wires to every switch and outlet you want to automate, an undertaking beyond the scope of most of us, especially in an existing house. Most of the devices for such systems are generally only available through commercial system installers at very high prices. (There are also powerline-based systems, such as X10 and Insteon, which are technically wired, but they don’t require any additional wiring, just your home’s existing electrical wires. However, these are considered fairly outdated by today’s standards, so we won’t be discussing them here.)

The four most common protocols in my experience are Z-Wave, Zigbee, WiFi and 433 MHz. There are also several proprietary protocols. Let’s talk about their pluses and minuses.

Z-Wave

Z-Wave is a wireless protocol designed for home automation. It uses a low-energy radio and a mesh network, which allows each device to re-transmit signals to a central control unit.

Advantages:

Due to it’s mesh architecture, having more devices makes the network stronger. In theory at least.

Z-Wave devices are available from many manufacturers, so you are not locked in to any one vendor. So far, all of them seem to be interoperable with one another if they use the same frequency.

Z-Wave is entirely local – the signals don’t go much past your house and don’t end up on the Internet, so it’s impossible to spy on or hack the Z-Wave system directly without being inside, or just outside, your house. (However, your hub is almost certainly exposed to the Internet in some way, so an attack is certainly possible).

Z-Wave operates on frequencies in the 869-926 MHz range which are not used by WiFi or most cell phone systems, so interference between them is less likely. (Note that the exact frequency depends on the country where it’s being used, so be sure to buy devices meant for your country of residence as they aren’t cross-compatible and it’s illegal to use the wrong frequencies.)

Z-Wave is a protocol designed for automation, and as such is very low powered. This means that you can have battery-powered devices whose batteries last for many months or even years.

Z-Wave will often work straight out of the box. Plug in (or wire-in) the device, pair it with the controller, and it should just work.

Disadvantages:

In my experience, Z-Wave is not very stable. I have devices constantly dropping off the network and coming back. Originally I was using an Aeotec Z-Stick as a controller with my Home Assistant system, and later switched to a Vera Pro. Vera has proven much more stable, but still far from perfect.

I also find Z-Wave to be very slow. With switches, from the time I issue a command in the HA UI, to the time they actually turn on/off, there’s a significant delay. There’s also a significant delay in the acknowledgment, from when the switch actually cycles to when it informs HA of the status change. Similarly, I have two PIR motion sensors, one Z-Wave, one WiFi. The reaction time of the WiFi one is much faster.

To be fair, I’ve been told that both of the above problems are configuration issues – that the devices have to be assigned to correct classes and have various settings tuned. That may be so – I’ve not investigated this deeper, yet. But if so, it significantly reduces Z-Wave’s ease of use advantage. Which also brings me to:

Configuring devices is annoying. While most work ok out of the box, changing a setting is complicated. You have to lookup which numbered parameter on a given device controls the setting you want, and then the value of the setting, and transmit that to the device from the controller. For example, I have some Aeotec Smart Plugs that by default have a bright green ring when “on” and bright “red” ring when off. Annoying in the bedroom. It’s configurable, but the code isn’t in the manual it ships with. You have to download the “Engineering Specifications and Advanced Functions for Developers” document, which tells you that you want to change parameter 0x51 (81 dec) from 0 to 1. But that’s only on that specific device. A different plug may have a different parameter.

The devices seem to learn their locations, so while more devices strengthens the mesh, moving a device to another locations seems to cause disruption of the network, as the routing information stored in the system is now incorrect. There are commands to repair the mesh, but they don’t seem to always work well.

Finally, there’s price. Z-Wave devices tend to be more expensive than many other types, especially WiFi ones.

Zigbee

Zigbee is a wireless protocol designed for various short-range, low-power applications, including home automation. It uses an ad-hoc network, which is similar to the mesh used by Z-Wave, but while in Z-Wave systems every device is a re-transmitter, in Zigbee systems only some devices have that function. Disclaimer: I’ve not used Zigbee so far, so my knowledge here is purely theoretical.

Advantages:

Like Z-Wave, Zigbee is designed for low-power usage and is thus well suited to battery-powered applications.

If enough devices can act as relays, then adding additional devices strengthens the mesh.

Zigbee devices vary in price, but some, like those from Xiaomi, are quite inexpensive compared to Z-Wave, though often more expensive than WiFi.

Disadvantages:

Unlike Z-Wave, Zigbee devices don’t interoperate well across brands. Most brands have their own hubs/controllers that only work with devices from that brand, and, conversely, devices only work with that brand’s hubs. There is some overlap, and there may be some devices that are more broadly compatible, but be verious cautious in your choice of brand if choosing to use Zigbee.

Most Zigbee units operate on 2.4GHz, which means they can interfere with WiFi or not work well in WiFi-heavy environments. I do not know if this is actually a problem in real applications.

WiFi

WiFi is familiar to most of us – 802.11(b) and it’s successors, WiFi is a general purpose protocol meant for connecting devices to local networks and the Internet. Unlike Z-Wave and Zigbee, it’s very high bandwidth, and fairly high power.

Advantages:

WiFi-based devices are generally among the cheapest IoT devices of their kind, with only 433MHz devices rivaling them in price.

WiFi devices are higher powered, allowing them to work at longer ranges than Z-Wave or Zigbee when no re-transmitters are present.

WiFi devices are very fast. In my experience, far faster than Z-Wave.

WiFi devices can be extremely stable, provided one has the right firmware and architecture.

Most WiFi devices on the market use an ESP8266 or ESP32 chip, which allow them to be fairly easily flashed with alternate firmware, giving one complete control over the device.

Especially with after-market firmware, WiFi devices can communicate with the system hub using several well-established, very efficient protocols such as MQTT.

Some WiFi devices can communicate directly, bypassing the system hub, which can allow a more distributed system. While I don’t find this of much use personally, it’s an interesting option.

WiFi devices are very DIY-friendly. A board like a NodeMCU or Wemos D1 Mini can be bought for $3-4 on Amazon or a fraction of that on Banggood or AliExpress, and can be connected to nearly any sensor, relay, or other component. The “Bruh multisensor” is a popular project that pairs a NodeMCU with a PIR motion sensor as well as temperature, humidity, and lux (light level) sensors in a very small package.

Disadvantages:

As the WiFi protocol was not designed for home automation uses, WiFi devices are power-hungry. While it’s possible to power them from batteries, the batteries rarely last long – days to weeks. An alternative, for some devices, is to shut down until needed, but that causes delays, as the amount of time necessary to re-connect to a WiFi network is significant. Therefore, if there’s no way to run power (5V via USB is usually sufficient) to a particular device, WiFi is a poor choice for the application.

Most WiFi devices, as purchased, use firmware that talks only to the manufacturer’s cloud server, leading to all of the problems discussed in my prior article. Flashing aftermarket firmware is a must, and that requires some computer knowledge and a willingness to tinker. There are two specific FOSS firmware projects that I use: Tasmota, which is the more established, and more universally usable firmware (but requires an MQTT server), and ESPHome, which is newer and is meant to work specifically with Home Assistant – not sure if it can be used with any other hub. This is one of the two places we lose the “Easy” designation. The other place is:

Most home WiFi infrastructure is insufficient for a WiFi-based smart home. If you are only using a handful of devices, you’ll likely be fine, but if you plan to have dozens, or hundreds, you’ll probably need to upgrade to better routers, switches, and access points. I am planning on another article dedicated to just this topic, but it’s the other major place where WiFi fails to be “Easy”. The good news is that putting in the right infrastructure is very doable and will not only give you the ability to use hundreds of WiFi IoT devices, but will also improve your WiFi performance in general and improve the security of your home WiFi network.

433MHz

Devices based around 433MHz are mostly used by older alarm systems, ceiling fan controllers, wireless doorbells, and other older and lower-end systems. However, this doesn’t make the outdated – for some applications they are still top notch. They will require a 433MHz receiver or transceiver to connect to the rest of your system.

Advantages:

Very cheap. Purchased on Banggood, a 433MHz door/window sensor can be had for as little as $1. Even on Amazon they can be had for $3-4.

Very low power usage. These devices are built to work with batteries and a single AAA battery can last for a year or more in a door sensor, even a frequently used one. In part, this is because the device often shuts down completely between uses.

Can sometimes be used to connect old, “dumb” devices to your smart home. A number of old systems that use 433MHz for internal communications can be repurposed. For example, some ceiling fans that came with proprietary remote controls use 433MHz signals. Your system can learn that signal and put the fan under the system’s controls without needing to rewire anything.

Uses the 433MHz frequency range which is fairly quiet and not subject to interference from WiFi or cell phones.

Disadvantages:

You will need a 433MHz transceiver. I am currently aware of three: the Broadlink RM Pro series of IR and 433MHz units, the Sonoff RF Bridge, and RFLink. I’ve not used the Broadlink units, but both the Sonoff and the RFLink have some issues. The Sonoff works very well, but is very limited in which devices it can work with. Unfortunately, there’s no way to tell ahead of time, except by asking around on the forums or trial and error. On receive, it just won’t “hear” some devices. On transmit, it’s limited to only some devices and also has a limit of a maximum of 16 commands it can send. There is a hardware modification that can be done to it that allows much broader capability, but I’ve not tried it yet. The RFLink is by far the most capable one, but the RFLink project itself is only the software – you need hardware to run it on. You can build it yourself, or there’s one vendor for ready to go kits, Nodo Shop. Unfortunately, both the project and the store are based in the Netherlands, and while there are English-language versions of most of the docs, some are missing and some are outdated. I got one working after a lot of poking at it, but never got sufficiently good performance to justify using it over the Sonoff. I might go back to it at some point.

You will also need the codes for the devices you use. Some can be found on-line, others you can try to capture by reading the raw data received by the transceiver. Not horrible, but a significant pain in the neck.

Lutron Caseta

Although not a standard protocol, this one deserves a mention for the sole reason that I use it. Lutron is a major lighting company with at least three lines of “smart” switches and dimmers, none of which work together. Caseta is their cheapest offering. While it violates some of my preference for FOSS and completely local control, the advantages of Caseta outweighed it for me.

Advantages:

Extremely reliable. All of my Caseta units have just worked. Every time. Well, except the one that arrived with physical damage to the unit, but even that one worked as a smart switch, it was just the physical button that didn’t function.

Extremely fast. Probably even more responsive than WiFi units, these change state and report changes back instantly.

Uses 431-437MHz frequency range which is fairly quiet and not subject to interference from WiFi or cell phones.

Works in some complicated 3-way and 4-way lighting situations. Many smart switches, and almost all smart dimmers require a neutral connection to function. In the US, many older houses (or poorly wired newer houses) don’t have a neutral wire in the electrical boxes holding some light switches. This can cause problems for any smart switch, but when you start talking about 3-way or 4-way switches, you often need neutral in every switch box to make things work. Additionally, there are some ways to wire a 3-way light switch that make it impossible to use with most smart switches. This page shows most of the different ways to do so. If your home, like mine, has one of the two bottom ways, the ones labeled “3-Way Switch (Power into Light) most smart dimmers just won’t work. Caseta dimmers, however, work because they “cheat”. Essentially, you can install just one Caseta switch or dimmer to control the light, and take the rest of the switches out of the circuit entirely. You can then use Caseta’s Pico remotes to fake the other switches. These little remotes can be installed into an electrical junction box and look and act just like the full-blown switch, but with no physical connection at all. The remotes are battery powered but the batteries are supposed to work for years.

Disadvantages:

Proprietary protocols. Nothing else uses Caseta signals and it can’t use any other devices.

You will need a Caseta hub. Specifically, you’ll want the Caseta Smart Bridge Pro hub, not the cheaper Smart Bridge. This is because only the Pro hub can be connected locally to other systems, like Home Assistant.

You are not completely independent of the manufacturer or the internet. With the Pro hub, all control is local, with Home Assistant (or other device) communicating directly with the hub, which then transmits to the switches/dimmers. However, to pair new devices with the hub you have to use Lutron’s mobile app. So if Lutron shuts down the product line, you’ll be unable to add new devices, but at least existing ones won’t stop working.

Expensive. At the time of this writing, the cheapest switches and dimmers are about $50 and $60 respectively, and the more expensive ones that handle LED lights better are $60 and $110.

Conclusion

So, what should you use? Well, that’s for you to decide. My personal approach is this: I use Caseta for in-wall switches and dimmers. This is partly because of the 3-way issue, partly because I didn’t know about the WiFi options when I started down the home automation road, and partly because my spouse has gotten used to them and wants all the switches to be consistent)

For all other applications, I use WiFi devices anywhere that I can run some power. Where I absolutely cannot run power, I use either 433MHz devices (mostly for door sensors and basement flood sensors) or Z-Wave (mostly multi-sensors).

You should choose what fits your needs best, but, as I’ve said several times now, if you decide to follow my path down the WiFi road, do it right. Upgrade your infrastructure and flash the devices with open source firmware.

Building Good Home Automation – Part I, The “Brain”

In my previous post, I explained my viewes on “Easy” vs. “Good” home automation and what it takes for a system to be “Good”. So how does one actually accomplish this? You need the right brain, the right protocols and infrastructure, and the right devices. In this article I’ll explore the first of these, and will address the rest in subsequent posts.

What is the “Brain” and Why Do You Need One?

Most home automation systems require some sort of centralized “Brain” as part of the system. The Brain is the software that takes in inputs from various sensors and controls, runs automations and other scripts, provides some sort of UI to the user, and sends commands to various devices and services.

While it’s possible to have a completely distributed system with no central brain, this is uncommon and, in my opinion, limiting.

In most “Easy” systems the Brains live in the cloud on one or more manufacturer’s server. In a “Good” system, the Brain, or, far more commonly, the “hub”, lives in your house/apartment/office/etc. where it can connect to things locally and is entirely under your control.

There are a number of options for the hub. Some are proprietary, some are Free Open Source Software (FOSS).

Proprietary Hubs

Among the proprietary ones, Hubitat appears to be popular, and most likely to meet our requirement of being entirely local. However, being proprietary, I don’t know whether the manufacturer can brick the device.

There are also hubs from Wink and Vera that appear to be mostly-local, and I’ve heard that Samsung SmartThings now allows automations to run locally – I don’t know if that means it can be disconnected from the internet entirely.

The advantage to proprietary hubs is that they are all-in-one device, hardware and software, and may be easier to set up and configure than FOSS hubs. They may form a decent middle ground between “Good” and “Easy”.

FOSS Hubs

There are several free, open source, hubs out there today. The ones I’m aware of are Home Assistant, openHAB, Domoticz, LinuxMCE, and Ago Control. All 5 are software, requiring you to provide your own hardware to run. Some can be run on a one-board computer like a Raspberry Pi, all can be run on a stand-alone PC or Virtual Machine, some can be run in a Docker container.

I first encountered the world of FOSS home automation by way of LinuxMCE. This seemed like an excellent platform, but when I tried setting it up I found it to be problematic. It’s very resource intensive, in part because its main focus is less automation and more media distribution. It requires a dedicated PC or VM to run on, and frankly, seems a bit outdated today. While it has a good community, I found that they were unable to help me overcome what appeared to be very basic issues with installation.

Ago Control was an outgrowth of LinuxMCE, intended to be a lighter-weight automation-only system, but I know little about it. It doesn’t appear to be very popular.

openHAB is one of the oldest FOSS home automation systems, and appears to be quite good. However, it seems to have lost some popularity of late, leading to fewer new integration and less community support.

Domoticz also looks like an excellent system, although it seems to be less capable than openHAB or Home Assistant. Its followers love it though, and if it fits your needs it’s probably an excellent choice.

Finally, the system I prefer and use is Home Assistant. It’s an incredibly powerful home automation platform, capable of integrating with thousands of different kinds of devices and services. It is in very active development, with new integrations appearing weekly and a very robust and supportive community that’s able to answer most questions very, very quickly.

Most of us started with Home Assistant (HA) running on a Raspberry Pi – a $35 single-board computer. This works, and some users have stuck with the Pi for years. It’s a very cheap and easy way to get into HA, get things set up and working. Many of us later moved our HA systems to more capable hardware. While it’s not necessary, I do recommend doing so if possible. It can run on a standalone PC (many use small PCs like the Intel NUC or old laptops), as a VM under ESXi, Proxmox, or Unraid, or as a Docker container.

For me, the choice came down to openHAB or HA. I chose HA because it seems to be more actively developed and have a more active community. Also, openHAB is written in Java, while HA is written in Python. As I don’t much care for Java, that affected my choice slightly, in case I ever want to contribute to the project in the future.

Conclusion

So which should you choose? Well, obviously, you should follow my lead and get Home Assistant 🙂

More seriously, while I do think it’s the best option out there, it may not be the best option for you. Do some research, figure out what fits your needs. I don’t think you can go too far wrong with HA, openHAB, or Domoticz. Hubitat is probably also an OK choice, though I strongly prefer FOSS solutions.

DIY Home Automation in Early 2020

I’ve not written anything on this blog lately, in part because I’ve not been working much on radio or electronics projects. Instead, I’ve spent a lot of my time on home automation, and I’ve now decided that it’s time to write up some thoughts on the matter.

Background

Home Automation isn’t new – I remember playing with Radio Shack Plug n’ Power (X10) devices back in the early 1990s and Insteon devices a decade and a half later.

Commercial home automation isn’t new either – companies like Crestron, Control4, Nortek, etc. have been around for quite a while, offering professionally-installed home automation and media distribution systems.

However, those early DIY technologies were limited, and the commercial systems are expensive – tens or even hundreds of thousands of dollars. The good news is that today, in the beginning of 2020, DIY home automation is cheaper, easier, and more capable than ever before.

Easy or Good – Pick One

Broadly speaking, I consider most modern DIY technology to fall into one of two categories. Unfortunately, I’ve yet to see anything on the DIY side that meets my criteria for being both Easy and Good at the same time, though the tech is getting better every day. Hopefully they’ll eventually meet in the middle somewhere.

Easy

The main advantage of this category is that it’s just that – easy to implement. This category uses off-the-shelf components with minimal setup, but relies on pre-programmed presets and cloud services to handle all of the automation and processing. This category includes things like Phillips Hue, Samsung SmartThings, devices advertised as working with Amazon Alexa/Google Home/Apple Homekit and used only with one of those systems, as well as any devices that work exclusively, or primarily, with their own mobile apps.

So why are these approaches easy? Because most of them require you to plug them in, do a little bit of configuration, and off they go. They aren’t effortless, you still need to do some research and configuration, but they require far less of your time, and far less knowledge, than the approaches in the “Good” category.

So why aren’t these devices good? Well, there are several problems:

Security

The first problem is security – this isn’t as much of a problem for devices that use ZWave, Zigbee, or proprietary protocols, but it’s a huge problem with WiFi devices in this category. Most off-the-shelf WiFi automation devices are built to talk over the internet to the manufacturer’s cloud servers, and that leaves a pathway for an intruder to hijack that connection and use it to cause all sorts of harm. Not only to that device, but possibly using it to to penetrate other systems on your home network, such as home computers. Additionally, many of these devices, especially the cheap ones, neglect security in the first place, and when security flaws are found don’t push firmware updates to the affected devices.

The cloud servers are another potential vulnerability. Especially with the cheap devices, you don’t know who’s running their servers and how good their security is.

If you take away anything from this article, please take away this: if you decide to go the easy route, stay away from WiFi devices, especially from cheap, no-name-brand manufacturers. Stick with ZWave/Zigbee/proprietary systems which suffer far less from this issue.

Privacy

The second concern is privacy. Most of the easy technologies rely heavily on cloud connections, and the companies operating those cloud servers are almost certainly collecting that data and mining it for their own purposes. I don’t just mean voice assistants like Amazon Alexa or Google Home that might record you (less of a threat than many believe, but beyond the scope of this article). I mean things like learning your daily patterns, knowing when people are at home or not, etc. At best that’ll be sold to advertisers. At worst, who knows?

Again, this is more of a problem for WiFi than the other protocols, but it’s applicable to all of them. If you use your Alexa Echo Plus to control a Z-Wave switch, the Z-Wave communication is local, but Amazon still knows that you turned the switch on, as the voice command goes through the Echo, to their servers, back to the Echo, and only then transmitted to the switch over the local Z-Wave connection.

Reliability

This one is pretty straightforward – even the best Internet connections sometime fail. Any device that relies on talking to a remote server, as most of these do, becomes inoperable unless it has a physical button you can press to operate it. Assuming it does have such a button, as most do, then it becomes a plain, old, “dumb” switch.

Bricking

One of the problems we keep hearing about with IoT/Home Automation devices is companies shutting down entirely, or shutting down their product lines. When this happens to a device that relies on a remote server to function, you now have a brick. Lowes’ entire Iris product line was bricked when they decided to get out of that business. Nest bricked their Revolv hubs. A quick Google search for “IoT bricking” will find many more examples, or check out this article.

Even more insidiously, companies are sometimes able to brick individual devices. For example, a few years ago, Garadget appears to have bricked a customer’s device in retaliation for a bad on-line review.

Good

So what does it mean to me for a home automation technology to be good? It means that all, or most, of the following conditions are met: a) the processing hub of the system resides locally at your home; b) all devices communicate with the hub locally; c) the hub and all devices can operate without a connection to the Internet; d) devices and hub do not send any data outside your home unless you allow it, and only what you want them to send; e) devices will continue to work even if the manufacturers shut down.

The advantages of this are obvious: they are the absence of the problems listed above, under “easy”. The disadvantage is that this approach takes more of your time, requires more knowledge (so you either need to already have a decent techie background, or be willing to go out and do a bunch of learning), and can be more frustrating at times, especially early on.

Conclusion

As you can probably guess, “Good” is my preferred approach.

But that’s me. I like to tinker, I have a background in IT, and I’m willing to spend the time to get things to work just “right”. But this path isn’t for everyone. Many (most?) people are perfectly content to have something that just works right away. Totally fair – my naming of the categories “Easy” and “Good” isn’t meant to be disparaging. Not everyone has the time and background to get into the techie side of things, and not everyone should have to. That being said, future posts in this blog will focus on the “Good” approach. Maybe I can show you that “Good” isn’t as hard as I made it seem 🙂