Rick's Wrist - Controls Your Smart Home From Anywhere

Every year, I host a Halloween party with a single word as its theme, where people dress up based on their interpretation of that word. This year's was "mad", and I couldn't think of a mad scientist that was also as angry-mad as Rick from Rick and Morty. Wearing a lab coat and dawning the blue hair doesn't cut it, though. It was time for epic gadgetry.

The idea of the gadget is to greatly expand on the setup that I had started in a recent project of mine with a custom smart home setup. The gadget would be able to control all these smart home devices, with a ton of extra custom functionality added, and be able to do so from anywhere using Blues. And, to address the fact that Rick's gadgets can seemingly do anything, I added the ability for AI to control these Smart Home devices as well.

We are using a UNIHIKER to make commands, Blues Wireless to make it so that we can make commands remotely, and a flask server. From there, we have the various smart home devices that we control, including custom made ones. We can have anything complicated run on the flask server because it's just chilling in my house with wifi connectivity. The Flask server acts as the command hub, managing device interactions, AI processing, and API integrations to keep everything centralized and easily customizable. That means the ai commands and more intricate logic all get handled on the server. We'll bring it back to all the neat new features added to our server, but in the meantime we need a way to make our commands in the first place.

The UNIHIKER is listening for voice commands that start with "butler" as our trigger word. I greatly enhanced the voice command processing logic here from my previous iterations. Previously, you had to speak fairly quickly or it would break the command into multiple segments. That is now resolved and working well.

When a command is recognized, we also display that command's text to make it more obvious that we did indeed have a command processed. This also helps clarify when the command actually sends, since longer commands sometimes take a bit to process.

Since there are built in buttons, I added a couple of built in commands to trigger from both of the side buttons. Since this all circles back to being part of my Rick costume for Halloween, I named them Halloween 1 and 2. We'll come back to activating devices with this later.

This brings us to the initial working title of this project - the "Remote Remote", where we are able to control our smart home from anywhere. This is where Blues comes into play. Using the Blues Notecard allows remote control of the setup without relying solely on local wifi, enabling true Remote Remote functionality. First, the python library is note-python so we'll need to include that in our Unihiker project.

For connecting the devices, simply connect them via usb. It's the easiest way and works right out of the gate.

Next up is getting this to work. I'm using a Notecarrier-F. If you don't already have your device setup or a project ready to go, you can find the quick start guide here. Step 1 is to create your project. You click your name in the top right, go to View Projects and click Create Project.

This gives you your project id that you'll need going forward. The quickstart guide gets the device connected and working with the project as well.

Within your Blues project, create a new route. We'll be using the first option, which includes webhooks.

You can name your route whatever makes sense to you. The main part we need to get working here is the url. Until now, the interactions with the flask server have been via local wifi. In fact, you'll notice in the code that we check for wifi before using Blues.

To be able to hit the the server externally, I tried a few options and ended up having the most success using something called Local Tunnel. It's free.

To install it, just run the following on your server. Mine is essentially a raspberry pi.

npm install -g localtunnel

Then you can start your server, but one feature that I really liked is that you can specify your subdomain. This makes it so that it's your own custom domain, but also ensures that you won't need to update the url on the Blues side if the Local Tunnel server stops.

lt --port 5000 --subdomain <your subdomain here>

And with that, you are up and running with a server that can be hit externally. We need to include the topic as well in the url for Blues, so it should look something like this:

https://<the subdomain you picked>.loca.lt/command

With that, you'll be able to hit the server with commands from anywhere without issue.

Considering that all of this leads to us being able to control smart home devices, we need smart home devices to interact with in the first place. Unluckily, setting up the ability to control store-bought Smart Home devices turned out to be notably more involved than one might hope.

One specific feature I want available in my home involves my nest thermostats, so I specifically wanted to ensure I could interact with them. Let's quickly go through how to setup a nest thermostat such that we can access it from our flask server.

Go to the Google Cloud Console and make a new project.

Navigate to APIs & Services > Library and enable Smart Device Management API.

From APIs & Services > Credentials, select Create Credentials and then OAuth client id. We'll need to be able to authenticate within our program and we'll do so using these credentials. You can download the client_secrets file to reference within the program and/or note down your client_id and client_secret.

Next go to the Device Access Console, setup a new project, and copy your Enterprise id.

This is the part that took me forever to get figured out. You need to grant permissions for your specific devices. The easiest way I found to do so was via the following url. Obviously, you'll need to fill in the url with your own project id, redirect uri, and client id, but then you'll be able to get the permissions setup and your project will actually work. Without doing this, it'll seem like commands are working correctly but that there are no devices to interact with, which is clearly not correct.

https://nestservices.google.com/partnerconnections/YOUR_PROJECT_ID/auth?redirect_uri=YOUR_REDIRECT_URI&access_type=offline&prompt=consent&client_id=YOUR_OAUTH_CLIENT_ID&response_type=code&scope=https://www.googleapis.com/auth/sdm.service

As noted, the server is where a lot of the hard work takes place. The main reason I wanted the nest to be able to be adjusted wasn't for simply changing the temperature with voice commands - that can already be done with existing devices. The goal is to add a very nice convenience feature.

The ac upstairs in my house is very loud and not conducive to recording music whatsoever. So, I put together a function where I say "I want to record in x time". It parses that time, including a bunch of possible options like "in half an hour", and then sets up a couple of timers. It blasts the ac until it's time to record, and it also turns on my computer a minute before it's time to record so I have a head start.

This feature really lends itself to the "remote remote" idea of the project. Being able to know you're 15 minutes away from home and have this working for you is really nice. It's becoming a lot less relevant now that it's no longer molten lava outside, but when next summer hits I can see myself getting a lot of use out of it.

That is the main custom feature I added, but I did also include a command to just turn on my computer as well as to set the temperature of a specified nest thermostat.

Whether this is a good idea is one thing. What I know is it's a fun thing, and that it's neat.

The reason for adding this is that Rick's gadgets can do seemingly anything. Realistically, I can't predict and program functionality for every possible situation. However, as I add more and more smart gadgets, custom and otherwise, AI could be utilized to command those devices in whatever way best suits what we ask it to do.

Prior to the actual AI part of this, one of the other custom functionalities I added to the server is a function that updates a list of smart devices. To actually leverage ai to control the smart home devices, we pass it its instructions, the command it's supposed to make happen, a list of all smart devices it has available to work with, and the exact format we expect back from it.

I'm using OpenAI's gpt-4o for this since it has the most up to date logic, and I want it to really consider the best approach in the case of complicated requests.

To use the feature, we start the command with ai. Including our trigger word of "butler", an example might be "butler ai, I want you to turn off all the lights downstairs". It should then identify the lights it thinks it should turn off, sends us the actions to execute, and then the flask server passes those commands to the relevant devices.

The Meeseeks!

We have the ability to control any device we put together, and an absolutely tragic lack of Meeseeks. It only makes sense that we create a simple smart device for us to control the Meeseeks' box. Custom devices, like the Meeseeks Box, use MQTT for communication, allowing the Flask server to dispatch commands reliably across the smart home setup. The device is a simple Esp8266 with a servo. When we get the command "Halloween 1", it opens. When we get it again, it closes. Rinse and repeat, and you have a cute little party gag.

The arduino code is pretty simple, but one thing that's worth noting is that I changed the topic that the custom devices listen for so that we don't end up in an infinite loop where the server publishes to a topic it's listening to.

I may not have built a space ship out of things I found in the garage, but our home is now smarter than ever, and we can now even let AI control it to account for as many use cases as humanly possible. Hope you enjoyed - have a good one!