I’ve had this idea forever
Well, for longer than you might otherwise imagine; just think of this as an idea that was formerly ahead of its time. And I know what you’re probably thinking: oh great, another smart home blog. Its not. I’ve had very specific requirements for a smart house that I’ve held onto for years. The time has come to finally build it.
A smart house should be, well …smart
Yes, it’s true that scheduling, monitoring, and automated control over house systems, and connecting data exchanges between IoT devices is very cool and useful. However, since our personal lives aren’t necessarily scheduled and automated, house systems should also have some basic common sense wired-in.
For me personally, although I work from home I travel occasionally on business, so am sometimes away for days at a time. There are often family and friends home when I’m not, and of course there are also times when everyone is gone. So although controlling house lighting to simulate presence is great, most of my interest in a smart home relates to monitoring and security.
The overarching point that I’m driving towards, is that these types of systems shouldn’t get in the way of living our lives, like some nuisance that needs constant tending. They should be invisible and unobtrusive, blending into the background and out of way; on the job when needed, and out of sight when not. And they should most definitely know when they are needed, without being told.
A house should recognize residents
It just seems to me that my house should know who lives there, and when they are home. That is why this project has a large computer vision component, including object detection, tracking, and facial recognition. There are of course, other easier ways of determining who’s home.
Why should the house care whether or not someone is home?
- Selected automation tasks, such as lighting, could be suspended when someone is present
- Personal preferences can be applied based on identity
- Perimeter security can be relaxed when someone is home, and tightened when everyone has left; or are all asleep
One simple solution might be to assign a unique address or name to a personal device, such as a phone, and associate that with each resident. If their device is connected to the house network, they must be home, right? Unfortunately this is not always reliable, as they can leave home without their phone, leave their phone out in the car, perhaps it was forgotten and left at work, lost, the battery might be dead, etc.
In fact, the main downside to any scheme requiring a physical token, is that everyone known to the house also needs a registered device; and that can be a hassle. Especially now that facial recognition tech is working so effectively, and can be easily deployed to the very edge of the network. It’s conceivable of course, that vision could also be just one aspect of a more complete solution designed to correlate presence detection factors from multiple sources.
“House, this is Bob. Bob is a house guest.”
Without being too draconian, a few well-placed cameras near doors and entries can track who’s coming and going. If desired, this could easily be extended to tracking movement within the home as well. A conservative approach to this would include just the main room, and one or more key hallways.
But the main foyer perhaps especially, needs vision: because we want to have a place to easily introduce new occupants.
No-hassle casual introductions
Enrollment needs to be simple and spontaneous. Obviously, family should be easy enough, most folks have lots of photographs that can be used to train a recognition model. You might want your house to be able to recognize not only family, but also friends, neighbors, and house guests. At your discretion, perhaps exterior doors should always be unlocked for house guests.
Ideally, this could be as simple as a voice command, with you standing next to the new individual while both of you are within view of the same camera “…House, this is Bob. Bob is a house guest”. The house would recognize you as someone who can make introductions.
I do feel just a bit weird talking to my house…
Confessional: I do feel just a bit weird talking to my house, so a console interface on a wall panel or handheld tablet could also be used for this. The idea here is to simply assign names to faces selected from a list of those that have been newly discovered. Ideally, video feedback should already be confirming that the house is correctly identifying your friend as “unknown 1”, or whatever you’re using for that.
For this to work, unrecognized new faces need to be enrolled automatically, by first capturing as many different views as possible for modeling. Ideally we would have both frontal and one or more angled views captured as an outcome from object tracking. For an unrecognized face, this would simply trigger a batch job for the deep learning needed to be able to subsequently recognize that individual. Until formally introduced, they get a default name, such as “visitor 1”, “visitor 2”, and so on. This should gives us a decent shot at ongoing recognition of them as a unique individual, ready for an introduction at your option.
A house should recognize the cars that belong there
The house should recognize my car, or any other car that might be known to come and go regularly. Whenever I pull into the driveway, I don’t need the house to take any notice of that. I already know that I just pulled in.
…could be just just a pizza delivery, or a friend dropping by
A strange car in the driveway might not be of much concern whenever someone is known to be home, because that could be just just a pizza delivery, or a friend dropping by to say hello.
However, a strange car in the driveway when no one is home, well, that might be an event you would want to know about. That’s also probably a video worth keeping for future reference.
Was there just a delivery? Has the postal carrier been by? The house should know these things. Once a model has been trained for it, the house could reasonably even be able to closely anticipate when the mail is expected to arrive based on historical measurements.
Safe, secure, contained
Safe and secure please, and isolated. A smart home should be self-contained, and able to operate effectively without a connection to the Internet. I know that we’ve taken to calling this stuff the “Internet of Things”, but none of this should require a reliance on cloud-based third party services, or vendor subscription schemes to make it work.
I strongly believe that internal house systems should be deployed on a separate network segment, with firewall-enforced isolation. We don’t want remote snoopers commandeering the cameras or gaining access to house systems.
Access can easily be granted to anyone, from anywhere…
Access can easily be granted to anyone, from anywhere, directly from laptop, tablet, or phone as desired… but only if they’re holding the key specifically assigned to them. Password authentication won’t be used.
An example proposed infrastructure for this project does allow internal systems to connect to the Internet as needed. Providing this opening simplifies such things as software updates and access to external services. For all the zealots out there, this security hole could be closed off by building a tightly-controlled caching server for distributing software updates internally, and at the very least, supplying an in-house time server.
Full encryption for all inbound connections
Inbound connections to house automation services are only allowed through encrypted tunnels, via a reverse proxy hand-off that has been restricted to known private network addresses. This allows us to avoid the use of dynamic naming and opening firewall pinholes into the home through the public-facing house connection. Fortifications with the fewest cracks in them are best. The idea here is to keep the attack vectors narrow, and few.
Freight paid against the infrastructure haul for this, is minimal.
Some folks might want to push copies of captured video, and system backups out to cloud storage for safe keeping. From a security perspective, this of course protects records in the event that house systems are destroyed or removed. Whatever floats your boat for that, whether its something like Dropbox or an Amazon S3 bucket, or perhaps just left on encrypted disk in a cloud-hosted virtual private server.
What’s next
An open source solution based on open source underpinnings. To build a distinct set of services and the application architecture needed to support this functionality, and allow for easy adoption by others. Heavy reliance on newer single board computers with the small form factors and low power requirements, such as the Raspberry Pi 4B and the NVIDIA Jetson Nano. Piling on GPU/TPU co-processors wherever helpful.