Sensory surveillance is the hidden price you pay for all the smart devices around you

Raju Vegesna Profile picture for user Raju Vegesna November 11, 2020
Today's smart devices come with a hidden price tag for all of us. Zoho's Raju Vegesna explains the concept of sensory surveillance and why you should care

Smart speaker in focus with blurred figure behind sensory surveillance © Andrey_Popov - shutterstock
(© Andrey_Popov - shutterstock)

Recently I was shopping around for a 'dumb' television that does not connect to the Internet, but I couldn't find one. In fact, it's difficult to find dumb devices or appliances these days. If you look at the top discounted consumer products during this year's Prime Day, for instance, they were all 'smart', meaning pre-enabled with an AI assistant. From smart smoke detectors to security systems to refrigerators to speakers to thermostats to practically any in-home consumer electronic, they're all collecting user data and monitoring customers mostly without their knowledge. This practice is what I call Sensory Surveillance, which is the act of tracking, storing, processing, and in some cases selling information about consumer behavior across all the human senses — vision, smell, hearing, taste, touch, even mood.

We at Zoho feel a moral obligation to educate consumers about surveillance so that they can make informed decisions on the products they buy and how their data should be used. Zoho takes a hard stance on privacy. We don't own customer data, we don't sell customer data, and we don't sell ads. That being said, we also don't make consumer products, we make business solutions. Thanks to the consumerization of business applications and office technology, however, we expect sensory surveillance to show up on the business side as well.

Take for instance smart televisions, many of which come pre-enabled with Automatic Content Recognition (ACR) technology. ACR takes snippets of content a consumer is watching every few seconds or minutes. That data is sold to marketers through ACR vendors who themselves are partnered with Smart TV manufacturers. Some ACR vendors go further and can target users on their mobile phones so long as they share an IP address with the user's smart TV.

These televisions have eyes on consumers and what they're watching. As many companies move away from projectors in the office and toward TVs, is business information and private work data being leaked and monitored as well? It's likely. And this trend will continue as more and more consumer devices enter the workplace.

So the first step is awareness on the consumer side. Surveillance companies consider your data to be money (because with ad-based revenue models, it is), but most consumers would not consent to handing over their wallets, so why should they consent to handing over their data? Let's take a look at the many ways surveillance companies are tracking consumers' senses.


Thanks to the meteoric rise of smart-home camera systems, surveillance companies 'see' what consumers see, inside or outside of their homes. They know what time a customer's kids get home from school, what time the mail is delivered, whether they have a dog, and all that data is being processed in the cloud. Advancements in face detection let these vendors know exactly who is visiting and when. 


As I mentioned before, it's nearly impossible to buy an electronic device nowadays that doesn't come integrated with a smart assistant. In exchange for voice command and app integration, consumers are being listened to around the clock. After all, smart speakers are always on, always listening. But it doesn't stop there. In some cases, consumers are not even aware that some of their devices contain microphones.

Surveillance companies may be using consumer data to improve their services, AI in particular. By listening to conversations in the home, these technology vendors can massively increase the data sets they use to train their AI engines. Just like with facial recognition, companies have made significant progress detecting an individual's voice (voice fingerprinting). These always-on microphones should be opt-in systems and not opt-out systems. This is not the case simply because opt-in systems are not in the best interest of surveillance companies.


Thanks to smoke and carbon monoxide detectors that are connected to the Internet, we have potentially opened the door to smell surveillance — olfactory data that is aggregated and could be packaged and sold. Today we have to be thankful(?) that somebody hasn't cracked the 'smell' puzzle for ads yet. When we get there, the systems and data are all in place. Fifteen years ago it would seem unfathomable that what we as consumers smell in our own homes would be of any interest to technology vendors, but it is on its way to becoming a monetizable asset.


Online grocery shopping, food delivery, recipe downloads — all of this taste information is now available to be aggregated (if it's not being done already) at the backend. In 2018, the average American household spent 13% of their income on food, which is more than they spent on entertainment, health care, personal insurance, pensions, and education. Targeting consumers based on their eating habits is huge business, and the collection of that data starts in the home through connected devices like refrigerators and follows customers into the grocery store or drive-through line or online reservation system. 


From the movies and shows you watch to the music you listen to, to the websites you visit (thanks to the routers and wi-fi devices provided by surveillance companies), there is a trove of information being monitored. When aggregated, this data paints a fairly clear picture of somebody's mood. In some cases, the companies might know how your feeling by what you're watching even before you do, and that type of early detection is a precious advantage for advertisers and marketers.

Surveillance is nothing new. It has, however, taken on new forms. Surveillance started in a straight forward way — the collection of user information with the user's knowledge. Soon we had adjunct surveillance, where tracking happens without users' permission. And today we have new tentacles of surveillance — like sensory — which may be in its infancy, but we have to recognize what is going on and understand what we are giving up. Your new 'smart' gadget might come with an attractive one-time price tag, but you may be paying a price every day for it. You are paying with your private data and you may not even know it.

A grey colored placeholder image