IoT devices can be most unfriendly -- from cheap components to bad sensors -- which means handling, consuming and processing IoT data is a key to edge device scalability.
It's a good time to be doing IoT development. At least that's how Pavel Hardak, the director of product management at Basho Technologies, sees it when he looks back at the previous 10 to 15 years of software development.
"Previously, you had to be a very rich company, or work for a very rich company, in order to build such applications," Hardak said when talking about the types of software programs and hardware capabilities that were historically required to only handle internet of things (IoT) data, let alone the cost of building and deploying IoT devices out in the wild. "You'd have to use very expensive software. You'd have to use very expensive hardware. Looking back, it seems like a totally new software development era that we are in today."
How to create scalable IoT applications
When developing IoT applications, Hardak advocated for the use of open source technologies, which he said he considers to be the industry's best, despite the presence of competing, for-profit vendors who are active in the IoT space. "You have best-of-breed open source technologies, like Spark, Mesos, Kafka [and] Riak, that you can experiment with, and most of the protocols are open and extensible."
Looking back, it seems like a totally new software development era that we are in today.
Pavel Hardakdirector of product management at Basho
Of course, that doesn't mean creating scalable IoT applications is necessarily easy. There are plenty of pitfalls that need to be avoided when supporting IoT applications, with the most common mistake being the underestimation of just how much data those IoT devices in the field are going to generate. "They don't understand that data volume is not going to be 20% more," Hardak said. "Instead, it's going to be 20 times, and then 50 times and, very soon, 100 times more in terms of volume."
The flipside of the IoT data load that inevitably gets passed onto data centers is the helpful fact that the data itself is relatively predictable, making it fairly easy to consume. "It's not going to be very complicated data," Hardak said. "The number of sensors, the number of configuration items, user profiles and devices profiles, all of these are pretty simple things."
IoT data-quality challenges
Unfortunately, that data isn't always crystal clean, as IoT devices deployed into the field aren't under the control of the data center. And many IoT devices are built to be disposable, which means cheaper components, cheaper sensors and cheaper software. Add in the fact that the data collected by these inferior components is delivered across an often unreliable network, and your data center has a pretty serious IoT data-quality issue with which to grapple.
Of course, it's not that these issues are insurmountable. Hardak and his team at Basho solve these types of challenges every day, which is exactly why he's been called upon to speak at QCon 2016 in San Francisco about the various ways organizations can build scalable IoT pipelines that an consume, store and process huge amounts of data efficiently and cost-effectively.
To learn more about creating scalable IoT apps, listen to the accompanying podcast with TheSeverSide's Cameron McKenzie and Basho'sHardak.
