The five keys to mastering software analytics, logging and reporting

The wealth of conclusions and extrapolations that can be obtained by sifting through Internet of Things data demands that analytics be tightly integrated into all IoT applications and devices.

The basic principles of analysis apply equally well to a world of drones, embedded devices and driverless cars as they do to business applications running on an enterprise server. The only differences, and they are potentially game-changing, are the scope of the data that can be obtained and the significance of the conclusions that can be extrapolated from it.

IoT analytics can be broken down into the following main points:

1. Report 

Reporting data is fairly straight forward in a controlled environment, where server-side software is running locally or an application is running on a user’s desktop. Reliable network protocols like User Datagram Protocol or TCP/IP can be used by desktop applications to deliver data back to a central host. Server-side software makes reporting even easier, as analytics software can be plugged right into the Web server or applications server, grabbing metrics and triggering alerts in real-time.

These types of luxuries do not exist in the world of embedded and IoT devices. Whether it’s a driverless car being tested in the Arizona desert or an autonomous marine robot collecting scientific data in the middle of the ocean, the availability of reliable networks is nascent. Even when reliable networks are available, they lack the bandwidth necessary to upload all of the information they acquire.

Therefore, new strategies for metrics reporting need to be employed. Data is typically prioritized. So when network latency is high, only the most critical pieces of data are uploaded. When latency improves, secondary priority data is reported. Only when an IoT device returns home will data with tertiary and quaternary priority be uploaded to the servers.

This, of course, presents problems. Real-time reporting is often not possible with IoT devices. When there is a delay in reporting, there will be a delay in aggregation and analysis. That postpones software adjustments and bug fixes, not to mention the stress it places on staff as their work schedules are dictated by the unpredictable reporting schedule of devices running live in the field. There is no question that IoT devices require an entirely new approach to data reporting and how organizations respond to that data.

2. Aggregate

The whole purpose of many IoT devices is to send data back to their home base about how they’re functioning and what they’re experiencing. The amount of data that an army of IoT devices might report back to a cloud server could easily overwhelm a relational database. Even if a traditional database system was sufficient for storing that information, querying or mining such giant clusters of data would be a frustratingly slow process.

We’re seeing a trend where there is just too much data and too many endpoints coming in.

Sean Bowen, CEO of Push Technology Ltd

“We’re seeing a trend where there is just too much data and too many endpoints coming in,” said Sean Bowen, CEO of Push Technology Ltd., a London-based company that develops software to improve the performance of Web and mobile apps.

As a result, new strategies are being employed in the aggregation and processing of IoT data. NoSQL databases that have looser transactional requirements but much better synchronization and storage capabilities are often a better choice for data storage and retrieval than traditional relational database management systems. Processing the large blocks of data that IoT applications collect is often a task better suited to a Hadoop cluster, a technology that was originally developed to help index the Internet, as opposed to standard query calls against a database.

3. Analyze

Tools that present aggregated data in a way that makes it easy to understand are paramount. Presenting meaningful data in graph-based structures or on dashboards that can be customized will always be a best practice, regardless of whether server-side or IoT data is being analyzed.

The big difference with IoT analysis is that the tools being used must be compatible with the underlying data aggregation tools. That means using analytics tools that work with technologies such as Hadoop, cloud data warehousing tools, Amazon Web Services-based big data solutions and various NoSQL databases. It also means many organizations will need to move away from what they’ve used in the past and acquire new tools that can process their data effectively.

4. Conclude

Analytics doesn’t end at analysis. If anything, the most important part only starts there. After all, the whole point is to look at your data and come to meaningful conclusions about your systems. Analysis tools help in this regard, and data mining tools that reveal trends or hidden correlations are vital to helping an organization come to conclusions and make decisions about how to update, change, adjust and improve a system. But, in the end, coming to conclusions about the data being analyzed will always remain a largely human endeavor.

5. Extrapolate

The truly exciting parts of IoT analytics are the extrapolation opportunities. Serendipity abounds, as information gathered from one application may inspire the development of a new or unrelated technology.

For example, Facebook check-ins were originally designed to help users share information about where they were and what they were doing. That data is now helping users figure out when the lines are longest at their favorite Starbucks or the local DMV. The exciting part of IoT analytics is the potential to discover unknown treasure in the data.

With the right analytics tools and the ability to report, aggregate, analyze, conclude and extrapolate, IoT opportunities abound.

What tips do you have for optimizing IoT and enterprise analytics? Let us know.