This IDC blog post points to an IDC Perspective that illustrates and discusses data integration patterns that have emerged in the IoT, the justification for using each pattern, and the implications of use that may derive additional requirements. It provides guidance to the IoT analytics architect on making decisions about which pattern to use under known constraints.
As the number of devices and sensors in the Internet of Things (IoT) grows, so do the questions about how to architect IoT data streams and integrations in support of continuous analytics and information management. Data from the IoT is being used for diagnostics, business analytics, logistics, and sense and respond business scenarios. Recently there has been a lot of discussion and debate in the industry about where analytics of IoT data should occur: at the edge or centralized. In reality, analytics can and will happen at the edge, or centralized, or both at the edge and centralized.
The use case and architectural constraints will drive out the most appropriate solution design. For example, some analytics need to happen at the edge as in an autonomous vehicle analyzing road and traffic conditions. Use cases that include integration of IoT data with business operations, sales, or marketing data will require moving data from the edge to a centralized location for integration and analysis. Architectural constraints also need to be considered, including connection reliability, network bandwidth, security, latency, and processing capacity at the edge and central locations. Once functional and non-functional constraints are understood, decisions can be made about where to place analytics and how to move data if required.
What's old is new again. Patterns have been a popular method for helping to define data movement, integration and processing problems and solutions. Common patterns have been observed in IoT data integration solutions using middle-tier components to collect, move, normalize and prepare data for analytics. Many of these patterns mirror and echo patterns that emerged in the client-server era and in multi-tiered architectures, but at levels of scale, volume, frequency, connectivity and security that drive new designs and decisions. IDC has documented these patterns, and justifications for using each pattern as driven by functional and non-functional requirements associated with the IoT.
Patterns of moving data between the edge and centralized locations using direct connections, or decoupled communications through databases and messaging, are justified by requirements, and have implications of use. Intermediaries are also being inserted into IoT data movement patterns as a method of distributing data filtering, normalization and cleansing closer to the edge as a way to reduce the volume of data being sent to centralized analytics applications.
Patterns can also help with deployment decisions: where and when is cloud appropriate for edge, intermediary, and messaging components. Selection is often driven by edge capacity, whether or not the edge is mobile or stationary, and connection reliability.
IoT offers tremendous opportunity to organizations looking to leverage the data and insights that can be derived. It also represents a problem of massive scale and complexity that could result in inefficient solutions that are costly to manage and maintain, especially if an architectural approach to solution design isn't used. For all this world has learned about the importance of architecture and design of physical things such as buildings and infrastructure, it is often put to the side in the virtual world under the banner of exploration, innovation, agility, or time and money pressures. An architect's response is often pay at the beginning to do it right or pay (more) later to clean up the problems and fix the mess.
IoT solution architecture will be critical, and as a result, there is a new role emerging of the IoT architect. Patterns have been effective in IT solution and system design, and will continue to be effective tools for IoT solution and system architects. In the IDC Perspective: Internet of Things: Data Integration Patterns, we document the patterns that have been seen to date. A constant in IT is change, and therefore we expect these patterns to change, evolve and we also expect to see new patterns emerge as the IoT expands and solutions mature.