Internet of Things think tank collation and release
For secondary reprinting, please contact the original author
Guide
In order to ensure the full flow and effective use of IoT data, in addition to the connectivity of communication technology itself, it is also necessary to break through the barriers between the various layers of the IoT architecture in order to realize the management of the whole life cycle of data collection, transmission, processing and analysis, so that data can truly play the value of "oil in the Era of the Internet of Things". EMQ Yingyun Technology is an enterprise that has been deeply engaged in the field of Internet of Things data infrastructure for many years, at the 2021 China AIoT Industry Annual Conference, EMQ Yingyun Technology Solution VP Yu Jielin brought a wonderful speech entitled "EMQ: AIoT Data Infrastructure". Here's what the full text of his speech was shared:
The topic I share today is "AIoT Data Infrastructure", which is my experience in the industrial Internet industry over the past decade and my judgment of industry trends, which I hope will be helpful to you.
EMQ is an open source Internet of Things data infrastructure software provider, we have been open source as the core, the product covers messaging and stream processing, as well as the ability of the edge end, the main purpose is to solve the real-time data movement, stream processing and analysis, hope to serve the digital, real-time, intelligent transformation of the service industry.
With the efforts of operators, the AIoT IoT architecture we are now facing basically solves the problem of connectivity. On the end side, there are a lot of heterogeneous devices generating data in real time, which is initially aggregated through a small gateway, connected to the core network through the edge base station, and then entered the cloud data center, where the data analysis is processed and fed back to each end – this is the model of most IoT architectures seen today.
But the problem of connection is solved, and there is still a big difference between data and information, that is, there is no data but does not produce value, and IoT data does not really feed back to the business layer. At present, the application of most AIoT enterprises in China is only in the stage of data, rather than digitalization and informatization.
EMQ believes that there are four trends in digital transformation in the era of the Internet of Things:
First, a comprehensive cloud-native shift, from the original business to the cloud, data to the cloud, to whether it is application services or data infrastructure application services, are gradually moving closer to cloud native. Whether it is research and development, deployment, operation and maintenance, and testing, the concept of cloud life has now penetrated into the industry.
Second, data gradually sinks from the cloud to the edge end, such as V2X with a relatively high amount of data and applications that require device data collection at the edge end, which makes edge computing begin to rise. But edge computing itself is not perfect, and its possible fragmentation also brings difficulties to cloud-edge collaboration.
The third is the development of distributed clouds, many developers of Internet platforms and AIoT applications will develop the original cloud native technology to the edge, and gradually put the content previously concentrated on the public cloud into the distributed cloud.
Fourth, the type of data is also changing, from transactional data and transactional data with obvious peaks and troughs generated by mobile phones in the mobile Internet era to continuous analytical data generated by machines.
Next, I will elaborate on the paradigm shift in data in the era of full connectivity of the Internet of Things.
- Data production subjects and production methods
Data-generating subjects are transformed from humans to machines. The transactional data generated by human behavior can have obvious peak and trough changes such as double eleven and double twelve, while for IoT devices, data is often reported every minute or even every second, there is no obvious peak and trough, and these data have very obvious spatio-temporal context characteristics.
- data type
For data in the AIoT space, the moment it is generated is the most valuable. For example, when you use the charging pile at a certain moment, the reason for the failure is the most valuable to the user, so it is necessary to change from transactional data to streaming data for real-time analysis and processing.
- Data volume
After the connectivity problem was solved, the amount of data connections ushered in a spurt of growth. According to the latest report, the global Internet of Things data will reach the ZB level in the future, which means that most large-scale IoT platforms will face the amount of data in the future from the original GB era to the PB and EB era.
- data processing
For the need for massive data concurrent connections, and at the same time to achieve real-time streaming computing, the original big data architecture is bound to change. Previously, the data was entered into a pre-designed table from the source of the data, and the data was read out from the database for analysis when it was needed for analysis. And now facing a lot of data is semi-structured, unstructured, such as video, audio and other data, for this type of data, often at the moment of production do not know how to analyze, need to have the ability to query, in order to maximize the value of the data, which requires the transition from the original batch processing method to streaming computing, naturally will produce a new database.
Combining the above points, we propose four principles, and our products are also designed and developed based on these four principles:
First, mobile computing takes precedence. In the development of the semiconductor industry, we see the design of shared memory, the cost of copying data at the information level is very low, but the cost of copying it at the physical level is very high, especially in the case of massive data. So we try to move high-value computations closer to data generation, so that computations and data can generate greater value.
Second, the data stream is reusable. In any AIoT device, a data source will not only dock with one system, it is likely that because of the development of the business, the data needs to be accessed once and reused multiple times.
Third, the whole scene can be adapted. In the face of the IoT device-side system, including the capabilities required in the device protocol, any suitable Internet IoT architecture needs to have a component to bridge the access of the fragmented protocol.
Fourth, distributed cloud native. In the future, much of the data will shift from a centralized cloud to a distributed edge cloud for processing.
The biggest demand AIoT places on data is interoperability. The cost of storing data before and then reading it out is very high, there may have been a lot of devices corresponding to an application, if more applications are added later, it will cause a very complex architectural coupling, and the result is that the architecture cannot be scaled horizontally.
If there is a unified data namespace, all the data is shared internally with each other, and different data models are isolated, so that true AI and IoT applications do not care about the internal routing of data, which is the most suitable architecture for the IoT era.
For this unified data access layer used as a namespace, we put forward four requirements:
The first is to be able to withstand a continuous and stable massive amount of ultra-high concurrent connections, and this connection volume is unprecedented in the previous mobile Internet era.
Second, for some scenarios that need to be prevented and controlled, it is necessary to ensure the QoS support of the link, especially in the OTA scenario, and it must be able to establish a reliable data transmission channel and two-way QoS support with the end side.
Third, for the amount of big data, it is necessary to meet the ultra-low latency stateful streaming and analysis data at the same time, so that when the data is generated, it can be directly calculated to generate value.
Fourth, the reusability of data requires stream virtualization, reduce data reuse costs, and quickly find the required data from massive data in order to develop more applications on the data platform.
Based on these principles, our products are constantly iterating. The first is to solve the architecture of massive data connectivity and removable storage in the cloud.
EMQ X is one of the first elastic, reliable, multi-protocol products we started to do, the front end can accept standard protocols, the internal is a message routing layer rule engine, can be seamlessly integrated with other open source components or application systems, so that data can be stored for rapid consumption many times, but also can be deployed in the cloud native, a variety of architecture deployment is very friendly.
The HstreamDB streaming database can synthesize a virtual stream of data for multiple data streams, which can generate a real-time view of the analysis model on it, help with task decision-making, and also provide plug-ins and AI application integration of big data architecture.
The part that talks about AIoT cannot but talk about edge computing. Because the data is at the edge, there is no connection and no data without edge computing, but edge computing has several challenges in the process of solution delivery:
First of all, there is the problem of heterogeneous devices and protocol access at the edge, followed by the lack of interoperability between heterogeneous devices and fragmentation of edge data, that is, some devices are Bluetooth access, and there are also 2G, 3G, 4G access, at the edge end, there needs to be a component that can support cloud-edge collaboration, can identify the state of the device, according to the current situation of edge device edge data autonomy.
For most cloud or software developers, the hardware environment at the edge is still unfamiliar, and this environment is very restrictive for them, and it is difficult to use the original development model. In various vertical industries, different protocols have their own place, and we must have an access layer at the edge to bridge different fragmented protocols and device access.
On the edge side, there are rising device connectivity and computing requirements on the one hand, and resource and cost constraints on edge hardware that are stretched thin on the other.
Combined with a background here, in the development trend of cloud data centers, multi-core processors are already mainstream, and we can be sure that this trend can be achieved at the edge. In the future, there will be more edge multi-core processors to support higher device connectivity and computing requirements. This presents new challenges to both the habits and programming models of traditional edge-side embedded developers.
Based on the problem of edge fragmentation protocol, we first developed the industrial protocol gateway software Neuron to meet the needs of unified access and protocol conversion of equipment in different vertical industries under the background of Industry 4.0.
Neuron 2.0 is very lightweight, easy to port on different hardware platforms, and even portable to modules, which can solve the problem of difficult interoperability. It can also be deployed on a variety of single-core, multi-core architecture-on-chip systems. We also balance the leverage of hardware multi-core performance at the edge with the guarantee of software product portability and multiloadability.
For the field of embedded programming, how to efficiently and correctly use multi-core hardware for the implementation of parallel algorithms, achieve high performance and take into account the characteristics of easy portability and easy adaptation, is a problem that has always existed. NanoMQ addresses this well with internally improved NNG asynchronous I/O.
NanoMQ is the next-generation lightweight high-performance MQTT messaging service engine for IoT & 5G edge computing scenarios just released by EMQ this year, and the first open source edge lightweight MQTT Broker in China. NanoMQ has high-performance message bus capabilities that bridge architectural differences between edge hardware and the cloud, connecting the physical world with digital intelligence. It enables edge message aggregation and redistribution capabilities to solve the massive concurrency problem of the edge access layer, thereby facilitating the development of edge computing applications.
In order to solve the ability to offload computing at the edge and interface with AI systems, we launched eKuiper, which made Flink a lightweight practice at the edge. For example, here is a simple application example: the front end is connected to a camera, and the data inference at the edge can be formed through a plug-in.
In summary, EMQ Cloud Technology provides a perfect product matrix on the connection, movement, storage, processing and analysis links of data, realizing the closed loop of unified data at the edge of the cloud, so as to create the greatest value for data in the closest place to data generation.
We have message data services Neuron and NanoMQ at the edge, and eKuiper for streaming analytics, EMQ X with massive connections in the cloud, and HSreamDB with streaming data processing analytics.
Based on these products, we can build an architecture for the industry. First of all, in the edge part, think of it as a gateway, connect different industrial protocols, if there is an EDGE-end AI application, you can use eKuiper for streaming, and then bridging through the edge of the cloud, so as to open up the data, forming a data from birth to the edge aggregation, storage and analysis of the full link coverage.
With the full link tracking and coverage capability of data, it is possible to form a closed loop of data and transform data into information and business insight capabilities, so as to realize the process of data feeding back into the business.
EMQ hopes to provide some help for the digital transformation and upgrading of enterprises in the IoT industry through our own world-class open source software, and truly serve the industry and society. Thank you!
March 31, Shenzhen
China Intelligent Connection of Everything Conference - Industrial Internet of Things Session
We sincerely invite your participation~
Copyright Notice: In addition to releasing original dry goods, the "Internet of Things Think Tank" is committed to the selection and intensive reading of excellent technology/research/investment articles. Some articles were not contacted by the original author when they were pushed. If copyright issues are involved, or if the source of the reprint is incorrect due to the secondary reprint, please contact us. Contact: WeChat wangsujing1314