Zhi DongXi (public number: zhidxcom)
Author | Yang Chang
Edit | Three Norths
Zhidong reported on April 29 that recently, Amazon Cloud Technology held a big data and artificial intelligence media communication meeting in Beijing, announcing the launch of a "cloud, digital, intelligent trinity" big data and machine learning integration service combination. Amazon Cloud Technology also shared amazon cloud technology cases in helping enterprises integrate data and intelligence with partners such as Joyme and Shanghai Convertlab.
Chen Xiaojian, general manager of the product department of Amazon Cloud Technology Greater China, said that Amazon Cloud Technology has two important concepts in the integration of digital intelligence: one is to achieve the integration of data and intelligence in the cloud will become the engine for enterprises to accelerate innovation, and the other is that enterprises should create a unified data foundation in the cloud to achieve the "double sword combination" of big data and machine learning, providing a new engine for enterprise development.
Wang Xiaoye, director of the technical expert team of Amazon Cloud Technology Greater China Product Department, analyzed in detail a number of upgrades of Amazon Cloud Technology's "Smart Lake Warehouse" architecture in the direction of deep intelligence.
How to integrate big data and artificial intelligence technology? What research and discussions has Amazon Cloud Technology conducted in this regard? After the integration of big data and artificial intelligence, what changes can it bring to enterprise users? This article explores these issues in depth.
First, the cloud digital intelligence integration service to meet the three major challenges of machine learning landing
Chen Xiaojian said that with more and more data in enterprises and more advanced machine learning models, many enterprises hope to further drive business innovation and enhance enterprise output through the integration of big data technology and machine learning technology.
However, enterprises often face the dilemma of having a large amount of data and analytical computing, trying a variety of advanced machine learning models, but it is difficult to have actual business output. From the perspective of technology development trends, big data technology and machine learning technology take different routes, big data attaches importance to the collection and optimization of data itself, while machine learning technology values the optimization and parameter adjustment of the algorithm itself.
Chen Xiaojian said that specifically, the production of machine learning in enterprises faces three challenges. First, big data and machine learning are divided and ruled, these two parts are often the responsibility of different teams, which are prone to data islands and technology islands, which restrict the agile iteration of related applications. Second, the ability of data processing is insufficient, and it is difficult to process massive amounts of business data, which restricts machine learning from experiments to practice. Third, the participation of data analysts is low, and the algorithm model that performs well in the research and development test stage may not produce satisfactory results in actual use, because the complexity of the real environment is higher.
To this end, Amazon Cloud Technology has launched a "cloud, digital, and intelligent trinity" service portfolio.
The first is to build a unified digital governance base in the cloud and break down data and skill silos.
By helping users build a unified digital governance base, Amazon Cloud Technology realizes the data sharing of big data and machine learning applications used by users, unified control of data permissions, and unified development and process orchestration of the two.
The second is to provide production-level data processing capabilities for machine learning, and help machine learning turn from experiment to practice.
Amazon Cloud Technology can provide a variety of flexible, scalable, and purpose-built big data services to help users perform complex data processing-level processing to cope with dynamic changes in data scale and optimize data quality.
The third is to make data analysis intelligent and empower enterprise business personnel to explore and innovate. Amazon Cloud Technology provides users with smarter data analysis services, allowing business people to complete intelligent analysis, model effect verification, and autonomous innovation.
Chen Xiaojian said hundreds of thousands of users around the world are using Amazon Cloud Technology's big data and machine learning services.
Second, the in-depth intelligent capability of "Smart Lake Warehouse" has been upgraded
Last year, Amazon Cloud Technology launched the "Smart Lake Warehouse" architecture to provide users with related digital intelligence services. Wang Xiaoye shared what new changes have been made in Amazon Cloud Technology's "smart lake warehouse" architecture since its release.
In terms of the unified data governance base in the cloud, Amazon SageMaker Studio of Amazon Cloud Technology can complete data development, modeling and related production tasks in one stop, providing a unified development platform for big data and machine learning.
Amazon Cloud Technology also offers Amazon Lake Formation, an app that adds several new features that help users share data assets across departments of the data grid, as well as the finest cell-based permission control mechanism.
In terms of providing production-level data processing capabilities for machine learning, Amazon Cloud Technology has Amazon Athena, a big data platform that can support multiple open source frameworks. Amazon Athena can perform federated queries on data from data sources such as Amazon EMR, Amazon Aurora, a high-performance relational database, Amazon DynamoDB, And Amazon Redshift, and other data sources to quickly complete data processing modeled by machine learning.
Amazon Cloud Technology has also built serverless analytics capabilities, including applications such as Amazon Redshift, Amazon Managed Streaming for Apache Kafka (Amazon MSK), and Amazon EMR. These allow users to process data of any size without having to configure, scale, or manage the underlying infrastructure, providing performance- and cost-effective feature data preparation for users' machine learning projects.
In terms of data analysis intelligence, Amazon Cloud Technology integrates machine learning model prediction capabilities into daily analysis tools, and also provides services such as Amazon Glue Databrew, a visual data preparation tool, and Amazon SageMaker Canvas, a zero-code machine learning model tool, allowing business people to explore machine learning modeling.
Third, cooperation in the development of new applications has been effective in many ways
Amazon Cloud Technology also invited the relevant responsible persons of le me unlimited and Shanghai Xin Zhaoyang to share the background, process and results of its cooperation with Amazon Cloud Technology.
Yang Fei, director of leme unlimited data research and development center, said that LiveMe, a global live broadcasting platform operated by Lemei unlimited, has users from more than 200 countries or regions, and the amount of data is very large, and it also needs to achieve compliance operations.
Based on Amazon's cloud technology solution, Lemi Unlimited has built a live content identification system and a fraudulent transaction identification system. The live content recognition system helps Le I infinitely improve the user experience and reduce the business cost of content management. Through the fraud identification system, Lemi Unlimited reduces fraud and chargeback transactions, and thus reduces economic losses by millions of dollars per year.
Li Zheng, co-founder and CTO of Shanghai Xinzhaoyang, said that Shanghai Xinzhaoyang and Amazon Cloud Technology have cooperated in marketing digital transformation. Shanghai Xinzhaoyang is a marketing cloud manufacturer that can provide integrated marketing cloud products.
Marketing based on data intelligence will bring more opportunities to enterprise users. Artificial intelligence allows user-centered marketing to be guided by thousands of people's information. But data intelligence marketing solutions also face multiple cloud challenges. First, data governance is weakly related to machine learning operation processes, second, data processing and analysis require a lot of energy, and third, the efficiency of management aspects such as model iteration and maintenance is low.
Based on the unified data foundation of Amazon Cloud Technology, Shanghai Xinzhaoyang has built an integrated data intelligent lake warehouse architecture Data Hub and an integrated and efficient machine learning platform AI Hub. These two applications can improve the timeliness of data circulation by 32%, and increase the efficiency of model launch by 30%.
Conclusion: Cloud-digital hybrid enables digital transformation of the industry
Based on its own practice and industry observations, Amazon Cloud Technology has created a set of "cloud, digital, and intelligent trinity" service portfolio to provide users with solutions that integrate artificial intelligence and big data.
Not only artificial intelligence and big data technology are empowering enterprises to digitally transform and develop more efficiently, but also intelligent technologies such as the Internet of Things, digital twins and so on to promote the better development of enterprises.