Written by | Wang Ruiping
Review | Yun Zhao
Open source, Github star soaring 20,000, CVP (i.e., ChatGPT+VectorDB+Prompt) architecture... Another database dark horse stands out.
It is the Milvus vector database, which was open sourced on Github a few years ago and instantly exploded around the world, not only perfectly interpreting the emerging CVP architecture, but also becoming a clear stream of "group model dance".
Image
Milvus aims to power AI applications and embedded similarity search, making unstructured data easier to search and access, enabling users to have the same experience regardless of the deployment environment.
1. The new era of AIGC moves from "vector database" to "large model memory"
Milvus entered the queue of Github's 20,000-star project at a rate of 5,000 stars in 4 months, and more and more developers are injecting more power into Milvus into the new era of AIGC.
In 2019, GitHub open-sourced Milvus' source code. At that time, there was no "vector database" in the industry, and the most concerned question for users was "what is a vector database".
In 2023, ChatGPT sparked the AIGC craze, and in this year of unbridled growth of large models, more and more people are concerned about "what can vector databases do?" ”
In the era of large language models, vector databases are indispensable, which is not only the memory of large models, but also makes large models knowledgeable through vector retrieval.
In this process, AIGC developers can see that Milvus has added three new functions of JSON, dynamic schema, and PartitionKey in version 2.2.9, simplifying the threshold for developers to use, and accessing the open source project GPTCache to improve the performance of LLM applications.
2. In the era of large models, databases also need to keep up
We are embracing a new era of connectivity for all, where e-mails, social media photo transfers, protein molecular structure elucidation and more generate massive amounts of unstructured data (text files, spreadsheets, etc.).
However, you must not let this data "fly"! So, they have to be processed by the computer on a regular basis! So, how to deal with it?
First, embedding technology can turn this data into a "vector." Milvus then stores and indexes these "vectors", forming the legendary "vector database".
Unlike existing relational databases, which can be used primarily to process structured data, Milvus is designed to handle Embedding vectors transformed from various unstructured data.
The database can connect with deep learning models including image recognition, video processing, voice recognition, and natural language processing, so as to provide search and analysis services for vectorized unstructured data.
You can use Milvus to build a vector similarity retrieval system that meets the needs of individual scenarios, which can be applied to scenarios such as image search, video deduplication, audio retrieval, and text search.
3. Quick start: Install Milvus
Milvus can be used for the vast majority of one-to-many retrieval of unstructured data, now you can get started quickly... In the open source files of Milvus, we found the detailed installation process, which is briefly introduced here:
Image
The first thing you need to make clear is that Milvus uses Docker Compose to control the startup and suspension of services, including:
- Create a working directory and download the YAML file
- Start the Milvus container:
sudo docker-compose up -d
- Stop Milvus vessel:
sudo docker-compose down
4. Use case: search for images by image
Milvus also officially gives the use case of image search, which is divided into building image vector libraries and performing searches:
Image
1) Build an image vector library
Firstly, the image dataset is prepared, the image features are extracted using the deep learning library, the feature vector of each image is obtained, and finally the feature vector is stored in the Milvus database.
2) Perform a search
Enter an image to be retrieved, obtain the feature vector corresponding to the image through the deep learning library, and then compare the similarity of this vector with the feature vector of all images in Milvus to obtain the closest image result.
5. The integrity can be hard, and the ecology has become a climate
From the above installation and use process, it is not difficult to see that Milvus, as a cloud-native vector database, has the characteristics of high availability, high performance, and easy expansion, and can be used for real-time recall of massive vector data.
Image
Github also has a full summary of its overall performance:
- High performance: High performance, which can perform vector similarity retrieval on massive data sets.
- Simplified management: Rich APIs designed for data science workflows;
- High availability and reliability: Milvus supports cloud scaling, and its disaster recovery capability ensures high service availability.
- Highly scalable and resilient: Component-level scalability makes it possible to scale up and down on demand.
- Hybrid query: Milvus supports scalar field filtering in the process of vector similarity retrieval to achieve hybrid query.
- Developer-friendly: Support for multi-language, multi-tool Milvus ecosystem;
- Community support, industry recognition: With more than 1,000 enterprise users and an active open source community.
6. System architecture: four levels
As a cloud-native vector database, Milvus adopts the architecture design of separation of storage and computing, and all components are stateless, which greatly enhances system elasticity and flexibility.
Image
The entire system architecture can be divided into four levels:
- Access layer: The façade of the system, consisting of a set of stateless proxy. The endpoint that provides user connections externally is responsible for validating client requests and consolidating the returned results.
- Coordination service: The brain of the system, responsible for assigning tasks to execution nodes. There are four roles for coordinating services: root coord, data coord, query coord, and index coord.
- Execution node: The limbs of the system, responsible for completing the instructions issued by the coordination service and the data manipulation language (DML) commands initiated by the proxy. Execution nodes are divided into three roles, namely data node, query node, and index node.
- Storage service: The skeleton of the system, responsible for the persistence of Milvus data, is divided into three parts: metadata storage (meta store), message storage (log broker) and object storage (object storage).
7. 2.0 heavy upgrade
On the official website of Milvus, we found what are the differences between Milvus 1.0 and 2.0, as shown in the following table:
Image
In fact, since Milvus upgraded to version 2.0, it has continued to improve and optimize, introducing new features such as batch import of data from files, disk-based approximate nearest neighbor (ANN) indexing algorithm, and improved performance such as metadata storage and batch import.
Far from it, what changes and upgrades have been made from Milvus 2.2 to Milvus 2.3?
Milvus 2.2 improves the stability and speed of vector search, has flexible scaling capabilities, and provides a series of new APIs to support more efficient batch import of data from files.
In addition, Milvus 2.2 includes bug fixes and improvements to enhance Milvus stability, observability, and performance.
In version 2.3, Milvus removed the Annoy index, reduced maintenance costs, and also supported the count interface for counting the number of rows in the collection. In addition to calculating the number of collection rows, the new count interface also supports query expressions for counting rows with conditions.
8. Users: Using Milvus, you are not alone
From the initial Milvus to version 2.3, every version upgrade is inseparable from user support and suggestions. Upgrade, iterate, and after each update, Milvus tends to perfection.
So far, GitHub has reached 20,800 stars on it. So you're not alone when using Milvus in an active open source community!
Resources:
https://www.yii666.com/blog/393941.html?actinotallow=onAll
https://github.com/milvus-io/milvus
https://blog.csdn.net/hello_dear_you/article/details/127841589
Source: 51CTO Technology Stack