ChatGPT plug-in system to build a new system in the AI era
In the past few days, new news based on OpenAI GPT4 has continued, first Microsoft's github released GPT4, upgraded github copliot to copliotX, and the original API will be offline on March 23, but old users are not affected. But yesterday OpenAI did unleash a nuclear bomb. OpenAI released a plug-in system based on the GPT4 model that can build everything connected. This is the era of intelligent operating systems that we have officially entered since this year. This is epoch-making, just like the Microsoft operating system and IPONE4 came along. Then we should fully understand, learn and apply new skills in such an era to make a difference in this new era. Then let's talk specifically about what functions the OpenAI plug-in system has and what impact it has on the future.
On March 23, 2023, OpenAI announced the official launch of the ChatGPT plug-in system with security as the core. The OpenAI plugin connects ChatGPT to third-party applications, after which ChatGPT can interact with developer-defined APIs to build new applications based on the ChatGPT model, which can build an open intelligent system, including:
- Retrieve real-time information such as sports results, stock prices, latest news, and more.
- Retrieve knowledge base information such as company documents, personal notes, and more.
- Perform actions on behalf of users, such as booking flights, ordering meals, and so on.
Currently, the plugin system is still in alpha testing, and the first vendors to participate in the creation of plugins include Expedia, FiscalNote, Instacart, KAYAK, Klarna Shopping, Milo Family AI, OpenTable, Shopify, Slack, Speak, Wolfram, and Zapier.
In addition, OpenAI has created two plugins, one for a web browser and one for a code interpreter. We also open source the code for a knowledge base retrieval plugin that can be hosted by any developer themselves and can use this information to enhance ChatGPT.
OpenAI opens the extension alpha access to users and developers who apply. OpenAI will initially prioritize a small number of developers and ChatGPT Plus users, but plans to roll out larger visits over time.
OpenAI says that while language models can perform in a variety of tasks, they still have limitations. And the only way for a model to improve itself is with its training data. But the content of the data may be outdated, and the models are often "generalized" and "rigid". In addition, the only output mode that a language model can use out of the box is text, and while the text contains practical instructions, it requires additional procedures to turn it into action.
OpenAI likens plugins to the "eyes and ears" of a language model, helping the model access information that is not appropriate to be included in the training data because it is newer, more personalized, or more specific. In response to explicit user requests, plug-ins also help language models actually perform secure, controlled operations, thereby increasing the usability of the entire system
OpenAI hopes that the emergence of open standards will unify the way applications access AI's open interfaces. They are working on an early attempt at such a standard and are looking for feedback from developers interested in building with us.
OpenAI began gradually enabling existing plugins for early collaborators for ChatGPT users, starting with ChatGPT Plus users. And start giving developers the ability to create their own plugins for ChatGPT.
Over the next few months, we will repeat this protocol as we learn from deployment and continue to improve our security systems, and we plan to enable developers using OpenAI models to integrate plugins into their own applications outside of ChatGPT.
What about the ChatGPT plugin system?
Let's first look at how OpenAI's own two plugins perform.
1. Browser plug-in
An experimental model that knows when and how to navigate the Internet.
In the past, ChatGPT politely pointed out that its training data did not have enough information to answer certain questions. But now the answer and the relevant link can be given.
In the example shown by OpenAI, ChatGPT retrieves information about the latest Oscars and adapts it with its best ability to "write poetry."
In addition to providing significant utility benefits to end users, OpenAI believes that this research into the full interpretability of language and chat models will also bring exciting prospects in terms of scalability and alignment.
2. Code interpreter
An experimental ChatGPT model that handles uploads and downloads using Python.
OpenAI developed a Python interpreter for the model that works in a sandbox, firewall execution environment, plus some temporary disk space. The code that the interpreter plugin runs is evaluated in a persistent session that remains valid for the duration of the chat (with an upper bound timeout), and subsequent calls can build on each other. There is also support for uploading files to the current conversation workspace and downloading work results.
For example, when asked: the weight of a brick is the sum of 30% of the weight of the brick itself and the weight of 5 GPUs, and it is known that each ton of weight contains 100 bricks and 200 GPUs, then what is the weight of a brick?
The answer given by ChatGPT is: the weight of a brick is 7.8125 kg.
A code interpreter plug-in that performs logical calculations
From early user research, OpenAI identified several important uses for code interpreters:
- Solve quantitative and qualitative mathematical problems
- Perform data analysis and visualization
- Convert files to different formats
OpenAI says it hopes the model will be able to use its programming skills to provide a more natural interface for basic functions in computers. It helps beginner programmers to be more productive, while also hoping to reach and master programming skills to a wider audience.
3. Third-party plugins
An experimental model that knows when and how to use plugins.
Third-party plugins in ChatGPT
Third-party plug-ins are described by manifest files that contain machine-readable descriptions of the plug-in's functionality and invocation methods, as well as user-facing documentation.
Example of a manifest file for a to-do management plugin
The steps to create a plug-in are as follows:
1. Users build APIs using the endpoints they want the language model to call (this can be a new API, an existing API, or an existing API packager designed for large language models).
2. Create an OpenAPI record specification for your own API, and create a manifest file that points to the OpenAPI specification and contains plugin-specific metadata.
When starting a conversation on chat.openai.com, users can specify which third-party plug-ins they want to enable. Documentation about enabled plugins is submitted to the Language model as part of the conversation context, enabling the model to call the appropriate plug-in APIs as needed to achieve user intent. Current plugins mainly consider calls to backend APIs, and OpenAI is also exploring how to make APIs call client APIs.
Of course, OpenAI focuses on security this time, but I won't discuss it here, and I think the most important thing now is that the feature is amazing, just like Microsoft disbanded its ethical security team.
OpenAI says it's working hard to develop plugins and bring them to a wider audience. They have a lot to learn and, with everyone's help, hope to build something that is both useful and safe.
In short, after reading through the official documentation, I feel that the OpenAI plug-in system is open, which is a nuclear bomb. It can also be seen that Microsoft once again dominates the core of the next era of intelligence after the failure of the mobile phone system. Recently, based on the capabilities of GPT4, we have built an AI ecosystem in an all-round way. It can be seen that Microsoft + OpenAI is the "Apple" of the intelligent era, so who will be the next "Android"? Finally, AI technology has developed rapidly recently, but it is difficult for Chinese users to experience and try it first, but there is too little innovation in China. Now the entire IT ecosystem is based on Microsoft Windows and the Apple ecosystem. There is no better ecosystem in China. Now under the leadership of Microsoft + OpenAI, there is no chance.