After a lapse of 9 months, meta has officially released the official version of Llama3. The model has been put on the shelves, the 8B and 70B models have been open sourced and can be commercialized for free (the limit of monthly active users does not exceed 700 million), what new information is there?
About Llama
Llama is an open-source project released by Meta (FaceBook) AI, which allows commercial use and has a huge impact. The previously released Llama 2, with support for 4096 contexts and excellent performance, is considered one of the biggest competitors (one of the GPT series).
Flame-3
Meta released the Meta Llama 3 series of language models (LLMs), including an 8B model and a 70B model, and the Llama 3 model performed quite well in the test protocol, and in the evaluation of usability and safety, it is on par with the popular closed-source models on the market.
Part 1 Llama-3 just released
At 0:00 on April 19, 2024 China time, Meta Llama 3 was released. Models are available in open-source form with 8B and 70B parameter scales, covering pre-trained and instruction-tuned variants. Llama 3 supports a wide range of commercial and research uses and has demonstrated superior performance in multiple industry-standard tests.
Technical Information
Transformer 架构
Meta Llama 3 uses an optimized autoregressive transformer architecture designed to handle complex text generation tasks, improving the coherence and relevance of generated text.
Hybrid tuning
The hybrid approach of the model, which combines supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF), not only enhances the helpfulness of the model, but also improves the security, making the model more reliable and in line with user expectations in real-world applications.
Superior performance
Meta Llama 3 outperformed many existing open-source chat models in multiple industry-standard benchmarks, especially in conversational applications, demonstrating its strong application potential, as detailed below.
Data training
Big data
Llama 3 is pre-trained using more than 15 trillion tokens of publicly available online data, which has been curated to ensure extensive model training and high-quality outputs.
New data
The 8B version is updated as of March 2023, while the 70B version is updated to December 2023.
30 languages
While predominantly English-based, the pre-training data includes high-quality, non-English data in more than 30 languages
* Big smart: Chinese data don't know whether to use the "mentally retarded" corpus
Political Correctness (Fog)
Carbon offsets
Meta is committed to offsetting all CO2 emissions generated during the pre-training process (2290 tonnes CO2 equivalent) through its sustainability initiative.
Very well-behaved
The use of Llama 3 strictly complies with laws and regulations, ensuring that it is not used for any illegal activities, while emphasizing the importance of intellectual property and compliance.
Part 2 Technical Performance
This time, Llama has shown a significant improvement in performance, including the most direct 8k context (previously 4k), and the ability to complete output tasks better.
Performance testing
Benchmarking
- Meta Llama 3's 70B model has shown excellent performance in several benchmarks, such as 89.7% accuracy in the TriviaQA-Wiki test, significantly outperforming other models of the same size.
- Llama 3 excels in real-world use cases of 1,800 prompts across 12 key use cases (including consulting, coding, creative writing, and more) in a high-quality in-house developed set of high-quality human assessments.
Here's another comparison of Llama 2 and 3:
Real-life scenarios
According to the human evaluators' preference rankings, Llama's 70B parameter model shows a significant advantage over other sizable models in real-world use cases, especially in terms of instruction following.
Architecture & Optimization
Model architecture
- Llama 3 uses an autoregressive transformer architecture, which is particularly suitable for complex text generation tasks, improving the coherence and relevance of text.
- The Grouped Query Attention (GQA) technology was introduced, which not only improved the efficiency of big data processing, but also accelerated the response speed.
Training and fine-tuning
- In the pre-training phase, Llama used a high-quality dataset of more than 15 trillion tokens, including text in multiple languages, to ensure broad applicability and excellent performance of the model.
- In the fine-tuning phase, Llama significantly reduced the false rejection rate and improved model alignment and response diversity through a hybrid approach of supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF).
Performance ImprovementsLlama 3 In this update, there are significant improvements in inference, code generation, and instruction following.
Part 3 Where to use
As an open source LLM, you can use it in a variety of ways: directly with someone else's product, find a deployed interface, or deploy it yourself
There are still problems with Chinese
Straightforward (easiest)
Hugging Face address is here, and you can cut the model directly after entering: https://huggingface.co/chat/
Replicate8B 模型:hat/https://replicate.com/meta/meta-llama-3-8b70B模型:https://replicate.com/meta/meta-llama-3-70b
(just put it on, only ran 8 times)
Meta AIMeta itself took Llama 3 to do, here to visit: https://ai.meta.com/ attention, this lock area.
Third-party APIs
The Microsoft Azure address is here: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/metagenai.meta-llama-3-8b-chat-offer?tab=overview
Replicate them good volume... 1 hour after Llama was released, they went live with the service, and these two addresses can also go API8B model: hat/https://replicate.com/meta/meta-llama-3-8b70B model: https://replicate.com/meta/meta-llama-3-70b
Self-department
The official website of the Meta project is located here: https://llama.meta.com/llama-downloads
Github project address: https://github.com/meta-llama/llama3
Part 4 Miscellaneous
Following the release of Llama 3, there is also the Meta AI series, including: a mobile app, a website, and a bunch of plugins in the Meta FaceBook family bucket
* Great Wisdom: Lessons from China, right?
What the app can do
能当 ChatGPT 用emmmmm...
What can a web app do?
还是能当 ChatGPT 用emmmmm...
What can the plugin do
It looks practical to be able to use this in a family bucket!
The above is reported by Ben "Big Smart". Next time it's me 🤔
Author: Cyber Zen Heart, WeChat public account: Cyber Zen Heart
This article was originally published by @ on Everyone is a Product Manager. Reproduction without the permission of the author is prohibited.
The title image is from Unsplash and is licensed under CC0
The views in this article only represent the author's own, everyone is a product manager, and the platform only provides information storage space services.