On Thursday, Meta open-sourced an AI code assistant code editor. It's only been a few days since its launch, and the github start star has soared to 4.4K.
Code Llama is a family of large language models based on Camel 2 that offers state-of-the-art performance between an open model, padding capabilities, support for large input contexts, and zero-sample instruction tracking capabilities for programming tasks. Multiple flavors are available to cover a wide range of applications: Code Llama, Python specialization (Code Llama - Python), and instruction following model (Code Llama - Instruct), each with 7B, 13B, and 34B parameters. All models were trained on 16k labeled sequences and showed improvements on inputs up to 100k per label. 7B and 13B Code Llama and Code Llama - Directive variants support padding based on surrounding content. Code Llama was developed by fine-tuning Llama 2 by using higher code sampling. As with Llama 2, we applied a number of security mitigations to the fine-tuned version of the model. For more information on model training, research papers. Output generated by the code-generating features of the Llama Materials, including Code Llama, may be subject to third-party licenses, including, but not limited to, open source licenses.
CodeLLama includes three models: the base model, a fine-tuned version of Python, and an Instagram instruction following model. Each model contains 7 billion, 13 billion and 34 billion parameter versions, and can be used for research or commercial use, CodeLLama is a code-specific model trained based on LLama2, Meta, on a specific code dataset, so CodeLLama can generate popular code including Python C++ Java PHP based on natural language prompts, and in multiple sets of coding benchmarks, the 34 billion parameter model has code generation capabilities comparable to chatgpt It is also the most powerful auxiliary programming model available.
Code Llama works
Code Llama is a code-specific version of Llama 2 that is created by further training Llama 2 on its code-specific dataset, sampling more data from the same dataset for a longer period of time. Essentially, Code Llama has enhanced coding capabilities and is built on top of Llama 2. It can generate code and natural language about the code based on code and natural language cues (e.g., "Write me a function that outputs a Fibonacci sequence. It can also be used for code completion and debugging. It supports many of the most popular languages in use today, including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash.
As mentioned earlier, there are three models released with 7B, 13B and 34B parameters. Each model is trained with a 500B code token and code-related data. The 7B and 13B base models and instruction models are also trained with intermediate padding (FIM) capabilities, allowing them to insert code into existing code, which means they can support tasks such as out-of-the-box code completion.
These three models meet different service and latency requirements. For example, a 7B model can run on a single GPU. 34B models return the best results and provide better coding assistance, but smaller 7B and 13B models are faster and better suited for tasks that require low latency, such as live code completion.
Evaluate the performance of Code Llama
The model uses two popular coding benchmarks: HumanEval and Mostly Basic Python Programming (MBPP). The HumanEval test model completes code based on a docstring and the MBPP test model's ability to write code based on descriptions.
Benchmarks show that Code Llama outperforms the open-source, code-specific Llama and outperforms Llama 2. For example, Code Llama 34B scored 53.7% on HumanEval and 56.2% on MBPP, the highest compared to other states. State-of-the-art open solution, comparable to ChatGPT.
Deployment and testing
It is deployed in the same way as the previous Llama 2 deployment. There is a text generation UI launch interface
At present, only this webUI interface style is supported, and there are no development tool plug-ins such as IntelliJ IDEA and VScode. However, I believe that the official should come out with this kind of development tool plug-in later, and the small partners who compare development prefer to use the programming code assistant on the IDE development tool.
Once the model is configured, it is generated using the code.
summary
In the future, code generation assistants such as Code Llama will play an important role in the field of software development. It provides the ability to quickly generate and explain code in detail, saving developers time and effort. This will speed up the software development process and improve development efficiency. In addition, Code Llama helps novice developers learn to program and provides the guidance and support they need.