With the popularity of ChatGPT, it provides fertile soil for the scene landing of large language models. At present, the fields of medical care, education, and finance have gradually developed their own models, but there are not many products related to the legal field.
Therefore, the Peking University team open-sourced the Chinese legal big model and gave a reasonable solution to the problem of combining the big language model and the knowledge base.
At present, the ChatLaw legal model is available in three versions: ChatLaw-13B, ChatLaw-33B and ChatLaw-Text2Vec, with Jiangziya-13B and Anima-33B on the base. A large number of original texts such as legal news, legal forums, legal articles, judicial interpretations, legal consultations, legal examination questions, and judgment documents were used to construct dialogue data.
Open source address: https://github.com/PKU-YuanGroup/ChatLaw
Online Test: https://chatlaw.cloud/lawchat/#/
Thesis: https://arxiv.org/pdf/2306.16092.pdf
ChatLaw-13B: This version is an academic demo version, based on Jiang Ziya Ziya-LLaMA-13B-v1 training, Chinese all the performance is very good, but the logical complex legal question and answer effect is not good, need to use a larger parameter model to solve.
ChatLaw-33B: This version is an academic demo version, based on Anima-33B training, logical reasoning ability has been greatly improved, but because Anima's Chinese corpus is too small, English data often appears in Q&A.
ChatLaw-Text2Vec: A dataset made of 93w judgment cases is based on BERT to train a similarity matching model, which can match the user's question information with the corresponding law. For example, what if the loan is not repaid?
ChatLaw's Answer: Contract Law (1999-03-15): Article 206 The borrower shall return the loan within the agreed period. If there is no agreement on the term of the loan or the agreement is not clear, and it is still uncertain in accordance with the provisions of Article 61 of this Law, the borrower may return it at any time; The lender may urge the borrower to return it within a reasonable period of time.
The AIGC Open Community uses ChatLaw through a free online experience address (https://chatlaw.cloud/lawchat/#/) that does not require registration, which shows a more professional and vertical legal effect than similar products, and supports multiple rounds of conversational interaction. (Not yet connected to the professional legal consultation function)
For example, if a couple divorces, what are the legal issues to pay attention to?
ChatLaw: Divorce is a very serious and complex decision that requires careful consideration of the following aspects of the legal issues.
1. Property division: During the divorce process, the spouses should negotiate what to do with jointly owned property. This includes assets such as houses, vehicles, bank accounts, etc. If no agreement can be reached, the court will rule in accordance with local law.
2. Child custody: If the couple has minor children, they must agree on the custody of the children. This may involve visitation, custody, payment of alimony, etc. If no agreement can be reached, the court will rule in accordance with local law.
3. Debt sharing: During the divorce process, the spouses should negotiate how to deal with the joint debt. For example, credit card debts, loans, etc. If no agreement can be reached, the court will rule in accordance with local law.
4. Insurance policy: During the divorce process, the spouses should discuss how to deal with the jointly purchased insurance policy. For example, life insurance, medical insurance, etc. If no agreement can be reached, the court will rule in accordance with local law.
5. Will: During the divorce process, the spouses should discuss how to renew or revoke a jointly made will. If no agreement can be reached, the court will rule in accordance with local law.
How to reasonably evaluate the performance of large vertical models has always been a problem, because there are differences between test data and real scenarios. The Peking University team simply collected more than a decade of questions from the National Bar Examination and compiled a test dataset containing 2,000 questions and their standard answers to measure the model's ability to handle multiple-choice questions: evaluation data demo.
However, the development team found that the accuracy of each model was generally low. In this case, it doesn't make much sense to just compare the accuracy rates. Therefore, drawing on the ELO matching mechanism of League of Legends, a model confrontation ELO mechanism is made to more effectively evaluate the ability of each model to deal with legal choice questions. Below are the ELO scores and win rate graphs, respectively
The following conclusions were obtained:
(1) The introduction of data on law-related questions and answers and regulatory provisions can improve the performance of the model on multiple-choice questions to a certain extent.
(2) Add data from specific types of tasks for training, and the performance of the model on such tasks will be significantly improved. For example, the ChatLaw model outperforms GPT-4 because it uses a large number of multiple-choice questions as training data;
(3) Multiple choice questions require complex logical reasoning, so models with a larger number of parameters usually perform better.
Improve logical reasoning ability and train Chinese model base above 30B: In the iterative process of ChatLaw, it is found that unlike vertical fields such as healthcare, education, and finance, real questions and answers in legal scenarios usually involve very complex logical reasoning, which requires the model itself to have strong logical capabilities, and it is expected that only the number of model parameters reaches 30B or more.
Safe and credible, reduce hallucinations: Law is a serious scene, we have made a lot of efforts to optimize the accuracy of legal provisions and judicial interpretations of the model reply content, and the current combination of ChatLaw and vector library can be further optimized, in addition to the deep combination with the ChatExcel team, in the academic field to study the hallucination problem of LLM, it is expected that there will be a breakthrough in two months, thereby greatly reducing the hallucination phenomenon.
Private data model: On the one hand, it will continue to expand the basic legal capabilities of the model, and on the other hand, it will explore the customized private needs of the B/G side.