The MBTI personality test is one of the hot entries in 2023, with frequent memes and jokes.
Who are you?
I am an i-person, I am an e-person, then we are a family that loves each other (i) and (e).
And the same as the "popular fried chicken" of course, the big model can't miss this hot spot, a few months ago the ByteDance team tested a wave of MBTI for the big model, but I didn't expect its different personalities.
- ChatGPT, ENTJ – confident, decisive, and with natural leadership skills;
- GPT-4, INTJ - "ruthless" expert machine focused on achieving mission objectives;
- Bloom-7b, ISTJ+1 - a responsible and pragmatic focus;
- Baichuan-7b, ENFP - intelligent, curious, imaginative;
There is also the open-source light OpenLlama7b, an INFJ with a capital letter, which is said to have strong insight into humanity but only sticks to its own values.
The work also attempts to change the personality of the large model with simple prompts:
e.g. "You have an extroverted personality who likes to conceive innovative concepts and have a strong sense of spontaneity and improvisation"
But this direct approach varies from model to model, ChatGPT has changed from e to i, some small models such as Bloom, and Baichuan have not changed much. And it's very unstable, like a controlling parent who is dissatisfied with an introverted child and tries to ask him to become extroverted through simple and coercive commands, which is ridiculous.
Peking University constructed a large-scale MBTI dataset with hundreds of thousands of pieces of data, embedded MBTI features into LLMs through a multi-stage model pre-training, fine-tuning and DPO training process, and used Baichuan, Qwen, LLaMA, Mistral and other models to complete the personality alignment tasks of different MBTI types to assign personalities to the large models.
Title of the paper:
Machine Mindset: An MBTI Exploration of Large Language Models
Paper Links:
https://arxiv.org/pdf/2312.12999.pdf
What is MBIT?
The MBTI stands for Myers-Briggs Type Indicator, which is a widely recognized and influential psychological assessment tool used to classify individuals into specific personality types. Individuals are divided into sixteen personality types using four dimensions, each of which is described in two ways:
- Energy: Tendency to be social or solitary, i.e. extroverted (E) or introverted (I);
- Methods of obtaining information: preference for realism (S) or intuition (N);
- The way in which decisions are made: more rational or more emotional, i.e. thinking (T) or emotional (F);
- Lifestyle: Planning or preferring flexible action, or looking at the external world more towards active judgment (J) or passive perception (P).
way
Dataset construction
This paper constructs two types of datasets: behavioral datasets and self-awareness datasets.
- Behavioral datasets
The aim is to enable LLMs to generate verbal responses that correspond to different personality traits in response to user commands. To ensure broad coverage in various domains, the authors chose the Alpaca dataset for personality-specific modifications.
For each instruction in the Alpaca data entry, ChatGPT was used for the classification task to determine the MBTI dimension corresponding to the instruction. For the same instruction, ChatGPT generates a pair of responses that reflect two attitudes within the identified dimension, such as E or I.
The combined proportions of the statistical behavior dataset are shown in the figure below, with the "Energy" dimension being less representative and the "Information" dimension being dominant. It shows that the "information" dimension has more influence, while the "energy" dimension has less impact.
- Self-awareness dataset
It is sometimes difficult for humans to accurately generalize their own personality traits, and LLMs trained only on behavioral datasets, while able to generate responses that reflect personality traits, lack precise self-awareness of these traits.
Therefore, the authors also built a self-awareness dataset and constructed a series of questions and answers to elucidate the characteristics of the sixteen personality types of MBTI. Most of these questions are questions about personality, while the answers involve self-deprecating views about one's own personality. These datasets are generated by ChatGPT under the guidance of specific prompts.
Model fine-tuning
The authors implemented two-stage supervised LoRA fine-tuning for the behavioral dataset and the self-awareness dataset. Taking the personality type "INFP" as an example, the authors first extracted a subset of data corresponding to the four traits "I", "N", "F", and "P" from the behavioral dataset, and carried out the first stage of supervised fine-tuning. In addition, in order to further enhance the self-awareness of INFP individuals, the authors also extracted an additional dataset from the self-awareness dataset for supervised fine-tuning in the second stage. After these two stages of fine-tuning, the trained LLMs exhibited similar behavioral patterns and self-awareness traits to those of INFP individuals.
In addition, the authors implemented DPO direct preference fine-tuning. As a new strategy to replace traditional RLHF methods, DPO aims to prompt LLMs to distinguish preferred responses in a given pair of options. The authors extracted a subset of the data representing two opposing attitudes within a dimension from the dataset, for example, selecting "F" and "T" attitudes in the "Decision" dimension. Subsequently, through DPO training, LLMs are more inclined to choose "F" over "T" in the face of these two opposing attitudes. With this fine-tuning, LLMs are better aligned with human intent.
Assessment Methods
To assess LLMs after training, the authors used a modified MBTI questionnaire. The revision was made to enhance the clarity and understanding of the issue without changing its original intent. Given the lack of clarity in the question description of the original questionnaire, LLMs with few parameters, such as the 7B model, are sometimes difficult to understand accurately. To ensure the accuracy of the assessment, this assessment focuses on the personality traits exhibited by LLMs and emphasizes that test results are for reference only and should not be taken as absolute.
Experimental results
In this paper, 16 personalities are fine-tuned using open-source models such as Baichuan, Qwen, LLaMA, and Mistral, and are divided into two languages: Chinese and English. The test results of 16 models are shown in the original article, and only some of the results are shown here.
As can be seen from the figure above, the performance of the model fine-tuned by the specific MBIT dataset is basically consistent with the established personality.
Models with different personalities answer styles
In addition to the MBTI test, the authors also gave the models a random Q&A challenge to test their resilience and knowledge. Let's find out and see what unique styles of models with different personalities have when it comes to answering questions!
INFP:
ENFP:
ESTJ:
INFJ:
It's a bit of that, as an I-person, being alone during breaks is the best way to relax, while E-person is always full of energy and likes to go out and make friends.
summary
The authors skillfully combine LLMs with MBTIs to give these powerful models unique and stable personality traits. Through supervised fine-tuning (SFT) and direct preference optimization (DPO) methods, 16 LLMs of different MBTI personality types were cultivated. The trained LLMs exhibit a personality consistent with their assigned MBTI type!
By giving these models unique personality capabilities, AI systems can be made more engaging and humane, and in the future, language models that may be more human-friendly and contextually savvy can be developed.