Mengchen originated from the Temple of Concave Fei
Qubits | Official account QbitAI
GPT-4 costs only 0.71% to replace a junior data analyst, and 0.45% to replace a senior data analyst...
You read that right, it's 0.71 percent, not 71 percent.
According to the Singapore market, a senior data analyst with an annual salary of 86,000-90,000 US dollars (600,000-630,000 yuan) only needs three to four hundred US dollars (more than 2,000 yuan) to change to GPT-4.
This conclusion comes from a new paper from the Ali Dharma Academy and Nanyang Technological University in Singapore, which has been evaluated by netizens as a must-read paper interested in the field of AI and data analysis.
Specifically, the conclusion of the senior analyst refers to the data analyst with many years of experience in the financial industry.
GPT-4's performance is comparable to that of a human with 6 years of experience in most indicators, with lower accuracy than humans, but higher complexity and consistency indicators.
In comparison with another analyst with 5 years of work experience, GPT-4 loses to humans in terms of the correctness of information, the aesthetics of charts, the complexity of insights, etc.
Compared to a junior analyst with 2 years of experience, GPT-4 performs better on correctness and accomplishes more work.
But GPT-4 can accomplish all types of tasks much faster than humans.
The final conclusion is drawn on the assumption that there are 21 working days per month, 8 hours a day, and wages are paid at the market price.
GPT-4 can do anything as a data analyst
The paper focuses on the following capabilities of GPT-4 as a data analyst:
Generate SQL and Python code
Execute code to get data and charts
Analyze data from data and external knowledge sources to draw conclusions
Experiments of 200 samples show that GPT-4 can understand the meaning of instructions for charting tasks, and has some background knowledge of chart types, so as to draw the correct chart.
The chart is mostly clearly visible without any formatting errors, with a perfect score of 3 on the Aesthetics indicator of the icons, with an average GPT-4 score of 2.73.
However, manual inspection can still find some minor errors, and the chart accuracy indicator has a full score of 1, and the GPT-4 average score is 0.78.
In particular, the paper states that their evaluation criteria are very strict, and points will be deducted whenever any data or any label on the x-axis or y-axis is wrong.
For data analysis tasks, GPT-4 scored full points on average for consistency and fluency, verifying that generating fluent and grammatically correct sentences is definitely not a problem for GPT-4.
Interestingly, the accuracy of the data analysis step is much higher than the accuracy of the chart information, indicating that GPT-4 draws the correct conclusions despite the incorrect graph.
In the case study, the research team also concluded three key differences between GPT-4 and human data analysts:
The human analyst can express it with personal thoughts and emotions, such as writing "Surprisingly is..."; Human readers tend to understand from such statements whether the data is expected or abnormal.
Human analysts tend to draw conclusions based on background knowledge, such as writing "... Common in..."; GPT-4 usually focuses only on the extracted data itself, and allowing GPT-4 to go online and search for real-time online information can improve this.
When providing insights or recommendations, human analysts tend to be conservative, such as stating "If there is no problem with the data..."; GPT-4 will give advice directly in a confident tone, without mentioning assumptions.
In addition, the team said that due to the limited budget, it was too expensive to hire a senior analyst to compare with GPT-4, and the amount of manual evaluation and data annotation was relatively small.
In the final conclusion:
Experimental results and analysis show that GPT-4 has comparable performance to humans in data analysis, but whether it can replace data analysts requires further research to reach a conclusion.
Thesis:
https://arxiv.org/abs/2305.15038