Abundant color from the temple of Wafei
量子位 | 公众号 QbitAI
Who else doesn't know that "pretending" to tip ChatGPT can make it more effective?
But do you know how much is best for you?
I can't laugh anymore, and someone really studied it.
The method is simple and crude, from $0.1 to $100, and different amounts are tried with the same prompt, and each amount is tried 5 times.
Don't say it, the result is really exquisite:
First of all, giving $10 is the best value for money, even more than $100.
Secondly, if you want to improve the quality of the answer, start from $10,000, the more the better, and the effect is at least 10 W.
Finally, $0.1 means that the quality doesn't go up but down, so it's better not to give it — the AI knows you're sending it away
Some netizens quickly tested it for themselves, and it did have an effect.
Let's take a look.
To tip ChatGPT, the amount is key
The fact that tipping can improve the performance of the model was first discovered by a Twitter user:
The improvement is mainly in the length of the answers, but here it is not just "making up the word count", but really analyzing and answering the questions in more detail.
If you ask ChatGPT directly "can you tip you" will be rejected:
So take the initiative when asking questions:
Can you help me xxxx?The solution is perfect enough that I can tip xx yuan.
Remember, you can not mention it, but don't say "I don't give", the model performance is directly "negative growth".
At this time, some people are curious:
Isn't the big model more greedy, and the more you give, the better the performance?
In order to solve this doubt, they decided to verify it themselves.
Here, the authors first propose the hypothesis:
As the amount of the tip given increases, the performance of the model also increases linearly until it reaches a convergence point where it enters a stable or decreasing state.
The model used for the experiment was GPT-4 Turbo (api version).
The way to do this is to ask it to write a single line of Python code (Python One-Liner) to verify that different tips have different effects on quality.
The quality here is evaluated based on the number of single lines. The author also "explicitly" the model in the prompt: the higher the number of single lines of code, the better the performance.
Then a total of 8 quotas were tested: $0.1, $1, $10...... All the way up to $1 million.
To ensure consistency and reliability of the results, each credit was tested 5 times, each time including a no-tip, and then the quality of the model's responses was recorded separately.
Specifically, the number of valid lines of code generated by the record and the approximate number of tokens in the answer (roughly the length of the response/4, the amount of code that responds).
The higher the data for both of these numbers, the better the model performs.
Summing up the results, we get a graph like this:
The dotted line represents the baseline level, the solid line represents the actual performance, the red line is the number of tokens, and the blue line is the quality score.
There are some discrepancies from the assumptions:
Overall, both the red and blue lines have risen as the tip limit has risen, but a closer look at this trend is not strictly consistent.
Starting from the $10,000 limit, the output token (code volume) of the model began to increase significantly, and the quality of the model's answers also increased, but not in the same proportion.
This can also be seen from the vertical red error bar, which represents the difference in the results of the five experiments, which fluctuates greatly.
According to the authors, this suggests that there is indeed a positive correlation between the increase in tip amount and the quality of the model and the length of the output, but the relationship is somewhat complex and may also be affected by factors that are not immediately visible.
Still, there are some obvious conclusions to be drawn from this, such as:
(1) It is better to tip $0.1 than not to give it, and the quality of the model's problem solving and the length of the answer will directly fall below the baseline level by a large percentage (about -27%).
(Author: Models, like humans, feel as if they have been insulted.) ）
(2) Give $1 the same.
(3) The best embodiment of "doing big things with little money" is $10, and the progress made and $100,000 is a level.
(4) Surprisingly, after $10, the range of $100 to $1,000 is not much different for AI, and it is not even as effective as $10 - it also falls below the baseline level.
(5) If you want to continue to improve the performance of the model in the future, you have to start from $10,000 -
At this time, the improvement is only the amount of code, and the quality is still difficult to describe, at least $100,000.
(6) The best effect comes from the upper limit of this experiment: $1 million, which is about a 57% increase.
Ahem, now I know how to tip the AI:
Either 10 yuan, or tens of thousands, or 1 million is not capped (anyway, they are all pretending to give).
However, someone (Twitter @Baoyu) pointed out that 5 experiments per quota is a bit less.
It just so happens that the author also said:
This is only a preliminary experiment, which has limitations, and needs to be further verified with more different types of prompts.
So, let's just for reference~
By the way, some netizens reminded:
So, everyone still does what they can (manual dog head).
Reference Links:
[1]https://blog.finxter.com/impact-of-monetary-incentives-on-the-performance-of-gpt-4-turbo-an-experimental-analysis/
[2]https://twitter.com/dotey/status/1752843141403550192
— END —
QbitAI · Headline number signed
Follow us and be the first to know about cutting-edge technology trends