關于神經網絡的輸出神經元個數的思考

部落客對于神經網絡的輸出神經元個數的問題，起源于“識别手寫數字的神經網絡為什麼需要10個輸出而不是四個？”.

實際上，這是兩種不同的編碼方式，兩種的網絡架構都是可行的，但是我們選擇十個神經元而不是四個神經元來表達各類别，是因為這是經驗上的選擇，從效果來說，輸出為十個的效果更好。

具體理由如下：

如果輸出為四個，那麼輸出層的每個神經元需要學習的是“1和2的手寫體之間的差別”之類的斷言；

如果輸出為十個，那麼輸出層的每個神經元需要學習的隻是“判斷一幅圖檔是不是1”這樣的斷言。

而描述一個圖檔是不是某個數字比描述兩個數字之間的差別容易的多。

（問題來自Neural networks and deep learning）

You might wonder why we use 10 output neurons. After all, the goal of the network is to tell us which digit (0,1,2,…,9) corresponds to the input image. A seemingly natural way of doing that is to use just 44 output neurons, treating each neuron as taking on a binary value, depending on whether the neuron's output is closer to 0 or to 1. Four neurons are enough to encode the answer, since 24=16 is more than the 10 possible values for the input digit. Why should our network use 10 neurons instead? Isn't that inefficient? The ultimate justification is empirical: we can try out both network designs, and it turns out that, for this particular problem, the network with 1010output neurons learns to recognize digits better than the network with 4 output neurons. But that leaves us wonderingwhyusing 1010output neurons works better. Is there some heuristic that would tell us in advance that we should use the 10-output encoding instead of the 4-output encoding?

……