Entropy Based Comparison of Neural Networks for Classification

This paper discusses some entropy based bounds for the case of real and limited integer weights neural networks. It is shown that a neural network using real weights can solve a dichotomy of m =m ++m_ patterns using a number of bits less than max (m +, m_) ·n {⌈log(D/d)⌉+2} where n is the number of dimensions and D and d are the maximum and minimum distance between patterns in opposite classes, respectively. In the case of limited integer weights, it is shown that a neural network using integer weight in the range [- p,p] can solve a dichotomy of patterns in general positions with a number of bits greater than #bits lt; min(m +,m_) · n · ⌈log 2pD⌉.