Gabor Filter showing a receptive field of a hypothetical simple-cell in the visual cortex (Wikipedia)
In 1972, Horace Barlow, great-grandson of Darwin, wrote an article in which he put forth “A neuron doctrine for perceptual psychology?” (The question mark at the end shows that Barlow was a contemplative guy.) With this doctrine, Barlow tried to relate the firing of neurons in sensory pathways with subjectively experienced sensation. One of the principles he introduces is: at progressively higher levels of sensory processing, information is carried by fewer neurons because the system is organized to a near complete a representation with the fewest active neurons. In other terms, the encoding of sensory information gets ‘sparser’ as one moves up into higher levels of sensory processing. Since then, a number of lines of evidence have converged, supporting Barlow’s proposition and the general value of ‘sparse codes.’
On the blog today I’ll be reviewing one such paper, “Sparse coding of sensory inputs” by Bruno Olshausen and David Field. This paper defines ‘sparse coding’ as a computational strategy where brains encode sensory information using a small number of simultaneously active neurons at a given time[1].
This paper puts forth four ideas for why sparse coding theoretically might be a good strategy:
- More memories can be stored
- Makes use of the statistical structure of natural signals
- Represents data in a convenient way for further processing
- Save metabolic energy, by decreasing neuronal firing rates
When model neurons are trained to optimize spare representations of natural scenes, the receptive fields that emerge represent the simple-cells of the primary visual cortex.
Learned receptive fields to maximize sparseness of natural scenes (Olshausen and Field, 2004, Current Opinion in Neurobiology)
One interesting feature of sensory representations that this paper points out, is that as visual signals move from the thalamus to the visual cortex, there is a 25:1 expansion (of axonal projects to cortex versus axonal projections to the LGN of the thalamus). They suggest that this 25:1 expansion possible emerges as a compromise between the trade-offs: with high sparseness eventually ending in ‘grandmother cells’ where a single unique neuron represents each element of a sensory[2], and low sparseness incurring developmental and metabolic costs of having to use many neurons to encode each element. Additionally, the more a neuron spikes the more energy is needed to maintain it’s electrochemical gradients through pumps like the sodium-potassium ATPase. From what we know about the metabolic use of the cortex, scientists have estimated that only 1/50th of neurons are active at any given time.
Another interesting point this paper makes, is that some experimental results show neurons with much higher firing rates than would be predicted by metabolic estimates. Because these experiments often involve searching for firing neurons with an electrode, we may be systematically biasing our studies towards a minority of neurons that fires “less sparsely” than the general populations. Solutions to this bias include chronically implanting electrodes where the positioning is set anatomically or using antidromic stimulation to identify neurons as opposed to stimulus elicited firing.
Sparse coding beyond sensory systems
This paper also points out that sparsely firing neurons are observed in motor cortex during movements, and that experimentally driving a single neuron can be enough to initiate whisker movements in rats (Brecht et al., 2004). In the zebra finch song production pathway, HVC neurons fire sparsely at precise points in song, and precise spike-timing in RA may be important as well.
How does one actually measure “sparseness”
Olshausen and Field, 2004, Current Opinion in Neurobiology
Olshausen and Field write that a standard measure of “sparseness” is kurtosis, with a larger value indicating a “sparser” distribution.
They also describe another method developed by Rolls and Tovee, the activity ratio, which is specialized for one sided distributions (and therefore good at modeling neurons since firing rate cannot drop below zero).
Finally, The activity ratio can then be scaled from 0-1, using Vinje and Gallant’s[3] sparse coding scale transformation:
S=(1-a)/(1-amin).
Conclusion
I think there is compelling evidence for brains using of sparse coding in sensory systems and there are good theoretical reason for why brains should use sparse coding. It will be interesting to see if these findings hold up for motor systems as well. If they find evidence of sparse coding in relatively simple and blunt movements like locomotion, I would guess that they will be more present in higher premotor areas encoding complicated and learned movements.
I big issue in machine learning is how to process the data before unleashing machine learning algorithms on it. I think neurophysiology shows that a good strategy might be to duplicate the data and turn it into over complete sparse representations. I assume scientists working with big data are already doing this, but honestly I have no idea.
References
Barlow HB: Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1972, 1:371-394.
Brecht M, Schneider M, Sakmann B, Margrie TW: Whisker movements evoked by stimulation of single pyramidal cells in rat motor cortex. Nature 2004, 427:704-710
Olshausen BA, Field DJ (2004). Sparse Coding of Sensory Inputs. Current Opinion in Neurobiology, 14: 481-487.
Rolls ET, Tovee MJ: Sparseness of the neuronal representation of stimuli in the primate temporal visual cortex. J Neurophysiol 1995, 73:713-726.
Vinje WE, Gallant JL: Sparse coding and decorrelation in primary visual cortex during natural vision. Science 2000, 287:1273-1276.
[1] I had always thought of sparse coding from the perspective of individual neurons, as a strategy where a given sensory neuron fires only at a very specific stimuli or aspect of a stimuli, so therefore it fires ‘sparsely.’ I realized in reading this, that really this paper’s definition and my mental understanding are two sides of the same coin if the neurons are firing independently. However, even with a population of neurons that only fire very rarely might not be ‘sparsely coding’ by Olshausen and Field’s definition if they are all firing together and then silent together.
[2] How does that neuron spur the rest of the brain into action and what occurs when that neuron dies?
[3] This is the same gallant that did some of the coolest experiments ever decoding neural activity from fMRI activation: https://neuroamer.wordpress.com/2011/10/31/scientists-record-lucid-dreams-with-eeg-and-fmri-simultaneously/
