Stimuli are represented in the brain by the collective population responses of sensory neurons, and an object presented under varying conditions gives rise to a collection of neural population responses called an “object manifold.” Changes in the object representation along a hierarchical sensory system are associated with changes in the geometry of those manifolds. To study this, we developed a statistical mechanical theory for the linear classification of these object manifolds, connecting the geometry of object manifolds with their perceptron capacity, as a measure of linear separability. Our theory and its extensions provide a new framework for characterizing high-dimensional population responses to objects or categories in biological and artificial neural networks. We demonstrate results from applying our method to neural networks for visual and auditory object recognition tasks. Exciting future work lies ahead as manifold representations of the sensory world are ubiquitous in both biological and artificial neural systems.