Computational models are increasingly used to explore possible mechanisms underlying infant capability in various tasks. Often, such models do not work directly on perceptual data, but on hand-computed features of images; such models are open to the criticism that these high-level features may not be what is actually computed in the neural computation. Here we explore the feasibility of the Serre-Poggio (S-P) model which emulates the early ventral stream of the primate visual cortex, and constructs a probabilistic model of the tuned cells of the V4-IT cortex. In experiment 1, we use this system to model asymmetry in visual category learning in early infancy (e.g. cats vs dogs), and show that surprisal for the novel category is higher when habituated on CAT than on DOG. In experiment 2, we show that face habituation can be used to discriminate on full bodies. Experiment 3 demonstrates that superordinate category discriminations are easier than for the basic level. These experiments agree with earlier psychological data and partially validate the S-P model for such tasks.