The use of vision allows us to guide and modify our movements by appropriately transforming external sensory information into proper motor commands. We investigated how people learned visuomotor transformations in different visual feedback environments. These environments presented perturbations of visual sense of movement direction. Across experiments and testing days, we altered the likelihood of visual perturbation occurrence and the distribution of sign and strength of visual perturbation angles. We then observed how transformation of sensed error into incremental adaptation depended on visual perturbation angle and on environmental experience. We found that environmental context affected adaptive responses within a day and across days. The across-day effect was profound enough that people exhibited very weak or very strong adaptive sensitivity to identical stimuli, dependent solely on prior days’ experience. We conclude that trial-by-trial adaptation to visual feedback is not fixed, but dependent on environmental experiences on both short and long time scales.