Empirical studies of landscape connectivity are limited by the difficulty of directly measuring animal movement. ‘Indirect’ approaches involving genetic analyses provide a complementary tool to ‘direct’ methods such as capture–recapture or radio-tracking. Here the effect of landscape on dispersal was investigated in a forest-dwelling species, the American marten (Martes americana) using the genetic model of isolation by distance (IBD). This model assumes isotropic dispersal in a homogeneous environment and is characterized by increasing genetic differentiation among individuals separated by increasing geographic distances. The effect of landscape features on this genetic pattern was used to test for a departure from spatially homogeneous dispersal. This study was conducted on two populations in homogeneous vs. heterogeneous habitat in a harvested boreal forest in Ontario (Canada). A pattern of IBD was evidenced in the homogeneous landscape whereas no such pattern was found in the near-by harvested forest. To test whether landscape structure may be accountable for this difference, we used effective distances that take into account the effect of landscape features on marten movement instead of Euclidean distances in the model of isolation by distance. Effective distances computed using least-cost modeling were better correlated to genetic distances in both landscapes, thereby showing that the interaction between landscape features and dispersal in Martes americana may be detected through individual-based analyses of spatial genetic structure. However, the simplifying assumptions of genetic models and the low proportions in genetic differentiation explained by these models may limit their utility in quantifying the effect of landscape structure.