We present a new, adaptive boundary integral method to simulate solid tumor growth in 3-d. We use a reformulation of a classical model that accounts for cell-proliferation, apoptosis, cell-to-cell and cell-to matrix adhesion. The 3-d method relies on accurate discretizations of singular surface integrals, a spatial rescaling and the use of an adaptive surface mesh. The discretized boundary integral equations are solved iteratively using GMRES and a discretized version of the Dirichlet-Neumann map, formulated in terms of a vector potential, is used to determine the normal velocity of the tumor surface. Explicit time stepping is used to update the tumor surface. We present simulations of the nonlinear evolution of growing tumors. At early times, good agreement is obtained between the results of a linear stability analysis and nonlinear simulations. At later times, linear theory is found to overpredict the growth of perturbations. Nonlinearity results in mode creation and interaction that leads to the formation of dimples and the tumor surface buckles inwards. The morphologic instability allows the tumor to increase its surface area, relative to its volume, thereby allowing the cells in the tumor bulk greater access to nutrient. This in turn allows the tumor to overcome the diffusional limitations on growth and to grow to larger sizes than would be possible if the tumor were spherical. Consequently, instability provides a means for avascular tumor invasion.