SHP2, encoded by PTPN11, is a non-receptor protein tyrosine phosphatase (PTP) containing two tandem Src homology-2 (SH2) domains. It is expressed ubiquitously and plays critical roles in growth factor mediated processes, primarily by promoting the activation of the RAS/ERK signaling pathway. Genetic and biochemical studies have identified SHP2 as the first bona fide oncoprotein in the PTP superfamily, and a promising target for anti-cancer and anti-leukemia therapy. Here, we report a structure-based approach to identify SHP2 inhibitors with a novel scaffold. Through sequential virtual screenings and in vitro inhibition assays, a reversible competitive SHP2 inhibitor (C21) was identified. C21 is structurally distinct from all known SHP2 inhibitors. Combining molecular dynamics simulation and binding free energy calculation, a most likely binding mode of C21 with SHP2 is proposed, and further validated by site-directed mutagenesis and structure-activity relationship studies. This binding mode is consistent with the observed potency and specificity of C21, and reveals the molecular determinants for further optimization based on the new scaffold.