Silver triangular nanoplates (STNPs) as a high efficient fluorescence quenching reagent of upconversion nanoparticles (UCNPs) was used to constract a novel label-free fluorescence nanosensor for ultrasensitive detection of protamine and trypsin based on fluorescence resonance energy transfer (FRET) between STNPs and UCNPs. In this assay, the negatively charged STNPs can bind with positively charged UCNPs through electrostatic interaction, and then quenched the fluorescence of UCNPs. When protamine was added to the mixture of UCNPs-STNPs, the STNPs interacted with protamine and then detached from the surface of UCNPs and aggregated, which result in the recovery of the fluorescence of UCNPs. Trypsin could catalyze the hydrolysis of protamine and effectively quench the fluorescence recovered by protamine. By measuring the changes of the fluorescence of UCNPs, the concentrations of protamine and trypsin were determined. Under the optimized conditions, the linear response range was obtained from 10 to 500ng/mL, 5-80ng/mL and with the low detection limit of 3.1ng/mL and 1.8ng/mL for protamine and trypsin, respectively. Meanwhile, the nanosensor shows good selectivity, sensitivity and can be successfully applied to detection of protamine and trypsin in serum samples.