The synthetic retinoid N-(4-hydroxyphenyl)retinamide (4-HPR) has been shown to induce apoptosis in various types of tumors, including prostate cancer. We sought to examine the key mechanisms affecting the resistance to 4-HPR-induced apoptosis in three human prostate cancer cell lines, PC-3, DU145, and LNCaP. Concentrations of more than 40 microM 4-HPR produced apoptosis to almost the same extent in all cell lines; however, only the LNCaP line remained highly sensitive to concentrations less than 10 microM. These differing sensitivities at low concentrations correlated well with the level of constitutive activation of nuclear factor kappa B (NFkappaB) in the individual cell lines. We found that NFkappaB activation inhibited c-jun NH(2)-terminal kinase and caspase 3 activation induced by 4-HPR and that NFkappaB inhibition by the I kappa B alpha phosphorylation inhibitor compound Bay 117082 resulted in increasing sensitization of both PC-3 and DU145 lines to apoptosis induced by 4-HPR at low concentrations. Furthermore, we found that inhibition of extracellular signal-regulated kinase (ERK) enhanced the suppression of NFkappaB by 4-HPR and also resulted in sensitization to apoptosis in the DU145 cell line, in which ERK is activated constitutively. It thus appears that mitogen-activated protein kinase associated with the activity of NFkappaB plays an important role in the degree of resistance to 4-HPR-induced apoptosis in human prostate cancer cells.