T cells play a crucial role in preventing the growth and spread of colorectal cancer (CRC). However, immunotherapies against CRC have only shown limited success, which may be due to lack of understanding about the effect of the local tumor microenvironment (TME) on T cell function. The goal of this study was to determine whether T cells in tumor tissue were functionally impaired compared to T cells in non-tumor bowel (NTB) tissue from the same patients. We showed that T cell populations are affected differently by the TME. In the tumor, T cells produced more IL-17 and less IL-2 per cell than their counterparts from NTB tissue. T cells from tumor tissue also had impaired proliferative ability compared to T cells in NTB tissue. This impairment was not related to the frequency of IL-2 producing T cells or regulatory T cells, but T cells from the TME had a higher co-expression of inhibitory receptors than T cells from NTB. Overall, our data indicate that T cells in tumor tissue are functionally altered by the CRC TME, which is likely due to cell intrinsic factors. The TME is therefore an important consideration in predicting the effect of immune modulatory therapies.