Relay stations (RSs) are usually used to enhance the signal strength of the mobile stations (MSs) close to the cell boundary. However, the introduction of RSs to the cellular networks increases the interference to the MS served by the base station (BS) under spatial multiplexing mode. In this paper, we propose a method which can cancel the interference through a novel use of dirty paper coding (DPC). Then, under a practical spatial multiplexing frame structure, we evaluate the cellularrelay system performance under three scheduling algorithms, Round Robin (RR) algorithm, Maximum Signal to Interference and Noise Ratio (MaxSINR) algorithm and Proportional Fair (PF) algorithm. Simulation and analytical results show that the spectral efficiency (SE) of cellular-relay system can be dramatically increased by 30% by the proposed interference cancellation method, and RR algorithm achieves the highest gain, while the gain of MaxSINR algorithm is limited gain in our scheme.