In this paper, a high efficient arbitrary transform blocks hardware allocation framework is proposed, which can adapt the prediction tree structure and then improve the utilization ratio as well as a parallel hardware design to improve the data throughput. This method will configure an appropriate combination of five different size inverse transform units: Fast IDST, 4×4 IDCT, 8×8 IDCT, 16×16 IDCT and 32×32 IDCT. If the input video stream changed, it will reconfigure the combination and allocated the hardware resources to retain a high utilization ratio in hardware framework. Experiments show that the performance of the proposed method is improved from 48.8% to 96.2% under various conditions. The proposed method can enhance the efficiency of H.265 decoder.