This study presents a hierarchical news video categorization architecture based on semantic analysis. First, news video is segmented into different news stories based on anchorperson shot detection. For each news story, we extract its caption information to perform semantic analysis. Theses stories are then semantically categorized as an adaptive hierarchy tree. This tree adaptive split its tree node in accordance with a term-based entropy measure. The superiority of the proposed system has been demonstrated through the three days news video obtained from EraNews in Taiwan.