Corpus ID: 61825275

Overview of DUC 2005

  title={Overview of DUC 2005},
  author={Hoa Trang Dang},
  • H. Dang
  • Published 2005
  • Computer Science
The focus of DUC 2005 was on developing new evaluation methods that take into account variation in content in human-authored summaries. Therefore, DUC 2005 had a single user-oriented, question-focused summarization task that allowed the community to put some time and effort into helping with the new evaluation framework. The summarization task was to synthesize from a set of 25-50 documents a well-organized, fluent answer to a complex question. The relatively generous allowance of 250 words for… Expand

Figures, Tables, and Topics from this paper

Overview of DUC 2006
The DUC 2006 summarization task was to synthesize from a set of 25 documents a wellorganized, answer to a complex question, and the overall responsiveness metric showed that readability plays an important role in the perceived quality of the summaries. Expand
DUC in context
This paper examines several major themes running through three evaluations: SUMMAC, NTCIR, and DUC, with a concentration on DUC. Expand
Proceedings of the Workshop on Task-Focused Summarization and Question Answering
The Task-Focused Summarization and Question Answering workshop, to be held on July 23, 2006 in Sydney, aims to bring together the two communities of summarization and question answering by examiningExpand
Question Answering as an Automatic Evaluation Metric for News Article Summarization
An end-to-end neural abstractive model is presented that maximizes APES, while increasing ROUGE scores to competitive results, and analyzing the strength of this metric by comparing it to known manual evaluation metrics. Expand
Googling answers ’ models in question-focused summarisation
This paper describes the techniques used for our system participating in the Document Understanding Conference 2006. We describe a new system, built from scratch, that focuses primarily on collectingExpand
Query-Focused Summaries or Query-Biased Summaries?
While query focus correlates with performance, it is shown that high-performing automatic systems produce summaries with disproportionally higher query term density than human summarizers do. Expand
Query Independent Sentence Scoring approach to DUC 2006
The task in Document Understanding Conferences (DUC 1 ) 2006 is to generate a fixed length, user oriented, multi document summary, which remains same as that of DUC 2005, and the use of web in scoring the sentences in a query independent manner. Expand
Re-evaluating Evaluation in Text Summarization
Assessing the reliability of automatic metrics using top-scoring system outputs on recently popular datasets for both system-level and summary-level evaluation settings finds that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems. Expand
Summary Cloze: A New Task for Content Selection in Topic-Focused Summarization
This work proposes a new method for studying content selection in topic-focused summarization called the summary cloze task and reports experimental results on this new dataset using various extractive models and a two-step abstractive model that first extractively selects a small number of sentences and then abstractively summarizes them. Expand
Automatic Summarization from Multiple Documents
This work formalizes the n-gram graph representation and its use in NLP tasks, and presents a set of algorithmic constructs and methodologies that aim to support meaning extraction and textual quality quantification. Expand


The Effects of Human Variation in DUC Summarization Evaluation
How the variation in human judgments does and does not affect the results and their interpretation of automatic text summarization systems’ output is examined. Expand
Evaluating DUC 2005 using Basic Elements
It is shown that this method correlates better with human judgments than any other automated procedure to date, and overcomes the subjectivity/variability problems of manual methods that require humans to preprocess summaries to be evaluated. Expand
Applying the Pyramid Method in DUC 2005
It is found that a modified pyramid score gave good results and would simplify peer annotation in the future and high score correlations between sets from different annotators, and good interannotator agreement, indicate that participants can perform annotation reliably. Expand
ROUGE: A Package for Automatic Evaluation of Summaries
Four different RouGE measures are introduced: ROUGE-N, ROUge-L, R OUGE-W, and ROUAGE-S included in the Rouge summarization evaluation package and their evaluations. Expand
An Empirical Study of Information Synthesis Task
This paper describes an empirical study of the "Information Synthesis" task, defined as the process of extracting, organizing and inter-relating the pieces of information contained in a set of relevant documents, in order to obtain a comprehensive, non redundant report that satisfies the information need. Expand
The effect of topic set size on retrieval experiment error
Using TREC results to empirically derive error rates based on the number of topics used in a test and the observed difference in the average scores indicates researchers need to take care when concluding one method is better than another, especially if few topics are used. Expand