Gold HITs — Human Intelligence Tasks with known answers — are commonly used to measure worker performance and data quality in industrial applications of crowdsourcing. We suggest adaptive exploration as a promising approach for automated, scalable Gold HIT creation. We substantiate this with initial experiments in a stylized model.
Most research in network pricing focuses on improving QoS for a single network. These approaches are not easily extensible to a set of interconnected networks, as the inter-ISP negotiations would be unwieldy. Our approach is to provide a general QoS-aware networking service. We first introduce the concept of QoS-transit services: transit services with QoS… (More)
The transmission of multimedia streams with a particular and guaranteed quality of service (QoS) is a major challenge in multimedia communications over IP networks. In this paper, we present the design and simulation results for a new distributed admission and adaptation controller for a set of interconnected networks. A new greedy admission heuristic and a… (More)
We have previously proposed the use of dynamically priced network services to provide QoS guarantees within a network. End-to-end QoS can be achieved by concatenating several of these services, perhaps from different ISPs. In this paper we consider the problem of a single ISP determining the bandwidth to allocate to each service, and on which path, in order… (More)
Crowdsourcing has been part of the IR toolbox as a cheap and fast mechanism to obtain labels for system development and evaluation. Successful deployment of crowdsourcing at scale involves adjusting many variables, a very important one being the number of workers needed per human intelligence task (HIT). We consider the crowdsourcing task of learning the… (More)
We consider the crowdsourcing task of learning the answer to simple multiple-choice microtasks. In order to provide statistically significant results, one often needs to ask multiple workers to answer the same microtask. A stopping rule is an algorithm that for a given microtask decides for any given set of worker answers if the system should stop and… (More)
We have proposed the use of dynamically priced network services to provide QoS guarantees within a network. End-to-end QoS can be achieved by using several of these services, perhaps from different ISPs. In this paper we consider the problem of determining the bandwidth to allocate each service in order to maximize revenue, assuming that a single ISP can… (More)