Analytic Quality: Evaluation of Performance and Insight in Multimedia Collection Analysis
Jan Zahálka, S. Rudinac, M. Worring
Oct 13, 2015
Citations
17
Citations
Journal
Proceedings of the 23rd ACM international conference on Multimedia
Full text
Semantic Scholar
Key Takeaway Key Takeaway: Analytic quality (AQ) is a novel evaluation paradigm for multimedia analysis methods, incorporating user insight gain and time needed to acquire it, providing more insightful information for method comparison and design improvements.
Abstract
In this paper, we present analytic quality (AQ), a novel paradigm for the design and evaluation of multimedia analysis methods. AQ complements the existing evaluation methods based on either machine-driven benchmarks or user studies. AQ includes the notion of user insight gain and the time needed to acquire it, both critical aspects of large-scale multimedia collections analysis. To incorporate insight, AQ introduces a novel user model. In this model, each simulated user, or artificial actor, builds its insight over time, at any time operating with multiple categories of relevance. The methods are evaluated in timed sessions. The artificial actors interact with each method and steer the course by indicating relevant items throughout the session. AQ measures not only precision and recall, but also throughput, diversity of the results, and the accuracy of estimating the percentage of relevant items in the collection. AQ is shown to provide a wide picture of analytic capabilities of the evaluated methods and enumerate how their strengths differ for different purposes. The AQ time plots provide design suggestions for improving the evaluated methods. AQ is demonstrated to be more insightful than the classic benchmark evaluation paradigm both in terms of method comparison and suggestions for further design.