Detection of Response Quality Degradation During Crowdsourced Annotation Tasks Using a Conversational Interface

Abstract

Crowdsourcing-based annotation tasks have been widely utilized as a cost-effective and scalable approach for collecting training data in machine learning. However, the quality of responses obtained through crowdsourcing often varies, posing significant challenges for quality control. To address this issue, this study aims to develop a method for preventing quality degradation by detecting signs of declining response quality in real time. We propose a binary classification model that estimates quality degradation by extracting features from device operation data-such as device orientation and screen interactions-collected during the annotation process and applying supervised machine learning. To evaluate the effectiveness of the proposed method, we developed a smartphone application that supports annotation tasks and continuously collects device operation data in the background. A user study was conducted in which participants were asked to evaluate the correctness of image captions, while their device operation data were continuously recorded during task execution. Using the collected data, we built and evaluated a binary classification model to estimate response quality. The model, trained with time-series validation on intra-individual data, achieved a precision of 0.723, a recall of 0.741, and an F1-score of 0.731. These results suggest that the proposed method is effective in estimating response quality degradation in crowdsourced annotation tasks.

Links

code:references.bib

@inproceedings{bib:taira_satisficeChat_ICMU2025,

author={Taira, Shigeyuki and Matsuda, Yuki and Suwa, Hirohiko and Yasumoto, Keiichi},

title={Detection of Response Quality Degradation During Crowdsourced Annotation Tasks Using a Conversational Interface},

booktitle={The 15th International Conference on Mobile Computing and Ubiquitous Networking (ICMU '25)},

volume={15788},

pages={x--x},

year={2025},

doi={},

url={}

}

https://scrapbox.io/files/688a428a7dcbb2ebb472c4d0.png

Category

Conference

Project

Keywords

Collaborating Organization