Published in the SIGDIAL 2023 proceedings, Prague, Czechia, pp. 242–254.
This paper investigates complex open-domain dialogue annotation using a multi-level speech-function taxonomy. Rather than classifying text directly into numerous categories, it uses a hierarchy of simple questions, understandable both to non-specialist annotators and to LLMs.
The study compares expert annotators, crowdworkers, and ChatGPT, as well as different prompt-engineering approaches. It shows that a multi-step, tree-like scheme can significantly improve automatic annotation and, in some settings, bring LLM performance close to human agreement.