This paper addresses the question of how to obtain consistent semantic annotation on the basis of a set of noisy texts. Many potential real-world applications of semantic computing are faced with the need to handle texts which are not well-edited, and for which a resource-intensive treebanking effort is not feasible. Student-produced short answers contain many grammatical and lexical errors, making consistent annotation a challenge. Nevertheless, this paper demonstrates that semantic role annotation can be done in a consistent and useful manner even under these constraints.