Evaluation consistency in iGAs: User contradictions as cycles in partial-ordering graphs

by Francesc Alías, Xavier Llorà, Lluís Formiga, Kumara Sastry, and David E. Goldberg (2006, accepted).

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2006). May 14-19, 2006. Also as IlliGAL TR No 2005022. Link to the PDF.

Abstract

Active interactive genetic algorithms (aiGAs) rely on actively optimizing synthetic fitness functions. In interactive genetic algorithms (iGAs) framework, user evaluations provide the necessary input for synthesizing a reasonably accurate surrogate fitness function that models user evaluations or, in other words, his/her decision preferences. User evaluations collected via tournament selection only provide partial-ordering relations between solutions. Active iGAs assemble a partial-ordering graph of user evaluations. In such a directed graph, any contradictory evaluation provided by the user introduces a cycle in the graph. This property is explored in this paper to measure the consistency of the evaluations provided by the user along the evolutionary process. The consistency measures are applied to a real-world problem, the weight tuning of the cost function involved in corpus-based text-to-speech synthesis. Results show the usefulness of such measures to identify inconsistent users during the evolutionary tuning process, and successfully the number of evaluations required by more than half.