D7.8: Final evaluation report

Project Reference

287607

Project Acronym

FIRST

Project Full Title

A Flexible Interactive Reading Support Tool

Distribution Level

Public                             

Contractual Date of Delivery

30th September 2014

Actual Date of Delivery

19th November 2014

Document Number

FIRST_D7.8_20141119

Status & Version

Approved v1

Number of Pages

100

WP Contributing to the Deliverable

WP7

WP Task responsible

Vesna Jordanova

Authors

Richard Evans and Arlinda Cerga Pashoja

Other Contributors

The whole consortium

Reviewer

Constantin Orasan

EC Project Officer

Krister Olson

Keywords:

evaluation, intrinsic evaluation, extrinsic evaluation, qualitative methodology, quantitative methodology, text accessibility, reading comprehension, text editing, language technology, readability

Abstract:

In the FIRST project, language technologies (LT) were acquired and developed to enable deployment of two services intended to convert texts into a more accessible form for people with autism. In one, the service is fully automatic, with end users directly consuming the output of the LT. In the other, the LT is exploited by intermediaries (carers) to assist in the manual conversion of texts into a more accessible form. In this context, conversion is a semi-automatic process. The set of language technologies integrated to support these text conversion services is called OpenBook. The quality of the text conversion services was assessed using two broad categories of evaluation: extrinsic and intrinsic. Extrinsic evaluation assesses the text conversion services from the perspective of users, intermediaries exploiting the LT to convert texts, and end users (people with autism), who are the final consumers of the text conversion services. Extrinsic evaluation includes an investigation of the impact of the service on the reading comprehension of end users and of the perceived usefulness of the LT for intermediaries. These aspects of extrinsic evaluation are conducted using quantitative methods (reading comprehension tests). Another important aspect of extrinsic evaluation involves the use of qualitative methods to investigate the impact of the text conversion services on quality of life, taking into account perceived changes in independence, education, and stigmatisation. Interviews with end users and intermediaries were conducted and the transcriptions analysed to support this aspect of extrinsic evaluation. Intrinsic evaluation involves quantitative assessment of the LT components supporting text conversion. This evaluation is made via standard methods used in the field of natural language processing (NLP) together with assessments of the readability of texts in their original forms and in converted forms generated by intermediaries exploiting OpenBook. Comparisons are drawn between the readability of texts converted by carers using OpenBook and a benchmark comprising the readability of texts produced by unaided conversion of texts (D7.2).