D7.2: Benchmark report (result of piloting task)

Project Reference

287607

Project Acronym

FIRST

Project Full Title

A Flexible Interactive Reading Support Tool

Distribution Level

Public                             

Contractual Date of Delivery

31st October 2012

Actual Date of Delivery

14th April 2014

Document Number

FIRST_D7.2_20140414

Status & Version

Approved (v1.2 modified to address evaluators’ comments from the second review meeting)

Number of Pages

40

WP Contributing to the Deliverable

WP7

WP Task responsible

Vesna Jordanova

Authors

Vesna Jordanova, Richard Evans, Arlinda Cerga Pashoja

Other Contributors

Eduard Barbu, Sandra Freire,Ana González, Martin Keats, Elena Lloret, Eugenio Martinez, Paloma Moreda, Sanja Stajner, Antoneta Shishkova

Reviewer

Constantin Orasan

EC Project Officer

Magdalena Szwochertowska

Keywords:

Benchmarking, texts, simplification, autism

Abstract:

Benchmarking involves structured comparisons to help define and implement best practice. Current practices of text simplification for people with Autistic Spectrum Disorders (ASD) involve manual processing of written information. Best practices are delivered by experienced special needs teachers and carers who have good understanding and knowledge about autism and its associated reading difficulties. The objective of the benchmarking exercise was to determine how long it might take for a special needs teacher or a professional carer to simplify written documents satisfactorily for adults with ASD so that a comparison might be drawn with the “Open Book” software as an alternative, digital service. The benchmarking task aims to provide a comparative measure of speed and accuracy provided by experienced professionals. In this way the software writers would know the degree of speed and accuracy that users of the software would have to exceed in order to justify its use. We recruited 15 professionals (special needs teachers and professional carers) with experience of working with people with ASD. They were given five different texts that they simplified manually to make them comprehensible for people with ASD. A total of 75 different texts covering a wide range of topics were analysed. The numbers of words in the original and simplified versions were recorded and compared as were the individual times taken as recorded by the teachers and carers who volunteered to take part in the study.

Professionals took just over 54 minutes in average to simplify a text of 250-350 words long. Simplifying text for adult users took 23 minutes, professionals working with children texts spent 97 minutes and those addressing both target groups spent 43 minutes. They worked at the average speed of 12 words per minute.

For all three languages, the use of established readability metrics demonstrated differences in the readability of the original and simplified versions of the texts. In several cases, these differences were statistically significant.