Towards Automated Web Accessibility Evaluation: A Comparative Study

Автор: Siddikjon Gaibullojonovich Abduganiev

Журнал: International Journal of Information Technology and Computer Science(IJITCS) @ijitcs

Статья в выпуске: 9 Vol. 9, 2017 года.

Бесплатный доступ

With each passing day, the Web is becoming increasingly important in our lives. Hence, the need of making it more accessible to everyone, especially for the disabled and elderly spurred a great interest in automated tools, the total registered number of which has been continuously increasing and reached from forty-five software bids in 2014 to ninety-three in 2017. The purpose of this empirical research is to assess and compare eight popular and free online automated Web accessibility evaluation tools (AWAETs) such as AChecker, Cynthia Says, EIII Checker, MAUVE, SortSite, TAW, Tenon and WAVE with regard to the WCAG 2.0 conformance. As a result, significant differences were observed in terms of tool’s coverage (a maximum of 32.4%), completeness (ranges between 10% and 59%), correctness (an average of 70.7%), specificity (reaches 32%), inter-reliability (lies between 1.56% and 18.32%) and intra-reliability (the acceptable score), validity, efficiency and capacity. These eight criteria can help to determine a new role played by modern AWAETs as dependent methods in Web accessibility evaluation. Moreover, consequences of relying on AWAETs alone are quantified and concluded that applying such approaches is a great mistake since subjective and less frequent objective success criteria (SC) failed to be automated. However, using a good combination of AWAETs is highly recommended as overall results in all the mentioned quality criteria are maximized and tools could definitely validate and complete each other. Ultimately, integrating automated methods with the others is ideal and preferably at an early stage of the website development life cycle. The study also provides potential accessibility barriers that make websites inaccessible, challenges AWAETs are currently facing, nineteen pros and fourteen cons and fifteen improvement recommendations for the existing and next generation of AWAETs. Fundamentally, achieving the objectives of this study was possible due to the elaboration and implementation of a new five-phased methodology named as “5PhM-for-AWAEMs” for successful selection, evaluation and/or comparison of AWAEMs. In addition to providing detailed descriptions of the estimation process, this methodology represents eleven key criteria for effective selection of suitable AWAEMs and necessary numbers of web pages and expert evaluators for acceptable, normal or ideal assessment.

Еще

Web accessibility, guidelines and standards, WCAG 1.0 and 2.0, automatic web accessibility evaluation methods and tools, TP, FP, FN, human-expert review, tool’s coverage, completeness, correctness, specificity, inter- and intra-reliability, validity, efficiency and capacity

Еще

Короткий адрес: https://sciup.org/15012679

IDR: 15012679

Список литературы Towards Automated Web Accessibility Evaluation: A Comparative Study

World Bank, Disability Overview, Apr 04, 2016. Available online: http://www.worldbank.org
Fogg B. J., Swani P., Treinen M., Marshall J., Osipovich A., Varma C., Laraki O., Fang N., Paul J., Rangnekar A. and Shon J., Elements that affect Web credibility: Early Results From a Self-Report Study, Proceedings of CHI'00, Extended Abstracts on Human Factors in Computing Systems, 2000, pp. 287-288.
Killam B. and Holland B., Position Paper on The Suitability of Task Automated Utilities for Testing Web Accessibility Compliance, Usability professionals’ association conference, 2001. Available online: http://www.upassoc.org/conf2001/
Lindenberg J. and Neerinex M.A., The Need for a “Universal Accessibility” Engineering Tool. Proceedings, Interact ’99 workshop: Making designers aware of existing guidelines for accessibility, August 1999.
Zeng X., Evaluation and Enhancement of Web Content Accessibility for Persons With Disabilities. Ph.D. Thesis, University of Pittsburgh, 2004.
Good A., An Investigation of a Method for Improving Accessibility to Web-Based information for Users with Impairments, Doctoral Thesis, University of Portsmouth, Portsmouth, 2008.
Letourneau C., Accessible Web Design – a Definitiion, 2001. Available online: www.starlingweb.com/webac.htm
World Wide Web Consortium W3C, Web Content Accessibility Guidelines – WCAG 1.0, 5 May 1999.
Paciello M., Web Accessibility for People with Disabilities. CMP Books, 2000, ISBN: 1929629087.
Jim Thatcher, Cynthia Waddell, Shawn Henry, Sarah Swierenga, Mark Urban, Michael Burks, Bob Regan and Paul Bohman., Constructing Accessible Web sites. Glasshouse 2002.
Slatin J. and Rush. S., Maximum Accessibility: Making Your Web Site More Usable for Everyone. Addison-Wesley, 2003.
Paciello M., Web Accessibility for People with Disabilities. CMP Books, 2000, ISBN: 1929629087.
Michigan State University, Web Accessibility, 2017. Available online: http://webaccess.msu.edu
Web Content Accessibility Guidelines 2.0, W3C Working Draft, 11 February 2005.
ISO IS 9241-171:2008 Ergonomics of Human-System Interaction - Guidance on Software Accessibility. (a restructured version of ISO TS 16071).
ISO TS 16071: 2002 Guidance on accessibility.
The order of the government of the Republic of Tajikistan, The State Program of Development and Implementation of Information and Communication Technologies in the Republic of Tajikistan, December 3, 2004, No. 468.
The order of the government of the Republic of Tajikistan, The State Program of Development and Implementation of Information and Communication Technologies in the Republic of Tajikistan for 2014-2017, July 3, 2014, No. 428.
National Action Plan on Disability 2012-2020, Strategy of the Austrian Federal Government for the Implementation of the UN Disability Rights Convention (Nationaler Aktionsplan Behinderung 2012-2020. Strategie der Österreichischen Bundesregierung zur Umsetzung der UN-Behindertenrechtskonvention), 2012.
UN, Convention and Optional Protocol Signatories and Ratification. Available online: www.europarl.europa.eu
Harper S. and Yesilada Y., Web Accessibility, Springer, London, United Kingdom, 2008.
Coyne K. and Nielsen J., How to Conduct Usability Evaluations for Accessibility: Methodology Guidelines for Testing Websites and Intranets With Users Who Use Assistive Technology, Nielsen Norman Group, Oct. 2001.
Brajnik G., Web accessibility testing with barriers walkthrough, March 2006.
DRC, Formal investigation report: Web Accessibility. Disability Rights Commission, April 2004.
Henry S.L. and Grossnickle M., Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004.
W3C/WAI, Conformance Evaluation of Web Sites for Accessibility, 2008. Available online: www.w3.org/
Dey A., Accessibility Evaluation Practices – Survey Results, 2004. Available online: http://deyalexander.com
Ivory M.Y. and Chevalier A., A Study of Automated Web Site Evaluation Tools. The Information School University of Washington, Aline Chevalier Department of Cognitive Psychology University of Provence, Technical Report UWCSE-02-10-01 October 8, 2002.
Brajnik G., Comparing Accessibility Evaluation Tools: A Method for Tool Effectiveness. Universal Access in the Information Society, Springer Verlag, Oct. 2004, 3(3-4), pp. 252-263.
Brajnik G., Beyond Conformance: the Role of Accessibility Evaluation Methods. In S. Hartmann et al., editor, WISE 2008: 9th Int. Conference on Web Information Systems Engineering – 2nd International Workshop on Web Usability and Accessibility IWWUA08, LNCS 5176, 2008, pp. 63–80, Auckland, New Zealand, Sept. 2008c. Springer-Verlag. Keynote speech.
Vigo M., Justin B. and Vivienne C., Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole Reliance on Automated Tests, In: Proceedings of 10th International Cross-Disciplinary Conference on Web Accessibility (W4A), ACM Press, May 13-15, 2013, Rio de Janeiro, Brazil.
Kaur A. and Diksha D., Comparing and Evaluating the Effectiveness of Mobile Web Adequacy Evaluation Tools. Article in Universal Access in the Information Society, May 2016. DOI: 10.1007/s10209-016-0466-z.2.
Pivetta E. M., Saito D. S., Flor C. S., Ulbricht V. R., Vanzin T., Automated Accessibility Evaluation Software for Authenticated Environments - A Heuristic Usability Evaluation. HCI (7) 2014, pp. 77-88
Al-Khalifa H. S., WCAG 2.0 Semi-automatic Accessibility Evaluation System: Design and Implementation. Computer and Information Science, 2012, Vol. 5, No 6.
Sukhpal K., An Automated Tool for Web Site Evaluation, International Journal of Computer Science and Information Technologies, 2012, 3 (3), pp. 4310 – 4313.
Molinero A. M. and Kohun F. G., Reliability in Automated Evaluation Tools for Web Accessibility Standards Compliance. In issues in Information Systems, Volume VII, No. 2, 2006.
Centeno V. L., Kloos C. D., Fisteus J. A. and Alvarez L. A., Web Accessibility Evaluation Tools: A Survey and Some Improvements. Journal Electronic Notes in Theoretical Computer Science(ENTCS) archive, 2006,157(2), pp. 87-100.
Xiong J., Farenc C. and Winckler M., Analyzing Tool Support for Inspecting Accessibility Guidelines during the Development Process of Web Sites. In Proceedings of 1st IWWUA workshop inside WISE. Springer LNCS 4832, 2007, pp. 470-480.
Ahmad Al-A., Ibraheem Y.Y. and Ahmaro M.M., Comparison Between Web Accessibility Evaluation Tools. Almadinah Islamic Studies, 2010, 1(66): Malaysian Studies.
Akgül Y. and Vatansever K., Web Content Accessibility of Municipal Web Sites in Turkey. February, 2016, 7(1), pp. 43-48. doi: 10.12720/jait.7.1.43-48
Hackett S., Parmanto B. and Zeng X., A Retrospective Look at Website Accessibility Over Time. Behaviour and Information Technology, 2005, 24 (6), pp. 407-417.
Chisholm W. Vanderheiden G. and Jacobs I. (Eds.), Web Content Accessibility Guidelines 1.0, May 5 1999.
International Organization for Standardization: ISO/IEC 40500:2012.
Caldwell B., Cooper M., Reid L.G. and Vanderheiden G. (eds.), Web Content Accessibility Guidelines 2.0, W3C Recommendation, 2008. Available online: www.w3.org
World Wide Web Consortium (W3C). Web Content Accessibility Guidelines (WCAG) 2.0, 2008. Available online: https://www.w3.org/TR/WCAG20/
Brajnik G., A Comparative Test of Web Accessibility Evaluation Methods. In Proceedings of the 10th international ACM SIGACCESS Conference on Computers and Accessibility (Halifax, Nova Scotia, Canada, October 13 - 15, 2008), Assets 08, ACM, New York, NY, pp. 113-120.
Brajnik G., Mulas A. and Pitton C., Effects of Sampling Methods on Web Accessibility Evaluations. In Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, 2007, Assets 07, pp. 59-66.
Sevilla J., Herrera G., Martínez and Alcantud F. Web Accessibility for Individuals with Cognitive Deficits: A Comparative Study Between an Existing Commercial Web and its Cognitively Accessible Equivalent. ACM Transactions on Computer Human Interaction, 2007, 14(3), p. 12.
Power C., Freire A., Petrie H., and Swallow D., Guidelines are Only Half of the Story: Accessibility Problems Encountered by Blind Users on the Web. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, CHI '12, 2012, pp. 433-442.
Gray W.D. and Salzman M.C., Damaged Merchandise: A Review of Experiments That Compare Usability Evaluation Methods, Human–Computer Interaction, 1998, 13(3), pp. 203– 261.
Microsoft and HiSoftware 2009, Microsoft Web Accessibility Handbook.
Petrie H. and Kheir O., The Relationship Between Accessibility and Usability of Websites. In: Proc. CHI’07, ACM, CA, 2007, pp. 397–406.
Mankoff J., Fait H. and Tran T., Is Your Web Page Accessible? A Comparative Study of Methods for Assessing Web Page Accessibility for the Blind. In Proc. SIGCHI Conference on Human Factors in Computing Systems, CHI’05, 2005, pp. 41-50.
Hertzum M. and Jacobsen N.E., The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. Int. Journal of Human-Computer Interaction, 2001, 1(4), pp. 421–443.
Lang T., Comparing Website Accessibility Evaluation Methods and Learnings from Usability Evaluation Methods, 2003.
Sears A.: Heuristic Walkthroughs: Finding the Problems Without the Noise. Int. Journal of Human-Computer Interaction, 1997, 9(3), pp. 213–234.
Kelly B., Sloan D., Brown S., Seale J., Petrie H., Lauke P. and Ball S., Accessibility 2.0: People, Policies and Processes. In: W4A 2007: Proc. of the 2007 international cross-disciplinary conference on Web accessibility (W4A), 2007, pp. 138–147. ACM, New York.
Carter J. and Markel M., Web Accessibility for People with Disabilities: An Introduction for Web Developers. IEEE Transactions on Professional Communication, 2001, 44 (4), 225- 233.
Christopher B. and Elaine P., Development and Trial of an Educational Tool to Support the Accessibility Evaluation. Proc. In: Proceedings of the 2011 International Cross-Disciplinary Conference on Web Accessibility W4A, 2011, pp. 2. Available online: http://dx.doi.org/10.1145/
Web Accessibility Evaluation Tools List, Updated June 2017 (first published March 2006). Available online: https://www.w3.org/WAI/ER/tools/
Lopes R. and Carric L., Macroscopic Characterizations of Web Accessibility. New Review of Hypermedia and Multimedia, 2010, 16(3), pp. 221-243.
Hackett S. and Parmanto B., Homepage Not Enough When Evaluating Web Site Accessibility, Internet Research, 2009, 19(1), pp. 78–87.
Patton M.Q., Qualitative Evaluation and Research Methods, Sage Publications, Newbury Park, California, 1990, p. 532 .
Web Tools for Quality, Accessibility, Standards Compliance. Available online: http://valet.webthing.com
Schiavone A. G. and Paterno F., An Extensible Environment for Guideline-Based Accessibility Evaluation Of dynamic Web applications. Univ Access Inf Soc, Vol. 14, pp.111–132, DOI 10.1007/s10209-014-0399-3.
Sean P. Aune, 12 Tools to Check Your Site’s Accessibility, July 06, 2009. Available online: www.sitepoint.com
Oleg Mokhov, 20 Наиболее Необходимых Инструментов для Проверки Отображения Сайта, 18 march 2011. Available online: habrahabr.ru/company/aiken/blog/
Justin Mifsud, 10 Free Web-Based Web Site Accessibility Evaluation Tools, August 22, 2011. Available online: http://usabilitygeek.com/10-free-web-based-web-site-accessibility-evaluation-tools/
Simon Heaton, 9 Tools for Website Accessibility Testing, Jun 29, 2016. Available online: shopify.com
Achecker (Web Accessibility Checker), Inclusive Design Research Centre, 2011. Available online: achecker.ca/
Gay G. and Li C.Q., AChecker: Open, Interactive, Customizable, Web Accessibility Checking. In: International Cross Disciplinary Conference on Web Accessibility-W4A, 2010, pp. 1–2.
Cynthia Says, HiSoftware Inc, 2003. Available online: http://www.cynthiasays.com
Koutsabasis P., Vlachogiannis E. and Darzentas J.S., Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility, Journal of Usability Studies, August 2010, 5(4), pp.157-171.
MAUVE (Version: 1.3), Human Interfaces in Information Systems Laboratory - ISTI-CNR. Available online: http://hiis.isti.cnr.it:8080.
SortSite, Power Mapper Software, 1996. Available online: http://www.powermapper.com.
Declaring Conformance on Web Accessibility, ANEC Print version 21, May 2011, pp. 1-44. Project Reference: ANEC-R&T-2009-ICT-001final
CEAPAT, Fundaci´on CTIC, Spanish Ministry of Employment and Social Affairs (IMSERSO), Online Web accessibility test. Available online: www.tawdis.net
Tenon (Version: 1.0), by Tenon. Available online: tenon.io
Quickly Check Your Website for Common Accessibility Problems with tenon.io. Available online: marcozehe.de
European Internet Inclusion Initiative (EIII Checker). Available online: http://checkers.eiii.eu/
Snaprud M., Benchmarking Results from 1000 European Websites. EIII supported by the European Union Seventh Framework Programme (Grant agreement no: 609667).
WAVE (Web accessibility evaluation tools), WebAIM, 1999. Available online: http://wave.webaim.org
Pivetta E. M., Flor C. F., Saito D. S. and Ulbricht V. R., Analysis of an Automatic Accessibility Evaluator to Validate a Virtual and Authenticated Environment. International Journal of Advanced Computer Science and Applications, 2013, Vol. 4, pp. 15-22.
Sears A., Heuristic walkthroughs: finding the problems without the noise. Int. Journal of Human-Computer Interaction, 1997, 9(3), pp. 213–234.
Gray W. and Salzman M. Damaged merchandise: a review of experiments that compare usability evaluation methods. Human–Computer Interaction, 1998, 13(3), pp. 203–261.
Hertzum M. and Jacobsen N., The evaluator effect: a chilling fact about usability evaluation methods. Int. Journal of Human-Computer Interaction, 2001, 1(4), pp. 421–443.
Lang T., Comparing website accessibility evaluation methods and learnings from usability evaluation methods, 2003. Available online: www.peakusability.com.au
Hartson H. R., Andre T. S. and Williges R. C. Criteria for Evaluating Usability Evaluation Methods. Int. Journal of Human-Computer Interaction, 2003, 15(1), pp. 145–181.
Diaper D and Worman L.: Two Falls Out of Three in the Automated Accessibility Assessment of World Wide Web Sites: A-prompt v. Bobby. In: People and Computers, Vol. 17, 2003, pp. 349-363, Springer.
Henry S.L. and Grossnickle M.: Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA; On-line book, 2004.
W3C/WAI., Conformance Evaluation of Web Sites for Accessibility. Available online: www.w3.org/WAI/eval/
Centeno V. L., Kloos C. D., Fernandez L. S. and Fernandez N. G., Device independence for Web Wrapper Agents Workshop on Device Independent Web Engineering (DIWE’04), 26 July 2004, Munich.
Cooper M., Limbourg Q., Mariage C. and Vanderdonckt J., Integrating Universal Design Into a Global Approach for Managing Very Large Web Sites. In Proceedings of the 5th ERCIM Workshop on User Interfaces for All, 1999.
Farenc C., Liberati V. and Barthet M.F., Automatic Ergonomic Evaluation: What Are The Limits? In Proceedings of the Second International Workshop on Computer-Aided Design of User Interfaces, CADUI '96, 1996, pp.159-170.
Ivory M.Y. and Hearst M. A., State of the Art In Automating Usability Evaluation of User Interfaces. ACM Computing Surveys, December 2001, 33(4), pp. 470–516.
Chevalier A. and Ivory M. Y., Web Site Designs: Influences of Designer’s Experience and Design Constraints. Submitted for Publication, 2002.
Lee A. and Hanson V., Enhancing Web Accessibility, Proceedings ACM Multimedia 2003, pp. 456-457, ACM Press.
Abduganiev S.G., Pils M. and Roithmayr F., Elicitation of Criteria Weights for the Web Quality Evaluation Method Universal Star: By Using Different Ranking Methods. The Strategies of Modern Science Development: Proceedings of the X International scientific–practical conference. North Charleston, USA, 12-13 April 2016. - North Charleston: CreateSpace, 2016, pp. 11-24.
Pils M., Ganglberger M. and Höller J., Barrierefreiheit von Behörden Websites - Anspruch Und Realität. in: H. Wandke /S. Kain/D. Struve (Hrsg.): Mensch und Computer 2009; Grenzenlos frei!? München Oldenburg Verlag, 2009, S.3-12.

Еще

Статья научная