Систематический обзор методов составления тестовых инвариантов

Систематический обзор методов составления тестовых инвариантов

Автор: Якушева С.Ф., Хританков А.С.

Журнал: Программные системы: теория и приложения @programmnye-sistemy

Рубрика: Методы оптимизации и теория управления

Статья в выпуске: 2 (61) т.15, 2024 года.

Бесплатный доступ

Тестирование инвариантами (metamorphic testing) - один из наиболее эффективных методов тестирования программ, для которых сложно подбирать тестовые примеры и формулировать тестовые оракулы. При тестировании инвариантами вместо проверки правильности вывода программы на отдельных наборах входных данных проверяется выполнение тестового инварианта (metamorphic relation) - функции от нескольких наборов исходных данных и соответствующих им ответов программы. Составление тестовых инвариантов требует понимания решаемой программой задачи и творческого подхода. Предлагаемый систематический обзор посвящён выявлению широкоприменимых методик получения инвариантов и повторяющихся приёмов составления инвариантов в разных научных областях. На основе проведенного анализа предложена классификация инвариантов на шесть основных типов, выявлены типовые преобразования исходных данных, используемые при составлении инвариантов в нескольких областях знаний. Результаты обзора будут полезны исследователям в примененении тестирования инвариантами на практике к верификации наукоемких программ и алгоритмов машинного обучения.

Еще

Тестирование инвариантами, тестовый инвариант, тестирование программного обеспечения, проблема формулирования тестового оракула

Короткий адрес: https://sciup.org/143183243

IDR: 143183243   |   DOI: 10.25209/2079-3316-2024-15-2-37-86

Список литературы Систематический обзор методов составления тестовых инвариантов

  • Wong W. E., Debroy V., Surampudi A., Kim H., Siok M. F. Recent catastrophic accidents: Investigating how software was responsible, 2010 Fourth International Conference on Secure Software Integration and Reliability Improvement (09-11 June 2010, Singapore).– 2010.– Pp. 14–22. https://doi.org/10.1109/SSIRI.2010.38
  • Howden W. Theoretical and empirical studies of program testing // IEEE Transactions on Software Engineering.– 1978.– Vol. SE-4.– No. 4.– Pp. 293–298. https://doi.org/10.1109/TSE.1978.231514
  • Barr E. T., Harman M., McMinn P., Shahbaz M., Yoo S. The oracle problem in software testing: A survey // IEEE Transactions on Software Engineering.– 2015.– Vol. 41.– No. 5.– Pp. 507–525. https://doi.org/10.1109/TSE.2014.2372785
  • Chen T. Y., Cheung S. C., Yiu S. M. Metamorphic testing: a new approach for generating next test cases.– 2020.– 11 pp. arXivarXiv 2002.12543
  • Chen T. Y., Tse T. New visions on metamorphic testing after a quarter of a century of inception // Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.– 2021.– Pp. 1487–1490. https://doi.org/10.1145/3468264.3473136
  • Chen T.Y., Kuo F. -C., Liu H., Poon P. -L., Towey D., Tse T., Zhou Z. Q. Metamorphic testing: A review of challenges and opportunities // ACM Computing Surveys.– 2018.– Vol. 51.– No. 1.– Pp. 1–27. https://doi.org/10.1145/3143561
  • Kitchenham B. Procedures for performing systematic reviews, Keele University Technical Report TR/SE-0401.– Keele, UK: Keele University.– 2004.– 33 pp.
  • Фальковский Р.Р. Метаморфное тестирование программ улучшения изображений // XIX Международная телекоммуникационная конференция молодых ученых и студентов «МОЛОДЕЖЬ И НАУКА», Тезисы докладов.– Т. 3, М.: НИЯУ МИФИ.– 2015.– ISBN 978-5-7262-2223-3%.– С. 176–177. hUtRtpLs://lib-repository.mephi.ru/conferences_mephi/2015_MOLODEZH_I_NAUKA_CHast3.pdf
  • Миронов А. М. Верификация программ методом инвариантов // Интеллектуальные системы. Теория и приложения.– 2017.– Т. 21.– №4.– С. 31–49. MhttNp://mi.mathnet.ru/ista27
  • Núñez A., Cañizares P. C., Núñez M., Hierons R. M. TEA-Cloud: A formal framework for testing cloud computing systems // IEEE Transactions on Reliability.– 2020.– Vol. 70.– No. 1.– Pp. 261–284. https://doi.org/10.1109/TR.2020.3011512
  • Pullum L. L., Ozmen O. Early results from metamorphic testing of epidemiological models // 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom) (14-16 December 2012, Washington, DC, USA).– IEEE.– 2012.– Pp. 62–67. https://doi.org/10.1109/BioMedCom.2012.17
  • Ramanathan A., Steed C. A., Pullum L. L. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics // 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom) (14-16 December 2012, Washington, DC, USA).– IEEE.– 2012.– Pp. 68–73. https://doi.org/10.1109/BioMedCom.2012.18
  • Ellis J. D., Iqbal R., Yoshimatsu K. Verification of the neural network training process for spectrum-based chemical substructure prediction using metamorphic testing // Journal of Computational Science.– 2021.– Vol. 55.– id. 101456. https://doi.org/10.1016/j.jocs.2021.101456
  • Sun C. -A., Dai H., Geng N., Liu H., Chen T. Y., Wu P., Cai Y., Wang J. An interleaving guided metamorphic testing approach for concurrent programs // ACM Transactions on Software Engineering and Methodology.– 2023.– Vol. 33.– No. 1.– id. 8.– 21 pp. https://doi.org/10.1145/360718
  • Yoo S. Metamorphic testing of stochastic optimisation // 2010 Third International Conference on Software Testing, Verification, and Validation Workshops (06-10 April 2010, Paris, France).– IEEE.– 2010.– Pp. 192–201. https://doi.org/10.1109/ICSTW.2010.26
  • Bozic J., Wotawa F. Testing chatbots using metamorphic relations // Testing Software and Systems, 31st IFIP WG 6.1 International Conference, ICTSS 2019 (15-17 October 2019, Paris, France), Lecture Notes in Computer Science.– vol. 11812, eds. Gaston C., Kosmatov N., Le Gall P., Cham: Springer.– 2019.– ISBN 978-3-030-31279-4.– Pp. 41–55.
  • Chen T. Y., Ho J. W., Liu H., Xie X. An innovative approach for testing bioinformatics programs using metamorphic testing // BMC bioinformatics.– 2009.– Vol. 10.– id. 24.– 12 pp. https://doi.org/10.1186/1471-2105-10-24
  • Giannoulatou E., Park S. -H., Humphreys D. T., Ho J.W. Verification and validation of bioinformatics software without a gold standard: a case study of BWA and Bowtie // BMC bioinformatics.– 2014.– Vol. 15.– id. S15.– 8 pp. https://doi.org/10.1186/1471-2105-15-S16-S15
  • Tian Y., Pei K., Jana S., Ray B. DeepTest: Automated testing of deep-neuralnetwork- driven autonomous cars // Proceedings of the 40th International Conference on Software Engineering (27 May 2018-3 June 2018, Gothenburg, Sweden), New York: ACM.– 2018.– ISBN 978-1-4503-5638-1.– Pp. 303–314. https://doi.org/10.1145/3180155.3180220
  • Wu C., Sun L., Zhou Z. Q. The impact of a dot: Case studies of a noise metamorphic relation pattern // 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET) (26 May 2019, Montreal, QC, Canada).– IEEE.– 2019.– Pp. 17–23. https://doi.org/10.1109/MET.2019.00011
  • Zhou Z. Q., Sun L. Metamorphic testing of driverless cars // Communications of the ACM.– 2019.– Vol. 62.– No. 3.– Pp. 61–67. https://doi.org/10.1145/3241979
  • Nakajima S., Chen T. Y. Generating biased dataset for metamorphic testing of machine learning programs // Testing Software and Systems, 31st IFIP WG 6.1 International Conference, ICTSS 2019 (15-17 October 2019, Paris, France).– Springer.– 2019.– Pp. 56–64. https://doi.org/10.1007/978-3-030-31280-0_4
  • Wang W., Huang J. -t., Wu W., Zhang J., Huang Y., Li S., He P., Lyu M. R. Mttm: Metamorphic testing for textual content moderation software // 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE).– IEEE.– 2023.– Pp. 2387–2399. https://doi.org/10.1109/ICSE48619.2023.00200
  • Srinivasan M., Shahri M.P., Kahanda I., Kanewala U. Quality assurance of bioinformatics software: a case study of testing a biomedical text processing tool using metamorphic testing // Proceedings of the 3rd International Workshop on Metamorphic Testing (27 May 2018-03 June 2018, Gothenburg, Sweden), New York: ACM.– ISBN 978-1-4503-5729-6.– Pp. 26–33. https://doi.org/10.1145/3193977.3193981
  • Troup M., Yang A., Kamali A. H., Giannoulatou E., Chen T. Y., Ho J.W. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline // Proceedings of the 1st International Workshop on Metamorphic Testing (14-22 May 2016, Austin, Texas), New York: ACM.– 2016.– ISBN 978-1-4503-4163-9.– Pp. 33–36. https://doi.org/10.1145/2896971.2896975
  • Méndez M., Benito-Parejo M., Ibias A., Núñez M. Metamorphic testing of chess engines // Information and Software Technology.– 2023.– Vol. 162.– id. 107263. https://doi.org/10.1016/j.infsof.2023.107263
  • Zhou Z. Q., Sun L., Chen T. Y., Towey D. Metamorphic relations for enhancing system understanding and use // IEEE Transactions on Software Engineering.– 2018.– Vol. 46.– No. 10.– Pp. 1120–1154. https://doi.org/10.1109/TSE.2018.2876433
  • Zhang J., Zheng Z., Yin B., Qiu K., Liu Y. Testing graph searching based path planning algorithms by metamorphic testing // 2019 IEEE 24th Pacific Rim International Symposium on Dependable Computing (PRDC) (01-03 December 2019, Kyoto, Japan).– IEEE.– 2019.– id. 158.– 9 pp.
  • Xie X., Zhang Z., Chen T. Y., Liu Y., Poon P. -L., Xu B. METTLE: A METamorphic testing approach to assessing and validating unsupervised machine learning systems // IEEE Transactions on Reliability.– 2020.– Vol. 69.– No. 4.– Pp. 1293–1322. https://doi.org/10.1109/TR.2020.2972266
  • Guderlei R., Mayer J. Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing // Seventh International Conference on Quality Software (QSIC 2007) (11-12 October 2007, Portland, OR, USA).– IEEE.– 2007.– Pp. 404–409. https://doi.org/10.1109/QSIC.2007.4385527
  • Xiao D., Liu Z., Yuan Y., Pang Q., Wang S. Metamorphic testing of deep learning compilers // Proceedings of the ACM on Measurement and Analysis of Computing Systems.– 2022.– Vol. 6.– No. 1.– id. 15.– 28 pp. https://doi.org/10.1145/3508035
  • Sun C. -a., Wang Z., Wang G. A property-based testing framework for encryption programs // Frontiers of Computer Science.– 2014.– Vol. 8.– Pp. 478–489. https://doi.org/10.1007/s11704-014-3040-y
  • Mouha N., Raunak M. S., Kuhn D. R., Kacker R. Finding bugs in cryptographic hash function implementations // IEEE Transactions on Reliability.– 2018.– Vol. 67.– No. 3.– Pp. 870–884. https://doi.org/10.1109/TR.2018.2847247
  • Iakusheva S., Khritankov A. Composite metamorphic relations for integration testing // Proceedings of the 2022 8th International Conference on Computer Technology Applications (12-14 May 2022, Vienna, Austria), New York: ACM.– 2022.– ISBN 978-1-4503-9622-6.– Pp. 98–105. https://doi.org/10.1145/3543712.3543725
  • Iqbal M. Metamorphic testing of advanced driver-assistance systems: Implementing Euro NCAP standards on OpenStreetMap // 2023 IEEE/ACM 8th International Workshop on Metamorphic Testing (MET) (14 May 2023, Melbourne, Australia).– IEEE.– 2023.– Pp. 1–8. https://doi.org/10.1109/MET59151.2023.00008
  • Luo G., Zheng X., Liu H., Xu R., Nagumothu D., Janapareddi R., Zhuang E., Liu X. Verification of microservices using metamorphic testing // Algorithms and Architectures for Parallel Processing.– V. I, 19th International Conference, ICA3PP 2019 (9-11 December 2019, Melbourne, VIC, Australia).– Springer.– 2020.– Pp. 138–152. https://doi.org/10.1007/978-3-030-38991-8_10
  • Segura S., Parejo J. A., Troya J., Ruiz-Cortés A. Metamorphic testing of RESTful web APIs // Proceedings of the 40th International Conference on Software Engineering (27 May 2018–3 June 2018, Gothenburg, Sweden), New York: ACM.– 2018.– ISBN 978-1-4503-5638-1.– Pp. 882. https://doi.org/10.1145/3180155.3182528
  • Brown J., Zhou Z. Q., Chow Y. -W. Metamorphic testing of navigation software: A pilot study with Google Maps, University of Wollongong Research Online.– 2018.– 12 pp. hUtRtpLs://ro.uow.edu.au/cgi/viewcontent.cgi?article=2349&context=eispapers1
  • Jia M., Wang X., Xu Y., Cui Z., Xie R. Testing machine learning classifiers based on compositional metamorphic relations // International Journal of Performability Engineering.– 2020.– Vol. 16.– No. 1.– Pp. 67–77. https://doi.org/10.23940/ijpe.20.01.p8.6777
  • Saha P., Kanewala U. Fault detection effectiveness of metamorphic relations developed for testing supervised classifiers // 2019 IEEE International Conference On Artificial Intelligence Testing (AITest) (04-09 April 2019, Newark, CA, USA).– IEEE.– 2019.– Pp. 157–164.
  • Shahri M. P., Srinivasan M., Reynolds G., Bimczok D., Kahanda I., Kanewala U. Metamorphic testing for quality assurance of protein function prediction tools // 2019 IEEE International Conference On Artificial Intelligence Testing (AITest) (04-09 April 2019, Newark, CA, USA).– IEEE.– 2019.– Pp. 140–148. https://doi.org/10.1109/AITest.2019.00017
  • Dwarakanath A., Ahuja M., Sikand S., Rao R. M., Bose R. J. C., Dubash N., Podder S. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing // Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (16-21 July 2018, Amsterdam, Netherlands), New York: ACM.– 2018.– ISBN 978-1-4503-5699-2.– Pp. 118–128. https://doi.org/10.1145/3213846.3213858
  • Zhu H., Liu D., Bayley I., Harrison R., Cuzzolin F. Datamorphic testing: A methodology for testing ai applications.– 2019.– 39 pp. arXivarXiv 1912.04900
  • Zhang M., Zhang Y., Zhang L., Liu C., Khurshid S. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems // Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (3-7 September 2018, Montpellier, France), New York: ACM.– 2018.– ISBN 978-1-4503-5937-5.– Pp. 132–142. https://doi.org/10.1145/3238147.3238187
  • Raif M., Ouafiq E. -M., El Rharras A., Chehri A., Saadane R. Metamorphic testing for edge real-time face recognition and intrusion detection solution // 2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall) (26-29 September 2022, London, United Kingdom).– IEEE.– 2022.– Pp. 1–5.
  • Zhang Z., Wang P., Guo H., Wang Z., Zhou Y., Huang Z. DeepBackground: Metamorphic testing for Deep-Learning-driven image recognition systems accompanied by Background-Relevance // Information and Software Technology.– 2021.– Vol. 140.– id. 106701. https://doi.org/10.1016/j.infsof.2021.106701
  • Xu L., Towey D., French A.P., Benford S., Zhou Z. Q., Chen T. Y. Enhancing supervised classifications with metamorphic relations // Proceedings of the 3rd International Workshop on Metamorphic Testing (27 May 2018-03 June 2018, Gothenburg, Sweden).– 2018.– Pp. 46–53. UhtRtpLs://ieeexplore.ieee.org/document/8457613
  • Yan R., Wang S., Yan Y., Gao H., Yan J. Stability evaluation for text localization systems via metamorphic testing // Journal of Systems and Software.– 2021.– Vol. 181.– id. 111040. https://doi.org/10.1016/j.jss.2021.111040
  • Park H., Waseem T., Teo W. Q., Low Y. H., Lim M. K., Chong C. Y. Robustness evaluation of stacked generative adversarial networks using metamorphic testing // 2021 IEEE/ACM 6th International Workshop on Metamorphic Testing (MET) (02 June 2021, Madrid, Spain).– IEEE.– 2021.– Pp. 1–8. https://doi.org/10.1109/MET52542.2021.00008
  • Ma P., Wang S., Liu J. Metamorphic testing and certified mitigation of fairness violations in NLP models, IJCAI’20: Twenty-Ninth International Joint Conference on Artificial Intelligence (7-15 January 2021, Yokohama, Japan).– 2021.– Pp. 458–465.– id. 64.
  • Chen S., Jin S., Xie X. Validation on machine reading comprehension software without annotated labels: a property-based method // Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (23-28 August 2021, Athens, Greece), New York: ACM.– 2021.– ISBN 978-1-4503-8562-6.– Pp. 590–602. https://doi.org/10.1145/3468264.3468569
  • Sun L., Zhou Z. Q. Metamorphic testing for machine translations: MT4MT // 2018 25th Australasian Software Engineering Conference (ASWEC) (26-30 November 2018, Adelaide, SA, Australia).– IEEE.– 2018.– Pp. 96–100. https://doi.org/10.1109/ASWEC.2018.00021
  • Gao W., He J., Pham V. -T. Metamorphic testing of machine translation models using back translation // 2023 IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) (15 May 2023, Melbourne, Australia).– IEEE.– 2023.– Pp. 1–8. https://doi.org/10.1109/DeepTest59248.2023.00008
  • Pesu D., Zhou Z. Q., Zhen J., Towey D. A Monte Carlo method for metamorphic testing of machine translation services // Proceedings of the 3rd International Workshop on Metamorphic Testing (27 May 2018, Gothenburg, Sweden), New York: ACM.– 2018.– ISBN 978-1-4503-5729-6.– Pp. 38–45. https://doi.org/10.1145/3193977.3193980
  • Sun Y., Ding Z., Huang H., Zou S., Jiang M. Metamorphic testing of relation extraction models // Algorithms.– 2023.– Vol. 16.– No. 2.– Pp. 102. https://doi.org/10.3390/a16020102
  • Raunak M. S., Olsen M.M. Metamorphic testing on the continuum of verification and validation of simulation models // 2021 IEEE/ACM 6th InternationalWorkshop on Metamorphic Testing (MET) (02 June 2021, Madrid, Spain).– IEEE.– 2021.– Pp. 47–52. https://doi.org/10.1109/MET52542.2021.00015
  • Donaldson A. F., Lascu A. Metamorphic testing for (graphics) compilers // Proceedings of the 1st International Workshop on Metamorphic Testing (14-22 May 2016, Austin, Texas), New York: ACM.– 2016.– ISBN 978-1-4503-4163-9.– Pp. 44–47. https://doi.org/10.1145/2896971.2896978
  • Le V., Afshari M., Su Z. Compiler validation via equivalence modulo inputs // ACM Sigplan Notices.– 2014.– Vol. 49.– No. 6.– Pp. 216–226. https://doi.org/10.1145/2666356.2594334
  • Zhou Z. Q., Tse T., Witheridge M. Metamorphic robustness testing: Exposing hidden defects in citation statistics and journal impact factors // IEEE Transactions on Software Engineering.– 2019.– Vol. 47.– No. 6.– Pp. 1164–1183. https://doi.org/10.1109/TSE.2019.2915065
  • Ahlgren J., Berezin M., Bojarczuk K., Dulskyte E., Dvortsova I., George J., Gucevska N., Harman M., Lomeli M., Meijer E., Sapora S. Testing web enabled simulation at scale using metamorphic testing // 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (25-28 May 2021, Madrid, Spain).– IEEE.– 2021.– ISBN 978-1-6654-3869-8.– Pp. 140–149. https://doi.org/10.1109/ICSE-SEIP52600.2021.00023
  • Rehman F. u., Izurieta C. Statistical metamorphic testing of neural network based intrusion detection systems // 2021 IEEE International Conference on Cyber Security and Resilience (CSR) (26-28 July 2021, Rhodes, Greece).– IEEE.– 2021.– Pp. 20–26. https://doi.org/10.1109/CSR51186.2021.9527993
  • Tambon F., Antoniol G., Khomh F. HOMRS: High order metamorphic relations selector for deep neural networks.– 2021.– 33 pp.
  • Zhang P., Zhou X., Pelliccione P., Leung H. RBF-MLMR: A multi-label metamorphic relation prediction approach using RBF neural network // IEEE Access.– 2017.– Vol. 5.– Pp. 21791–21805. https://doi.org/10.1109/ACCESS.2017.2758790
  • Zhang J., Chen J., Hao D., Xiong Y., Xie B., Zhang L., Mei H. Search-based inference of polynomial metamorphic relations // ASE ’14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (15-19 September 2014, Vasteras, Sweden), New York: ACM.– 2014.– ISBN 978-1-4503-3013-8.– Pp. 701–712. https://doi.org/10.1145/2642937.2642994
  • Sun C. -A., Fu A., Poon P. -L., Xie X., Liu H., Chen T. Y. Metric ++: A metamorphic relation identification technique based on input plus output domains // IEEE Transactions on Software Engineering.– 2019.– Vol. 47.– No. 9.– Pp. 1764–1785. https://doi.org/10.1109/TSE.2019.2934848
  • Kanewala U., Bieman J. M., Ben-Hur A. Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels // Software: Testing, Verification and Reliability.– 2016.– Vol. 26.– No. 3.– Pp. 245–269. https://doi.org/10.1002/stvr.1594
  • Blasi A., Gorla A., Ernst M. D., M. Pezzè, Carzaniga A. MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation // Journal of Systems and Software.– 2021.– Vol. 181.– id. 111041. https://doi.org/10.1016/j.jss.2021.111041
  • Wu P. Iterative metamorphic testing // 29th Annual International Computer Software and Applications Conference (COMPSAC’05).– V. 2 (26-28 July 2005, Edinburgh, UK).– IEEE.– 2005.– ISBN 0-7695-2413-3.– Pp. 19–24. https://doi.org/10.1109/COMPSAC.2005.166
  • Yang Y., Li Z., Wang H., Xu C., Ma X. Towards effective metamorphic testing by algorithm stability for linear classification programs // Journal of Systems and Software.– 2021.– Vol. 180.– id. 111012. https://doi.org/10.1016/j.jss.2021.111012
  • Xie X., Wong W. E., Chen T. Y., Xu B. Metamorphic slice: An application in spectrum-based fault localization // Information and Software Technology.– 2013.– Vol. 55.– No. 5.– Pp. 866–879. https://doi.org/10.1016/j.infsof.2012.08.008
  • Liu H., Liu X., Chen T. Y. A new method for constructing metamorphic relations // 2012 12th International Conference on Quality Software (27-29 August 2012, Xi’an, China).– IEEE.– 2012.– Pp. 59–68. https://doi.org/10.1109/QSIC.2012.10
  • Qiu K., Zheng Z., Chen T. Y., Poon P. -L. Theoretical and empirical analyses of the effectiveness of metamorphic relation composition // IEEE Transactions on software engineering.– 2020.– Vol. 48.– No. 3.– Pp. 1001–1017. https://doi.org/10.1109/TSE.2020.3009698
  • Hui Z. -W., Huang S., Chua C., Chen T. Y. Semiautomated metamorphic testing approach for geographic information systems: An empirical study // IEEE Transactions on Reliability.– 2019.– Vol. 69.– No. 2.– Pp. 657–673. https://doi.org/10.1109/TR.2019.2931561
  • Iakusheva S., Khritankov A. Metamorphic testing for recommender systems // Analysis of Images, Social Networks and Texts, AIST 2023, Lecture Notes in Computer Science.– vol. 14486, eds. Ignatov D.I. et al., Cham: Springer.– 2024.– ISBN 978-3-031-54533-7. https://doi.org/10.1007/978-3-031-54534-4_20
  • Hui Z. -w., Wang X., Huang S., Yang S. MT-ART: A test case generation method based on adaptive random testing and metamorphic relation // IEEE Transactions on Reliability.– 2021.– Vol. 70.– No. 4.– Pp. 1397–1421.
  • Sobania D., Briesch M., Röchner P., Rothlauf F. MTGP: Combining metamorphic testing and genetic programming, Genetic Programming. EuroGP 2023, Lecture Notes in Computer Science.– vol. 13986, eds. Pappa G., Giacobini M., Vasicek Z., Cham: Springer.– 2023.– ISBN 978-3-031-29572-0.– Pp. 324–338.
Еще
Статья обзорная
QR-код для установки приложения SciUp Наведите камеру и скачайте бесплатное приложение SciUp