Обзор результатов международного экзамена IELTS в техническом университете за восьмилетний период

In the global higher educational landscape, the issues concerning English proficiency are being widely discussed as the language of instruction in many universities worldwide is English. It imposes certain language demands on non-native English speakers majoring in their disciplines in the EMI (English as a medium of instruction) environment. The academic performance expressed as Grade Point Average (GPA) is in most cases higher if students have B2 or higher level of English proficiency [9, 18] as compared with intermediate students. Though this correlation might be weak for higher levels, for students with intermediate English it can become a major hindrance in their studies [5, 7, 16].

There are two internationally recognized tools to measure students’ proficiency in academic English and their ability to study in an EMI university, which are the Test of English as a Foreign Language (TOEFL) and the International English Language Testing System (IELTS). IELTS is taken by people who aim at working or studying abroad or in an English-speaking environment at home. It was designed by the examination committee of the University of Cambridge in cooperation with the British Council and IDP Education Australia, and currently totals more than 3.5 million test takers in the world (IELTS Guide for teachers, 2022). In anglophone universities IELTS plays the role of a gatekeeper, letting in only entrants with a suitable level of English. Although still debatable, there seems to be a consensus view that Band 6.0 (B2) is the minimal requirement, enabling students to study and graduate in the chosen field successfully.

The situation in our country differs from that in Australia, the UK, the USA, the UAE, New Zealand and even some former Soviet republics such as Kazakhstan [1], Uzbekistan or Ukraine as universities here operate in the Russian-speaking environment (Russia was not even represented in the 2014 British Council survey of countries where EMI was reported as allowed) [6]. In view of this, though IELTS used to be the most popular exam in academic English in Russia until recently, its role in students’ motivation to master English is not so obvious.

This research is conducted in the National University of Science and Technology MISiS (MISiS), which is one of the leading higher educational institutions in the fields of material science, nanotechnology mining, and metallurgy in Russia. The university had been the part of the Competitiveness Enhancement Program 5-100 and had an opportunity to become a part of international educational environment. The state project was aimed at the internalization of the education in the Russian Federation, so that Russian universities will be able to become top world institutions, attracting students, scientists and academics from other countries, producing high quality research and demonstrating good standards in teaching.

MISiS is a technical university, so our applicants do not have to submit the results of a state exam in English and are not selected by the language proficiency. Yet, the involvement into the Competitiveness Enhancement Program 5-100 led the university to revise many aspects of teaching, and the English language program in particular. Now the course is unified across all specialist departments, covers seven semesters and ends with a standardized international exam IELTS.

The article presents the experience of the Department of Modern Languages and Communication (IYAKT) with the introduction of the compulsory IELTS exam for all undergraduates over the period from 2014 to 2021. As all MISiS students regardless of their major have to take IELTS at the end of a 4-year course, our department receives a unique tool to assess the efficiency (KPI) of the English language program.

Having such a large dataset, we were able to analyze the results with respect to objective factors such as the entry level of English, and other impacts including the volatility of the student population and their motivation to have an internationally recognized certificate in English proficiency.

Literature Review

The International English Language Testing System (IELTS) is an English language proficiency test that serves as a gatekeeping mechanism for students to enter an international university in many countries where English is used as the language of instruction. Petrishcheva [14] writes about international students from China, Vietnam, Japan, Portugal and Russia who came to New Zealand to continue their tertiary studies, Thorpe et al [18] explore the wide geography of students in UK universities, which comprises Africa, Americas, East, West and South Asia, and Europe.

University departments set a certain proficiency level at which the risk of failure is minimized, and in this regard much research deals with the predictive validity of IELTS for the future academic success expressed in GPA [7, 9, 16, 18].

There are no set requirements for the acceptable level expressed in IELTS bands, and each institution makes its own decision about admission expectations. However, higher IELTS bands (7.5 or higher) are generally recommended for the ‘linguistically more demanding’ courses, such as business, law, arts, humanities, and technical disciplines, such as engineering, are considered ‘less demanding’ with Band 6.5 as a pass. Lower entry IELTS score (5,0 or lower) might lead to a poor academic record in an EMI university even in a non-English-speaking country [16].

In Russia the incentive for students to study English and to pass IELTS successfully is more ambiguous as the language of instruction in the overwhelming majority of non-linguistic universities remains Russian. In the article on needs analysis conducted at MISiS in 2015–2016, we note that in some specialties students can get a degree without ever using English in their studies or work [15]. Having said this, most of students admitted using English on a regular basis for reading research papers and listening to lectures and tutorials online.

Meanwhile, the role of the English language in the Russian educational landscape has increased dramatically since the Russian government announced the course to the internationalization of university education while bolstering their positions in the global education market. First, The Russian Academic Excellence Project launched in 2013 gathered twenty-one top Russian universities, then in 2021 the list was expanded to 43 participating members under the federal academic leadership program Priority 2030. One of the stated aims of both projects is ‘to become the leaders in creating new scientific knowledge, technology, and developments for introduction into the economy and social sphere in Russia’.

After 2013 all top universities introduced master’s degree programs in English. For example, in 2017 MISiS announced nine EMI postgraduate courses in materials science, engineering materials, nanotechnology and public relations (MISiS). Many universities offered double degree master programs, predominately with European and Asian universities ( www.hse.ru ; www.dvfu.ru ; https://itmo.ru ; www.tpu.ru ; www.etu.ru ). The essential prerequisite for entering international programs is a certain level of language proficiency, and the most acceptable certificate is IELTS score 5.5 or higher.

On a brief examination of the 5\100 members websites, one can find information about six international programs in collaboration with Finland and Germany in ITMO university, or several double degree projects with leading French engineering schools ParisTech in Novosibirsk State University ( www.nsu.ru ).

Undergraduate programs in EMI are much less common in the Russian Federation and very small cohorts of bachelors have taken part so far. In this regard we can note London Educational Project, which Financial University in Moscow implemented jointly with the London School of Economics [10], and three double degree programs realized in Higher School of Economics together with London and Seoul universities. The language requirements in both courses are IELTS Bands 6.0 or higher.

Despite the fact that IELTS is a gatekeeper for many international educational projects, to the best of our knowledge only two tertiary institutions in Russia make IELTS test compulsory for all undergraduates (HSE). In other universities those students who need the exam certificate either attend fee-paying extracurricular courses, or prepare by themselves. The two universities where IELTS is taken by all bachelor’s degree students are Higher School of Economics and MISiS. The declared aim of both English language programs is achieving B2 level CEFR, or Band 5.5 or higher IELTS.

In Higher School of Economics (HSE) the exam is taken in the second year of studies, in our university IELTS culminates a 4-year long course of English as a final external standardized control. According to the publication of Baidel-dinova [2], the results in HSE and Financial Academy are high, with around 90% of testtakers achieving at least Band 5.5, but some issues need clarification. The initial levels of students who study in the program are not specified, but it may affect the score dramatically; what is more, the results are not presented in dynamics, which can lead to overlooking some important trends.

It would be interesting to examine the issue of motivation as in Russia the relationship between effective language skills and academic achievement or career prospects is not always obvious. Motivation is widely acknowledged as a significant factor in learning, alongside with hard work, effective study strategies and supportive environment [8, 20]. Researchers distinguish two motivational orientations, namely intrinsic and extrinsic. The intrinsic motivation refers to doing something because of the satisfaction derived from the process, while the extrinsic motivation refers to doing something in order to get external outcomes [4, 17].

Many students derive enjoyment from the process of learning English, the knowledge they acquire or the sense of accomplishment, that is they have the intrinsic motivation to advance [4, 17]. External regulation comes from external rewards or punishments and is displayed when a student sees the tasks as a way to the goal to be reached. It seems obvious that the necessity to ‘have one eye on the grade’ in the view of increasing competition for jobs after graduation remains one of the main incentives [19]. Such categories as finance, career, job or educational program were distinguished as codes by Kember [13].

If we accept the view of a tertiary student as a ‘conscientious consumer’ who weighs the benefits and efforts [11], they have a good idea of what they are capable of achieving without jeo- pardizing other things. The final IELTS exam can be considered as such a goal, and the motivation to succeed in it is a significant factor affecting the results.

In this article the authors present the reflection on a longitudinal study of the results of IELTS taken at the end of the English course by all bachelor’s degree students at a technical university. The proportion of B2/C1 graduates transcribed in IELTS bands serves as a clear-cut KPI for the English course. The research investigates the processes underlying the final achievements in the English language course with the view of enabling course designers to set feasible goals and considering possible ways to improve the results.

Methodology

Participants

The data were obtained from the entire cohorts of students who entered the bachelor’s program in the years from 2011 to 2018. All entrants were streamed into four trajectories, A1, A2, B1 and B1+/B2, and they moved along their trajectory during seven semesters, or 600 contact hours, of English studies.

According to CEFR Guided Learning Hours [3], this number is sufficient for a beginner to achieve B2. All MISiS institutes, College of Engineering, College of New Materials, College of Computer Technologies and College of Economics, have the same English language program, which ends with the final IELTS exam.

IELTS procedure

In 2014 and 2015 all bachelors regardless of their English achievements were examined by an official test center BKC and, consequently, received certificates with IELTS bands ranging from 2.0 to 7.0. Achieving of B2 level (Band 5.5) was stated by the department as a compulsory requirement of the English program, and underachievers had to go through retake procedures.

Starting from 2016 and up to 2021 students with preliminary scores 5.0 and lower were excluded from the certified exam and took a mock IELTS exam administered completely by the IYAKT teachers. In 2017 the status of IELTS was downgraded to a certain number of points in the score-rating system, so a student could be attested without achieving B2.

The other change in the IELTS procedure occurred in 2020 and 2021 due to COVID-19 pandemic restrictions, when all classes and, more importantly, exams, were taken online. Those candidates who were selected for a certified exam did a computer-based test in the premises of IELTS Students International, while the others took a test from home on the university platform Canvas. Needless to say, the results from a certified center were credible as they eliminated cheating, whereas the online in-house test could not guarantee complete integrity.

Quantitative analysis

For statistical analysis of the data, we used a free open-source program JASP. The strength of the relationship between the placement test and IELTS results by years was measured by means of the correlation analysis. Both data sets were not normally distributed, so we used Spearman’s rank correlation coefficient rho. The p -value was used in order to quantify the statistical significance of a claim, with p ≤ 0,05 regarded as statisically significant. The same indicators were calculated for the number of students who missed the final exam and its overall results.

Mean values for different trajectories were compared using descriptive statistics methods, and the paired samples t-test allowed us to prove that there are statistically significant differences between the final IELTS results for trajectories A1, A2, B1 and B2. The test of normality demonstrated that the difference scores were not normally distributed, so the Wilcoxon signed-rank test was applied, with p ≤ 0,05 signified that the differences were reliable.

Qualitative data

The factors which affected students’ motivation to take IELTS or the reasons to abstain from it were examined with questionnaires. High achievers, who took the certified exam in IELTS Students International test center, were asked by their English teachers to answer a few questions:

Why did you take a certified IELTS?

How are you going to use your certificate?

We received 72 responses out of 281 (25%), in which we identified the categories and counted the number of voices in each.

In the group of underachievers, who failed to get Band 5.5, 167 students did not attend any part of the exam, and 29 skipped the oral part. It turned out that 41 were factually expelled or on the academic leave, but still were registered for IELTS on paper. A small proportion from this group were attested in the semester despite IELTS failure, and the others had to take IELTS on a MISiS electronic platform.

The tests opened only upon completion of a short questionnaire, and this way we obtained

69 answers to open and closed questions. The ones to be discussed further are as follows:

Why did you fail to attend the exam?

How do you assess your chanced to get a passing score now?

Results

IELTS against placement test

The situation at our university was unique for Russia, as all undergraduates in a technical university had to take a certified IELTS exam at the end of the English language course. As there were no ‘shoulders to stand on’ and to base predictions on the proportion of graduate with B2 or C1, at first we were guided by CEFR, which states that 600 contact hours is sufficient for an A1 to get to B2 (Cambridge English Support Site, 2019). Consequently, in the ideal conditions we could expect at least up to 80 percent of upper-intermediate graduates. The actual results of IELTS for all years are shown in Fig. 1. As can be seen, the proportion of B2/C1 students had never been higher than 50%, with the best score in 2019 of 49,2% and the lowest number in 2014 of just 28%.

From 2014 to 2019 the proportion of B2/C1 graduates gradually increased alongside with the betterment of student intake. The placement test graph shows the percentage of entrants with B1 and B2 levels by year.

If for the 2014–2016 cohort the overwhelming majority of entrants were beginners or elementary (71 and 66%, respectively), for 2020 and 2021 graduates only around half were A1/A2 at the start of the program. A noticeable decline is observed in the COVID years, when the figures were no more than 45% in 2020 and 38% in 20211. The trend looks like a pullback to the early years of the program, when we taught much less proficient students, and the teachers at IYAKT were less experienced in IELTS training. Whatever the reasons, it can be clearly seen that in the COVID pandemic years, with learning online and taking an exam online, fewer opportunities for academic mobility and employment abroad, the efficiency of the program suffered significantly. At the same time, the proportion of high-achievers (C1) rose throughout the whole period, allegedly because of the increasing number of B1+/B2 students at the starting point.

Fig. 1. IELTS against Placement Test over the period from 2014 to 2021

Table 1

Correlation between final IELTS and placement test results

Variable		Placement	B2/C1 in 2014–2021	B2/C1 in 2014–2019
Placement	Spearman’s rho	–
	p-value	–
B2/C1 in 2014–2019	Spearman’s rho	0.893	1.000	–
	p-value	0.012	< 0.001	–
B2/C1 in 2014–2021	Spearman’s rho	0.717	–
	p-value	0.037	–

Visually there is a strong correlation between the proportion of upper-intermediate and advanced students at the end and at the start of the English course. This trend persisted from 2014 to 2019, but then the results deteriorated despite the rising trend for the placement test.

In order to check the hypothesis that the correlation is strong, we calculated Spearman’s coefficient for nonparametric data. Table 1 shows that there is a strong correlation for 2014–2019 (ρ = 0,893; p-value 0,012), but the last two years disrupted the linear trend and resulted in a lower coefficient (ρ = 0,717; p-value 0,037).

The conclusion here is that in the current situation, with existing students’ intake and administrative procedures, the realistic KPI for the final IELTS exam is just 50%. The first explanation might be derived from looking into a ‘grey zone’, namely the population of students who simply skipped the exam. For traditional paper IELTS, which the department organized in 2014–2019, sitting an exam meant coming to exam premises at the appointed time. In COVID-19 years the exam performed on Canvas LMS required even much less effort as it could be done from any place with the Internet access.

However, Table 2 demonstrates that not all students managed to take the exam, with onefifth on average skipping it. These absentees, who amounted in some years to a quarter or more (Table 2), is a reason for lower than theoretically possible KPI. However, if we compare results by year, there is no statistically significant correlation between IELTS score and absentee percentage.

Trajectories for each starting level

IELTS as a standardized objective tool for assessing individual achievements opens many doors for researchers. For instance, we were able to trace each student from the placement test to his or her final IELTS and draw trajectories for each CEFR entrance level. Two sets of trajectories for a year with low final IELTS scores (2015) and the most successful year (2019) are shown in Fig. 2 and Fig. 3.

Table 2

Correlation between the number of students who did not appear in the exam and the final IELTS results

Year	2014	2015	2016	2017	2018	2019	2020	2021
Absentees,%	112.7	15.0	9.5	16.0	15.8	21.1	16.4	25.6
IELTS B2/C1,%	28.0	35.0	34.0	43.7	45.8	49.2	45.0	38.0
Spearman’s rho	0.433
p-value	0.250

Fig. 2. Trajectories for students who started the program in 2012 at A1, A2, B1 or B2, and completed in 2015

Fig. 3. Trajectories for students who started the program in 2016 at A1, A2, B1 or B2, and completed in 2019

In 2015 low level students basically failed to achieve B2, or Band 5,5 IELTS, and the peaks were at Band 4,0 for both A1 and A2. The difference between the two levels was that some students who started at A2 were able to achieve B2, but A1 starters generally finished with elementary level bands. Those who already had B1+ or B2 at the start typically achieved C1, and the starters with B1 demonstrated the whole array of final scores, but with a peak at Band 6.0.

In the 2019 graph the peaks for all levels except A1 shift towards higher bands. The most perspective trajectory seems to be A2 as it is feasible for the majority to achieve Band 5,5 or even higher. Unfortunately, 2019 saw a large percentage of absentees, and that fact pulled down the results for all trajectories.

Overall, students who started with B1 and B2 could all achieve the target level, provided they did not miss the whole exam or a speaking part. The area of concern is beginners, who struggle to move beyond the intermediate level. The most promising cohort is A2 starters, who demonstrated the potential for achieving the target level and beyond.

Despite the fact that IELTS results fluctuate for each trajectory, the mean values have a rising trend (Table 3). According to t-test, the differences between trajectories are significant with a high probability value of p = 0,008 (Table 4).

Table 3

B2/C1 achievers (%) in each trajectory

Trajectory	Mean	SD
A1	11.59	6.18
A2	37.21	13.51
B1	66.65	12.39
B2	81.27	14.07

Note. N = 8 years.

Table 4

Paired samples t-test

Trajectory 1	Trajectory 2	W	p
A1	A2	0.000	0.008
A2	B1	0.000	0.008
B1	B2	0.000	0.008

Insight into the program efficiencyin a COVID year

The standard external international exam IELTS is a unique objective instrument for assessing the program KPI. According to it, the program efficiency in 2020 and 2021 declined as compared to the previous years (45% and 38% B2/C1, respectively). Although there might be various reasons for that, the main change in COVID period was online format of studies. 2021 graduates spent two out of four years in front of their computers, without live interaction, and they took IELTS in an online format too. The diagram in Fig. 1 is a vivid proof that face-to-face classes are more efficient.

In order to understand what happened in 2021, especially in view of high starting results in 2018 (51% B1/B2), we conducted two surveys: with high achievers, who took a certified exam, and underachievers, who came to the retake in order to be attested.

Seventy-two students (25%) from the first group completed a questionnaire specifying the reasons for taking IELTS. The typical responses to the question “Do you think you can use the certificate soon?” were as follows:

• •    In the current situation, no. It's hard to get a visa in any desirable country now. Maybe few months later it will change. Although that's a good proof of my English level to include in my resume.

• •    It is difficult to answer, but the certificate will not be superfluous.

• •    I suppose yes. I have plans to work in a country where it is quoted.

• •    Unfortunately, in the current situation I doubt it, but the experience was interesting anyway.

• •    I very much hope so.

The quotes above imply that IELTS certificates had rather a potential than practical value for their holders, and nobody was going to apply them just after graduation.

However, 281 students took and passed the exam successfully, and it is unlikely that they were motivated only by marks in their diploma. Fig. 4 represents responses to the question, “Why did you take IELTS?”

Although ‘getting a certificate’ was the most popular answer, this response does not look like a strong incentive, but more like ‘just in case’. Surprisingly, only two students reported ‘mark’ as the only reason, while others outlined factors commonly associated with the intrinsic motivation:

• •    I took IELTS in order to prove myself that I studied English for four years not in vain.

• •    I was interested to check my real knowledge and know my real level of proficiency.

• •    For the sake of experience.

  

  Responses,%

  Getting a certificate Getting experience Getting a good mark Checking knowledge Getting a bonus for master's programs abroad Getting a bonus for master's programs in Russia

  Sprucing up a in a resume Might be useful somehow

  
  

  Fig. 4. Incentives to take IELTS in 2021

  
  
  

Responses, %

Fig. 5. The reasons for not attending the exam

In the second group, hereafter called ‘underachievers’, 155 students failed to attend the whole exam or an oral part. The department ceased the opportunity to question them. Before getting access to IELTS retake materials on Canvas electronic platform, each student had to complete a short compulsory questionnaire on the same platform. We have collected and analyzed 69 answers, which are grouped into six categories in Fig 5.

Low English proficiency is cited as a reason by 26% of respondents:

• •    The reason is my individual poor preparation during the course, lack of knowledge.

• •    I blame lack of self-preparation and skipping many English classes, lack of time for preparing.

• •    I realized that I do not have enough points for a satisfactory mark anyway. Now I understand that it was worth trying to pass IELTS.

Among other reasons a popular one was work: ‘I have to work on Saturdays and thus did not have an opportunity to come’, ‘The time for the exam was inconvenient because of my work’.

Some respondents were quite critical of themselves: ‘I have a bad habit of doing everything at the last moment. I have learned a lesson thanks to IYAKT department’.

However, ‘lack of knowledge’ did not prevent students from feeling confident about an online retake. Thus, 25% assessed their chances as 100%; 59% as 80%, 30% as 50%, and nobody thought that their chances were lower than 50%. Such optimism can be caused by different opportunities for cheating while doing a test online.

The findings presented in this section indicate that ‘unsuccessful’ students did not have the intrinsic motivation to succeed in IELTS, and this fact can also be the cause of poor results in 2021.

Conclusion and discussion . The article describes unconventional experience in large-scale assessing the English program efficiency. The measuring tool was a standardized external exam IELTS, the participants were all bachelor degree students, and the duration of the observations was eight consecutive years. Conditions for admission, as in any other technical university in Russia, did not involve English proficiency, and applicants ranged from A1 to B2.

The main result of this study is that the target set by MISiS at IELTS Band 5,5 was achieved only by half of graduates at best, and higher proportion of upper-intermediate levels seems unrealistic under the present conditions. Much research on a link between language proficiency and academic success claim that a minimum prerequisite for studying in EMI is Band 6,0 [5, 9, 16, 18]. Consequently, under the existing state of things only 50% of graduates have the potential ability to participate in academic program in EMI. However, even a 50% of B2/C1 graduates can be considered as success, given the abundance of low levels upon admission.

The proportion of B2/C1 levels increased constantly from 28% in 2014 to 49,2% in 2019, in accordance with the rise in the entrance proficiency. There is a strong correlation between IELTS bands and placement test results (ρ = 0,893) in pre-COVID years. There should be other factors contributing to this relative success, for instance, better teaching, as IYAKT department conducted a series of training with external experts on IELTS teaching and assessment. The teachers from the department also regularly participate in in-house sessions, do team teaching and peer observations to improve their understanding of IELTS, but the effect of teaching is not in the scope of this paper.

In the last two years IELTS results went down, and this trend can be a proof that the dis- tant format affects the studies and motivation in a negative way.

The trajectories drawn for every level CEFR support the fundamental role of secondary school language education in progressing further at university. Thus, students who arrive with A1, rarely achieve B2 (only 11,6 % on average), and almost never C1; students with A2 with proper teaching and motivation were able to achieve B2 (54%) and even C1 in best years (3%). This category of learners should be the focus of close attention in the future. Students who arrived with B1 or B2 generally did not struggle with IELTS. The defining role of school versus pre-sessional university courses was also reported by Thorpe et al [18].

Surprisingly, students who did well at school and entered with B2 failed to demonstrate 100% of B2/C1, though many did achieve C1 (47% in 2019). One reason for that might be that IELTS format is different from the State Exam in English [2, 12]. IELTS tests academic English, and as such requires the ability to read academic texts, listen to subject lecturers and, what is more, writing academic texts in a limited time-frame. Students who skip classes and home assignments generally demonstrate results lower than their actual score in General English.

Speaking about absenteeism, this is a significant issue at a technical university. Although this research indicates that the proportion of students who did not show up for the exam did not correlate directly with total IELTS results for each year, 15 or even 25% of absentees definitely worsen statistics. This category remains a ‘grey zone’ for the department due to practical impossibility to trace them accurately.

Our university is not a closed system, where we assess exactly the same population as we started to teach. Some students are transferred from other institutions in the middle of the course, others are expelled or take an academic leave on the way, some continue studying after expulsion. These fluctuations, treated by the university as necessary evil, nonetheless affect the program KPI.

Finally, an important role in a non-English speaking university with limited opportunities for academic mobility is played by learners’ motivation to study English profoundly. Unlike the situation in universities with EMI environment [9], in MISiS English is just a subject in a curricular rather than a vital need.

Surveys revealed that while good students find intrinsic motivational factors to take IELTS, for example, to get an objective measure of their achievements, or to have a nice line in a resume; underachievers do not even appear on the exams, giving various pretexts. IYAKT department constantly makes efforts to increase the intrinsic motivation by making an English class an attractive place to visit with communicative activities, media resources and all-department events. These drivers work to a certain extent, but they are not a universal solution.

A considerable decline in the proportion of B2/C1 graduates in COVID years may indicate lower efficiency of online studies as compared to the face-to-face format. Students’ answers to questionnaires also demonstrated that the overall motivation to study English decreased in these years.

The article gives a snapshot of the current situation of applying IELTS on a large scale at a technical university, and does not offer quick solutions on how to achieve higher than 50% B2/C1 results. However, reflection on this experience may provide teachers and course designers with new ideas on revision of English language programs.