A meta-analysis on educational technology in English language teaching

Abstract

As more various types of computer-assisted language learning (CALL) programs have been incorporated into language classrooms over the recent decades, it has become more important to uncover whether, to what extent, and under which moderator variables CALL can be yield more effective outcomes than traditional language instruction. The issue of education is one of the most important materials addressed by technology. Instead, meta-analysis is a statistical and quantitative method that leads us to a general conclusion by integrating the results of different researches. In this study, researchers worked on the impact of educational technology in English language teaching by studying 67 articles and theses (from 1000 studies that were relevant in title and abstract). All articles and theses were included from 2009 to 2020 and 7 articles were excluded from this study due to insufficient information. Furthermore, two instruments, SPSS (mainly its sub-branch Kruskal-Wallis test) and CMA were used to calculate and evaluate data in this research. The total effect size calculated for studies under both fixed and random models was statistically significant and also the study of effects by year of publication, instruments used in research and research methods showed that their effect size was significant. Teaching English with the help of technology has an effective effect size and has shown the success of this technology in language learning.

Introduction

Due to the rapid advances in Information and Communication Technology (ICT) in the world, there is growing attention to combine technologies into the classrooms to prepare learners to meet the needs of a progressively technological-dependent culture (Bond). The presence of technology and its constant advances have been disclosed into society by shifting the way how people cooperate with technology itself and through technology devices (Hollands & Escueta, 2020; Gonzalez-Acevedo, 2016). Warni, Aziz, and Febriawan (2018) believe that technology allows students to study independently and cooperate with their peers. This is possible because technology inspires students to reflect and analyze where these two capabilities are at the basics of developing autonomy. Since the 1960s, educational technologists have tried to make this image become a reality through emerging programs based on computer-assisted instruction (CAI) to drill, train, and test students (Andone & Frydenberg, 2019). According to Xiao (2019), every educator must utterly think about, update concepts, be courageous in innovation, let advanced science and technology assist college English education, and familiarize multimedia technology with a large amount of information, closeness and interactivity into college English teaching. Many educational researchers believe that computer-assisted language learning (CALL) would prove to be effective because it would decrease educational costs and increase learning outcomes in the long period (Atabek, 2020; Oz, Demirezen, & Pourfeiz, 2015).

Research in English language teaching sometimes contradicts differences in educational measures, situations, measurement instruments, and research methods that make it difficult for the researcher to easily compare the findings (Ozkale & Koc, 2020).

The disagreement between the results means that there is no acceptable answer to guide policymakers, and there is always an endless demand for re-research. There is a danger that the sponsors of social and educational research would conclude that this research is unproductive and unscientific. In addition, reviewing the sources of empirical research is usually not helpful, and because it depends so much on the opinion, judgment, preferences, and tendencies of the reviewers, conflicting interpretations of a piece of evidence are not uncommon.

However, examining the effectiveness of CALL is not easy for a number of reasons. First, the effect of any CALL program on learning outcomes is some way related to its uses. A specific CALL program may have great educational potential not revealed until it is used properly. Hence, evaluating the effectiveness of the CALL in language education is evaluating its uses rather than the CALL program itself. Second, the effectiveness of CALL is affected by some other moderator variables such as the learners, tasks, the educational setting conditions, and the assessment instruments. Third, CALL can be used either in isolation, as the sole instrument to convey language knowledge to the students, or in combination with traditional, face-to-face teaching methods (Sadeghi & Dousti, 2013). In addition to the above obstacles, no individual study by itself can show whether CALL programs are actually effective or not. In most countries, the use of educational technology is a headway and a national movement, and many organizations and educational institutions have been created in order to properly use educational technology and find better and more complete ways and techniques.

General objectives of the article: The role of educational technology in teaching English in Iran

Literature review

Educational technology

An accurate understanding of the definition, subject, and scope of educational technology depends largely on the root meaning of the word technology. The word is derived from the word technology in Greek, meaning systematically performing art or profession. The first part of this word (technologia) is a combination of performing art and a technique involving knowledge of the meaning of the principles and the ability to achieve the desired results. In other words, logos mean practical things like knowing and doing. The word root means reasoning, explanation, principle, and ratiocination. So technology means the rational application of knowledge. The word consists of two words “technique” and “logic.” “Technique” means skillfully doing any work and “logic” is equivalent to the “knowing” suffix and means “science and knowledge.” Technology can, therefore, be regarded as methodology or knowledge and science in subtle ways of doing things. The second meaning is what the word “technology” is mostly used to express (Faradanesh, 2001).

Concept of educational technology

There are many definitions of educational technology, each referring to its various aspects. Before the application of technology in its new sense, planners helped improve the teaching and learning outcomes of audiovisual cases and devices. Thus, it can be concluded that the contributions of this branch are summarized from education to the use of purely educational items. But Brown (1972) has defined educational technology differently: Educational technology goes beyond the use of instruments. Educational technology is thus more than just a set of components (Ipek & Ziatdinov, 2018). It is a systematic way of designing, executing, and evaluating the whole process of research and learning to use specific goals, utilizing research findings in psychology and human communication, and employing a combination of human and non-human resources to create more effective learning, more reliable, and more deeply. In-depth attention to the above definition leads the reader to several basic conclusions:

• 1. The first conclusion to be drawn from the first part of the definition is that educational technology is not just about the use of educational instruments, but the broader scope of the use of educational instruments and the use of educational materials as part of it.

• 2. As educational technology is considered to be a systematic way or method, so it is more like an empire than its constituent parts because they are actions and reactions. Because the action, reaction, or interaction between its constituents lead to effects and results that are greater than the sum of its constituent elements.

• 3. Educational technology uses scientific findings such as psychology and the humanities to design and implement the whole process of teaching and learning.

• 4. Educational technology employs a good mix of human and non-human resources. In other words, unlike the use of educational materials or audio-visual training in which the use of material instruments is concerned, in technology, human resources are appropriately used.

The most recent definition agreed by educational technology experts, The American Association of Educational Communication & Technology (AECT) stated that educational technology is the theory and practice of designing, producing, using, and evaluating learning processes and resources (Spector & Yuen, 2016). Caffarela and Fly (1992) define this as considering that in any field of science philosophical issues such as epistemological epistemology and methodology are raised and that experts in different disciplines present theories in that field.

Application of educational technology in English language teaching

The application of educational technology in English language teaching includes any possible means and information that can be used in language teaching. It deals with language teaching instruments such as television, language labs, and a variety of designed media. In other words, the use of educational technology in language teaching is the same folk concept of educational technology as the use of audiovisual devices, monitors, and computer keyboards. The public domain of its audiovisual equipment consists of two distinct parts: the hardware and the software. The hardware talking part deals with physical and real equipment, such as projectors, sound recorders, TV sets, microcomputers, etc., and the software part includes many items used in connection with such equipment and devices like slide, audiotapes, videotapes of computer programs, written languages, and more (Ahdian, 2007); (Xu, Banerjee, Ramirez, Zhu, & Wijekumar, 2019).

Research in the field of education is sometimes contradictory. Differences in educational measures, situations, measuring tools, and research methods make it difficult for the researcher to compare the findings (Rai'i, Farzaneh, & Delavar, 2013). The contradiction between the results leads to no acceptable answer to be a guidance for policy makers (Talan, 2021). It also means that there is always an endless demand for research and re-research. There is a risk that research sponsors may conclude that this research is confusing, unproductive, and unscientific (Asgharpour, 2006).

Considering the research done in the field of language learning with the help of technology, it can be seen that there are a lot of disagreements about the success rate of teaching English with the help of technology. Some researchers such as Sung, Cheng and Liu (2016) and Lee (2010) are its defenders, while other researchers such as Lipsey and Wilson (2001), Norris and Ortega (2000), and Oswald and Plonsky (2010) have expressed doubts about its success.

Proponents of using technology for language learning, giving the learner freedom of action, access to a variety of language content, ease of access and its inherent attractiveness, opponents of factors such as lack of infrastructure, lack of software and hardware to especially in developing countries, students and teachers are not familiar with this technology, teachers do not master technology to produce appropriate educational content, superficial and unrealistic interactions in existing software, too much emphasis on the use of multimedia, heavy volume Content for language learners, lack of appropriate feedback and finally receiving superficial and unrealistic feedback as reasons for their opposition to using technology for language learning. These contradictory reasons led us to perform meta-analysis to determine the effectiveness of technology-assisted language learning and, in general, whether it was successful or not. These reasons became contradictory in order to determine the effectiveness of language learning through meta-analysis with the help of technology and to reach a general conclusion whether it was successful or not.

Research methods

Considering that the purpose of this research was to describe, analyze, and combine the studies presented in the field of educational technology in English language teaching based on research; the method of this research was a meta-analysis. Meta-analysis is a set of statistical methods that are performed in order to combine the results of independent experimental and correlation studies that have the same research questions on a single topic, and leads to a single estimate and result. Unlike traditional research methods, a meta-analysis uses statistical summaries of individual studies as research data.

Based on the main assumption of this method, each study provided different estimates of the underlying relationships in society. Therefore, by combining the results of these studies, a more accurate view of these relationships could be provided, which was provided by estimating individual studies. Meta-analysis research was applied type and was among the few pieces of research. The method used to collect data in this research was the library.

General objective subgroups: variables

Hypothesis 1 — There is a significant difference between the years of publication in research on the application of technology in English language teaching.

Hypothesis 2 — The research method used in the research has been used in the field of application of educational technology in English language teaching.

Hypothesis 3 — Research tools have been used in the application of educational technology in English language teaching.

Hypothesis 4 — There is a significant difference between the effect size of different statistical methods in research in the field of technology application in English language teaching.

Hypothesis 5 — There is a significant difference between the size of the work based on the gender of the sample in research on the application of technology through English language teaching.

Eligibility criteria and exclusion criteria

In this research, 67 articles or theses out of 1000 articles or theses which were related to technology in English language teaching in Iranian cites, like Tehran, Shahr-e- Qods, Yasouj, Shahrood, Mazandaran, Bandar Abbas, Alborz, Shahrekord, Ahvaz, Qeshm, Guilan, Semnan, and Chabahar, were randomly chosen from Iran Doc, google scholar, and science direct websites.

It is important to mention that 7 out of 67 articles and theses were ignored in this thesis based on the table that has been mentioned (Additional file 1: Appendix A). Topics in selected articles were completely consistent and had a relatively high and appropriate subject similarity for meta-analysis were from the years between 2009 and 2020 (Additional file 1: Appendix B). Conditions were detected and meta-analysis tests had been performed on them. It should be noted that in the meta-analysis method, there is no specific limit on the number of studies.

Method of data collection

To perform meta-analysis, the specifications of all theses in the field of educational technology in English language teaching, which are the year of publication, sampling method, statistical method, research method, and gender of the sample were studied. These data were then used in analysis, syntheses, and comparison.

Instruments

Meta-analysis is the statistical method which was used in this study. The SPSS (Sciences Statistical Package for the Social) software (SPSS Statistics 26) that researchers examined the frequency and statistical significance. The research hypotheses were also tested by SPSS software. CMA 2 software (Comprehensive Meta-Analysis version 2) was used to calculate the effect size for each study, the overall size effect, and the size of the discriminant effect to test their statistical significance in this research.

In addition, the research hypotheses were tested using SPSS software and the Kruskal-Wallis nonparametric test. The effect size in this r was calculated using the Hex method.

The Egger regression method has also been used to evaluate the homogeneity of the studies. The advantage of this method compared to other tests is that it is stronger. This method uses real effect size methods for prediction.

Results

Data analysis

Part 1: Descriptive analysis

Description of general characteristics of the studied samples

Descriptive information about the year of publication of the studies was examined in this study.

As can be seen in Table 1, the highest percentage is related to research published during the years 2015 to 2017 with a rate of 51.7% and the lowest percentage is related to research published in the years 2018 to 2020 with a rate of 6.7 percentage.

Table 1 Frequency distribution of researches by year of publication

Descriptive information about the research methods used in the studies reviewed in this study

As can be seen in Table 2, the highest percentage is related to the quasi-experimental research method with 36.7% and the lowest percentage is related to the qualitative research method with 3.3%.

Table 2 Frequency distribution of researches by research method

Descriptive information about the instruments used in the studies examined in this study

As can be seen in Table 3, for the instruments used, the highest percentage is related to the Questionnaire instrument with 37% and the lowest percentage is related to the Observations instrument with a rate of 3.3%.

Table 3 Frequency distribution of researches by research method

Descriptive information about the statistical method used in the studies examined in this study

As can be seen in Table 4, the highest percentage is related to the method of using pre- and post-tests with 56.7% and the lowest percentage is related to the statistical method of ANOVA with 1.7%.

Table 4 Frequency distribution of researches by statistical test

Descriptive information about the sex of the sample in the studies

As can be seen in Table 5, for the sample gender, the highest percentage is related to mixed-gender with a rate of 48% and the lowest percentage is related to female gender with a rate of 20%.

Table 5 Frequency distribution of researches by gender

Homogeneity of studies

In order to check the homogeneity of the studies, the Eger regression test is used and the results of this test are summarized in the following table:

As can be seen in Table 6, due to the value of Sig, which is greater than 0.05, the assumption of homogeneity of studies at an error level of 0.05% is accepted.

Table 6 Frequency distribution of researches by homogeneity

The following Fig. 1 is used to determine whether the initial studies are biased and their impact on data analysis.

Fig. 1

Funnel diagram shape the size of each study with the effect size accuracy

If the initial studies do not have a diffusion bias, they should be distributed symmetrically around the average effect size, as shown in the diagram above.

Overall effect size

Before examining the effect size separately for the variables in this study, the overall effect size is calculated in two modes: a model with random effects and a model with fixed effects, and the results are recorded in the table below.

It should be noted that due to the homogeneity of the initial studies in this study, the model with fixed effects is more efficient than the model with random effects.

As can be seen in Table 7, considering that the sig value for both models is less than 0.01, it can be accepted that the total effect size in both models is significant with random effects and fixed effects at the error level of one percent.

Table 7 Total effect size results

Effect size by year of publication

The following table records the results related to the effect size by year of publication of studies in both model modes with random effects and fixed effects.

According to the Sig values obtained in Table 8, the size of the effects in all the studied years is significant in both types of models with fixed effects and random effects.

Table 8 Effect size by year of publication

The size of the work is used separately according to the research method

In the table below, the results related to the size of the effect are recorded separately by the research method used in the studies in both models with random effects and fixed effects.

According to the Sig values obtained in Table 9, the size of the effects in all research methods used in the studies under study in both types of models with fixed effects and random effects are significant.

Table 9 Effect size by research method

Effect size by the instrument used

In the table below, the results related to the size of the effect by instruments used in the studies are recorded in both model modes with random effects and with fixed effects.

According to the Sig values obtained in Table 10, the size of the effects in all instruments used in the studies under study in both types of models with fixed effects and random effects are significant.

Table 10 Effect size by instrument

The size of the work is separated by statistical methods

In the table below, the results related to the effect size are recorded separately by the statistical methods used in the studies in both models with random effects and with fixed effects.

According to the Sig values obtained in Table 11, the size of the effects in all statistical methods used in the studies in both types of models with fixed effects and random effects, except the random effects model in the case where the statistical instrument is used Qualitative Have been meaningful.

Table 11 Effect size by research method

Effect size by sample gender

In the table below, the results related to the effect size by sample gender in both models with random effects and fixed effects are recorded.

According to the Sig values obtained in Table 12, the size of the effects on the sex of the sample in both types of models with fixed effects and random effects are significant.

Table 12 Effect size by gender

Part II: Inferential analysis

Hypothesis 1

There is no significant difference between the size of the effect of years of the publication on research in the application of educational technology in English language teaching.

To test the above hypothesis, the Kruskal-Wallis test was used and the results of this test are recorded in the following tables:

As can be seen in Table 13, considering the value of Sig = 0.151, which is greater than 0.05, the assumption of zero, i.e. the assumption that the size of the work is the same according to the year of publication is not rejected at the level of 5% error.

Table 13 Kruskal-Wallis test

The effect of years of publication in research in the field of technology application in English language teaching is not significantly different.

Hypothesis 2

There is no significant difference between the size of the work and the research method used. Research conducted in the field of technology application in English language teaching.

To test the above hypothesis, the Kruskal-Wallis test was used and the results of this test are recorded in the following table:

As can be seen in Table 14, considering the value of Sig = 0.302, which is greater than 0.05, the null hypothesis, i.e. the assumption that the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work and the research method used in the field of technology application in English language teaching.

Table 14 Kruskal-Wallis test

Hypothesis 3

There is no significant difference between the size of the work and the instruments used in research in the field of technology application in English language teaching.

To test the above hypothesis, Kruskal-Wallis test was used and the results of this test are recorded in the following table:

As can be seen in Table 15, considering the value of Sig = 0.830, which is greater than 0.05, the null hypothesis, i.e. the assumption that the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work and the instruments used in research in the field of technology application in English language teaching.

Table 15 Kruskal-Wallis test

Hypothesis 4

There is no significant difference between the size of the work according to the statistical method used in research in the field of technology application in English language teaching.

To test the above hypothesis, Kruskal-Wallis test was used and the results of this test are recorded in the following tables:

As can be seen in Table 16, considering the value of Sig = 0.814, which is greater than 0.05, the null hypothesis, i.e. the assumption that the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work according to the statistical method used in research in the field of technology application in English language teaching.

Table 16 Kruskal-Wallis test

Hypothesis 5

There is no significant difference between the size of the work by gender of the sample in research in the field of technology application through English language teaching.

To test the above hypothesis, Kruskal-Wallis test was used and the results of this test are recorded in the following table:

As can be seen in Table 17, considering the value of Sig = 0.819, which is greater than 0.05, the null hypothesis, i.e. the assumption which the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work by gender of the sample in research in the field of technology application through English language teaching.

Table 17 Kruskal-Wallis test

Discussion

In this part, researchers describe the collected results in general and discuss the statistical results obtained. The present study includes 67 studies out of 1000 theses and articles which 7 of them were excluded from this study due to a lack of sufficient information (Appendix A).

The main purpose of this study was to investigate the impact of educational technology on English language teaching. The optimal research method to achieve this goal was meta-analysis. In this method, “each research” was a unit of study, furthermore, the amount of effect size was calculated for each research in order to obtain the effectiveness of each research.

Our results indicate that technology applications have a large effect (1.68 and 0.91, fixed effect model and random effect model respectively) on English language teaching. This proposes that the use of technology is more effective than traditional teaching methods without technology for English language teaching quality.

Overall effects of educational technology on English language teaching

The result of a medium-sized overall positive effect of educational technology on English language teaching confirmed that the use of a computer, telegram, mobile, laptop devices, and software could facilitate language learning. These results were consistent with other research findings regarding the effects of different devices and software on English language teaching.

Related to the first research question: Year of publication

This research question was in line with Sung, Yang, and Lee (2017) and Chauhan (2017), which both had the same experimental results show that their meta-analysis was not substantially affected by publication bias. The most obvious finding to emerge from this research question was that years of publication did not have a significant result in this research.

Based on the fact, the year of publication was selected for research as a variable; if years are considered differently, that is, for example, the year 2009 is assumed alone, they are meaningful. They also have the same feature for the year of publication until 2020. However, based on the research question of how much the effect of the year of publication affects educational technology, it should be noted that this variable is not recommended for future research because it changes every time based on advances in technology and different methods for research. Considering the year of publication, it will not have a significant effect as a whole on the effect size of the work.

Related to the second research question: research method

It is in line with Farzaneh Shakki (2015). There is no significant difference between the effect size by the research method used and the research conducted in the field of technology application in the English language teaching.

In fact, the research method as a whole depends on the researcher and the type of research that is being done. In this research, we conclude that if we want to consider the research methods one by one we can claim that they all have a significant effect but when we want to consider all of them relative to each other, they do not have a significant effect. Therefore, this research shows us that the required research methods or resources required as well as different goals can be variable, so it depends on the researcher in what circumstances, in what environment and with what tools they can choose the research method. Of course, a single research method may not be used in an article, and several types can be used.

Related to the third research question: instruments

It was in line with Fazeli (2016). It was not in line with Pourtayebi (2015), Alinejad (2015), Sadeqi (2015), Rastegar (2014), Shahkooei (2016), and Parinaz (2010). There is no significant difference between the effect size and the instruments used in research in the field of technology application in English language teaching

As we have seen, among the number of theses and articles we reviewed, a variety of instruments were used. In the meantime, the questionnaire was used more than other instruments, but this does not mean that this instrument is superior to others in research instruments. In this study, in each of the articles and theses, one or more instruments were used, which were significant, but in general, they were not significant in comparison to each other. This means that we cannot say which tool is better than other instruments so it depends on the researcher which instrument to choose over the research.

Related to the fourth research question: statistical method

It was not in line with Shahkooei (2016). There is no significant difference between the size of the work according to the statistical method used in research in the field of technology application in English language teaching. Although the number of statistical methods used in these studies was different, in the ranking, they did not have a significant difference.

To check the quantitative research data, the use of statistical tests is mandatory. A statistical method is necessary to use for each research. Reviewing all statistical tests can be a good guide for analyzing data in an article. Meanwhile, it may not always be enough to use a test.

Statistical methods are one of the practical ways to identify problems and provide solutions to managerial, social, and psychological problems, etc. that, if implemented correctly, can provide real data for our research.

In other words, there may be different ways of doing research or how we can collect our data to prove or answer questions. At this point, having high analytical power, problem-solving ability. And sufficient experience can help you to know the correct method of research. Because it directly uses people’s opinions, it can solve society’s problems, and these studies are often very practical and can be cited.

Related to the fifth research question: gender

It was in line with Alipour (2017), Sadeqi (2015), Nateghi (2018), Dayani (2014), Ghazavi (2017), and Aliakbari (2013). It was not in line with Mohammadi (2014), Alashti (2013), AsgharHeidari (2014), and Nakhaei (2017) found the result of the study that the attitudes of English teachers or students regarding their gender towards the use of the Internet, mobile or other devices were positive and high.

According to the statistical part of this study, the participants were mixed in most of the articles, but in some of them, exclusively female participants or in some other male participants were used to conduct the research. Based on the findings, we conclude that there is not much difference between men being superior to women or vice versa.

Based on the availability of technology in education, educational technology has caused many changes in the field that meet the needs of students in different ways. With the provision of software that teaches students with special needs, the appropriate educational equipment is designed to make learning easier for the individual.

With the use of technology, the concept of education is changing for both students and teachers to progress. Therefore, the introduction of technology in education is very important.

Research limitations

The present meta-analysis, like many others, has its limitations and forces the researcher to interpret the findings with extreme caution.

• Lack of access to some articles and dissertations that did not receive a response from the authors despite sending an email.

Suggestion for further study

Due to the limitations that researchers applied in this research, 67 theses and articles were selected from different cities of Iran that had a topic related to the subject of this research, but it should be noted that due to development and progress in recent years, the importance of this thesis is observed. It is better to select researches that have been published in reputable publications all over the world, in addition to this, it is suggested to work on other various variables.

Conclusion

A meta-analysis of research on the application of technology in English language teaching, which was published in valid journals in this field and examined, showed that the application of information technology in this field has an acceptable impact factor.

In this section, the overall findings of the current study were presented. According to the studied variables, we conclude that the five variables studied and researched, according to their statistical information in this study, did not have a significant effect size. And in response to the overall purpose of the article, how much technology can affect English language teaching, it can be concluded that, initially, compared to the variable of the year, 2017 to 2020, the size of the work was more representative than previous years, so technology has been effective. Other variables, such as tools, research methods, statistical methods, and gender, have had a smaller effect than size that we can ignore.

The results showed that all chosen variables in this study, considering every unique thesis or article, were significant, but as the whole consideration of each variable to 60 theses and articles, they were not significant.