Skip to main content

Adaptation and validation of a critical thinking scale to measure the 3D critical thinking ability of EFL readers

Abstract

Thinking has always been an integral part of human life, and it can be said that whenever humanity has been thinking, it has been practicing a kind of criticizing the issues around. This is the concept of critical thinking that enhances the ability of individuals to identify problems and find solutions. Most previous research has focused on only one aspect of critical thinking that is critical thinking skills, while two other dimensions of criticality and critical pedagogy should have also been included. In order to assure of the validity of the instrument designed by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review), it was first adapted and then SEM modeling was used. Examination of the results of factor analysis and modeling of SEM showed that the model satisfied the fit indices (χ2/df, CFI, TLI, RMSEA), and all the factor loads are greater than 0.4 which represent that the items are defined properly. This research suggested a SEM modeling of critical thinking skills, composed of six factors measured by twenty-two indices. The results of the PLS-SEM CFA represented that it is a valid structural model to measure a critical thinking of EFL readers at three levels.

Introduction

Recent research on reading has represented that, although it is generally established as the first skill in language learners, it is a complex cognitive activity for individuals to perform well in learning and obtaining sufficient information from the target community (Shang, 2010). According to Krathwohl (2002), the cognitive domain is divided into two parts: first is the knowledge (including real, theoretical, procedural, and metacognitive knowledge) and then the cognitive process (including recalling, comprehending, applying, examining, evaluating, and creating). In defining this skill, Chamot (2004) holds that reading is the process of activating language-acquired knowledge and skills to access information and transfer them. Swallow (2016) looks at it as a three-dimensional construct including content, perception, and understanding through thinking, metacognition, and meaning construction (Gear, 2006).

According to Rashel and Kinya (2021), the focus of education in this competitive period of time is on higher-level thinking skills (including critical thinking) rather than low-level thinking skills, and research into measuring critical thinking skills is growing. In the eyes of Ennis (2011), critical thinking ability is defined as clear and rational thinking that includes engaging in reflective and independent thinking. Moon (2008) and Willingham (2007) emphasized that the development of critical thinking in individuals is the goal of higher education and can be recognized as the primary goal of learning. Paul and Elder (2002), in describing a critical thinker, stated that in the eyes of such a person, all four skills of reading, writing, speaking, and listening are methods of skilled higher-order thinking. Such a person, while reading the text, finds it a representation of the author’s thinking and therefore tries to align with his point of view. In this regard, Din (2020) emphasizes that since a critical thinker has the ability to understand beyond the content of a text, they tend to react to the content being studied. Moreover, the tendency towards implementing critical thinking programs in the English language teaching context has increased as well (Heidari, 2020; Liu & Stapleton, 2014).

Beside the theory-wise investigations, there are a couple of studies with practical direction. Some research has examined the role of critical thinking in learning a language (e.g., Akyuz & Samsa, 2009; Willingham, 2007), others focused on the thinking strategies used by language learners in improving reading skills (Shang, 2010) or the relationship between critical thinking and language learning strategies (Nikopour et al., 2011). A few studies confirmed the relationship between critical thinking ability and reading comprehension (e.g., Eftekhary & Besharati Kalayeh, 2014). In such area, a limited number of studies have relied on the participation of the academic community (Hawkins, 2012; Hosseini et al., 2012), and this study is also innovative in this respect. It can be inferred that in most of these studies, critical thinking is limited to the use of definite basic skills (compare and contrast, conclusion, inferencing, etc.). According to Facione (1990) and Rashel and Kinya (2021), most research on this topic has focused on general critical thinking skills (but not expertise), although these skills have been of interest for years. But, is it enough to just use these skills to understand a content? Is critical thinking summarized in terms of several sub-skills? Where and how is the role and impression of society reflected in critical thinking or critical reading? Does learning these sub-skills alone indicate the internalization of critical thinking and reading in individuals? These key questions have been left intact mainly due to a lack of specific and valid instrument, as a rationale behind this very study.

The novel point in the present study is that, despite the existence of the one-dimensional attitude towards critical thinking (Facione, 1992; Kember et al., 2000), it tries to highlight the concept of a three-dimensional critical thinking in academic context and in this regard developed a tool for measuring its subscales (and not just individual skills). Such a tool can measure the real needs of the next generation with evidence of real-life multifaceted critical thinking issues. The purpose of this study was to evaluate the validity of the questionnaire developed for assessing three-dimensional critical thinking skills in EFL readers. Moreover, the application of the partial least squares method (PLS-SEM) in the development and validation of the proposed model has also made this research innovative. The objectives of this study were (1) to assess the validity of the items introduced in the questionnaire, (2) to investigate the relationship between and among the identified components, and (3) to determine the validity and reliability of the questionnaire designed to assess three-dimensional critical thinking skills in EFL readers. The contribution of this article in the literature is to illustrate the importance of critical thinking both in personal and sociocultural aspects, to evaluate and validate the tool that was developed to measure the components of three-dimensional critical thinking (proposed by the same researchers), to provide the model fit indices for factor analysis, and to adapt the instrument to the conditions of English language readers. Therefore, an attempt was made to briefly introduce the components of the proposed model, and then to discuss the validation method of the developed instrument to measure these components, and finally to report the validation results of the introduced instrument. The pedagogical implications of this study include the following: using the presented concepts in research centers to identify and introduce the method of teaching and developing each of the sub-skills of critical thinking in different societies; identifying differences in instructional approaches for each of the sub-skills; applying both concepts (i.e., three-dimensional critical thinking and reading) in other areas and assessing the generalizability of findings; and reviewing the previous literature by looking at all three dimensions introduced and evaluated in order to identify their strengths and weaknesses in this regard.

Literature review

Today that critical thinking is more prominent in language teaching than ever (Li, 2016; Van Laar et al., 2017), there is a wealth of research on the need and importance of fostering such thinking in language classrooms (Zhao et al., 2016), representing that developing such thinking facilitates the language acquisition (Wang & Henderson, 2014; Wu et al., 2013), and equips learners with such self-criticism that it develops analytical and reflective view of themselves and their environment in learners (Moghadam et al., 2021). Brookfield (2019), Dekker (2020), Haji Maibodi and Fahim (2012), and Zou and Lee (2021) acknowledged that teachers who emphasize the education and application of critical thinking increase awareness and understanding of socio-cultural concepts in learners. In this regard, Crenshaw et al. (2011) stated that encouraging language learners to participate actively in thinking activities is essential, and McGregor (2007) and Rezaei et al. (2011) emphasized that engaging teachers and language learners in thinking and reflecting on the views and assumptions presented in a text are among the essential steps in the development of critical thinking in individuals. Rezaei et al. (2011) acknowledged that learners’ participation in critical thinking processes during teaching is done through asking questions and providing answers, discussing topics, asking for explaining or elaborating on opinions, and so on. They also emphasized the need to provide teachers with accurate and comprehensive knowledge of critical thinking before attending such classes. In addition, Tehrani and Razali (2018) and (Li, 2016) have suggested that critical thinking training should begin at early ages and in the natural process of learning the target language. However, despite the importance and emphasis on its development, little progress has been made in its application and integration in education (Li, 2011; Pica, 2000), whose reasons, according to Lin et al. (2016) can be found in its challenging-widespread nature and ambiguous details of its components.

The traditional definitions of critical thinking by philosophers do not necessarily assist individuals to become a critical citizen/being. However, the core characteristics of critical thinking introduced in these definitions remain fundamental to what is meant by critical thinking on (Bali, 2015; Davies, 2015; Davies & Barnett, 2015; Renatovna & Renatovna, 2021; Widyastuti, 2018; Wilson, 2016). Considering critical thinking as a very pivotal learning skill, the acquisition of related skills in the traditional attitude was limited to practices of certain types of skills such as inferencing, reasoning, and analyzing (Davies, 2015). He emphasizes that one of the weaknesses of the traditional sense of critical thinking, which is crystallized in the critical thinking movement, is the lack of formation of the important component of action. This is worth mentioning that paying less attention to the topics related to critical thinking in higher education may result in a lack of having a proper and well-defined practical (and even theoretical) instruction, and as it was mentioned by Paulsen (2015), little advancement can be formed if critical thinking remains vague.

A model of critical thinking in higher education is suggested by Davies (2015) in which the basic focus is on critical rationality and critical character of individuals. He presumes six distinct dimensions for critical thinking including critical argumentation, critical judgment, critical dispositions, critical actions, critical social relations, and critical creativity or critical being. Each of these dimensions plays a significant role in the comprehensive model of critical thinking (Davies, 2015; Davies & Barnett, 2015).

There are many well-developed models of critical thinking which might be called “philosophical” models of critical thinking. These models might be dispersed on a continuum from the taxonomy of pedagogical objectives (e.g., Airasian et al., 2001; Bloom, 1956) to the APA Delphi Report and Paul-Elder models (e.g., Paul & Elder, 2002; Sadler, 2010) and also to the model of critical thinking by Ennis (1991) according to which the main emphasis is on cognitive decision-making. However, Davies (2015) represented that these models are utilized mostly in the case of educating for critical thinking in which the main goal is providing learners with activities based on which they can improve their basic judgment and decision-making ability, while critical thinking is a multidimensional concept containing both personal and social aspects. In endorsing and supporting the use of the term multidimensional in relation to the concept of critical thinking, some of the existing challenges can be mentioned. Lun et al. (2010) and Manalo and Sheppard (2016) stated that a specific level of language proficiency is expected to accomplish such thinking. Similarly, Peng (2014) stated that for students, language deficiency is one of the main reasons of cognitive barriers that prevents them from practicing critical thinking. Explaining the other challenges, Liang and Fung (2020) and Merrifield (2018) stated that the subject of culture is effective in applying and practicing such thinking. For example, factors such as a significant decline in the quality and quantity of social interactions and intense attention to the social status of an individual in a group (Suryantari, 2018), some considerate social standards explicitly in eastern setting (Bag & Gürsoy, 2021), socio-cultural factors (Imperio et al., 2020; Shpeizer, 2018), fear of being ridiculed during expressing an opinion (Tumasang, 2021), epistemic belief in the certainty of knowledge (Kahsay, 2019), the emphasis on teacher-centered language classes (Fahim & Ahmadian, 2012; Hemmati & Azizmalayeri, 2022; Khorasani & Farimani, 2010), or weakness in CT experiences due to the lack of inductive education in Iranian context (Birjandi et al., 2018), reduce the natural learning ability of developing such skill as well as the power of induction—especially in adults (Dornyei, 2010). Therefore, the subject of language learning, whether in a foreign or a second language context, complicates the issue of cultivating critical thinking in such a way that its development cannot be limited to learning a few individual skills. In this regard, Davies and Barnett (2015) attempted to bring together a set of perspectives, thus identified three broad perspectives on critical thinking in the literature. These perspectives are often opposed to each other, while overlapping and significantly merging with each other (Frykholm, 2020; Ledman, 2019; Shpeizer, 2018; Wilson, 2016; Wilson & Howitt, 2018). Shpeizer (2018) also emphasized that this mutual influence and the lack of transparency in the boundaries of each of the three areas have made the concept of critical thinking confusing and perhaps daunting for English teachers.

In addition, understanding the nature and dimensions of critical thinking in order to evaluate it is also of crucial importance. Assessing an individuals’ critical thinking requires a measuring instrument that can precisely and perfectly determine the true conditions. From the result of the literacy study, one can find some instruments to measure critical thinking skills and abilities of students each with their specific perspectives, definitions of criteria, and priorities. Among these instruments are California Critical Thinking Skill Test (CCTST) by Facione (1992); Critical Thinking Questionnaire by Kember et al. (2000); Ricketts (2003) questionnaire; Critical Reading Scale by Zhou et al. (2015); and Critical Thinking Inventory by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review). The designed questionnaire by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review), unlike previous tools, addresses all the three dimensions of critical thinking (i.e., individual skills, criticality, and critical pedagogy).

Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review), getting insights from Davies (2015) and Davies and Barnett (2015), represent that critical thinking is composed of both personal critical thinking skills, and those skills gained at the criticality level and critical pedagogy level. The levels, movements, and skills of each of the levels introduced in their study are presented in the figure below.

As shown in Fig. 1, as one moves from the center to the outside (the surface), the stages of critical thinking development appear, according to which this process begins with the development of individual critical thinking skills, the criticality movement, and then the critical pedagogy movement. This figure includes the XY plane (page drown on x and y diagrams), indicating the measurement subscales; YZ plane (page drown on y and z diagrams) represents individual and socio-cultural dimensions; and the XZ plane (page drown on the x and z diagrams) is different movements.

Fig. 1
figure 1

The model of critical thinking movements, skills and abilities, and assessing criteria extracted from Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review)

The model represents that in order to improve critical thinking in a person, it is necessary to consider both individual and socio-cultural aspects. In this figure, the X-Z page represents various dimensions of critical thinking, the Y-Z page represents cognitive-developmental skills, and the X-Y page shows sub-skills of each layer (i.e., assessing criteria in this study). Aspects and skills of the three-dimensional critical thinking which were previously introduced by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) are briefly explained in Table 1.

Table 1 Assessing criteria of CTI proposed by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review)

Critical thinking and criticality are the most interwoven concepts with language skills acquisition in general and processing and development of reading skills in particular. And of course, developing skills related to each of these two movements requires critical pedagogy. According to Haji Maibodi (2014), reading comprehension refers to the ability to construct meaning through thinking, before, during, and after reading the text, as well as integrating the information presented in the text with one’s prior knowledge. She also stated that different types of texts with different levels of difficulty and various topics are available to people to be encouraged to read and thus gain new knowledge and strengthen their reading skills. As people go through this process, they realize that in order to understand the texts they read as much as possible, they have to use thinking skills (Haji Maibodi, 2014), and this thinking takes different forms and the implementation of each and requires skills that are called critical thinking skills. Haji Maibodi (2014) also emphasized that practical teaching of reading comprehension requires the development of the ability to understand, analyze, and recognize various components of a text.

Reading is viewed as the most crucial academic skill for foreign language learners which can facilitate their professional progression, social success, and personal development. Reading skill is defined by Berardo (2006) as a dynamic and complex phenomenon and is considered as a source of gaining language input since it is a receptive skill based on which there should be an interaction among the author of the text, his/her message, and the reader in order to comprehend it. Therefore, in order to read, comprehend, and respond to a written content, the reader is expected to have some certain skills and abilities including reading to grasp the message of each line and paragraph, reading to find the existing relationship between the paragraphs, understanding the basic message of the author, and finding the most appropriate answer to the idea of the writer (Berardo, 2006). According to Berardo (2006), stages of reading require readers to apply a higher order of thinking called “critical reading” by Bloom (1956). According to Hall and Piazza (2008), critical reading skill is still one of the skills which helps learners gain success in academic courses whilst it is still vague to many teachers and they usually fail to develop such skill in their students. They represent that if students lack the skill to analyze and challenge written content in the classroom environment, then they will face many problems in understanding and questioning their living environment and society.

Wallace (2003) and Sweet (1993) approach the critical reader as an active reader who is able to ask questions, to recognize, analyze, and confirm evidences; to detect the truth; to understand tone, bias, and persuasion; and to judge them throughout the reading process. Khonamri and Karimabadi (2015) state that in order to have an effective reading, readers should have the ability to read with their critical eyes, i.e., they have to read and evaluate a text for its intentions and the reasons behind it, that is the ability to think critically.

Critical reading, as the key player in the development of core language skills, involves activities such as reasoning, questioning, evaluation, comparison, and inference (e.g., Adalı, 2010; Adalı, 2011; Söylemez, 2015). Regarding critical reading, Nemat Tabrizi and Akhavan Saber (2016) emphasized that this skill plays an important role in the formation of democratic societies since it makes people decide what they accept as reality, only after reviewing, analyzing, and comparing the content presented with their knowledge and values of their internal-external worlds.

Instrument validation

Measurement validation in the eyes of Zumbo (2005) is a continuous process in which evidence is collected to support the appropriateness, significance, and usefulness of the inferences derived from scores obtained from a sample. He also emphasizes that the method and process of validation is important in the construction and evaluation of tools related to social sciences, behavioral, health, and humanities, since without the implementation of this process, any conclusions or inferences from the obtained scores are meaningless.

Many have argued that in the contemporary view, the main purpose is to extend the conceptual framework and power of the traditional vision towards validity (Johnson & Plake, 1998; Kane, 2001; Messick, 1989; Messick, 1995), according to which validity is not one of the characteristics of measuring tools anymore, but the characteristics of inferences made on scores that can be examined in the form of a continuum (valid/invalid dual is no longer considered). In this view, construct validity is the only and the most important feature in validation, and there are only different sources of evidence to prove the validity of inferences. Zumbo (2005) stated that the calculation of validity using statistical methods such as correlation is not acceptable, and it is necessary to provide a detailed theory and support for it, including analysis of covariance matrices between experimental data and covariance structure model. From the study of previous research, it can be seen that the two categories of models are introduced as key for validation, which are confirmatory factor analysis (CFA), which has a lengthy and rich history in research (for example, Byrne, 1998; Byrne, 2001; Kaplan, 2000) and Multiple Indicators Multiple Causes (MIMIC) that have been generalized to linear structural equation models by integrating structural equation modeling and item response theory (Ullman, 2001). The multidimensional and hierarchical representation of the skills needed for critical thinking at each level is primarily based on theoretical reasoning (by Davies, 2015; Davies & Barnett, 2015; Frykholm, 2020; Ledman, 2019; Shpeizer, 2018), as mentioned in the previous paragraphs.

Accordingly, this study was an attempt to adapt and assure of the validity of the questionnaire proposed by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) in order to measure criteria introduced in Fig. 1, XY plane (see Appendix A for the validated version). A review of previous studies showed that previous research has only examined individual skills and examined various subskills in this area. None of the studies have provided a comprehensive scale, consisting of both individual and socio-cultural factors, and the validation of a common scale for measuring the set of factors. Regarding this, the present study assessed the three-level scale of critical thinking and validates the proposed model. In this study, a measurement and structural model according to the previous literature and the method of factor analysis is proposed. This research is innovative because it uses the partial least squares method (PLS-SEM) and various software to validate the proposed model. The PLS method relies on a series of consecutive ordinary least square (OLS) regressions; thus, it eliminates the necessity of having a normal distribution of observations. OLS indicates the compatibility of the partial least squares method with small samples and is suitable for the conditions of this research (Farahani, 2010). On the other hand, given that PLS assumes that all blocks are linear combinations of their reagents, common problems such as nonlinear solutions and uncertainty of the factors that occur in covariance-based structural equation modeling (CB-SEM) techniques do not occur (Pirouz, 2006). Researchers aimed to answer the following question:

  • RQ. To what extent is the newly developed rating description a valid measure of critically thinkers’ reading ability?

Methodology

In this study, an attempt was made to validate the three-dimensional critical thinking instrument developed by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) to assess critical thinking in English as a Foreign Language (EFL) readers (Tables 2, 3, 4, 5, and 6).

Table 2 Model Parameters and their Significance
Table 3 Model Eevaluation Indicators
Table 4 Validity and reliability investigation
Table 5 Estimation of model parameters and their significance
Table 6 Model evaluation indices

Participants

In order to answer the research question, 89 Iranian EFL under-graduate students (age range 18 to 35) were selected for the development and validation of a reading skill-oriented critical thinker measurement instrument. The participants were members of intact classes (with the aim of involving individuals with diverse abilities), and the homogeneity of the classes was also assessed via Preliminary English Test (PET score above 147). Due to the fact that the participants cooperated with the researchers during different phases of the study, the implementation steps were introduced to them, ethical approval was given, participants were assured of not publishing personal opinions to the third person/parties, and the final results were communicated to them.

Instruments

  1. 1.

    Critical thinking inventory: The CTI, by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review), contains 59 items of 5-point Likert type to measure the factors of argumentation (15 items), judgment (5 items), disposition (9 items), criticality (12 items), social cognition (9 items), and creativity (9 items number) in 50 min. The minimum score of the questionnaire is 59 and the maximum is 295, and the participants were asked to respond within 60 min. The CR and AVE were reported in the work as 0.97 and 0.687 (see Table 7).

  2. 2.

    Preliminary English Test (PET): This test was used to select groups of participants who have similar language proficiency. It is an official intermediate English language test (designed by Cambridge ESOL examinations) with the maximum achievable score of 170. This test includes sections of reading (five parts, thirty-five items, scoring range of 0–35), writing (three parts, seven items, scoring range of 0–15), listening (four parts and twenty-five items, scoring range of 0–25), and speaking (four parts of face-to-face interview questions, scoring range of 0–25). Two raters were asked to assess the test to be assured of interrater consistency of scores. The intra-class correlation coefficient (ICC) test was run to determine if there was an agreement between raters’ judgment on the scores. A high degree of reliability was found between the scores (F(88, 88)= 146.08, p < .000) with the average measure ICC of .982).

Table 7 Reliability results review

Procedure

Initially, the written informed consent was obtained from the participants. Then, PET test was used to ensure the homogeneity of the participants and those with similar performance were selected for this study. Next, participants were asked to respond questions to assess CTI validity. After collecting data, the relationships between the elements, skills, and concepts introduced in the questionnaire (see Table 1) were assessed. For this purpose, the validity testing of the model was conducted through CFA method of evaluating and comparing alternative models: CFA of the measurement model (first-order model) and CFA of the structural model (second-order model). In this study, in order to increase statistical power, researchers tried to use predictor variables (i.e., AWC, QAR, classic instructions), considering less operating levels for continuous variables, utilizing continuous variables instead of polarizing or grouping them, defining focused hypothesis tests, crossing the extracted factors, etc., which are described in Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review). The scale validation in this study included a PLS-SEM analysis technique due to the abnormal distribution of the data gathered (Afthanorhan, 2013) and the model validation included the following tests:

  • Analysis of the convergent validity

  • Test of discriminatory validity

  • Test of construct validity.

Data analysis

After collecting the data of the designed inventory in SPSS, the collected data related to the validity of the questionnaire were transferred to SmartPLS software to validate the proposed model through model validation techniques (FIT, GFI, RMR, etc.), SEM, CR, and AVE estimation. The datasets generated and analyzed during the current study are not publicly available due to ethical issues and privacy restrictions.

Results

In order to find the answer to the research question, a CFA-based approach was used as an MTMM technique to estimate the validity of the designed instrument (Bentler, 2000; Byrne, 2006; Koch et al., 2018). For this purpose, different types of validity of the developed inventory were evaluated.

  1. 1.

    Internal validity

    1. 1.1

      Face validity: Face validity depends on the judgment of the constructor of the test and was approved according to the Advisor’s opinion.

    2. 1.2

      Content validity: Various aspects of the structure are examined. Content validity was confirmed by the Advisor.

  2. 2.

    Criterion-related validity (both concurrent validity and predictive validity): In order to appraise the predictive validity, this instrument should be evaluated over a long period of time, for example, once at the beginning of the undergraduate course and then, again at the end of the fourth year, and then compare its performance in predicting the results with current results. To measure concurrent validity, it is necessary to examine this tool in a completely different content and on a completely different group of learners (at the same time).

  3. 3.

    Construct validity: The category is focused on the structure of the questionnaire. In order to measure the next three criteria, Smart PLS software was used.

    1. 3.1

      Convergent validity: Estimation of CR and AVE

    2. 3.2

      Discriminate (Divergent) validity: Confirmatory factor analysis (t value)

    3. 3.3

      Construct validity: Model validation (SRMR)

In examining the introduced validity criteria, the results of (a) checking the suitability of factor loads, (b) investigating structural equation model, and (c) estimating Goodness of Fit were investigated as follows:

At the beginning, in order to investigate the effect of items and factor loads in measuring the desired structure, the model parameters and their significance were calculated (Fig. 2).

Fig. 2
figure 2

Measurement model test

It is observed that all factor loads are more than 0.4 and are significant. Therefore, the studied items have a significant effect on the measurement of the structure (Table 2).

The model parameter table accurately shows that the p value and t value measures are respectively, less than .001 and more than 1.96, representing a good value. In the following table, the measures of the overall hypothetical fitted model (i.e., goodness-of-fit indicators) are calculated (Table 3).

According to the results, both GFI and AGFI value are more than 0.80; RMR values are close to .00; X2/df ratios are less than 5; and RMSEA estimates are less than 0.08 indicating reasonable errors for approximation in the society. Therefore, all indicators are in the desired range, so the results of the model are trusted and valid and can be used, in general. It should be noted that variables with less than three items cannot be fitted and accurate calculation of their indicators are not possible. In the following, the results of detailed analysis of the model and determination of validity indicators are presented.

Next, the data analysis algorithm in Smart PLS software is displayed. In this algorithm, after model formation and confirmatory factor analysis, it is the time to examine the structural model in three areas:

  1. a.

    Measurement model test: To evaluate the validity and reliability of each structure, the AVE (average variance extracted) and CR (composite reliability) are calculated, respectively (Table 4).

Therefore, according to the results, the validity criterion is more than 0.4 and the reliability criterion for this structure is close to 0.7, so it can be said that in terms of convergent validity criteria, all structures are in the desired range (Fig. 3).

  1. b.

    Structural equation modeling: The results of confirmatory factor analysis of the model represented that:

Fig. 3
figure 3

Structural equation modeling results

It can be seen that all items have a significant effect (p <0.001) on the structure. This shows that the items related to each structure measure the desired structure well (Table 5).

The estimation of the model parameters represents that p values are lower than .001, and t values are greater than 1.96, meaning that the path is significant at the .05 level, meaning that its estimated path parameter has a significant effect on the structure (Ullman, 2001). This shows that the items related to each variable measure the desired structure well.

  1. iii.

    Goodness of fit: For the purpose of conducting confirmatory factor analysis (CFA), as an MTMM technique to assess divergent validity of the model, goodness-of-fit indices were estimated as follows (Table 6):

According to the obtained indicators, it can be seen that AGFI is greater than 0.80, x2/df ratio is less than 3, RMSEA value is less than .08, and CFI is greater than .95 which means that there is a great satisfactory fit. All in all, this can be concluded that the indicators are in the desired range and the results of the model are reliable. Finally, the results of confirmatory factor analysis confirm the relationships and structure of the model, investigating the validity and reliability of the structure (Table 7):

Investigation of the significance of covariance relations also shows that all covariance relationships between structures have a p value less than the error level of 0.05, and the relationships are significant. The advantage of composite reliability over Cronbach’s alpha is that the reliability of structures is not computed definitely; rather, it is obtained through evaluating the correlation of existing structures with each other. In this method, indicators that have a higher factor load are more important. Therefore, both criteria are used to better measure the reliability of this type of models. Moreover, the common measure for creating convergent validity at the structural level is the mean extracted variance (AVE). This criterion is defined as the equivalent to the share of a structure. Acceptable values for CR is over .70, and the excellent value for AVE is over .50.

Considering that the second generation of structural equation modeling is based on the variance approach, and in order to ensure the values of covariance and provide a complete report, the covariance relationships in this model were also examined and the results were reported (Table 8).

Table 8 Significance of covariance relations

As it turns out, all covariance relationships between structures have a p value less than the 0.05 error level and a t value greater than 1.96, meaning that the relationships between latent variables are meaningful.

Campbell and Fiske (1959) and Langer et al. (2010) stated that CFA is an analysis for construct validity. Putting the results observed in steps 2 and 3 together, it can be concluded that all the three absolute fitness indices, parsimony fit indices, and incremental fit indices have desirable values in the model, and this theoretical model is consistent with its experimental model, and therefore, the divergent validity of this structure is confirmed. The results of calculating the reliability of the inventory were also presented in “instrumentation” section. Therefore, combining the results of covariance analysis and the three-level analyses, it can be seen that this questionnaire is valid and reliable.

Discussion

Since there is little agreement on the nature and dimensions of the term critical thinking (Facione et al., 2000; Frykholm, 2020), the researchers of this study decided to provide a comprehensive picture of its various dimensions and develop a valid tool for its measurement. Frykholm (2020) believes that no educator has proposed a comprehensive definition and model of critical thinking, and it can be said that most previous studies have focused only on a few limited skills of critical thinking. However, the results of the interviews in the first phase of this study (Mohammadi et al., Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) clearly showed that the socio-cultural dimensions—if not more—are as important as the individual skills dimension. And by approaching the proposed model of the present study to the model of Davies (2015) and Ledman (2019), it can be inferred that the comprehensive model is well suited to the set of skills, judgments, and activities (especially for investigating and questioning tasks of receptive skills) as well as expressing desires or attitudes (expressing ideas, creativity, analysis, and other productive skills). In review, the main objectives of this study were to investigate the validity of items and components of the model and also the validity of the tool designed by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) to assess three dimensional critical thinking in EFL readers based on which the following results were identified.

Examining the values obtained from the data, it was observed that the data distribution was not normal. Therefore, in order to assess the validity of this tool, confirmatory factor analysis (CFA) and PLS-SEM was used in SmartPLS software because this method is suitable for abnormal data (Hair et al., 2014) and makes it possible to examine complex models with multiple exogenous and endogenous constructs as well as index variables (Hair Jr. et al., 2010; Hair et al., 2014; Hair et al., 2019). The study of structural equation modeling and covariance relationships and also model evaluation indices clearly showed that the components were selected correctly, the relationships between the components of the model were defined properly and the questionnaire items were well designed, and in this way, the study has reached its objectives.

The six-factor and twenty-two items scale that was proposed by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) has been validated using a hybrid technique mainly due to the existence of abnormally distributed data. Results indicated that the PSL-SEM CFA represented the best fit to the proposed model, in terms of factor loadings. The findings of the first phase of this study indicated the existence of validity between the factors introduced in the three-level model of critical thinking. From the results obtained in this phase, it can be seen that focusing on all the skills and abilities introduced (i.e., argumentation, judgment, disposition, action, social cognition, and creativity) is important in developing critical thinking in English readers.

Discussing the elements of the first movement, a comparison on the criteria was introduced in this study with the ones mentioned in Kaewpet (2018); this can be said that the same measures were mentioned by EFL learners. Focusing on factors of judgements, the elements of buck-passing and vigilance were extracted which were also mentioned by Mann et al. (1997). They also referred to hypervigilance and defensive avoidance which were not mentioned by EFL learners. The last skill of the first movement was disposition which was assessed based on innovativeness, maturity, and engagement as introduced by Ricketts (2003).

In the second movement of developing critical thinking, it was referred to criticality which was mentioned by learners in terms of habitual action, understanding, reflection, and critical reflection. These factors were also used by Kember et al. (2000). The findings of this section, contrary to the view of Shpeizer (2018), in which the two concepts introduced in the first and second movements were considered the same without considerable distinctions, clearly showed that the second movement involves the development of critical actions (and the introduced sub-actions) in individuals, while the first movement does not focus on the development of action skills in individuals. The findings of this study also confirm the views of Wilson and Howitt (2018) based on which they acknowledged that critical thinking in this movement is self-centered and manifests itself in the form of introspection, self-adjusting, and metacognition. The set of abilities acquired at this stage will make a person a prosperous learner, specialist, and scholar, while the first movement focuses on the application of rational-argumentative thinking in the form of training methods and with the aim of improving exactness, proficiency, and creativeness in individuals. Similarly, Ledman (2019) considers this dimension as disciplinary criticality based on which the thinking tools and habits of mind promote epistemological structures.

And the third movement in this study, namely critical pedagogy movement, was composed of the two layers of social cognition and creativity. The first layer was assessed based on factors such as social competence, literacy, cultural competence, and extraversion. The findings of this section are very similar to Pishghadam et al. (2011) criteria in which factors of social competence, social solidarity, literacy, cultural competence, and extraversion were introduced as basic criteria in measuring social cognition. But these findings are in contrast with criteria introduced by Pishvaei and Kasaian (2013) among which are tenets of monolingualism, monoculturalism, native-speakerism, native teacher, native-like pronunciation, and authenticity of native-designed materials quantitatively. Reasons for such a difference may include the nature of the classes, the objectives of the courses, and the interlocutors/participants. These findings are consistent with the works of Davies (2015) and Davies and Barnett (2015) who predicted that critical thinking is not only limited to individual critical thinking skills but also other dimensions such as socio-cultural dimensions and critical pedagogy should also be considered. The last layer was creativity which was assessed based on factors of fluency, flexibility, originality, and elaboration which were also mentioned by O’Neil et al. (1992) and Abedi (2002).

Discussing this movement, the introduced elements of this dimension confirmed the orientations taken by Davies (2015), Davies and Barnett (2015), Rahimi and Asadi Sajed (2014), and Shpeizer (2018) based on which critical pedagogy have impact on critical thinking. According to Shpeizer (2018), the fundamental difference between the two schools of critical thinking discourse and the critical pedagogy is in the contrast between the sociocultural as well as political and moral tendencies in this school and the apparent neutral tendencies of the school of critical thinking. According to Shpeizer (2018) and Freire (1993), in the former, it is not possible to intercept epistemology and politics, and if there is a critical approach, then people’s awareness of power relations and structural inequalities of the societies will be aroused. Shpeizer (2018) adds that, advocates of critical thinking believe that this approach is incompatible, inconsistent, and hazardous since it initially creates uncertain assumptions about a society and thus diverts us from the path of truth-seeking and enlightenment required by a critical thinker. And perhaps the main reason for the slow and one-dimensional movement of critical thinking during all the years can be found in this point. According to Shpeizer (2018) and Rahimi and Asadi Sajed (2014), the proponents of critical pedagogy development argue that since social, political, and educational structures in different societies hitherto run in an inequitable and oppressive manner, disregarding such conditions (which undoubtedly construct the lives and thoughts of individuals) makes objective critical development—and consequently, the progress of community members and communities—impossible. They emphasized that to develop critical pedagogy, it is not possible to teach rational and critical thinking skills and tendencies in individuals without regard to other dimensions such as awareness of cultural, political, and religious. The findings are also in line with Ledman (2019), who states that moral education (the name chosen for the third dimension) emphasizes the need to develop the capacity for moral thinking and judgment independent of official orders and requirements. Finally, by matching the findings of this study with the study of Davies (2015) and Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review), it can be concluded that critical thinking can be defined in three complementary layers; critical thinking skills, criticality, and critical pedagogy. And the more one strives to become more capable in thinking critically, the more s/he moves from gaining initial-personal skills (critical thinker) to socio-cultural skills (critical being).

Regarding the methodology of the study, as explained, due to the fact that the distribution of data obtained from the questionnaire was not normal, the PLS-SEM method was used as a confirmatory factor analysis (CFA) technique. The validation of the model used in this study is based on theoretical and experimental concepts developed in the previous study (Mohammadi et al., Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review). The model validity test was performed in the framework of SEM approach of CFA based on which the investigation of the first degree model (same as the measurement model), and the second degree model (same as the structural model) was conducted. Examination of the absolute values of skewness and Kurtosis as well as data distribution showed that the distribution was not normal; therefore, PLS-SEM confirmatory factor analysis was performed to determine the structural validity of the scale (Mâtă et al., 2020). In addition, the modeling approach is suitable for complex models with multiple endogenous and exogenous index structures and variables (Hair et al., 2014). Also, due to the fact that the sample size in this study is more than the minimum recommended value (i.e. 50), so the most appropriate method for model analysis was considered (Mâtă et al., 2020).

The results of this study provided the next implications: this study investigated a framework of assessing EFL readers who has critical thinking in the three main streams of individual skills, critical pedagogy, and criticality. The results showed that in each of these three main streams, there are criteria that can be used to assess learners’ abilities; Students were interviewed in different phases of the study and offered a variety of views not only on their attitudes toward critical thinking, but also on their perceptions of teaching instructions and the strengths and weaknesses of each, which can provide insights towards designing and implementing critical thinking training sessions; a review of previous literature on three-dimensional critical thinking provided a comprehensive overview of its strengths and weaknesses, as well as the supporters and opponents and finally, the findings of this study were a true validation of the studies confirming the views of all those who agree with the three-dimensional approach to critical thinking under any heading; using the presented concepts in research-academic institutions to identify the most suitable training methods of each of the sub-skills of critical thinking in different societies is very helpful. Given that this study was conducted only in the field of English language and in the university context, its application in other educational spaces and for people with different academic backgrounds and identifying differences in the application of various instructions for each of the sub-skills will be very effective. It is possible to apply both concepts (i.e., three-dimensional critical thinking and reading) in other areas to assess the generalizability of findings. An interesting finding was that in some cases, students engaged in group discussions sometimes returned to their first language, which could be a consequence of poor language proficiency. In such circumstances, Lun et al. (2010) have suggested that in order to promote critical thinking, the emphasis on language processing should be reduced or, on the recommendation of Ko (2013), teachers should first describe the task in order to prepare students and initialize the analysis and then ask them to complete it. The validity of the criterion proposed in the previous study (Mohammadi et al., Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) was evaluated through structural equation modeling, which is a new method and has a very limited history in language studies. This study showed that the method can be used to evaluate path analysis/regression, repeated measures analysis/latent changes modeling, and confirmatory factor analysis.

Conclusion

This study was designed and conducted to confirm the subscales introduced by Mohammadi et al. (Characterization and Development of Critically-thinker EFL Readers’ Reading Ability: Asynchronous Webbased Collaborative vs. Question-Answer-Relationship Instructional Approach, under review) in determining the critical thinking ability in three different layers (i.e., individual critical thinking skills, criticality, and critical pedagogy) through assessing the validity of the proposed questionnaire. The model studied in this study well confirmed the relationship between the factors identified in previous studies and the proposed model with six scales and twenty-two subscales showed a good fit representing that argumentation, judgment, disposition, action, social cognition, and creativity are proper components for measuring three-level critical thinking in language learners.

The results of assessing the validity of CTI through CFA represented that all the three absolute fitness indices, parsimony fit indices, and incremental fit indices have desirable values in the model, and the proposed model is consistent with its experimental model; meaning that the divergent validity of the structure is confirmed. Therefore, combining the results of covariance analysis, the three-level analyses, and the reliability calculations, it can be seen that the questionnaire is valid and reliable. This represents that a critical thinker EFL reader is an individual with the ability to make argumentation (i.e., to find relevance, provide reasoning, recognize language use, comprehend the text’s organization, and distinguish author’s voice), to make judgement (i.e., to pass the buck and vigilant), to provide dispositions (i.e., to innovate, be mature, and engage in doing activities), to act (i.e., to form habitual actions, to understand, to be reflective, and to have critical reflection towards issues), to have social cognition (i.e., to have social competence, literacy, cultural competence, and be extrovert), and to be creative (i.e., to be able to elaborate, be flexible, have fluency, and propose original ideas).

Future research can introduce the extent and manner of internalization of the introduced skills and the effectiveness of different internalization methods. In addition, it should be noted that in this study, the views of language learners were examined. It is necessary to examine the introduced criteria also from the point of view of teachers and administrators in order to answer questions such as the following: Are teachers’ perceptions different from students? If so, what are the differences? What are the effective strategies in teaching these criteria? This type of research can also determine whether students, teachers, and planners have the same understanding of the concepts as well as the strategies used in the classroom. And whether their understanding of the criteria introduced in the first language is the same as in the second language? Moreover, due to the distribution of the gathered data in this study, the factor analysis method with partial least squares (PLS) approach was used. Subsequent researchers can use other analysis programs, such as LISREL or AMOS, for structural analysis relying on larger communities.

Finally, it is necessary to mention that the generalization of the results of this study to other fields and research communities is not possible due to the limited number of participants and its specific field, and it is recommended that first the necessary research efforts be made to apply this scale in different educational fields and societies in order to have more strength and generalizability.

Availability of data and materials

The datasets generated and analyzed during the current study are not publicly available due to ethical issues and privacy restrictions, but are available from the corresponding author on reasonable request.

Abbreviations

3D:

Three dimensional

AGFI:

Adjusted Goodness of Fit Index

APA:

American Philosophical Association

Argum.:

Argumentation

AVE:

Average variance extracted

CB-SEM:

Covariance-based structural equation modeling

CFA:

Confirmatory factor analysis

CFI:

Comparative Fit Index

CR:

Composite reliability

CTI:

Critical thinking inventory

CulComp.:

Cultural competence

EFL:

English as a Foreign Language

GFI:

Goodness of Fit Index

MIMIC:

Multiple indicators multiple causes

OLS:

Ordinary least square

PET:

Preliminary English Test

PLS-SEM CFA:

Partial Least Squares-Structural Equation Modeling Confirmatory Factor Analysis

Reflec.Think:

Reflective thinking

RMR:

Root mean squared residual

RMSEA:

Root mean square error of approximation

SEM:

Structural equation modeling

Social.:

Social cognition

Soci.Comp.:

Social competence

SPSS:

Statistical Package for the Social Sciences

SRMR:

Standardized root mean squared residual

TLI:

Tucker-Lewis Index

References

  • Abedi, J. (2002). Standardized achievement tests and English language learners: psychometrics issues. Educational Assessment, 8, 231–257. https://doi.org/10.1207/S15326977EA0803_02.

    Article  Google Scholar 

  • Adalı, O. (2010). Interactive and critical reading techniques. Toroslu Library.

    Google Scholar 

  • Adalı, O. (2011). Understand and tell. Pan Publishing.

    Google Scholar 

  • Afthanorhan, W. M. A. (2013). A comparison of partial least square structural equation modeling (PLS-SEM) and covariance based structural equation modeling (CB-SEM) for confirmatory factor analysis. International Journal of Engineering Science and Innovative Technology, 2, 198–205.

    Google Scholar 

  • Airasian, P. W., Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Longman.

    Google Scholar 

  • Akyuz, H. I., & Samsa, S. (2009). The effects of blended learning environment on the critical thinking skills of students. Procedia Social and Behavioral Sciences, 1, 1744–1748.

    Article  Google Scholar 

  • Bag, H. K., & Gürsoy, E. (2021). The effect of critical thinking embedded English course design to the improvement of critical thinking skills of secondary school learners. Thinking Skills and Creativity, 41. https://doi.org/10.1016/j.tsc.2021.100910.

  • Bali, M. (2015). Critical thinking through a multicultural lens: cultural challenges of teaching critical thinking. In M. Davies, & R. Barnett (Eds.), The Palgrave handbook of critical thinking in higher education, (pp. 317–334). Palgrave Macmillan.

    Chapter  Google Scholar 

  • Bentler, P. M. (2000). Rites, wrongs, and gold in model testing. Structural Equation Modeling, 7, 82–91. https://doi.org/10.1207/S15328007SEM0701_04.

    Article  Google Scholar 

  • Berardo, S. (2006). The use of authentic materials in the teaching of reading. The Reading Matrix, 6(2), 60–69.

    Google Scholar 

  • Birjandi, P., Bagheri, M. B., & Maftoon, P. (2018). The place of critical thinking in Iranian Educational system. Foreign Language Research Journal, 7(2), 299–324. https://doi.org/10.22059/JFLR.2017.236677.353.

    Article  Google Scholar 

  • Bloom, B. S. (1956). Taxonomy of educational objectives, handbook 1: Cognitive domain. Longmans Green.

  • Brookfield, S. (2019). Using discussion to foster critical thinking. In D. Jahn, A. Kergel, & B. Heidkamp-Kergel (Eds.), Kritische Hpschullehre. Diversitat und bildung im digitalen zeitalter, (pp. 135–151). Springer. https://doi.org/10.1007/978-3-658-25740-8_7.

    Chapter  Google Scholar 

  • Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS and SIMPLIS: basic concepts, applications and programming. Lawrence Erlbaum.

    Google Scholar 

  • Byrne, B. M. (2001). Structural equation modeling with AMOS: basic concepts, applications, and programming. Lawrence Erlbaum Associates Publishers.

    Google Scholar 

  • Byrne, B. M. (2006). Structural equation modeling with eqs: basic concepts, applications, and programming, (2nd ed., ). Lawrence Erlbaum Associates Publishers.

    Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.

    Article  Google Scholar 

  • Chamot, A. U. (2004). Issues in language learning strategy research and teaching. Electronic Journal of Foreign Language Teaching, 1, 14–26.

    Google Scholar 

  • Crenshaw, P., Hale, E., & Harper, S. L. (2011). Producing intellectual labor in the classroom: the utilization of a critical thinking model to help students take command of their thinking. Journal of College Teaching & Learning, 8(7), 13–26. https://doi.org/10.19030/tlc.v8i7.4848.

    Article  Google Scholar 

  • Davies, M. (2015). A model of critical thinking in higher education. In M. B. Paulsen (Ed.), Higher Education: Handbook of Theory and Research, (vol. 30, pp. 41–92). Springer International Publishing.

    Chapter  Google Scholar 

  • Davies, M., & Barnett, R. (2015). Introduction. In M. Davies, & R. Barnett (Eds.), The Palgrave handbook of critical thinking in higher education, (pp. 1–26). Palgrave Macmillan.

    Chapter  Google Scholar 

  • Dekker, T. J. (2020). Teaching critical thinking through engagement with multiplicity. Thinking Skills and Creativity, 37. https://doi.org/10.1016/j.tsc.2020.100701.

  • Din, M. (2020). Evaluating university students’ critical thinking ability as reflected in their critical reading skill: a study at bachelor level in Pakistan. Thinking Skills and Creativity, 35, 1–11. https://doi.org/10.1016/j.tsc.2020.100627.

    Article  Google Scholar 

  • Dornyei, Z. (2010). Researching motivation: from integrativeness to the ideal L2 self. Introducing applied linguistics. Concepts and skills, 3(5), 74–83.

    Google Scholar 

  • Eftekhary, A. A., & Besharati Kalayeh, K. (2014). The relationship between critical thinking and extensive reading on Iranian intermediate EFL learners. Journal of Novel Applied Sciences, 3(6), 623–628.

    Google Scholar 

  • Ennis, R. (1991). Critical thinking: a streamlined conception. Teaching Philosophy, 14(1), 5–24.

    Article  Google Scholar 

  • Ennis, R. H. (2011). Critical thinking: reflection and perspective, Part I. Inquiry: Critical Thinking across the Disciplines, 26(1), 4–18. https://doi.org/10.5840/inquiryctnews20112613.

    Article  Google Scholar 

  • Facione, P. A. (1990). Executive summary of critical thinking: a statement of expert consensus for purposes of educational assessment and instruction. The California Academic Press.

    Google Scholar 

  • Facione, P. A. (1992). Critical thinking: what it is and why it counts. Insight Assessment and the California Academic Press Retrieved May 2019 from http://www.student.uwa.edu.au.

    Google Scholar 

  • Facione, P. A., Facione, N. C., & Giancarlo, C. A. (2000). The disposition toward critical thinking: its measurement, and relationship to critical thinking skill. Informal Logic, 20(1), 61–84. https://doi.org/10.22329/il.v20i1.2254.

    Article  Google Scholar 

  • Fahim, M., & Ahmadian, M. (2012). Critical thinking and Iranian EFL context. Journal of Language Teaching and Research, 3(4), 793–800. https://doi.org/10.4304/jltr.3.4.793-800.

    Article  Google Scholar 

  • Farahani, H. A. (2010). A comparison of partial least squares (PLS) and ordinary least squares (OLS) regressions in predicting of couples’ mental health based on their communicational patterns. Procedia Social and Behavioral Sciences, 5, 1459–1463. https://doi.org/10.1016/j.sbspro.2010.07.308.

    Article  Google Scholar 

  • Freire, P. (1993). Pedagogy of the oppressed. Continuum.

    Google Scholar 

  • Frykholm, J. (2020). Critical thinking and the humanities: a case study of conceptualizations and teaching practices at the Section for Cinema Studies at Stockholm University. Arts and Humanities in Higher Education, 20(3), 253–273. https://doi.org/10.1177/1474022220948798.

    Article  Google Scholar 

  • Gear, A. (2006). Reading power: teaching students to think while they read. Pembroke.

    Google Scholar 

  • Hair, J., Hult, G. T. M., Ringle, C., & Sarstedt, M. (2014). A primer on partial least squares structural equation modeling (PLS-SEM). SAGE Publications.

    Google Scholar 

  • Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of PLS-SEM. European Business Review, 31(1), 2–24. https://doi.org/10.1108/EBR-11-2018-0203.

    Article  Google Scholar 

  • Hair Jr., J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis, (7th ed., ). Pearson Prentice-Hall.

    Google Scholar 

  • Haji Maibodi, A. (2014). The effect of critical thinking skills on reading English novels. Research in English Language Pedagogy, 2(2), 97–108.

    Google Scholar 

  • Haji Maibodi, A., & Fahim, M. (2012). The impact of critical thinking in EFL/ESL literacy. The Iranian EFL Journal, 8(3), 24–44.

    Google Scholar 

  • Hall, L., & Piazza, S. (2008). Critically reading texts: what students do and how teachers can help. Reading Teacher, 62(1), 32–41. https://doi.org/10.1598/RT.62.1.14.

    Article  Google Scholar 

  • Hawkins, K. T. (2012). Thinking and reading among college undergraduates: an examination of the relationship between critical thinking skills and voluntary reading (Unpublished Doctoral Dissertation) University of Tennessee, USA.

    Google Scholar 

  • Heidari, K. (2020). Critical thinking and EFL learners’ performance on textually-explicit, textually-implicit, and script-based reading items. Thinking Skills and Creativity, 37. https://doi.org/10.1016/j.tsc.2020.100703.

  • Hemmati, M. R., & Azizmalayeri, F. (2022). Iranian EFL teachers’ perceptions of obstacles to implementing student-centered learning: a mixed-methods study. International Journal of Foreign Language Teaching and Research, 10(40), 133–152. https://doi.org/10.30495/JFL.2022.686698.

    Article  Google Scholar 

  • Hosseini, E., Bakhshipour Khodaei, F., Sarfallah, S., & Dolatabad, H. R. (2012). Exploring the relationship between critical thinking, reading comprehension and reading strategies of English university students. World Applied Sciences Journal, 17(10), 1356–1364.

    Google Scholar 

  • Imperio, A., Staarman, J. K., & Basso, D. (2020). Relevance of the socio-cultural perspective in the discussion about critical thinking. Journal of Theories and Research in Education, 15(1), 1–19. https://doi.org/10.6092/issn.1970-2221/9882.

    Article  Google Scholar 

  • Johnson, J. L., & Plake, B. S. (1998). A historical comparison of validity standards and validity practices. Educational and Psychological Measurement, 58, 736–753. https://doi.org/10.1177/0013164498058005002.

    Article  Google Scholar 

  • Kaewpet, C. (2018). Quality of argumentation models. Theory and Practice in Language Studies, 8(9), 1105–1113. https://doi.org/10.17507/TPLS.0809.01.

    Article  Google Scholar 

  • Kahsay, M. T. (2019). EFL students’ epistemological beliefs and use of cognitive and metacognitive strategies in Bahir Dar University. International Journal of Foreign Language Teaching & Research, 7(26), 69–83.

    Google Scholar 

  • Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38, 319–342. https://doi.org/10.1111/j.1745-3984.2001.tb01130.x.

    Article  Google Scholar 

  • Kaplan, D. (2000). Structural equation modeling: foundations and extensions. Sage Publications.

    Google Scholar 

  • Kember, D., Leung, D. Y. P., Jones, A., Loke, A. Y., McKay, J., Sinclair, K., & Yeung, E. (2000). Development of a questionnaire to measure the level of reflective thinking. Assessment and Evaluation in Higher Education, 25(4), 382–395. https://doi.org/10.1080/713611442.

    Article  Google Scholar 

  • Khonamri, F., & Karimabadi, M. (2015). Collaborative strategic reading and critical reading ability of intermediate Iranian learners. Theory and Practice in Language Studies, 5(7), 1375–1382. https://doi.org/10.17507/tpls.0507.09.

    Article  Google Scholar 

  • Khorasani, M. M., & Farimani, M. A. (2010). The analysis of critical thinking in Fariman’s teachers and factors influencing it. Journal of Social Science of Ferdowsi University, 6(1), 197–230.

    Google Scholar 

  • Ko, M. Y. (2013). Critical literacy practices in the EFL context and the English language proficiency: further exploration. English Language Teaching, 6(11), 17–28.

    Article  Google Scholar 

  • Koch, T., Eid, M., & Lochner, K. (2018). Multitrait-multimethod-analysis: the psychometric foundation of CFA-MTMM models. In P. Irwing, T. Booth, & D. J. Hughes (Eds.), The Wiley Handbook of Psychometric Testing: A Multidisciplinary Reference on Survey, Scale and Test Development, (vol. VII, pp. 781–846). Wiley-Blackwell Publishing Ltd..

    Chapter  Google Scholar 

  • Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: an overview. Theory into Practice, 41, 212–218.

    Article  Google Scholar 

  • Langer, D. A., Wood, J. J., Bergman, R. L., & Piacentini, J. C. (2010). A multitrait–multimethod analysis of the construct validity of child anxiety disorders in a clinical sample. Child Psychiatry Hum Dev, 41, 549–561. https://doi.org/10.1007/s10578-010-0187-0.

    Article  Google Scholar 

  • Ledman, K. (2019). Discourses of criticality in Nordic countries’ school subject civics. Journal of Humanities and Social Science Education, 3, 149–167.

  • Li, L. (2011). Obstacles and opportunities for developing thinking through interaction in language classrooms. Thinking Skills and Creativity, 6(3), 146–158. https://doi.org/10.1016/j.tsc.2011.05.001.

    Article  Google Scholar 

  • Li, L. (2016). Thinking skills and creativity in second language education: where are we now? Thinking Skills and Creativity, 22, 267–272. https://doi.org/10.1016/j.tsc.2016.11.005.

    Article  Google Scholar 

  • Liang, W., & Fung, D. (2020). Fostering critical thinking in English-as-a-second-language classrooms: challenges and opportunities. Thinking Skills and Creativity, 2020. https://doi.org/10.1016/j.tsc.2020.100769.

  • Lin, M., Preston, A., Kharrufa, A., & Kong, Z. (2016). Making L2 learners’ reasoning skills visible: the potential of computer supported collaborative learning environment. Thinking Skills and Creativity, 22, 303–322. https://doi.org/10.1016/j.tsc.2016.06.004.

    Article  Google Scholar 

  • Liu, F., & Stapleton, P. (2014). Counterargumentation and the cultivation of critical thinking in argumentative writing: investigating washback from a high-stakes test. System, 45, 117–128. https://doi.org/10.1016/j.system.2014.05.005.

    Article  Google Scholar 

  • Lun, V. M.-C., Fischer, R., & Ward, C. (2010). Exploring cultural differences in critical thinking: is it about my thinking style or the language I speak? Learning and Individual Differences, 20(6), 604–616. https://doi.org/10.1016/j.lindif.2010.07.001.

    Article  Google Scholar 

  • Manalo, E., & Sheppard, C. (2016). How might language affect critical thinking performance? Thinking Skills and Creativity, 21, 41–49. https://doi.org/10.1016/j.tsc.2016.05.005.

    Article  Google Scholar 

  • Mann, L., Burnett, P., Radford, M., & Ford, S. (1997). The Melbourne decision making questionnaire: an instrument for measuring patterns for coping with decisional conflict. Journal of Behavioral Decision Making, 10(1), 1–19 https://doi.org/10.1002/(SICI)1099-0771(199703)10:1<1::AID-BDM242>3.0.CO;2-X.

    Article  Google Scholar 

  • Mâtă, L., Clipa, O., & Tzafilkou, K. (2020). The development and validation of a scale to measure university teachers’ attitude towards ethical use of information technology for a sustainable education. Sustainability, 12(15), 1–20. https://doi.org/10.3390/su12156268.

    Article  Google Scholar 

  • McGregor, D. (2007). Developing thinking; developing learning. A guide to thinking skills in education. Open University Press.

    Google Scholar 

  • Merrifield, W. (2018). Culture and critical thinking: exploring culturally informed reasoning processes in a Lebanese university using think-aloud protocols (Unpublished Doctoral Dissertation). George Fox University, College of Education.

    Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement, (pp. 13–103). American Council on Education and Macmillan Publishing Co., Inc.

    Google Scholar 

  • Messick, S. (1995). Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749. https://doi.org/10.1037/0003-066X.50.9.741.

    Article  Google Scholar 

  • Moghadam, Z. B., Narafshan, M. H., & Tajadini, M. (2021). Development of a critical self in the language reading classroom: an examination of learners’ L2 self. Thinking Skills and Creativity, 42, 1–29. https://doi.org/10.1016/j.tsc.2021.100944.

    Article  Google Scholar 

  • Moon, J. (2008). Critical thinking: an exploration of theory and practice. Routledge.

    Google Scholar 

  • Nemat Tabrizi, A. R., & Akhavan Saber, M. (2016). The effect of critical reading strategies on EFL learners’ recall and retention of collocations. International Journal of Education & Literacy Studies, 4(4), 30–37. https://doi.org/10.7575/aiac.ijels.v.4n.4p.30.

    Article  Google Scholar 

  • Nikopour, J., Amini, F. M., & Nasiri, M. (2011). On the relationship between critical thinking and language learning strategies among Iranian EFL learners. Journal of Technology & Education, 5(3), 195–199. https://doi.org/10.22061/TEJ.2011.283.

    Article  Google Scholar 

  • O’Neil, H. F., Abedi, J., & Spielberg, C. D. (1992). The measurement and teaching of creativity. In H. F. O’Neil, & M. Drillings (Eds.), Motivation: Theory and Research, (pp. 245–264). Lawrence Erlbaum.

    Google Scholar 

  • Paul, R., & Elder, L. (2002). Critical thinking: tools for taking charge of your professional and personal life. Financial Times Prentice Hall.

    Google Scholar 

  • Paulsen, M. B. (2015). Higher education: handbook of theory and research. Springer.

    Book  Google Scholar 

  • Peng, J. (2014). Willingness to communicate in the Chinese EFL university classrooms: an ecological perspective. Multilingual Matters.

    Book  Google Scholar 

  • Pica, T. (2000). Tradition and transition in English language teaching methodology. System, 29, 1–18.

    Article  Google Scholar 

  • Pirouz, D. M. (2006). An overview of partial least squares (unpublished doctoral dissertation). University of California.

    Google Scholar 

  • Pishghadam, R., Noghani, M., & Zabihi, R. (2011). The construct validation of a questionnaire of social and cultural capital. English Language Teaching, 4(4), 195–203. https://doi.org/10.5539/elt.v4n4p195.

    Article  Google Scholar 

  • Pishvaei, V., & Kasaian, S. A. (2013). Design, construction, and validation of a CP attitude questionnaire in Iran. European Online Journal of Natural and Social Sciences, 2(2), 59–74.

    Google Scholar 

  • Rahimi, A., & Asadi Sajed, M. (2014). The interplay between critical pedagogy and critical thinking: theoretical ties and practicalities. Procedia - Social and Behavioral Sciences, 136, 41–45. https://doi.org/10.1016/j.sbspro.2014.05.284.

    Article  Google Scholar 

  • Rashel, U. M., & Kinya, S. (2021). Development and validation of a test to measure the secondary students’ critical thinking skills: a focus on environmental education in Bangladesh. International Journal of Educational Research Review, 6(3), 264–274.

    Google Scholar 

  • Renatovna, A. G., & Renatovna, A. S. (2021). Pedagogical and psychological conditions of preparing students for social relations on the basis of the development of critical thinking. Psychology and Educational Journal, 58(2), 4889–4902. https://doi.org/10.17762/pae.v58i2.2886.

    Article  Google Scholar 

  • Rezaei, S., Derakhshan, A., & Bagherkazemi, M. (2011). Critical thinking in language education. Journal of Language Teaching and Research, 2(4), 769–777. https://doi.org/10.4304/jltr.2.4.769-777.

    Article  Google Scholar 

  • Ricketts, J. C. (2003). The efficacy of leadership development, critical thinking dispositions, and students academic performance on the critical thinking skills of selected youth leaders (unpublished doctoral dissertation). University of Florida.

    Google Scholar 

  • Sadler, D. R. (2010). Beyond feedback: developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35(5), 535–550. https://doi.org/10.1080/02602930903541015.

    Article  Google Scholar 

  • Shang, H. F. (2010). Reading strategy use, self-efficacy and EFL reading comprehension. Asian EFL Journal, 12(2), 18–42.

    Google Scholar 

  • Shpeizer, R. (2018). Teaching critical thinking as a vehicle for personal and social transformation. Research in Education, 100(1), 32–49. https://doi.org/10.1177/0034523718762176.

    Article  Google Scholar 

  • Söylemez, Y. (2015). Development of critical essential language skills for middle school students (unpublished doctoral dissertation). Ataturk University Educational Sciences Institute.

    Google Scholar 

  • Suryantari, H. (2018). Children and adults in second language learning. Tell Teaching of English and Literature Journal, 6(1), 30–38. https://doi.org/10.30651/tell.v6i1.2081.

    Article  Google Scholar 

  • Swallow, C. (2016). Reading is thinking. BU Journal of Graduate Studies in Education, 8(2), 27–31.

    Google Scholar 

  • Sweet, A. P. (1993). Transforming ideas for teaching and learning to read. Office of Educational Research and Improvement CS 011 460.

    Google Scholar 

  • Tehrani, H. T., & Razali, A. B. (2018). Developing thinking skills in teaching English as a second/foreign language at primary school. International Journal of Academic Research in Progressive Education and Development, 7(4), 13–29. https://doi.org/10.6007/IJARPED/v7-i4/4755.

    Article  Google Scholar 

  • Tumasang, S. S. (2021). How fear affects EFL acquisition: the case of “terminale” students in Cameroon. Journal of English Language Teaching and Applied Linguistics, 63–70. https://doi.org/10.32996/jeltal.

  • Ullman, J. R. (2001). Structural equation modeling in using multivariate statistics. In B. G. Tabachnick, & L. Fidell (Eds.), Understanding multivariate statistics, (4th ed., pp. 653–771). Allyn & Bacon.

    Google Scholar 

  • Van Laar, E., Van Deursen, A. J., Van Dijk, J. A., & De Haan, J. (2017). The relation between 21st-century skills and digital skills: a systematic literature review. Computers in Human Behavior, 72, 577–588. https://doi.org/10.1016/j.chb.2017.03.010.

    Article  Google Scholar 

  • Wallace, C. (2003). Critical reading in language education. Palgrave Macmillan.

    Book  Google Scholar 

  • Wang, Y., & Henderson, F. (2014). Teaching content through Moodle to facilitate students’ critical thinking in academic reading. The Asian EFL Journal, 16(3), 7–40.

    Google Scholar 

  • Widyastuti, S. (2018). Fostering critical thinking skills through argumentative writing. Cakrawala Pendidikan, 37(2), 182–189. https://doi.org/10.21831/cp.v37i2.20157.

    Article  Google Scholar 

  • Willingham, D. (2007). Critical thinking why is it so hard to teach? Arts Education Policy Review, 109(4), 21–32. https://doi.org/10.3200/AEPR.109.4.21-32.

    Article  Google Scholar 

  • Wilson, A. N., & Howitt, S. M. (2018). Developing critical being in an undergraduate science course. Studies in Higher Education, 43(7), 1160–1171. https://doi.org/10.1080/03075079.2016.1232381.

    Article  Google Scholar 

  • Wilson, K. (2016). Critical reading, critical thinking: delicate scaffolding in English for Academic Purposes (EAP). Thinking Skills and Creativity, 22, 256–265. https://doi.org/10.1016/j.tsc.2016.10.002.

    Article  Google Scholar 

  • Wu, W. C. V., Marek, M., & Chen, N. S. (2013). Assessing cultural awareness and linguistic competency of EFL learners in a CMC-based active learning context. System, 41(3), 515–528. https://doi.org/10.1016/j.system.2013.05.004.

    Article  Google Scholar 

  • Zhao, C., Pandian, A., & Singh, M. K. M. (2016). Instructional strategies for developing critical thinking in EFL classrooms. English Language Teaching, 9(10), 14–21. https://doi.org/10.5539/elt.v9n10p14.

    Article  Google Scholar 

  • Zhou, J., Jiang, Y., & Yao, Y. (2015). The investigation on critical thinking ability in EFL reading class. English Language Teaching, 8(1), 84–93. https://doi.org/10.5539/elt.v8n1p83.

    Article  Google Scholar 

  • Zou, M., & Lee, I. (2021). Learning to teach critical thinking: testimonies of three EFL teachers in China. Asia Pacific Journal of Education. https://doi.org/10.1080/02188791.2021.1982674.

  • Zumbo, B. D. (2005). Structural equation modeling and test validation. In B. Everitt, & D. C. Howell (Eds.), Encyclopedia of Statistics in Behavioral Science, (pp. 1951–1958). Wiley.

    Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

The research received no specific grant from any funding agency in the public, commercial, or non-profit sectors.

Author information

Authors and Affiliations

Authors

Contributions

M.M., G-R. A., and M. S. contributed to the design and implementation of the research, to the analysis of the results, and to the writing of the manuscript. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Gholam-Reza Abbasian.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohammadi, M., Abbasian, GR. & Siyyari, M. Adaptation and validation of a critical thinking scale to measure the 3D critical thinking ability of EFL readers. Lang Test Asia 12, 24 (2022). https://doi.org/10.1186/s40468-022-00173-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40468-022-00173-6

Keywords