How sociocultural factors mediate washback effect: a study of the Hong Kong Diploma of Secondary Education Examination English Language Paper 4

Abstract

The study explores the unaddressed washback on the English-speaking test of the Hong Kong Diploma of Secondary Education Examination (HKDSE-English Language Paper 4). The HKDSE is a university entrance exam in Hong Kong which includes a group discussion speaking test known as English Language Paper 4. To identify ongoing washback effects, including sociocultural perspectives, the study analyses ongoing student washback of test-specific and non-specific preparation and how the mediating factors affect these preparations. One-hundred and ninety-nine secondary 4th to 6th grade students provided the questionnaire data in two rounds. Results from exploratory factor analysis indicate six major types of test preparation, which are categorised as two test specific, three non-specific, and one integrated type of preparation. Mediating factors comprise five categories of learner characteristics and stakeholders in and out of school. The cluster analyses identified four students’ groups intertwined with the involvement of washback and the mediating factors. A strong relationship between extrinsic factors and explicit learning and intrinsic factors and implicit learning was found. The study concludes that students’ washback is affected by mediating factors aside from the test; learning method choice by students (e.g. learning from entertainment content) has a strong relationship with specific mediating factors (e.g. interest in English).

Introduction

The examination of the sociocultural contexts around test-takers and how they mediate students’ learning have been widely adopted in recent washback studies (e.g. Cheng et al., 2011; Tsang & Isaacs, 2022; Zhan & Andrews, 2014). From the first washback model purposed by Alderson and Wall (1993), decades of ongoing studies (e.g. Bailey, 1996; Booth, 2012; Green, 2006; Saville, 2010; Shih, 2007) have broadly extended the aspects of the washback model. It is suggested that washback should be viewed inclusively, from micro aspects such as ‘test, learner, and teaching’, as well as macro level impact aspects of ‘culture, family and community’ (Saville, 2010, p.4). With regard to washback on learning, the surface of washback (i.e. the contents and methods to learn by test influence) has been the major focus (e.g. Cheng, 1998; Qi, 2004; 2005), but the attention on underlying factors (i.e. the influence of students’ beliefs and sociocultural context) shaping such washback is growing (e.g. Tsang & Isaacs, 2022; Xie and Andrews, 2012). As Shih (2007) proposed, students’ extrinsic factors (e.g. friends, family, school) and intrinsic factors (e.g. students’ interests and beliefs) are nested within their learning actions. Thus, comparing these sociocultural contexts with washback is essential to identify how they mediate students’ learning (e.g. Tsang & Isaacs, 2022).

The present study analyses the relationship between washback on learning and mediating factors (i.e. sociocultural context), following previous studies (e.g. Booth, 2012; Shih, 2007; Tsang & Isaacs, 2022; Zhan & Andrews, 2014) in the high-stakes university entrance test in Hong Kong, the Hong Kong Diploma of Secondary Education Examination (HKDSE). HKDSE became the sole high-stakes test for secondary school students in 2012, replacing two former public examinations (HKALE & HKCEE). English language is one of the core subjects in the HKDSE assessment and has a major impact on stakeholders (e.g. students, teachers, and family) in washback literacy. While previous research has identified washback in the two former and replaced examinations (e.g. Qian, 2008), the HKDSE English language has received limited research focus. This study analyses how mediating factors (e.g. family, classmates, students’ beliefs) influence students’ test preparation by exploring the washback of speaking assessment (Paper 4) in the HKDSE.

Literature review

Washback on learning

Alderson and Wall (1993) proposed the first preliminary washback model on how testing influences teaching and learning, drawing from the empirical studies over the course of three decades. Before and during this period, teaching was the sole focus of washback studies (e.g. Andrews, 1994; Lam, 1994; Watanabe, 1996), while studies focusing on students were scarce (e.g. Tsagari, 2007a, 2007b; Watanabe, 1992). In an inquiry on validity in washback (Messick, 1996), Bailey’s (1996) ‘basic model of washback’ provided an engaging perspective founded on the micro aspects of washback, which shifted the focus of washback studies from teaching to learning, including out-of-class learning. Alderson and Hamp-Lyons (1996) student-centred study was the first to provide empirical evidence of students’ reliance on test material implying teacher influence of introducing test material to students. This study, among others (e.g. Andrews et al., 2002; Green, 2006) provided insight for later studies (e.g. Shih, 2007; Xie & Andrews, 2012) which focused on sociocultural factors shaping students’ learning choice.

Students’ learning is considerably prompted by the test; however, out-of-class learning methods can be varied and can occasionally be non-test-specific (e.g. Pan, 2014; Stoneman, 2006). Washback on learning was considered to be test oriented. Cheng (1997; 1998) provided evidence of students’ preference for test-specific activities over other activities in the classroom environment. This test-specific practice focus was confirmed by subsequent literature in Hong Kong (e.g. Stoneman, 2006; Tsang & Isaacs, 2022; Qian, 2008) and across Asia (e.g. Watanabe, 2004; Xie & Andrews, 2012). However, the use of non-test-specific methods has been observed concurrently with test-specific methods in out-of-class learning (Stoneman, 2006; Pan, 2014; Xie, 2013). This includes general daily practices (e.g. watching movies and listening to English songs), online content (e.g. learning English using software), and interaction with native speakers. Students’ beliefs (e.g. non-test-specific methods can help them improve), time constraints (e.g. using test-specific methods for short-term improvement), and opinions from others (e.g. following the learning method of their friends) are various factors that determine the degree of (non-)test-specific and method adoption (Baba, 2019; Green, 2007; Tsang & Isaacs, 2022; Pan, 2014). In addition, the growing use of out-of-class online learning (Zhan and Andrews, 2014) has been observed with methods such as test-oriented (e.g. test preparation websites), entertainment content (Stoneman, 2006), and game-play contents (Sylvén & Sundqvist, 2012). The variety of out-of-classroom learning can make it difficult to determine the nature of positive or negative washback (i.e. consider as good or bad learning methods); factors such as stakeholders’ (e.g. students, test-makers) intention toward these learning methods (Green, 2007), long- and short-term learning goals (Maxmudjonova, 2023), and internalisation and results of various methods (Andrews et al., 2002) need to be examined. This study embraces the term ‘observable washback’ (Tsang & Isaacs, 2022) as the primary focus to describe learning actions, as positive or negative notions, should be examined carefully and used only after thorough investigation.

Impact and sociocultural washback

Washback on learning is not only affected by the test itself but also driven by the agents and power with and without the students (Andrews et al., 2002; Cheng et al., 2011; Chik & Besser, 2011; Tsang & Isaacs, 2022). The underlying interaction between multiple aspects other than the test itself is often associated with ‘impact’ research (e.g. Booth, 2012; Shih, 2007; Shohamy, 2007; Stoneman, 2006; Tsang, 2017; Xie & Andrews, 2012). The impact research explores mediating factors from various stakeholders (e.g. students, teachers, and family) or involved elements (e.g. test, environment, and society). Shih (2007, 2010) proposed a sociocultural washback model of students’ learning based on various types of mediating factors. His model suggests that students’ learning is not only affected by the test but also determined by the extrinsic and intrinsic sociocultural factors. His study on existing tests in Taiwan, using classroom observations and interviews, found a small but varied washback in parts of the school. Utilising various types of washback, he proposed a model with external (e.g. family and teachers), intrinsic (e.g. language proficiency and motivation), and test (e.g. stakes and content) factors to explain his findings. According to the results, these factors interact differently with the washback effect. This provided insight into how washback is sharpened by these mediating factors rather than solely observing stakeholders’ actions.

An awareness of these mediating factors has led others to focus on selected factors. (e.g. Cheng & Deluca, 2011, Fox and Cheng, 2007; Xie & Andrews, 2012). Fox and Cheng’s (2007) study, utilising class observations and interviews, investigated test preparation for the Ontario Secondary School Literacy Test in Canada. Students’ intrinsic factors were the reason for lower self-perceived language proficiency of English as a second language (ESL) students compared with their native peers. Alternatively, Xie and Andrews (2012) examined other aspects of intrinsic factors in the Chinese College Entrance Test (CET). Their study proposed four intrinsic factors from expectancy-value theory (Jacob & Eccles, 2000) and two related aspects (test design, test use) and their influence on students’ test preparation. These studies, among others, indicated possible intrinsic factors such as language proficiency (e.g., Pan, 2014) and test-taking experiences (e.g. Cheng & Deluca, 2011; Stoneman, 2006) for future research.

In addition to intrinsic factors, Green’s (2006) impact study on International English Language Testing System (IELTS) in a Chinese setting revealed extrinsic factors at the school level. This study was quantitative and revealed test-specific preparation methods adopted by students through rounds of questionnaires. He concluded that teachers’ expectations were behind students’ adoption of test-specific preparation methods. Teachers’ language assessment literacy (Tsiplakou & Tsagari, 2023), teachers’ viewpoints toward the test (Tsagari, 2020), their feedbacks given to students (Khan et al., 2023; Samaie & Valizadeh, 2023), and score-related pressure given to teachers (Ali & Hamid, 2023) can reciprocally determine students’ washback. Chik and Besser’s (2011) study on preparation for an international language test in Hong Kong exhibited other extrinsic factors. Multiple phases of data collection in several settings (e.g. interviews, classrooms, and online observation) suggest that school environment, cram schools, and the Hong Kong media influence Hong Kong learners to some degree. These studies, among others, indicated possible extrinsic factors such as peer information (e.g. test score) from private tutoring (Allen, 2023), peer evaluation (e.g. Black et al, 2003; Cowie, 2005), teacher belief and evaluation (e.g. Gu & Lyu, 2023; Khan et al., 2023), and social language attitude (Shum et al., 2023) for future research.

Similar to Shih (2007), Tsang and Issacs (2022) examined the washback effect in a sociocultural approach that focused on conclusive factors from stakeholders in and beyond the classroom. They found negative washback in learning from the introduction of the new test design, the graded approach in the HKDSE English writing test. They concluded that learners’ perceived washback effects were affected by mediating factors from multiple stakeholders (e.g. teachers, classmates, and family members). For example, learner test preparation using intensive paper and pencil drills was linked to teacher evaluation and learner’s knowledge of the test. All types of test preparations were affected by at least one of the mediating factors, which affirmed the influence of intrinsic and extrinsic factors on HKDSE. He concluded that ‘washback on learning is fundamentally a construct driven by an array of intertwining forces’ (p.59).

HKDSE-English Language Paper 4

The Hong Kong Diploma of Secondary Education Examination (HKDSE) was introduced in 2012 as a high-stake and standard reference test for senior secondary schools conducted annually in Hong Kong. Of the three compulsory subjects offered in the HKDSE, English (HKDSE-English) is administered for assessing four objectives: reading, writing, listening, and speaking. The assessment scheme (a total of 100 points) assigns the following weights, 20% to reading, 25% to writing, 30% to listening and integrated skills, 10% to speaking (0% in the 2020 HKDSE), and 15% to school-based assessment (25% in the 2020 HKDSE owing to the cancellation of the speaking test) (Hong Kong Examinations and Assessment Authority, 2020). The focus of this study is HKDSE-English Language Paper 4, which consists of two parts: (A) group interaction (8 min per group of four candidates) and (B) individual response (1 min per candidate). In group interaction, the candidate is required to discuss an assigned topic (10-min preparation time given) with three other test-takers. Little instruction is given during the test, such that the discussion is unrestricted and extemporaneous as ‘you will be… completely responsible for the progress of the discussion’ (Hong Kong Examinations and Assessment Authority, 2019, p.1). The examiner poses questions to individual candidates for which no time or preparation is given before answering. This unique and close to real-life method in a high-stakes test is believed to have sound validity and facilitates positive learning and teaching (Qian, 2008). Furthermore, school-based assessment (SBA) that aligned with the test was used for continuous evaluation and improve school learning (Davison & Hamp-Lyons, 2009; Yu, 2010).

To the best of the researcher’s knowledge, no research has been conducted on the washback on learning from HKDSE-English Language Paper 4. Several researchers have examined previous speaking tests before the new reforms (Andrews, 1994; 2002; Cheng, 1998, 1999; Qian, 2008) or related school assessment section (e.g., Yu, 2010). These have neither focused on teaching nor have examined these tests inclusively with other test areas (i.e. reading and writing). The focus on HKDSE has been scarce for a decade; the latest research has investigated other sections (i.e. writing section) of the test (Tsang, 2017; Tsang & Isaacs, 2022). Thus, there is a significant need for research that reaffirms the washback on learning from the HKDSE-English Language Paper 4 and further examines the mediating factors underlying it.

Washback studies, conducted in Hong Kong (e.g. Chik & Besser, 2011; Stoneman, 2006; Tsang, 2017), provided insights into test-specific and general preparation methods used by local students. Tsang and Issacs (2022) used the term ‘observable washback’ and classified it into two categories: explicit ways of practising (target skills) and implicit ways of practising (target skills). Similar to test-specific and general preparation, explicit ways of practising indicate that learner action or preparation is directly centred, whereas implicit ways of practising indicate that learner action or preparation is indirectly focused on the test. The term ‘observable washback’ is adopted as a neutral category term, aside from positive–negative trichotomy.

In summary, Shih (2007) proposed a sociocultural washback model that provided the fundamental mediating factor theory in this research. Previous washback studies showed evidence that learner perspective, attitude, and their surrounding affect washback equally with the test introduction. Thus, learning toward test does not stand alone but is sharpened by mediating factors. In other words, beyond the test itself, mediating factors for individual learner and society should be examined. This research seeks to examine the relationship between washback and mediating factors (Shih, 2007) empirically then further affirm the previous findings in the HKDSE environment (Tsang & Isaacs, 2022). Under Shih’s model, Tsang and Isaacs’s study using simultaneous multiple regression (SMR) provided a separated but apparent relationship of each types of explicit test preparation to mediating factors. While their result provides meaningful insights into mediating factors, two empirical gaps remain: (1) what types of students exist in the current test environment and how they choose from different test preparation (i.e. assuming students would not only use exclusively one type but also multiplied types of preparation during the test period), reciprocally affected by mediating factors, are unknown. (2). Unlike other studies in Hong Kong (e.g. Stoneman, 2006), general preparation methods were not examined. This raises the question of how students’ implicit learning functions in HKDSE. By filling the research gaps, a broader and student-oriented perspective of washback and mediating factors can be expected.

Research questions

To address the lack of attention into HKDSE and affirm the findings from previous studies, the current study is based on a conceptual replication approach (Tsang, 2017; Tsang & Isaacs, 2022). The following research questions are framed:

• 1. What are the observable washback effects of students (test-takers of HKDSE English Language Paper 4) when preparing for the test outside the classroom?

• 2. How do the mediating factors from previous findings (Tsang, 2017; Tsang & Isaacs, 2022) shape students perceived observable washback effects?

Methods

Participants

The survey data was collected in June, 2020. Two-hundred and thirty-five students from a local secondary school were selected, and using purposeful sampling due to the nature of the school is relevance to the research purpose. The school was chosen as it is government-aided institution and also uses Chinese as the medium of instruction. This is representative for identity of the majority of HKDSE test-takers characteristic, where approximately 71% of schools are Chinese medium-of-instruction (CMI) government aided (Education Bureau, 2022, p.63). Two-hundred and seventeen completed questionnaires were received (39.3% males and 59.7% females) with a response rate of 92.3%. For eliminate false data skewing the results, eighteen questionnaires with identical answers (the same responses for all items) were excluded. Table 1 shows the summary of one-hundred and ninety-nine participants (36.2% Form 4, 27.1% Form 5, and 36.7 Form 6). Prior to the main study, a pilot study (n = 32) was administered in March 2020. These students were also included in the main survey.

Table 1 Demographics of participants

Research instrument

A 5-point Likert scale questionnaire with 52 items was administered (see Appendix A). For test preparation methods, 26 items measuring various extracurricular learning activities were selected from multiple studies (Pan, 2014; Stoneman 2005; Sylvén & Sundqvist, 2012; Tsang, 2017; Tsang & Isaacs, 2022). These included both test-specific (i.e. explicit ways of practising) and test-unspecific (i.e. implicit ways of practising) learning activities. For sociocultural factors, 26 items were selected from the questionnaire in Tsang and Issacs (2022) study to measure mediating factors.

For example, the use of tutorial schools from Pan’s (2009) and Tsang and Issacs’s (2022) findings, preparation through communication from Stoneman’s (2005) finding, or the use of entrainment from multiple findings (Pan, 2009; Stoneman, 2005; Sylvén & Sundqvist, 2012) were included. The siblings mediating factors were excluded from the study because of the unfitting factor analysis in our pilot state. This instrument aims to reveal the relationship between observable washback (i.e. test-specific and test-unspecific extracurricular activities) and mediating factors (Shih, 2007; Tsang & Isaacs, 2022).

Data collection and analysis

A pilot test was conducted with 32 Form 5 student participants in late March 2020 to ensure reliability, item readiness, and wording appropriateness. The overall result indicated high reliability (Cronbach’s alpha = 0.955) and acceptable item readiness (three comments about item readiness in 32 questionnaires).

The main survey was conducted with 235 student participants through Google Forms in mid-May, 2020. This was an intense period for test preparation, as the cancelled 2020 speaking test (Hong Kong Examinations and Assessment Authority, 2020) was originally arranged to take place in May 2020. Owing to the relatively larger number of participants and the cancelled test, the data was further collected online for 2 weeks until late May. The adjusted questionnaires in the main survey were provided in both Chinese and English languages and included 52 items.

Data analysis was performed using IBM SPSS Statistics version 25 (SPSS 25). Two separate sets of exploratory factor analyses (EFA) were conducted: (1) observable washback effects and (2) mediating factors utilising the principal axis factoring method and the Promax rotation method. Sampling adequacy was confirmed using the Kaiser–Meyer–Olkin (KMO) index (both > 0.5) and Bartlett’s sphericity test (both < 0.05). Item extractions were based on Kaiser’s criterion of retaining factors with eigenvalues greater than 1, given that each factor accounts for at least 7% of the total variance (Fields, 2013; Stoneman, 2006, Tsang & Isaacs, 2022). Consequently, six clusters of observable washback and five clusters of mediating factors were identified. The results from the two sets of EFA were further compared using cluster analysis. This study aims to statistically analyse the relationships between the variables.

Results

Research question 1: What are the observable washback effects of students (test-takers of HKDSE English Language Paper 4) when preparing for the test outside the classroom?

To answer RQ1, exploratory factor analysis (EFA) with 26 extracurricular test preparation items was conducted. Sampling adequacy was achieved with a KMO value of 0.850 and significant fit in Bartlett’s test of sphericity (chi-square = 3471.52, p < 0.001). Factor analyses obtained clusters based on Kaiser’s criterion for eigenvalues > 1, and six clusters were extracted, with 70.61% of the variance explained. Table 1 shows the results of exploratory factor analysis (EFA) with 26 items on the extracurricular test: (1) integrated practising English daily and online content (α = 0.88), (2) practising English with test-specific material and drilling (α = 0.87), (3) practising English with communicative purposes (α = 0.84), (4) practising English in digital gaming (α = 0.94), (5) practising English in tutorial classes (α = 0.86), and (6) practising English in entertainment content (α = 0.79).

Five of the six factors were organised into two groups of observable washback: implicit and explicit ways of practising. The first factor, containing both test specific (e.g. using online test preparation website) and test unspecific (e.g. reading English novels, listening to radio programmes), was listed as ‘integrated practice in daily and online contents’, as its nature could not be clearly defined. Table 2 shows that the first group represented the implicit way of practising the target language and included three out of six factors (Factors 3, 4, and 6). The other group represents the explicit ways of practising the target language and included the last two factors out of six (Factors 2 and 5). Factor 1 was placed in the middle because of its integrated nature.

Table 2 Factor analysis of students’ observable washback

Further analysis was conducted on six factors of observable washback to compare the differences. Table 3 presents descriptive statistics for the six factors with Factor 1 as the average. The contents included multiple learning methods (e.g. finding test-related information online, listening to radio programmes, reading newspapers) that correspond to the parallel use of test-related and non-test-related learning. Factor 6, as an implicit factor (practising in entertainment content), was rated as the highest, compared to Factor 2 (practising with test-specific material and drilling), both in the value of mean (mean = 3.87 > 3.55) and the rank test (Z =  − 0.4.83, p < 0.000). This suggests that in HKDSE-English Language Paper 4, students favoured implicit practice methods more than traditional test preparation methods. The comparison of differences in observable washback provides insight into students’ preferences, as the main purpose of the results was cross-analysis with mediating factors.

Table 3 Factor groups of the six observable forms of washback

Research question 2: How do the mediating factors from previous findings (Tsang, 2017; Tsang & Isaacs, 2022) shape students perceived observable washback effects?

The EFA was conducted with 26 mediating factors (Tsang, 2017; Tsang & Isaacs, 2022). The items in the mediating factor effects were adopted from a previous study in an identical environment. Similar to RQ1, an EFA using the extraction method of principal axis analysis with Promax rotation was performed on the 26 items. Sampling adequacy was achieved based on a KMO value of 0.899 and a significant fit on Bartlett’s test of sphericity (chi-square = 4311.68, p < 0.000) in the preliminary analysis. The factor analyses resulted in five clusters which were extracted based on Kaiser’s criterion of eigenvalues over 1, with 72.08% of the variance explained. Table 4 presents the factor loadings after the Promax rotation. Based on the nature of clusters and referring to patterns from the Tsang and Issacs (2022) study, five factors were extracted as follows: (1) family and tutorial influence (α = 0.94), (2) interest in the language (α = 0.89), (3) evaluation made by teachers (α = 0.87), (4) language proficiency (α = 0.81), and (5) peer influence (α = 0.92). The five factors hold a pattern similar to Tsang and Issacs’ findings (eight factors were found); however, some mediating factors were either combined with other factors (e.g. family and tutorial school) or had a weak loading to be independent in EFA (e.g. society influence). The findings from the EFA provided the foundation for research question 2. Combining the analyses with findings from research question 2 (i.e. six types of observable washback), correlation was performed using cluster analysis (Table 5).

Table 4 Descriptive statistics of the six factors of observable washback

Table 5 Factor analysis of mediating factors

Cluster analysis was conducted to identify the patterns in factors of students’ learning and mediating. Six types of students’ washback effect and five categories of mediating factors were obtained based on the clustering. As shown in Fig. 1, the 11 factors are presented in a left-to-right order similar to the previous corresponding order. The results indicate four types of HKDSE test-takers under both washback and mediating factor. The mean of the group provides the degree of the student’s dedication while preparing for the test.

Fig. 1

Cluster analysis of six types of washback and five types of mediating factor

The four clusters are arranged in a descending order based on the number of participants and are characterised as follows:

• 1. Cluster 1 (n = 81) shows balanced means for all factors, exhibiting an above average score. The balanced mean value of learning methods and mediating factors indicates no strong preference or influence. The less significant family mediating factor is similar for all the clusters.

• 2. Cluster 2 (n = 71) shows the lowest average among all clusters. It is characterised by higher preparation using test-specific material and entertainment, corresponding to a higher mediating factor in language interest and self-perceived language proficiency. The results suggest that students are less dedicated to test preparation, and their learning preference is in using test-specific and entertainment content. Strong intrinsic mediating factors suggest that students decide their learning choices.

• 3. Cluster 3 (n = 30) is characterised by higher game-based and entertainment test preparation, a higher mediating factor in language interest, and self-perceived language proficiency. The cluster also has a strong lean, with the lowest tutorial test-preparation and mediating factors in family and peers. The results suggest that students prefer enjoyable daily content and consider it useful for their English learning and test preparation. These results are similar to Cluster 2 in mediating factors. Such implicit learning choices are decided by intrinsic factors, such as students’ language interest and proficiency.

• 4. Cluster 4 (n = 17) is characterised by higher tutorial and test-specific material test-preparation, higher mediating factors, with peer influence being the highest mediating factor. The cluster has relatively lower learning use in game-based and entertainment. The results suggest that the student group is most dedicated to test preparation. The noticeable use of explicit methods (i.e. test-specific material and tutorial school) suggests the students group prefer the ‘traditional way’ in test preparation. Apart from explicit methods, all other learning methods were favoured to some degree. Thus, test-dedicated students use all learning methods in their learning strategy. The extremely strong peer influence suggests that classmates’ opinions affect students’ learning with considerable involvement of study groups.

Discussion

Regarding RQ1, six types of learning method clusters were adopted by the students. Students engaged in autonomous learning using non-test-specific methods (i.e. focus on language and less relevant to test) slightly more than test-specific methods. One of the favoured method clusters was of practising using entertainment content and digital gaming.

Regarding RQ1, learning with test-specific material was favoured by the most and the least dedicated group. Tutorial schools were the least favoured method.

Regarding RQ2, evidence was provided by four student groups of test-specific (explicit) methods being driven by extrinsic factors. Strong peer influence was found in most dedicated and test-specific groups in contrast to groups that preferred non-test-specific methods. The use of test-specific methods was considerably driven by extrinsic mediating factors.

Regarding RQ2, non-test-specific (implicit) methods were driven by intrinsic factors. Groups which used entertainment and game-based methods had high correlation with language interest and proficiency. The extrinsic factors had lesser influence on non-test-specific groups, such as family, teachers, and peers.

This study aims to reveal the underlying sociocultural factors that affect students’ observable washback following the theoretical works of Tsang and Issacs (2022). The results exhibited two major correlations between the students’ observable washback (implicit/explicit) and sociocultural factors. First, students who preferred test-specific methods considered that other opinions (e.g. peers, family, and teachers) affected their choice. This indicates that the influence of extrinsic mediating factors can lead to explicit learning. Second, students who preferred non-test-specific methods made a choice based on their own individual factors. Thus, the use of implicit learning is driven by intrinsic factors and vice versa. The results suggest that students’ learning choices are influenced not only by the test but also the sociocultural factors which greatly affect how students learn.

Tsang and Issacs (2022) had prioritized on explaining the reasons for certain learning actions. This study further distinguishes which student groups could be predicted with mediating factors, in the HKDSE English environment: the most dedicated ‘top students’ groups (Cluster 4), the average group (Cluster 1), the self-driven entertainment group (Cluster 3), and the least dedicated ‘self-driven but do what is necessary’ group (Cluster 2).

The two self-driven groups (Clusters 2 and 3) show a major influence of sociocultural factors, specifically in their language interest and proficiency. The use of entertainment content as a learning method was found in studies in Hong Kong decades ago (Cheng et al., 2011; Stoneman, 2006). Similarly, the focus on test-specific material used was considered an effective score gaining method in the Asia environment (Pan, 2014; Stoneman, 2006). Interestingly, the use of entertainment and test-specific methods for learning in Cluster 2 is different from the others. Thus, lesser-dedicated students select their strategy practically to achieve what is necessary, one ‘enjoyable’ way to learn, and one ‘practical’ way to scoring gaining. This result is consistent with Stoneman (2006) that students preferred such non-test-specific methods and considered them ‘enjoyable’. In the manner of improving the test scores as their short-term (immediate) learning goal, students tend to employ test-specific learning strategies (Dong, 2023; Maxmudjonova, 2023). On the contrary, digital gaming as implicit in English learning has not yet been reported but is emerging from the European environment (e.g. Sylvén & Sundqvist, 2012). The high preference for game-based learning methods reflects that a growing number of students selected practising from games that required English skills as their learning strategy. The game-based learning use in Cluster 3 provides enjoyable ways to learn; this may help serve learners’ long-term goal (Stoneman, 2006). The differences in game type and content can vary the learning quality significantly (Gee, 2007). Students consider it helpful in their test preparation, but the games played should be carefully analysed.

In terms of mediating factors, the two groups prefer entrainment or gaming content based on their language interest and proficiency. In Tsang and Issacs’s (2022) study, these two factors were associated with activities outside the classroom. This suggests that such activities and learning practices of the two groups are autonomous. This preference for autonomous practising, unlike the use of traditional test requires preparation (e.g. the use of test-specific material and tutorial school), is not unique to HKDSE but matches previous washback studies (e.g. Pan, 2014; Stoneman, 2006). Based on the nature of the cluster, such autonomous practice is for learners who are associated with intrinsic factors. Thus, to achieve autonomous learning as washback, increasing students’ strong language interest and understanding their own proficiency level are essential. On the other hand, the result reveals that both groups exhibit lower extrinsic factors, such as family and teachers influence. It thus seems that family and teachers influence are ‘negative’, as they somehow against students choosing the ‘positive’ autonomous practising. It is important to note that such tendency may stem from the discrepancy between the entrainment-gaming contents learning and ‘traditional learning’, while their influence does not necessary against the novel learning approach. For example, teachers influence can improve students’ learning, such as encourage peer/self-evaluation and give constructive feedback (Khan et al., 2023; Samaie & Valizadeh, 2023) and assume teachers to crucial role of monitoring students’ entrainment-gaming learning strategies (Sylvén and Sundqvist, 2012).

The most dedicated student group (Cluster 1) and their noticeable use of test-specific materials and tutorial schools show preferences for traditional methods (Stoneman, 2006; Zhan & Andrews, 2014). Explicit learning in test-oriented drilling, material, and tutorial schools was the most common in Hong Kong and Asian environments (e.g. Allen, 2016, 2023; Stoneman, 2006; Tsang & Isaacs, 2022). This indicates that students consider traditional preparation to be the most useful method for HKDSE. Unlike other groups, dedicated test-takers use implicit and explicit learning inclusively, suggesting they consider both improving their test preparation. The inclusive use of multiple methods, including intergrade learning and game based, reflects the adoption of online learning, as useful test website and enrolling in school are commonly provided to students (Pan, 2014). The result of Cluster 4 resembles Pan’s findings (2014), as students adopted both implicit and explicit learning during their initial years of high school, while during final years (with higher test intensity), highly motivated (dedicated) students adopted more test-specific methods for gaining scores.

In terms of mediating factors, the most dedicated group is affected by language interest and proficiency but largely by extrinsic factors such as peer influence. Strong language interest is common among dedicated test-takers. In contrast, extremely strong peer influence suggests that students share their preferences or even engage in dedicated group to seek the most sophisticated learning methods, thus affecting their preference (i.e. test-oriented material and tutoring). This strong peer influence on extrinsic learning is consistent with Tsang and Issacs (2022); also, student’s preference on relying on peer helps and ideas was suggested (e.g., Black et al, 2003; Cowie, 2005). Furthermore, the result aligns with Allen (2023) that private tutoring in Japan provides academic performance information relative to their peers. This interplay between the uses of private tutoring, peer influence, and tutoring influence exemplifies that mediating factors affect beyond students’ learning choices, with the reciprocal effect can potentially amplify their own influences through the learning process.

Thus, we conclude that our results have been demonstrating the critical role of implicit and explicit mediating factors in determining observable washback, together with the test itself. The widely held belief that the test alone determines its washback is challenged by our finding. We have revealed that mediating factors carry greater influence over students than the test itself, especially if a period has elapsed since the introduction of the test. Our study has illuminated that these washback and mediating factors engage in a reciprocal relationship, rather than a casual one. Such as, students who enjoy entertainment content will have a higher language interest which may lead to demand for more entertainment content or extend implicit learning. These findings confirm the influence of extrinsic and intrinsic mediating factors to students’ learning choices. Importantly, our study advances the understanding of the existing students characteristics by offering a model of where learning and sociocultural contexts engages in high-stakes test environment. This interactive dynamics of mediating factors and student learnings suggests careful analysis of stakeholders elements such as belief and opinions, and their surrounding is needed to achieve positive washback. Future research should examine the complex structure of such reciprocal relationship. To summarise, by redefining the washback concept within the sociocultural context, our study adds to the growing body of research (e.g. Cheng et al., 2015; Shih, 2007; Tsang & Isaacs, 2022) which acknowledges the complex interplay between testing, learning, and social influence. Our findings provide a novel and student-oriented perspective for facilitating positive washback in HKDSE English.

Conclusions

This study has several implications in the field of washback and for Hong Kong educators. Initially, using the classification of observable washback, which objectively observes the nature of students’ test preparation, it contributes to revealing the notion of washback by (a) addressing the lack of studies that investigate HKDSE-English Language Paper 4 after the revision of 2012 and (b) addressing the relationship between washback and mediating factors in the student groups.

This research provides new insights on out-of-class learning methods among current Hong Kong secondary students. The two major results of the study are high preference for implicit (non-test related) methods adopted by majority of students and strong peer influence (extrinsic factors) associated with explicit methods. In the school environment, implicit methods have been extensively adopted by students. This indicates that teachers or course makers can adopt entertainment-related activities for test preparation. New learning methods for test preparation such as digital gaming may extend the possibilities of future washback studies. However, the connection between explicit methods and peer influence should consider how they shape students’ inner groups. Future research can draw conclusions on what constitutes ‘negative washback’ from test-specific methods because the group discussion test method is considered closer to daily speaking than other tests. For students to understand the test, test-related preparations should match to daily speaking, thus raising the question, ‘Is studying for the test (explicit methods) equal to negative washback?’ For defining these test-related preparations in a positive–negative dichotomy of washback, examining only ‘what’ students do (as conducted in the study) is not enough. In a type of learning action (e.g. drilling students on past paper), ‘how’ refers to how they prepare (e.g. is the procedure/material test-specific and authentic to daily content) and how they feel when they prepare (e.g. do they feel bored, anxious, or more motivated) and the ‘why’ behind the selection of a particular method (e.g. is it suggested by teacher/peer?). Further research that conclusively examines all facets is required. To facilitate learning of general English, results from clusters can be improved, which suggests that the use of implicit learning is linked to students themselves, while explicit learning is linked to extrinsic factors such as peers.

It is important to note that the HKDSE also includes a school-based assessment which shares a similar scoring system (Yu, 2010), and a lack of examination of SBA may not fully reveal the ongoing washback.

This study has three major limitations inherent to the quantitative research design, which limits the data use and generalisation. First, the data collection method of using questionnaires limited the representation of the participants. As the questions were close ended, the students could not voice any opinions other than the selected items. There was also a lack of integration as these selected items were adopted from separate literature. Therefore, any correlations or relationships drawn across different sections of the study may not be valid or fully representative. Second, as a conceptual replication, the items of EFA and cluster analysis are partially adopted from previous research. Thus, the validity of the variables may not hold true as in the original research. Finally, the reciprocal relationship between washback and mediating factors requires a structural investigation. A thorough qualitative research involving interviews and continued observation should be conducted in the future to identify the progress and degrees of certain mediating factors influencing students’ learning.

Future studies should employ a mixed-method approach to examine the washback phenomenon. The nature of ‘negative’ washback, to reveal multiple facets underlying learner actions, requires diversified research and a mixed research approach which should include data collection from interviews and class observations and not rely on a self-reported method. A follow-up research with a better and more sophisticated research design is required to confirm research washback finding for HKDSE-English Language Paper 4.