Consequences of test use: a case study of employers’ voice on the social impact of English certification exit requirements in Taiwan

Abstract

Background

This study investigates the social impact of a policy requiring university graduates to pass an English proficiency test by examining the consequences of test use in the workplace in Taiwan.

Methods

Interviews were conducted with 19 business people in charge of recruiting potential employees in 17 industries across Taiwan. All these 19 employers hired the gratudes from a technological university in southern Taiwan. These interviews sought to discover the importance of English certification as an element of job hunting,the opinions of businesses regarding various certification tests, and their attitudes towards the exit requirement.

Results and conclusions

Findings indicate that although these employers were favorably disposed towards this policy, only 13 % of them required English certificates as a hiring criterion. Another finding was that 53 % of employers regarded the certificates as evidence that applicants who possessed them were diligent and likely to be hard-working employees. These informants interpreted tests differently from testers, focusing on cultural notions of what personal qualities tests highlight rather than on language ability. Due to this and other factors, the impact of the test remained weak.

It is widely recognized that tests exert consequences at both the micro level of the individual test taker and the macro level of society as a whole. For example, tests are used not only for selecting candidates for education, employment, and promotion but also for immigration, citizenship or asylum, monitoring the performance of schools and colleges, implementing educational policies, reforming educational systems, and deciding on the distribution of funding (Cheng, 2005; Eades, 2005; McNamara, 2012). The importance of tests in all these areas has led to the investigation of the consequences of test use and the justification of that use to be regarded as vital steps in validating a test (Bachman, 2005; Kane, 2006; Messick, 1989). However, current research on test washback and test impact has focused primarily on teaching and learning in educational contexts conducted within classroom settings (e.g. Cheng, 2005; Green, 2007; Saville & Papp, 2009; Shohamy et al., 1996; Shohamy et al., 2009; Wall, 2005), devoting little attention to broader social consequences. This study aims to explore what consequences an English certification exit requirement policy in Taiwan has had on the workplace.

Background

The economic development and stability of Taiwan is dependent mainly upon international business transactions. According to World Trade Organization (WTO) international trade statistics, Taiwan is the 20th largest exporter in the world (cited in 2013 International Trade Development in Taiwan, Bureau of Foreign Trade, 2013, p.4). In addition, the Taiwan Bureau of Foreign Trade reports that in 2013, export trade contributed up to 60 % of Taiwan’s GDP (ibid., p. 4). Except for the nearly 40 % of its export trade that is conducted with Mainland China, which shares with Taiwan Mandarin as its primary language, the remaining 60 % of Taiwan’s export trade was conducted with the rest of the world, where English is considered a critical medium for global communication (ibid., p. 9).

Given the importance of English for Taiwan, the Taiwanese government has made it clear that students enrolled in institutes of tertiary education should reach a satisfactory level of English proficiency prior to graduation. In 2001, former Taipei mayor and President of Taiwan since 2008 Ma Ying-Jiu¹ asserted that university students should not be allowed to graduate if they possess Chinese and English proficiency deemed to be unsatisfactory due to the fact that Taiwan’s global competitiveness demands mastery of both languages (Chen & Han, 2001). Furthermore, in the same year, the Minister of Education at the time, Tzen Zhi-Lang, called for an improvement in the basic English proficiency of university students and urged interested parties to conduct an investigation into the appropriateness of establishing exit requirements for English (Chen & Han, 2001). Two years later, Huang Rong-Cun, the subsequent Minister of Education, issued a recommendation that such exit requirements be implemented. The stated purpose of these requirements was to ensure that all university graduates attain at least a minimum level of English proficiency before entering the workforce (Yang, 2003).

A large number of tertiary institutions in Taiwan have taken up this call and nearly 90 % of technical universities/colleges in Taiwan have established English certification exit requirements as of 2012. Generally students must choose from an array of English proficiency tests such as the GEPT (General English Proficiency Test), TOEIC (Test of English for International Communication), TOEFL (The Test Of English as a Foreign Language), and IELTS (International English Language Testing System) in order to graduate. If they do not pass a certified test, they are required to take a supplementary course. The most widely taken test is the GEPT, a locally developed English language test, which is skills based and assesses learners’ receptive (reading, listening) and productive (speaking, writing) abilities. Test takers obtain a certificate for passing the GEPT’s elementary, intermediate, high-intermediate, advanced, or superior level (Wu, 2012).

Because more and more universities in Taiwan are adopting English proficiency certification exit requirements, it is an interesting test case for investigating what consequences this policy has had on society at large. In language testing, the examination of social consequences is part of validity research.

Literature review

A historical perspective on test effects

Test effects have been most commonly investigated as “washback” (Alderson & Wall, 1993; Buck, 1988) or “backwash” (Biggs, 1995, 1996; Hughes, 1993) with a focus on teaching and learning. For example, Hughes’s (1993) trichotomy of backwash model describes test effects in terms of “participants”, “process(es)”, and “product(s)”. A test could affect “participants” such as teachers and students, “all of whose perceptions and attitudes toward their work may be affected” (p. 2). Bailey’s (1996) basic model of washback explores washback effects under two categories : “washback to the learner” and “washback to the program” (pp. 264–265). The former refers to the test effects on learners, such as what and how learners learn and the rate/sequence and degree/depth of learning that is influenced by tests. The latter mainly refers to the test effects on teachers, such as what and how teachers teach and how the rate/sequence and degree/depth of teaching is influenced by tests.

However, these models do not take into account how washback impacts society at large. “Impact” (Bachman & Palmer, 1996; Hamp-Lyons, 1997, 2000; McNamara, 2000; Shohamy, 2001; Taylor, 2005; Wall, 1997) has therefore been proposed as a perspective to account for the influence tests exert not only within the classroom but also on the educational context and on society as a whole. As Bachman and Palmer (1996) have indicated, when exploring test impact, both micro effects in the classroom and macro effects on educational systems and society have to be examined. This investigation forms part of a validation of test score use.

The broader concept of validity

Validity is the central concern in any effort to develop a test. Prior to the 1980s, test validation was undertaken by examining the psychometric qualities of a test, utilizing content, construct and external criteria for validation (Chappelle 1999). During the past three decades, the focus of validity has shifted from a wholly technical view to one that encompasses test-use perspectives. Messick (1989, 1996) as well as Kane (2006) stress the importance of test use and emphasize that validity resides in inferences or decisions based on scores, and that testing is not an isolated, value-free matter. “Consequential validity” (Messick, 1989, 1996) has therefore been proposed to evaluate the intended and unintended social consequences of test use when evaluating the validity of a test.

To better understand both the scope and mechanism of test consequences, Pan (Pan: Consequences of test use: Educational and societal effects of English certification exit requirements in Taiwan, Unpublished) combined Bachman and Palmer's concept of micro and macro test effects, Bailey's (1996) basic model of washback, and Hughes's backwash model (1993) to create the model depicted in Fig. 1.

Fig. 1

A holistic model of test effects proposed by Pan (Pan: Consequences of test use: Educational and societal effects of English certification exit requirements in Taiwan, Unpublished)

In the model, at the micro level, test effects consist of washback on teaching, learning, teaching materials, and learning outcomes, whereas at the macro-level test effects consist of impact on educational goals and society. We can consider both “test washback” and “test impact” to be “consequences of test use”. In this study, the terms “consequences of test use,” “impact,” and “washback” are used interchangeably to denote test effects, though the focus of the study is on societal (rather than educational) effects of test use. All of the stakeholders engaged in the test or the test-driven policy are viewed as “participants”. “Mechanisms” refers to the procedures of how a given test and its mediating factors interact with participants.

A number of studies on test consequences have taken an educational perspective and focused on teachers and students in various contexts, such as the US (Alderson & Hamp-Lyons, 1996; Saif, 1999, 2006; Stecher et al., 2004); Australia (Burrows, 1998, 2001, 2004); Europe (Oerke, et al., 2011; Smyth & Banks, 2012; Wall & Horak, 2006); China (Cheng, 2004, 2005; Deng & Carless, 2010; Xie & Andrews, 2012); Japan (Watanabe, 1996, 2004), and Taiwan (Chu, 2009; Pan, 2011; Pan & Newfields, 2012; Shih, 2007). A far more limited number of studies have investigated test effects at the macro level as the next section demonstrates.

Consequences of test use on society

Previous studies on the social dimensions of washback of large-scale tests beyond educational settings have focused on the economic burdens tests can create on families. Wall and Alderson’s (1993) and Wall’s (2005) Sri Lanka washback studies showed that students from poorer families and in schools with fewer resources were unable to prepare well for tests, because books were too expensive or unavailable. Bray (1999), in his review of studies on private tutoring, concludes that tests have resulted in a boom for the private tutoring industry. In addition, private tutoring has led not only to a financial burden on families but also to social inequalities. He explained that richer families can afford one-on-one or small group tutoring for their children, where the tutors usually come to their homes and their children can receive more attention. On the other hand, children from poor families usually have to travel long distances to attend large tutoring classes.

Hamp-Lyons (1997) provides a broader perspective of the social consequences of test use, suggesting that a variety of stakeholders must be investigated to generate a better understanding of how tests affect different strata of society probably because “washback will be most intense where participants work in a context where the perceptions are shared (or dictated) by other participants” (Green, 2007, pp. 24–25). The aforementioned empirical studies lack an understanding of those groups in society that use language test scores as at least one component of driving decisions but who are not actively involved in the construction of test materials and who “may make assumptions about tests, testing processes, and outcomes that are at odds with what is intended or can be endorsed by the language testing community” (Pill & Harding, 2013, p. 312). For example, Nkosana’s (2008) and Akiyama’s (2004) showed that socio-cultural values can influence stakeholders’ perspectives in regard to what a test should or should not contain. In their studies, stakeholders considered assessment of oral/aural skills irrelevant because societal values regarded reading skills as “academic” and indicative of test takers’ diligence whereas speaking skills were considered “too basic to assess” (Nkosana, 2008, p. 305).

This study, therefore, focuses on a stakeholder group that has not been previously researched—namely, businesses. As potential employers of university graduates, businesses are end users of test scores, but very little is known about how they actually use those scores, which is problematic from a validity and a policy perspective.

The validity of these test score uses needs to be investigated since it is not a given that the infererences that these users draw from test performance and the inferences envisioned by test developers are aligned. Misalignments may be due to differences in test makers’ and end users’ understanding of the meaning of scores, which is often conceptualized as end users’ (lack of) assessment literacy (Pill & Harding, 2013). At the same time, assessment literacy is only one possible explanation for valid or non-valid score use as “a range of educational, public, and political influences” (Black & William, 2005, p. 258) also affect how scores are used in specific settings.

From a policy perspective, the overall impact of test score use by employers is particularly relevant in a situation like that in Taiwan, where the explicit focus of the exit requirement policy is to increase the English proficiency of the workforce and thereby the competitiveness of the nation as a whole. Bluntly put, if English proficiency makes no difference to end users, the policy is likely to have little effect. Conversely, if end users place great emphasis on English proficiency, the policy is likely to work as intended.

This study is guided by the following research questions:

• 1. How important do potential employers consider English proficiency to be for applicants?

• 2. How do potential employers assess applicants’ English proficiency?

• 3. What are potential employers’ perceptions of English certificates that applicants hold?

• 4. To what extent do English proficiency certificates sway hiring decisions?

• 5. How do potential employers view the policy of requiring university graduates to have English proficiency exit requirements?

Methods

Subjects

Relying on the lead researcher’s professional networking connections in business, employers were recruited based upon the number of employees at their organizations, the types of industries in which their companies operated, and the location of their businesses to ensure the representativeness and variety of the sample. Each participant was told that his/her name would not be used, and the data collected would be used solely for the purpose of this research. All participants in this study were then asked for their consent to participate by obtaining their signatures on the “Consent form for participants being interviewed” (see Additional file 1), which was also explained in Mandarin by the researcher.

A sample of 19 potential employers at 17 companies were the informants in this study. Among those 17 companies, 12 had branch offices across Taiwan, four were located in the south, and one was in the north. Six of the business representatives interviewed were females, and 13 were males. The number of employees at their companies varied from 10 to 30,000. Nine different types of industry were included ranging from technology, accounting, and hospitality to education. Nearly 100 % of the recently recruited employees were university/college graduates. These companies are categorized in Table 1 by type, number of employees, and location.

Table 1 Type, number and location of industries surveyed in this study

The participants were senior managers in the personnel and management divisions. They had between six and twenty years of experience in their positions. They were interviewed to discover whether students who had passed the exit requirement and received their certificates were preferred, and for what reason(s). Their perceptions of the exit requirements were also investigated.

Instrument

Semi-structured interviews were chosen as the key instrument because it was the intent of this study to collect “in-depth information” from these insiders based on their emic understandings, conceptions, and attitudes about how English certification tests are viewed and used in the workplace. The major purpose of the interviews was, therefore, to explore the 19 potential employers’ opinions on 1) the necessity of English for fulfilling job responsibilities, 2) the criteria for hiring decisions, and 3) the attitude towards English certificates and certification exit requirements (see Appendix 1 for the interview guide).

Data collection

The interviews were conducted in Mandarin (the lead researcher’s and interviewees’ native language) to ensure smooth communication and to minimize any confusion that may have arisen from interviews held in English. Each interview lasted between 15–40 min. One interview was conducted by phone due to a scheduling conflict, while the remaining interviews were conducted in the interviewee’s office. Several follow-up phone interviews took place when clarification of incomplete or confusing responses was necessary. Each interviewee was told the purpose of the research and then was asked for his/her consent to be audio-recorded. Two did not grant such permission, so hand-written notes were taken at those interviews. The electronically recorded dataset was translated from Chinese to English by the lead researcher for further analysis, as described in the following section.

Data analysis

Interview transcripts were analyzed according to the following broad analytical steps proposed by Schidmit (2004). First, the researcher conducted “intensive and repeated reading” (p. 254) of the transcribed interviews. Second, main categories and subcategories were constructed by extracting the themes that emerged from the intensive and repeated reading of the transcripts. Third, using NVivo (Ver. 8), all data were sorted into categories/subcategories to provide responses to the research questions. NVivo is software that supports the sorting and organization of qualitative data such as interviews that were collected for this study. In this study, NVivo was utilized mainly to code the interview data by topic or themes in order to source classifications.

More specifically, the leader researcher first transcribed and translated the transcripts orthographically into nearly 30,000 words in English. In order to ensure the highest level of precision, both Chinese-speaking teachers of English and native English speakers were consulted for sections of the transcriptions that presented a higher degree of difficulty in terms of translation to English. All transcripts were then loaded into NVivo and read repeatedly and intensively before the determination of the main categories that might be able to provide answers to the research questions posed in this study. For example, a main category under the name “assessment methods of English ability” was created to generate an understanding of how the participants evaluated their potential employees’ English levels in terms of their ability to do work that involved English. After a careful reading of the transcript under that main category, five subcategories were defined: applicant’s self-reports, English certificates, oral interviews, internal written tests, and informal corporate assessments.

For some main categories, subcategories were hierarchically ordered, for example, for the overarching category of employers’ perceptions of English certificates held by applicants. The transcripts related to the employers’ perceptions were first marked with the main category: employers’ perceptions of English certificates. Next, through an intensive reading of the transcript under this category, two subcategories were identified: 1) certificates represent English proficiency, 2) certificates do not necessarily represent English proficiency. Following that, the subcategory of “certificates represent English proficiency” was further divided into additional sections: a) certificates indicate English proficiency, and b) certificates indicate work ethic and improve corporate image. Finally, with the assistance of NVivo, the number of different assessment methods of English ability and the frequencies of different types of employers’ perceptions of English certificates could be calculated, and examples could be searched for and provided under the main and subcategories.

Results

Q1. How important do the informants in this study consider English proficiency to be for prospective applicants?

About 53 % (n = 9) of the informants stated that all their employees must hold at least basic English proficiency because their job responsibilities involve some knowledge of English. Table 2 breaks down the responses by industry type, level of English deemed necessary, and the reasons they thought that English was needed.

Table 2 A profile of 9 Taiwanese companies requiring English skills among all employees according to interviews from representatives of each company

Most respondents did not pinpoint the English proficiency required but gave a broad range from basic to advanced, with the specific requirements depending on the demands of the position. Considering that over half of company representatives said that English was a prerequisite to hiring, it seems beneficial to enhance students’ English proficiency so they can be more competitive in the job market.

The remaining 47 % of the businesses (n = 8) required English for some positions. Table 3 provides some general information about those industries in a format similar to Table 2.

Table 3 A profile of 8 Taiwanese companies requiring English among some employees according to interviews from representatives of each company

Many positions at the companies listed in Table 3 require employees to communicate with overseas clients and foreign visitors in English. The ability to read and write in English is considered essential in more professional or supervisory/managerial positions. Although some positions in the companies listed in Table 3 do not require English proficiency, the interviewees in general found English very beneficial for the future development of their companies and it was a criterion for evaluating personnel work performance or for considering promotions.

To sum up, many businesses in Taiwan believe that English is essential and beneficial for the smooth management of some jobs, for corporate development, to enhance corporate image, and to earn promotions, even though English may not be needed on a daily basis for all positions. The ability to communicate is a basic requirement for positions involving contact with overseas clients, and many professional finance and service positions require the ability to read and write in English. This finding is in line with the government’s policy on enhancing English proficiency and points to a likely noticeable impact of the policy.

Q2. How do potential employers’ assess applicants’ English proficiency?

When asked how they assessed job applicants’ English competency for certain posts, the 17 interviewees reported adopting the following methods:

• a. Applicant self-reports of English competency and English certificates: (18 %, n = 3)

In some cases, applicants are required to indicate their levels of English competency on their resumes by utilizing a simple 4-point rubric of “excellent, good, fair, or poor” for each of the four language skills. Applicants are also asked to indicate which English certificates they hold.

• b. Informal corporate assessments: (12 %, n = 2)

A retail manager (B11) stated that he informally assessed the English proficiency of applicants by the universities they had graduated from, and the English proficiency certificates they held.

In a related case, an employer (B6) in a high-technology company also reported using such informal, subjective means of assessing an applicant’s English proficiency by looking at which university he/she had graduated from and what his/her major was. In addition, that informant mentioned that applicants’ English self-introductions provided another rough barometer for him to understand their proficiency.

These informal and admittedly subjective evaluations are given by businesses where English is not in big demand and customers are mostly local people. Such ways of assessing of English proficiency offer an approximate estimate, rather than an objective picture.

• c. Oral interviews (29 %, n = 5)

Some informants mentioned that oral English tests were given during the interview process at their companies. The structure and content of such tests varied from company to company. Informant B15, a general manager at a nation-wide bookshop, asked the applicants to introduce themselves and read the materials her company published to obtain a rough idea of their English proficiency.

B2, a president of a high-technology company, had applicants read invoices or commercial documents. “For me, a written test is not important.” he added. He explained that a person who was good at taking a paper-and-pencil test does not necessarily do well at the work. In his opinion, a written test is no reliable indicator of one’s real ability.

In the same vein, B10 - a general manager at a banking and mortgage company - stated that applicants were given questions in English and required to answer them orally so that their English level could be assessed. Interestingly, this informant also stated that although his company welcomed applications from graduates of local universities/colleges for English-related positions, they preferred applicants with study abroad experience. Such applicants, in his view, would presumably have fewer problems communicating with foreign clients or suppliers at work.

A similar view was expressed by the general manager of a human resources employment agency (B7). This company hired two staff members to take care of overseas clients because they had overseas study experience. This may be an indicator that employers not only value strong language abilities, but also that studying overseas provides applicants with valuable socio-cultural knowledge and experience. This emphasizes the need for language teachers to also develop students’ awareness of foreign cultures and customs and for language tests to assess pragmatic and intercultural skills.

Another interesting finding is that all of these interviewed employers indicated that the English proficiency demonstrated by applicants at the interview was not a major consideration as to whether they would be hired. Their interest in and passion for the position, along with their attitude, influenced hiring decisions much more. This implies that the recruitment process is not primarily based on foreign language ability. One hotel manager (B1) stated flatly that he would not hire a person with very good language ability if he/she did not also possess certain character traits desired among service industry employees. Another employer in a high tech field (B2) echoed this by stating:

Ideally our employees should understand English, but if they are not good at it they can learn, but their professional knowledge and attitude is essential, and we hire them on that basis.

This type of evaluation predominates in businesses where English is a component of job responsibilities, and employers’ subjective impressions determine the levels of the applicants’ English proficiency.

• d. Internal English written test: (29 %, n = 5)

Five informants at companies ranging in size from 250 to 13,000 reported using internally developed written English tests to assess prospective employees’ English. Most informants were reluctant to explain the contents of their tests due to security concerns but Informant B18 mentioned that her company’s test consisted of multiple-choice reading and listening questions.

• e. English certificates: (13 %, n = 2)

Two of the informants said that English certificates were a prerequisite for employment. A general manager (B4) at a bank stated that applicants needed to show proof of their English ability by providing TOEIC or GEPT certificates before applying to become a financial management specialist, a position in which English skills were deemed indispensable. One manager of an educational training institute (B8) indicated that the intermediate GEPT was required for administration and marketing positions because such people needed to handle questions about learning English and preparing for the GEPT from students.

This indicates that graduates with English certificates have an advantage in certain businesses. Large corporations in particular have adopted the certificates as a benchmark for recruitment, while other businesses use certificates solely as additional information. Again, this is in line with policy and suggests that the exit requirement has an impact on the job market.

Q3. What are potential employers’ perceptions of English certificates that applicants hold?

The opinions of 19 business informants regarding English certificates were solicited, and their responses have been grouped into three categories.

• a. Certificates indicate English proficiency

Nine out of 19 (47 %) of the business informants contended that the English certificates help to assess English proficiency. As a personnel leader in the manufacturing industry stated (B18):

How can one know if a person possesses good English ability without any proof? However, by showing a recognized proficiency certificate, or saying how many points on the TOEIC or TOEFL one has achieved, one’s ability is affirmed. This is what many companies seek.

Another manager (B17) shared a similar opinion. He said that the certificates represent a standard to judge English proficiency. In his view, if someone has a certificate in English, it means that the person has reached a certain level of English proficiency. This informant felt that certificates do not fully guarantee that individuals will be able to handle all the tasks they confront that require English. However, he maintained that those with certified levels of English proficiency are easier to train or better able to acquire the English language competency needed at work. On the contrary, without the certificates, the employers have no idea what level of English proficiency an applicant has.

In the view of these informants, English certificates represent a convenient yardstick to gauge whether or not a person’s English has reached a particular standard.

• b. Certificates may not indicate proficiency well

Three informants (16 %) indicated that they had little faith that English certification tests demonstrated English proficiency. They were concerned that passing scores could be attained primarily through test preparation, and that those who did well on tests would not necessarily be able to utilize English in real life situations. In view of this, they administered internal tests or oral interviews to determine the English proficiency of applicants. This group of informants only considered certificates as rough reference points. A president of a high-technology firm (B2) said:

We don’t trust certificates, but we trust specialized knowledge. It’s like a person with a driver’s license who can't drive…. Some people spend a lot of their time preparing for certificate tests, but once something comes up at work that needs to be done, they might not be able to do it. For example, some people with certificates as mechanics are still not able to fix automobile circuit boards… It’s good to get English proficiency test certificates, but having them does not necessarily mean one can communicate well.

This group of employers believes certificates demonstrate the applicants’ ability to perform on paper, but they do not necessarily reflect whether they can apply what they (supposedly) know.

• c. Certificates indicate work ethic and improve corporate image

53 % of the interviewed informants (n = 10) said that they regarded the certificates not only as proof of English proficiency, but also as indicators of positive attributes such as diligence and discipline.

One general bank manager (B10) said that if applicants applied for jobs requiring a good command of English, it was both essential and beneficial for them to hold certain English proficiency test certificates. However, he associated English certificates with a strong work ethic by saying:

We can assume, although not 100 % correctly, that these people [with English certificates] will be more likely to devote themselves to their jobs. Based on my experience, those with good English skills also possess superior comprehension and learning ability in other areas.

Similarly, one general manager of a human resources service company (B7) stated that possessing certificates was an indicator of a person’s commitment to high levels of initiative, as opposed to doing only what was required. In her opinion, all certificates imply a degree of diligence. She considered those with double majors or several certificates as being more diligent and active compared to those without any certificates and only a single major. According to this informant, the more certificates one has earned, the more diligence one is likely to possess.

A general manager of a hotel (B1) said that certificate ownership is a gauge of a person’s capacity for self-development and learning. He explained:

In my case, I’m a certified accountant. Personally, the certificate entails a measure of self-assertion and self-discipline simply by being willing to invest the time to earn this certificate. English proficiency certificates have two meanings: they measure language talent, and they also measure person’s self-discipline and their capacity for self-development. If the person possesses these qualities, they are the sort of talented people we need.

Finally, one general manager of a retail company (B11) indicated that her company encouraged its employees to gain certificates in order to enhance the company’s image.

It is interesting that more than half of the employers seem to regard certificates as evidence that applicants were “diligent” students and hence likely to be hard-working employees. This is far from the purpose of the certificates as envisioned by the test designers. In other words, many business representatives seem to interpret tests differently from test designers. Business representatives seem to be interested in the personal qualities they believe that the tests highlight, rather than only the linguistic ability of the certificate holders. This “folkloric” belief about the meaning of certificates is interesting and problematic from a validation and policy perspective, which we will consider further below.

Q4. To what extent do English proficiency certificates sway hiring decisions?

Two responses patterns to this question were observed: (a) certificates carried some weight in hiring decisions, and (b) certificates carried no weight in hiring decisions, but influenced other corporate decisions.

• a. Certificates carry some weight in hiring decisions

When asked whether applicants with English proficiency certificates had a greater chance of being hired than their counterparts with no certificates, 82 % of interviewees (n = 14) responded positively provided that the applicants were evenly matched in other respects. This finding suggests that English certificates are not the primary criterion for recruitment, although it might be advantageous to have one. For example, a general manager of a human resources service (B7) described the benefits of English certificates in job searches by stating that if two otherwise equal candidates applied for the same position, the person with better English proficiency would get the job. However, she stated that businesses would not hire job applicants only because they are good at English. Instead, educational background, personality, and special skills are taken into account as well. So, a score of 550 on the TOEIC, she said, was not the major factor in hiring, but rather, only one of many factors. A general manager of a hotel (B1) gave a similar example. He said that “it [recruitment] should take into account two factors: one’s personality and character account for 50 % or more, and one’s professional knowledge and language proficiency account for 30–40 %.”

• b. Certificates carry no weight in hiring decisions, but influence other corporate decisions

More than half (58 %, n = 11) of the interviewed employers claimed that English certificates did not sway hiring decisions, but they were beneficial in helping applicants obtain promotions or further training. The following example illustrates this point:

A general manager in a high-technology company (B13) indicated that although his company does not force applicants to pass certain English proficiency tests, those who did so had better opportunities at work, and employees were well aware of the importance of enhancing their English proficiency. For this reason, some ambitious employees went to language institutes or grouped themselves into a class and hired an English teacher to help them with their English. Even though most new college/university graduates at his company do not use much English, acquiring more English proficiency certificates is one way for an employee to demonstrate ambition.

In addition, two informants in the manufacturing and financial fields (B16 & B19) respectively asserted that employees with certificates are given more opportunities to take enrichment courses in Taiwan or overseas. For example, one company offers employees whose TOEIC scores reach a certain range a three-month study abroad option in which salary, tuition and living expenses are provided. These people are also able to take training courses held in different countries, such as Singapore, Australia and America.

One employer (B18) also stated that achieving an unspecified TOEIC score was a requirement for those who wanted to be dispatched overseas for business trips. Another employer (B6) of a high-technology company stated that employees with an unspecified TOEIC score might have the opportunity to participate in overseas conferences in which English is required for communication.

A minority of employers (16 %, n = 3), however, did not consider English certificates to be a factor for promotion; instead, they looked exclusively at employees’ work performance. However, even they believed that people with a satisfactory level of English proficiency had better opportunities at work.

It can be concluded that most employers consider English certificates a positive asset for those seeking job placement, promotion, and further training although English certificates are not the major criterion for recruitment. This use of certificates is likely to greatly weaken policy impact: if English proficiency plays a minor role in hiring, job applicants are not likely to make it an educational priority beyond attaining passing scores.

Q5: How do potential employers view the policy of requiring university graduates to have English proficiency exit requirements?

All 19 business interviewees said they were in favor of the exit requirement policy. 8 informants elaborated, giving two general reasons for their positive views.

• a. The policy enhances students’ English proficiency

Four (21 %) business representatives underscored the importance of using English to communicate with clients and colleagues from abroad. These interviewees contended that satisfactory English competence would help future employees to deal with work related issues more easily, or foster future company development.

A general manager in accounting and finance (B16) lamented how disappointing it was that so many Taiwanese accounting majors did not possess sufficient English proficiency to address work requirements. If their English were better, they would perform better and benefit the company more.

A hotel general manager (B5) indicated that every company wants to hire people with high potential because they wish to improve their public image both domestically and internationally. In light of this, he wants to hire employees who speak good English to promote international business.

• b. The policy establishes objective standards for evaluating English

Due to the high admission rate of Taiwanese tertiary education (almost 98 % of high school students in Taiwan obtain some kind of post-secondary degree), many business informants voiced concern about the English level of most graduates. Three business representatives supported the government’s exit certificate requirement policy because they felt it would ensure that university/college graduates had reached a certain level of English. As the general manager of one hotel (B5) remarked, there is a surplus of universities/colleges in Taiwan, so each year there are more than enough graduates entering the job market. In the absence of an objective standard, proficiency certificates were needed to ascertain graduates’ English ability.

Similarly, a manager of an educational and training institute (B15) noted that many applicants claim to have intermediate level English proficiency on their resumes, but during their interviews, they are often unable to say or write even very simple English words correctly. This suggests they either inflated their language ability to look better than they actually are, or their working English proficiency level is much lower than their passive understanding of the language.

Two business informants claimed that the policy not only helped them be more selective about hiring, but also saved their companies money because less time was needed to hire candidates with suitable English ability. Moreover, in their experience, the task of training employees in foreign language skills was made easier by the requirements in that training in job-specific English skills could reliably assume that candidates already possess a certain level of competence.

Interestingly, a case of using test scores to draw inferences outside those covered by the test construct became apparent in this area as well. A general manager of a high-technology company (B6) regarded the English proficiency exams as a sort of ‘filtering mechanism’ by stating, “If applicants had not been able to learn English well within four years of college, how well can we expect them to perform in the workplace?” This informant added that the level of the proficiency exams was quite basic, and those who had not even passed this were not qualified to be college/university graduates.

Overall, businesses regard the exit requirement policy favorably, but as shown above, its impact on their real-world decisions is limited.

Discussion

The simple first question is: did the policy work? Did the introduction of an English exit requirement for university graduates lead to an increased level of English proficiency in the Taiwanese workforce? Or put in terms of impact: was desired impact generated?

It is impossible to answer this question for the workforce as a whole due to the limited number of participants in this study. However, from employers’ views of exit requirements and the use they make of English certificates, it is apparent that this is a case of weak impact. Job applicants from institutions with exit requirements are not fundamentally advantaged over applicants from institutions without such a requirement. Holding a certification of English proficiency does not necessarily guarantee a job; rather, it is a bonus in cases of all other things being equal. Not surprisingly, perhaps, employers focus primarily on directly job-related skills and knowledge in applicants, such as their accounting, technology, or sales abilities. Except for the few cases where English proficiency is essential for carrying out the responsibilities of a position, it is only one extra qualification that increases the employee’s value for the company, but it cannot replace abilities directly relevant to a position.

Possible factors for this weak test impact on the workplace can be attributed to the conflicts between employers’ perceptions of the proficiency tests and the avowed aims of the test-driven policy. For example, all of the business representatives claimed to support this policy, but not all of their companies actually used English certificates to assess applicants.

This is not to say that the policy has had no effect: there will be cases where having certified English proficiency makes the difference in getting hired or rejected, and employers indicated that employee’s English abilities can have an impact further down their career trajectory when it comes to additional training or international postings. It is just not the case that the exit requirement is a hurdle which only enables highly English proficient applicants to enter the workforce.

And presumably, it was never primarily intended as such a hurdle. The main point of the policy was to raise English levels overall. In the current situation, where some institutions have imposed exit requirements and others have not, having certified proficiency is a distinguishing characteristic in job applications. Once all institutions have exit requirements, this distinction falls away, and applicants can only distinguish themselves through the level of English, i.e., the level of GEPT or the TOEFL score they attained. Given the low proficiency levels required to pass the university exit requirement in most cases, this situation may lead to noticeable increases in overall English proficiency, but such a process would happen over the long term. Washback is not a stable, immediate effect but changes over time as circumstances, test uses and societies as a whole change.

An interesting finding in this study was employers’ interpretation of the meaning of exit certificates as being indicative of employees’ work ethic or contributing to a positive image of the company, which was not a construct that test developers intended. This is reminiscent of Nkosana’s (2008) and Akiyama’s (2004) studies, which demonstrated the effect of socio-cultural values on stakeholders’ interpretations of test scores. Similarly, in the case of Taiwanese employers, passing an English exit requirement was viewed as evidence of tenacity and willingness to do hard work, which are valued personal attributes in employees but again, not as essential as in-depth subject knowledge.

The conflicting score interpretations between test makers and test users can be attritubed to a gap in test users’ language assessment literacy. According to Pill and Harding (2013), the misconceptions of non-testers’ language assessment literacy include their misunderstanding or lack of knowledge of the constructs the tests assess, and of how the tests operate. Some employers’ misconceptions of language assessment concepts in this study might lead to their associating certificates with character traits such as diligence and discipline and could therefore contribute to unintended consequences resulting from the English certification exit requirement policy. From a validation perspective, the use of English test results to draw conclusions about test takers’ diligence is not justified by the information tests provide. To which extent it is still a valid inference would require empirical investigation.

This mismatch between the interpretation of scores and the inferences intended by test developers poses an interesting problem for validity research. Of course, unintended score use would be easy to dismiss as “inappropriate” and irrelevant to the test construct. Such an approach views assessment literacy as something that end users are lacking and experts have, but it is not a very productive stance. End users will make sense of test scores from their socio-historical-cultural perspective and this sense-making exercise cannot be ignored by test providers and assessment researchers. As a first step, we need to understand what unintended interpretations and uses might result from scores, which was one goal of this study. Investigating end users is essential for gaining such an understanding since interpretations of scores are context and value bound, and neither end user contexts nor their values are intuitively available to test developers. Another step is to work for closer alignment between test users’ and test makers’ score interpretations through “greater engagement between language testing professionals and policy makers or other decision-making non-practitioner stakeholders” (Pill & Harding, 2013, p. 399). Holding professional conferences, seminars or workshops that concern the principles and practice of language assessment for these non-practicioners might assist them to have a more thorough understanding of language testing, such as the construct of English proficiency tests, or what proficiency tests aims to measure. This makes validation a process that includes end users as stakeholders, and requires ongoing monitoring of the real-world uses made of test scores. Furthermore, it is suggested that a needs analysis be conducted by test developers to investigate the levels of English proficiency required by different jobs and then build the relationship between the test scores and the levels of English proficiency. This will also help the test users appropriately utilize certified test scores. Alothough there have been a number of investigators into the language of business, such as Forey & Nunan (2002), Gray (2010), Jin and Hamp-Lyons (2015), and Nickerson (2005), none of these researchers explored the English needs of business professionals in the Taiwanese workplace.

Conclusion

Policies using tests as levers to implement educational and political goals do not always work as smoothly and directly as policy makers might intend. In the case of exit requirements in Taiwan the possible coercive strength of the policy is weakened at the essential point of using certifications of English ability for a real-world purpose, i.e., getting a job. If proving their English proficiency was an indispensable step for applicants in obtaining employment, the policy might lead to increased nationwide English levels over time (or just to a great deal of teaching to the test). But since candidates can get jobs without demonstrating high levels of English, the policy’s effect is greatly reduced. An intended tidal wave of washback is reduced to a ripple, and that ripple might even run as a cross-current since employers may not actually look for English skills but rather use certifications as a further piece of evidence that a candidate is hard-working, diligent and motivated. This demonstrates that test consequences can be unintended and unforeseen.

Though this study is exploratory in nature, the data analyzed can serve as a useful baseline study for future research to test the hypothesis that the strength of washback depends on the degree to which the interpretation of scores and the inferences intended by test developers match. Furthermore, it remains to be seen if greater future spread of exit requirements to all institutions leads to a stronger policy effect. An interesting longitudinal research project awaits in investigating changes in washback over a long period. At this point, the leverage hoped for by policy makers remains limited.

Appendix 1

Interview guide for business representatives

• (A) Business representatives’ personal information in regard to their job positions, and the history of their companies.

  • 1. How many employees are there in your company?

  • 2. What does your company do?

  • 3. How long has your company been in operation?

  • 4. What is your position?

  • 5. What percentage of your employees is college or university graduates?

• (B) The necessity of English for completing the job responsibilities

  • 1. Which sectors or divisions require employees with a certain English proficiency, and which sections do not have such a need for their staff?

  • 2. There are a number of English proficiency test certificates such as the GEPT, TOEIC, and TOEFL. With which of these are you the most familiar? Which would you recommend your employees or prospective employees to take?

• (C) Criteria for hiring decisions

  • 1. How do you evaluate your employees’ English levels in terms of their ability to do work that involves English?

  • 2. What do you consider to be very important attributes, such as one’s skills, personality, and English proficiency, when recruiting new employees?

  • 3. Is it an advantage for an applicant to hold an English test certificate?

  • 4. Are employees whose English is better or who have been awarded English proficiency test certificates given more opportunities for promotion?

• (D) The attitude toward English certificates and certification exit requirements

  • 1. An increasing number of universities and colleges have set English exit requirements. Do you support the establishment of English exit requirements? And why?

  • 2. Have you seen any impact on your company brought about by the exit requirements?

  • 3. It is now an age of certificates. Do you recommend that your employees take part in the English proficiency tests and receive certificates such as TOEIC or GEPT?

  • 4. What do you think about employers’ or applicants’ English certificates? Do they demonstrate English abilities at work? Or do these certificates have other meanings?