Test review: French examination of the College Scholastic Ability Test in Korea

Abstract

This study reviews the French examination of the College Scholastic Ability Test (CSAT), the most influential and highest-stakes test in Korea. The CSAT score has considerable influence on university admissions, and admission to a highly ranked university is generally a prerequisite for socioeconomic success. From this perspective, the significance of the CSAT is enormous. In general, the assessment is influenced by the goals and content of the curriculum, and it also plays an important role in curriculum implementation. In Korea, the national curriculum has been recently reformed. The main goal of the reformed French curriculum is to promote communicative skills and cultural literacy. However, the examination has been criticized for not uniformly or adequately evaluating overall communicative skills. Considering the importance of the CSAT and the interdependence of assessment and curriculum, the CSAT French examination has to be reviewed from the perspective of test quality. Due to its limited assessment of communicative skills, the validity of the French examination is debatable. The problem of low reliability also arises due to the uncertainty of assessment criteria and guidelines. Authenticity and interactiveness could be improved by avoiding fragmentary knowledge and memorization-oriented assessment. The negative washback effect needs to be addressed to resolve the low relevance of the curriculum to the test and the relatively low applicability of the CSAT French score in university admission policy.

Introduction

The purpose of this paper is to review the French examination of the College Scholastic Ability Test (CSAT) in Korea, one of the multiple foreign language subjects on the CSAT. The CSAT, known as Suneung, is a high-stakes test that evaluates students’ academic abilities required for university in Korea. The test has a major impact on Korean society at large in that people’s academic background has a great influence on their social and economic success. This national-level test is administered by the Korea Institute for Curriculum and Evaluation (KICE) and is implemented once a year, generally in November of the last year of high school. There are six evaluation areas: Korean language, mathematics, English, Korean history, social studies and science, and foreign languages. The foreign language subjects include nine languages: French, German, Spanish, Russian, Japanese, Chinese, Arabic, Vietnamese, and Classical Chinese-language literature in Korea.

The foreign language test was first included in the CSAT in 2001 (Lee, 2006). While English language consistently occupies a predominant position as an independent subject in the CSAT, the status of the nine (other) foreign language subjects have reflected contemporary political and social perspectives on internationalization; their status largely depends on foreign policy (Kim, 2018). Since 2016, Korea has joined the International Organization of the Francophonie (IOF) in a bid to enhance international cooperation with Francophonic countries and to encourage the use of the French language in social and economic domains. This means that French language learning and assessment should be considered important from a systemic perspective in public education in Korea and should be strengthened.

Moreover, the national curriculum was recently reformed in 2015, (thus, henceforth, the 2015 curriculum), with the goal of nurturing future-oriented talents suitable for the globalized era (Ministry of Education, 2015). The 2015 curriculum is reflected in the CSAT beginning with the 2021 academic year. While this assessment is influenced by the goals and content of the curriculum, it also plays an important role in curriculum content and implementation (Au, 2007; Das et al., 2014; Islam et al., 2021). It is thus necessary to examine how well the CSAT French examination conforms to the 2015 curriculum that it is meant to evaluate. Furthermore, the nine foreign language subjects in the CSAT operate under the same policy tool of implementation. Therefore, a critical review of the French subject can throw light on the matter of the assessment of other languages. Considering the interdependence of curriculum and assessment, the primary aim of this article is to critically examine and review the characteristics of the French section of the CSAT, within the framework of test usefulness proposed by Bachman and Palmer (1996).

Test formats of the CSAT French examination

The CSAT French examination consists of 30 questions, broken into areas as follows: nos. 1–2, Pronounication and Spelling (2 items); nos. 3–5, Vocabulary (3 items); nos. 6–21, Communicative Functions (16 items); nos. 22–26, Culture (5 items); and nos. 27–30, Grammar (4 items). It is noted that considering that pronunciation, vocabulary, grammar, and even culture can also be parts of communication, it is somewhat unclear by what standards they are classified into different areas of the test (Cho, 2009; Jang, 2006; Porcher, 1994). All questions are multiple-choice questions (MCQs) with five options. A total of 50 points are awarded, with 10 items scored at 1 point and 20 items at 2 points depending on their intended level of difficulty. The duration of the exam is 40 min. The range in terms of topics, vocabulary, and grammar is based on materials covered within the high school curriculum for French 1 as set by the 2015 curriculum. Passages or dialogues for questions can be presented with supporting pictures or drawings, and in the case of words that do not belong to the 2015 curriculum materials, the meanings must be explained in footnotes in Korean.

The CSAT French examination is designed as follows. The pronunciation part consists of questions choosing the same or different spelling pronunciation from five given words. The questions deal with French vowels, consonants, and silent phonemes. Questions on the spelling part are one of the following two types: choosing a sentence including a wrongly spelt word from five given options or those creating a five-letter word by inferring each blanked letter in five words based on the context of passages, then combining them. The vocabulary part consists of questions that involve choosing proper words or expressions for completing sentences in the passage. The questions are generally related to a word’s meaning in context or an idiomatic expression rather than the dictionary meaning. The amount of vocabulary that can be used in the French CSAT is around 800 as stipulated by the 2015 curriculum (Ministry of Education, 2015). The communicative function part contains questions in areas such as choosing proper expressions or idioms for the completion of dialogues, matching information, summarizing, and finding the main idea of the passage. While the types of passages include dialogues, emails, announcements, SNS messages, or commercial messages consisting of 3–4 sentences, all questions are related to the assessment of reading skills. The culture part includes questions about Francophone cultural knowledge, covering a wide range of topics such as language, life, tradition, education, society, art, geography, history, and nature. For instance, for the CSAT French examinations for the 2021 and 2022 academic years, the topics were food, perfume, media, prominent people, nature, social systems, geography, and art and architecture. The questions mostly involve matching information based on passages or dialogues, thus used to assess reading skills as well. The grammar part contains questions such as choosing grammatical or ungrammatical sentences or choosing grammatical items suitable for the blanks in a sentence. The grammar items that can be used in the French CSAT are also specified by the 2015 curriculum. However, since there can be multiple correct answers, this part has the highest level of difficulty and the highest wrong answer rate.

Analysis of the CSAT French examination

This area analyses the CSAT French examination based on Bachman and Palmer’s (1996) test usefulness framework, which reinterprets Messick’s (1989) “one unitary conception, with construct validity as central” (Chapelle, 2012, p. 24). The framework has six criteria: construct validity, reliability, authenticity, interactiveness, impact, and practicality. These are described below.

Validity and reliability

Validity is defined as “the extent to which [a construct] measures what is supposed to measure and nothing else” (Heaton, 1975, p. 153; italics in original). For the present test, validity refers to “the extent to which we can interpret a given test score as an indicator of the ability(ies), or construct(s), we want to measure.” (Bachman & Palmer, 1996, p. 21). In this light, the CSAT French examination has low validity. What the examination constructors want to measure corresponds to the learning goal of the 2015 curriculum for French, which is “to cultivate French communicative skills and Francophonic cultural literacy” (Ministry of Education, 2015, p. 159). However, the CSAT French examination does not properly assess listening, speaking, or writing skills, which are essential for communication. In addition, questions assessing cultural literacy are not meaningfully different from those assessing reading skills, as both include, for example, items summarizing the passages. Although the issue of low validity continued to exist until the 2015 reform, limited studies were conducted (Park, 2006; Yoon, 2012). Even previous studies on cultural literacy focused on a range of topics but not the validity (Jang, 2006; Lee, 2020). Ultimately, the validity of the CSAT French examination, reflected in the 2015 reform, did not improve significantly. Regarding this, the Korea Institute for Curriculum and Evaluation (2020b, 2021d) mentions that a question, such as selecting a suitable expression in a given dialogue or text might assess listening, speaking, and writing skills “indirectly.” However, the score of the indirect assessment cannot be an indicator of the test-taker’s actual skill. Even if the indirect assessment is to forestall the subjectivity in scoring these skills, “the exclusion of these skills due to the subjectivity” […] may cause concern regarding validity (Khodi et al., 2021).

Another concern related to the test’s usefulness is the issue of reliability. Reliability refers to “the consistency of measurement [...] across different characteristics of the testing situation” (Bachman & Palmer, 1996, p. 19). Regarding the reliability of the CSAT French examination, the test consistency fluctuates greatly across an actual test and two mock tests taken before the actual one. For the 2022 academic year, the ratio of total grades accounted for by grades 1 and 2, where top students are distributed, illustrates the concern: 8.04% for the actual CSAT, 14.99% for the first mock exam, and 22.88% for the second mock exam (Korea Institute for Curriculum and Evaluation, 2021a, 2021b, 2021c). In other words, the 2022 CSAT French examination has low reliability. The reason for this is the KICE’s failure to adequately control the difficulty level. This issue concerns the uncertainty of the assessment criteria. The examination guideline states that only the vocabulary and the grammar presented in the 2015 curriculum French 1 can be used in the CSAT French examination. However, there is a huge gap between the curriculum and the CSAT; while around 800 words can be used in the test, test-takers should have knowledge of different meanings or different grammatical usages for individual words, since derivatives, homographs, verb conjugations and declensions applying to nouns, adjectives, and articles are counted as one word. For instance, if grammatical words, such as articles, are presented in the curriculum for French 1, their homographs, with different grammatical usages, can be also used in the test, even if they are not presented in the curriculum (Lee, 2018; Lee, 2022). Consequently, a particular fluctuation is observed in the scores of the grammar part, resulting in the low reliability of the CSAT French examination.

Authenticity and interactiveness

Authenticity refers to the degree of correspondence between a test and an actual language use tasks (Bachman & Palmer, 1996, pp. 23–25). The 2015 curriculum excludes some important grammar frequently used in daily conversation, such as passive voice and relative pronouns (among others). Accordingly, these grammar items cannot be assessed in the CSAT French examination. However, it is questionable whether or not actual communicative skills can be authentically evaluated while excluding important grammar items. Furthermore, as mentioned above, listening, speaking, and writing skills are never assessed in reality. As a result, the authenticity of the CSAT French examination is low in relation to communicative skills.

Meanwhile, for high interactiveness, the test items should activate the test-taker’s individual characteristics, such as language ability, metacognitive strategies, and topical knowledge (Bachman & Palmer, 1996, p. 25). The interactiveness of the CSAT French examination is questionable, as simple memorization can play a crucial role. Due to the importance of the CSAT, the Ministry of Education provides online lectures that focus on preparation for the CSAT and also supports two types of CSAT workbooks designed and written by the KICE. According to the Ministry of Education (2018, p. 7), 50% of the material and vocabulary on the French CSAT examination should be directly or indirectly linked with the two French CSAT workbooks, meaning that the texts, dialogues, multiple-choice options, diagrams, etc. of the workbooks should be presented as they are or in modified or reconstructed forms in the test. This high linkage rate has been the subject of constant controversy, as test-takers can get good grades by simply memorizing entire texts, dialogues, or questions from the workbooks rather than learning them deeply in class (Ministry of Education, 2018). Given that the test only harnesses the learner’s language ability to a limited extent and that minimal metacognitive strategies and topical knowledge are involved, the French CSAT examination has low interactiveness.

Washback

Washback is “the effect of testing on teaching and learning” (Bachman & Palmer, 1996; Hughes, 1989). Since the 2015 curriculum emphasizes communicative skills and cultural literacy, the CSAT French examination should be to bring a positive washback on these points (Khodi et al., 2021; Lee, 2020; Sultana, 2018). However, the French CSAT examination mainly consists of reading comprehension questions. This leads to both teachers and learners intensively focusing their teaching and learning toward reading skills, which in turn means they do not teach or learn such as to improve actual communicative skills or cultural literacy. In addition, the direct or indirect linkage between the CSAT French examination and the CSAT workbooks creates a deeper problem, as noted: Students can get good grades by easily memorizing the materials and types of questions presented in the two workbooks. Actually, due to the low relevance of the curriculum to the test, some teachers teach with workbooks instead of textbooks at school. In other words, the motivation to obtain a good score in the CSAT French examination, which measures limited ability, blurs the essence of teaching and learning emphasized in the 2015 curriculum. The “test based on a narrow definition of language ability,” thus, may cause negative washback (Taylor, 2005).

On the other hand, washback can also be considered in terms of its impact on society and education system (Bachman & Palmer, 1996, p. 35; Das et al., 2014; Islam et al., 2021). In Korea, the CSAT score has a considerable influence on university admission, and admission to a highly ranked university is a prerequisite to and to some extent a guarantee of social success (Park, 2006; Yoon, 2012). In this sense, the CSAT examination might induce a positive washback effect that motivates students to learn. However, this test has a negative washback effect in terms of the evaluation system. The Ministry of Education (2018, 2019) changed the CSAT evaluation in foreign language areas from relative to absolute evaluation starting with the CSAT for the 2022 academic year. The reason for the change in evaluation is that the relative evaluation had problems such as a rush of test-takers to a few specific languages the scores of which can be easily “exploitable” for university admission (Kim, 2019). One case is Arabic: 69.6% of foreign language test-takers chose the Arabic examination among the nine languages in the CSAT for the 2021 academic year (see Table 1). In the relative evaluation, they were evenly distributed across the whole range from 0 to 100 regardless of the overall average of the CSAT score. Thus, the more test-takers there are, the relatively higher the grades they can attain (see Tables 2 and 3).

Table 1 Changes in the number of test-takers for second foreign language subjects in 2021 and 2022 (Korea Institute for Curriculum and Evaluation, 2020a, 2021c)

Table 2 Share of each grade in relative standard (normal distribution)

Table 3 Number of points per grade under absolute standard (total 50 points; Ministry of Education, 2019)

The change in the evaluation system to absolute evaluation had an immediate impact on the admission systems of many major universities for the 2022 academic year, when they decided not to require the scores of the CSAT foreign language examination in the admission process. This is due to the problems of fairness resulting from differences in the difficulty level among the nine languages in the CSAT (Kim, 2019; Ministry of Education, 2018, 2019). The implementation of absolute evaluation had an overall negative effect on the testing rate in the second foreign language area of the CSAT; the overall application (that is, exam-sitting) rate in this area on the CSAT for the 2022 academic year decreased by 5.6% compared to the previous year (Korea Institute for Curriculum and Evaluation, 2020a, 2021c). The uselessness of the test scores in the university admission system undermines students’ motivation to learn French, thus leading to negative washback.

Practicality

Despite problems, such as low validity and reliability, low authenticity and interactiveness, and negative washback, the CSAT French examination is still difficult to change for reasons of practicality (Yoon, 2012). Here, practicality refers to “the relationship between the resources that will be required in the design, development, and the use of the test and the resources that will be available for these activities” (Bachman & Palmer, 1996, p. 39). In other words, in order for the CSAT French examination to attain practicality, the required resources for test development should not exceed the available resources. The CSAT is a paper-based test; however, even if the listening, speaking, and writing parts, which are the most problematic in constructing validity, are tested with a computer, as in CEFR-based exams, there will be practical limitations considering time allotment for administering and scoring, the qualifications and availability of test administrators and scorers, and budgetary issues. Here, the most significant problem common to both options is the short scoring time. In order for the scorers to evaluate the test fairly and objectively, much time will be required to discuss the scoring criteria for the listening, speaking, and writing parts. However, to proceed with university admissions on schedule, the CSAT score report must be issued within 3 weeks of the CSAT. In other words, the required resources are much larger than the available resources. The CSAT French examination thus has low practicality and does not necessarily address the problems with the described test qualities. Nevertheless, it is not impossible. The CSAT French examination can ameliorate the low practicality problem by developing a multiple-choice listening comprehension part and short-answer questions in the listening and speaking parts by using recording equipment or computers. An examination with listening, speaking, and writing will also increase the validity of the test.

Conclusion

This study has reviewed the CSAT French examination through Bachman and Palmer’s (1996) test usefulness concept. Regarding the validity of the test, listening, speaking, and writing parts should be developed using equipment such as recordings and/or computers in accordance with the 2015 curriculum that emphasizes communicative skills. This test format could also resolve the low practicality issue. The problem of low reliability could be solved by establishing clear assessment criteria. The concerns in terms of authenticity and interactiveness should be addressed by including grammar frequently used in daily conversation in the test and by reducing the CSAT’s linkage rate with the workbooks. Lastly, universities should establish more inclusive admission policies which weigh CSAT foreign language examination scores in their consideration of candidates. The overall reconsideration will have a positive washback effect on both teaching and learning motivation and the education system. Reconsidering the solutions described above to the difficulties related to test qualities, the main challenge of the CSAT French examination should be evaluating uniformly and adequately communicative skills and cultural literacy. As far as we know, however, little previous research on the CSAT French examination has been carried out with respect to the theoretical framework of the usefulness concept. This review could have implications for non-English foreign language learning and testing in Korea and other countries, in that it contributes to the normalization of foreign language education by establishing the elements of an appropriate alignment among the curriculum, university entrance examination, and university admission system.