Diagnostic assessment of novice EFL learners’ discourse competence in academic writing: a case study

Abstract

Discourse competence is an essential component of communicative competence and central to the mastery of academic writing. This article reports a three-tiered diagnostic assessment of two English as a foreign language (EFL) student writers’ discourse competence in terms of textual features, composing strategies, and knowledge about academic writing. Drawing on extensive analyses of their writing samples, writer logs, and semi-structured interviews, we identified their discourse-level weaknesses and examined their writing strategies and academic writing knowledge status to ascertain their relationships with the observed weaknesses. Global coherence, which conveys the macros-level structure of the whole discourse, was found to be a shared problem of the two students. An investigation into their writing processes found that both students were deadline fighters whose poor time management seemed to account for the problems in their texts. Both spent too much time researching content knowledge but did not reserve sufficient time for writing and revision. Having limited knowledge of the disciplinary content and the required genre type and lacking adequate writing strategies were three salient blocks that impeded their composition of disciplinary essays. Based on the above, we propose a three-stage instructional procedure to facilitate novice EFL learners’ development of discourse competence in English academic writing.

Introduction

Discourse competence (DC), with its significant contribution to coherence, cohesion, and overall flow of the text, is considered an essential component of communicative competence and central to the mastery of academic writing (Bachman and Palmer, 1996; Bruce, 2008; Canale and Swain, 1980; Evans and Morrison, 2011; Widdowson, 2015). In linguistics and applied linguistics studies, DC encompasses the knowledge and skills necessary for managing linguistic features and their semantic functions in constructing a connected and coherent discourse within a specific linguistic context. The development of adequate DC is important yet challenging as it entails skilful management of multiple linguistic and semantic variables corresponding to specific writing tasks. When writing in an English as a second (L2) or foreign language (EFL) context, first-year university students often have considerable difficulties at the discourse level as they newly transition from the secondary to the tertiary level and have little knowledge of the academic genres required in university study. They are likely to suffer severe frustration in meeting the demands of extended academic writing. Hence, it is of great value to understand L2/EFL learners’ specific difficulties in developing DC in academic writing to better facilitate their acquisition of this ability.

In the context of China, prior to university study, a large portion of students’ English writing in schools involves the guided practice of timed short compositions (e.g., 250 words) on general topics. Upon entering an English-medium university, however, they are required to write extended academic essays (e.g., 1500 to 2500 words) focusing on discipline-specific topics. The challenges the first-year students face could be overwhelming. Given the conventional length and complexity of academic papers, features at the discourse level are particularly difficult. Instruments purposefully developed to evaluate discourse-level features would be helpful. Diagnostic assessment, i.e., the act of precisely analyzing a problem and identifying its causes for effective treatment (Rupp et al. 2010), provides meaningful guidance for the construction of such tools, especially for an activity as complex as L2/EFL academic writing.

In understanding university students’ writing problems and difficulties, many existing studies have focused on the analysis of written scripts (Bruce, 2016; Bruce and Hamp-Lyons, 2015; Toraskar and Lee, 2016). There are also needs analyses conducted via surveys or interviews (Evan and Morrison, 2011; McNamara et al. 2018). However, there are still few attempts specifically focusing on DC and still less research tracking the actual composing process to identify possible causes and factors for the textual problems observed. Relating students’ problems with potential causes are, however, necessary for planning targeted teaching and learning activities (Alderson, 2005; Harding et al. 2015; Zhao, 2022). In light of the above gap in the literature, the present study conducted a three-tiered diagnostic assessment of student writers’ textual features, composing strategies, and knowledge about academic writing to identify their weaknesses in writing and to investigate potential influencing factors.

Theoretical review and research goals

Murray (1982) said it is as hard to deduce the process from the product as it is to deduce a field of wheat from eating a slice of bread. Thus, effective diagnosis of writing needs to examine both the written products and the composing processes. This section first reviews how DCs, i.e., the discourse level qualities, have been conceived in L2/EFL writing literature and then provides a summary of studies on composing processes in L2/EFL academic writing. Afterwards, a broader metacognitive perspective of writing is introduced for a holistic understanding of potential interactions among discourse features, processing strategies, and other influencing factors in the context.

Theories about DC components

DC is an integral component of intercultural communicative competence. Taken literally, DC means the knowledge and skills necessary for constructing an effective discourse. Byram (1997) elaborated that DC mainly attends to “the ability to use and discover strategies for the production and interpretation of monologue or dialogue texts which follow the conventions of the culture” (p. 84). Byram’s elaboration stressed the conventions of a specific culture in relation to the texts to be produced. With regard to the event of writing, Grab and Kaplan (1996) proposed a model of writing as communicative language use, consisting of three competencies: linguistic, discourse, and sociolinguistic. Their discourse knowledge refers to knowledge of the ways in which cohesive text is constructed, such as recognizing main topics, organizing schemes, transition markers, informational structure, and semantic relations across clauses. Based on Byram (1997) and Grab and Kaplan (1996), it is perceived that recognizing the main topic plays a leading role in building global and local coherence in a discourse. Regarding academic writing in English, topic recognition and its effective building in writing are closely related to the conventions of a specific discipline or subject domain, where discipline or domain conventions are essential. Thus, the incorporation of topic building in DC is a concrete means for writing teachers and researchers to promote “legitimate peripheral participation” (Flowerdew, 2016, p. 9) in the process of academic writing. Hence, the present study conceives topic building as the first component of DC.

Situated in the communicative language perspective, Bachman and Palmer (1996) proposed two types of language competence: organizational competence and pragmatic competence, with the former focusing on grammatical accuracy and the latter on appropriateness in context. Organizational competence is further divided into grammatical (vocabulary, syntax, and phonology/graphology) and textual competence (cohesion and rhetorical organization). The “textual competence” of Bachman and Palmer (1996) contributed to the major components of DC as conceived in the present study, except for topic building (as explained in the previous paragraph) and reader–writer interaction (as will be shown in the following paragraph). In other words, cohesion and rhetorical organization are at the core of DC. These two core components have individual priority functions but are interdependent and mutually inclusive. In general, rhetorical organization can be specified at both the global and local levels, i.e., global and local coherence. In English, cohesion is mainly achieved by three types of cohesive devices: grammatical cohesion, lexical cohesion, and conjunctive cohesion (Halliday and Hasan, 1976). Together, the three devices contribute to local coherence at the paragraph level, disclosing thematic progression patterns concerning the distribution and connection of given/new information between sentences (Weigle, 2002). Global coherence, on the other hand, pertains to the overall conceptual structure of a text, conveying the rhetorical pattern characteristic of the target communicative purpose (Bachman and Palmer, 1996; Lee, 2002; Tardy and Swales, 2008). In the model of Bachman and Palmer (1996), the teaching of global structure and cohesive devices was depicted as showing students how to organize information in a paragraph in terms of a topic sentence, supporting sentences, and conclusions.

In addition, reader–writer interaction is critical in the creation and interpretation of textual features (Bruce, 2016; Bruce and Hamp-Lyons, 2015; Hyland, 2008; Knoch, 2011). Reader–writer interaction mainly informs how writers position themselves and their readers through linguistic devices, such as self-pronouns, reader-pronouns, hedges, and boosters. Self-pronouns and reader-pronouns can convey interactional engagement, while hedges and boosters convey expressions of attitude and degree of commitment to a proposition. These reader–writer interaction features carry considerable information in creating and interpreting individuals’ points of view. Writing can be conceived as an imagined conversation/interaction or a meaning construction between the assumed writer and the intended reader/audience (Bazerman, 1988; Hyland, 2008; Johns, 1986). This suggests that besides text/writer-based features, coherence is also reader-based, intending to meet the intended audience’s expectations. In relation to studies on writing assessment, Bruce and Hamp-Lyons (2015) reinforced the significance of reader–writer interaction features and grouped them into the components of DCs in their rubrics for assessing undergraduate academic writing. Considering the increasing expectation of clear and consistent presentation and interpretation in academic writing, this study also incorporated reader–writer interaction as an important element of DC.

Through synthesizing existing theories in writing research (Grabe and Kaplan, 1996; Hyland, 2008; Lee, 2002; Weigle, 2002), language assessment (Bachman and Palmer, 1996; Bruce and Hamp-Lyons, 2015; Knoch, 2011), and linguistic studies (Byram, 1997; Halliday and Hasan, 1976), Wang and Xie (2022) conceived DC as consisting of five operational and observable categories: topic building, global coherence, local coherence, logical connectives, and reader–writer interaction. Topic building refers to the specification of a research focus and the establishment of an authorial opinion or a main argument to suggest an interesting and meaningful direction. Global coherence conveys the rhetorical pattern or schema characteristic of and appropriate to the target communicative purpose. Local coherence typically involves thematic progression patterns in relation to the distribution of given/new information between sentences. Logical connectives delve into the use of conjunctions and adverbial adjuncts to show the logical or semantic relationships between sentences and paragraphs. Reader–writer interaction informs how writers position themselves and their readers through typical linguistic devices such as hedges, boosters, self-mentions, and reader-pronouns. Based on these five categories, Wang and Xie (2022) further validated 10 specific discourse features for understanding EFL undergraduates’ DC in academic writing. Given its specific focus on the assessment of DC, Wang and Xie (2022) provide the main constructs for the diagnostic assessment of students’ discourse level problems in the present study. Nevertheless, a fine-grained assessment of written features is not sufficient to fulfil the learning-oriented objectives of diagnostic assessment. For the provision of effective treatment measures, it is necessary to examine the writing processes to uncover related contributing factors. A systematic review of L2/EFL writers’ composing processes is presented in the next section.

Studies on L2/EFL learners’ writing processes

L2/EFL writers share composing processes and textual features similar to L1 writers in many important ways (Silva et al. 2001). Both L1 and L2/EFL lines of research found composing to be a recursive process, which normally consists of planning, formulating, and revising (Flower and Hayes, 1981; Roca de Larios et al. 2008; Stapleton, 2010). While keeping its recursiveness and simultaneity features in mind, each of the three phases can be further broken down into sub-entities for a more tangible description and explanation of the composing process. The work of Flower and Hayes (1981) is often considered seminal for the detailed study of the thinking processes in writing. They employed think-aloud protocols to examine native English speakers’ writing process and proposed a cognitive model of planning, translating, and reviewing in composition writing. These composing processes have a hierarchical structure and can occur at any time; each process component can be embedded within another process.

Using a similar think-aloud method but focusing on English as L2 Spanish writers’ timed argumentative writing, Roca de Larios et al. (2008) conceptualized the composing process in terms of seven activities (reading the prompt, task conceptualization, planning, formulation, evaluation, revision, and meta-comments). They found that formulation took up the largest portion of composition time and writers’ L2 proficiency significantly influenced the time allotment across different activities; high-proficiency writers showed a more balanced time distribution than low-proficiency writers.

Using in-depth logs, a questionnaire, and interviews, Stapleton (2010) explored the composing process of an L2 learner writing a 4000-word English essay. He proposed two additional activities in composing an academic essay: research (sourcing, reading, and/or copying information pertaining to the composing task at hand) and collaboration (consulting others). He then analyzed students’ composing behaviors in these six main categories (research, planning, formulation, revision, evaluation, and collaboration). This categorization provided a systemic framework for understanding novice L2 writers’ composing behaviors in academic writing. He found that students allocated considerable time to research and much less to formulation, revision, and evaluation. However, unlike the focus of the present study on first-year undergraduate students, the focus of Stapleton’s study was on a master’s student who has relatively higher language proficiency and a certain amount of research experience as the research was conducted at a time when the participant had already completed her master’s study. While their composing process of writing an untimed English academic essay may be similar, an examination of L2 first-year undergraduate students’ writing process may reveal more of the difficulties they encounter upon entering academia and be of importance to L2 academic writing instructors.

More recently, Tabari (2022) conducted a mixed-methods study to examine the effects of task planning on the accuracy, complexity, and fluency of the written product. He invited a group of undergraduate EFL learners in a local non-English university to perform a picture-based narrative writing task in four conditions with different planning times. He found that task planning could mitigate the content and organization demands on L2 writers’ cognitive processes and consequently enable the allocation of proper attentional resources to other aspects of the writing system and processes, hence enhancing writing products.

Altogether, existing research on the writing process has produced rich and diversified outcomes by integrating both L1 composition studies and L2 studies, by investigating both timed and extended writing, and frequently by investigating individual cases. With the major concern of understanding the process in academic writing, the present study adopts Stapleton’s (2010) model as the analytical framework to probe into EFL learners’ processing strategies. Meanwhile, as effective development of particular competence entails both processing skills and an adequate knowledge base, it is indispensable to understand target EFL learners’ academic writing knowledge status for the provision of effective writing pedagogy. Hereby, metacognition may shed light on the interaction between textual problems, processing activities, and academic writing knowledge. The following section will present the main constructs of academic writing knowledge from the metacognition perspective.

A metacognitive perspective of academic writing knowledge

Metacognition is considered essential to understanding factors influencing L2/EFL learning processes and strategies (Zhang and Zhang, 2018). Writing can be conceived as an applied metacognition where the production of a text results from a person’s goal-directed monitoring and control of their cognitive and affective states (Hacker, 2018). Definitions of metacognition vary, but all definitions involve metacognitive knowledge, experience, and strategies. Considering writing as an applied metacognition, metacognitive knowledge can be understood as corresponding to knowledge about academic writing, metacognitive experience to actual writing practice/processes, and metacognitive strategies to typical writing strategies. Empirical research has consistently found the impact of metacognitive knowledge on writing achievement (Knospe, 2018; Qin and Zhang, 2019; Ruan, 2014). As the previous session explored typical processing activities and writing strategies, this section focuses on academic writing knowledge.

Due to different perspectives on understanding the purposes and contexts of academic language, the conceptualization of academic writing knowledge also varies. A recent theoretical conceptualization of academic writing knowledge is the four-dimensional model: academic linguistic features, genre mastery, reasoning/argumentative strategies, and disciplinary knowledge (Snow and Uccelli, 2009; Zhao and Lyu, 2019). Adopting this four-dimensional model, Zhao and Lyu (2019) examined 177 Chinese undergraduate English major students’ metacognitive knowledge status regarding the four main constructs of academic writing in the EFL context. They found that these students conceived academic writing as mainly consisting of two general dimensions: content knowledge and language skills. These students could not discriminate fine-grained dimensions of content/disciplinary knowledge, genre knowledge, and reasoning and argumentation skills; they particularly lacked sufficient awareness of the subtle aspects of academic genre knowledge, such as citation styles, discourse patterns in particular communities, and voice and identity construction features (confident, objective, prudent, etc.).

The notion of “genre” has multiple interpretations, of which purpose is the central determinant criterion (Bazerman, 1988; Tardy et al. 2020; Tardy and Swales, 2008). Specifically relating to academic genres in university studies, Nesi and Gardner (2012) identified five main purposes and functions of university writing and thereby 13 types of academic assignments (genres) written by undergraduates in EMI (English Medium Instruction) universities in the UK. Among these genres, academic essays are considered the most typical. An academic essay refers to genres whose central purpose is to allow writers to demonstrate their informed and independent reasoning. They also noted that an academic assignment may sometimes be categorized into more than one genre family because it has writing features shared by several types of genres. Gardner et al. (2019) further identified different clusters of linguistic features in relation to academic disciplines, disciplinary groups, level of study, and genre families.

However, the four-dimensional model is derived from an academic language perspective, and its operationalization in practical writing assessment remains unclear. From a writing assessment perspective, rating scales act as the de facto constructs of writing abilities (Knoch, 2011). In L2/EFL writing assessment studies, most evaluation constructs and criteria are developed for short essays written in test conditions (Brown and Abeywickrama, 2013; Jacobs et al. 1981; Kim, 2019; Knoch, 2009). There are yet relatively limited publications on presenting validated assessment rubrics specifically related to academic writing, among which are Bruce and Hamp-Lyons (2015) and Kim (2019). Bruce and Hamp-Lyons (2015) proposed an assessment rubric specially designed for academic writing in EMI universities. Their rubrics assess four main constructs: task fulfilment, DCs, language competencies, and source integration. Kim (2019) developed and validated a set of 35 binary descriptors at five dimensions for the assessment of ESL (English as a second language) academic writing: content fulfilment, organizational effectiveness, grammatical knowledge, vocabulary use, and mechanics. However, her study did not specify source integration, which is an important component, especially in extended academic writing. Incorporating the academic language perspective (Nesi and Gardner, 2012; Snow and Uccelli, 2009; Zhao and Lyu, 2019) and writing assessment framework (Bruce and Hamp-Lyons, 2015; Kim, 2019), the present study conceives academic writing into five main constructs: academic linguistic features, discourse features, disciplinary knowledge, source integration, and mechanics.

Research on the development of DC remains inadequate. There is also a lack of investigation into the interrelations among the production of discourse-level features, the composing processes, and writers’ academic writing knowledge status. The majority of writing studies on EFL learners focus on timed short-essay writing. Studies on longer academic writing are needed to develop a better understanding of EFL learners’ writing processes for authentic writing purposes.

Considering the intimacy between product and process and the intricacy of interaction between processing skills and academic writing knowledge status, we investigated the discourse features in EFL writing via textual analysis of written scripts, investigation of the composing process, and exploration of writers’ academic writing knowledge status. The research was guided by three research questions (RQs):

• (1) What discourse problems do novice EFL undergraduates display in their English essays?

• (2) How do these problematic features arise in the composing process?

• (3) How may students’ academic writing knowledge status affect their choices and decisions in producing these discourse features?

Methodology

Design

This research adopted a case study approach to investigate two mainland Chinese undergraduate students studying in an English as the medium-of-instruction university in Hong Kong. There are three strengths of the case study approach, making it well suited for the present research. First, the case study process allows multi-faceted explorations of complex issues in real-life settings (Yin, 2018). The complexity of discourse-level qualities in extended academic writing is likely to be presented meaningfully within the fullness of cases and the triangulation of multiple sources of data. Second, the unit of analysis in the case study allows in-depth exploration of instances. Our study adopted a homogeneous sampling strategy in recruiting participants in terms of mother tongue, age, major, and years of study in university. This allowed us to gather sufficient in-depth information about the interactions among the textual features, composing processes, and metacognition status and to build a nuanced picture of individual students’ academic writing experience. Third, the case study approach allows qualitative exploration of the underlying reasons contributing to certain problems, which fits well with our research needs to identify factors that influence writers’ creation of discourse-related features.

Focal participants

The participants were recruited from a WeChat group consisting of 500 mainland students currently studying in Hong Kong. The two cases presented in this paper were two year 2 undergraduate psychology students (Lucia and Wei) selected from six case study participants. In the participant recruiting stages, both stated having genuine writing problems/difficulties and communicated strong motivation to improve their writing. Their problems and struggles are very interesting and indicative of students of their like. The two participants were 20-year-old females studying at the same university. They had achieved scores of 126 and 128 (out of a total score of 150), respectively, in the English subject of China’s matriculation examination. Among the volunteers from the WeChat group, we selected two year 2 students majoring in psychology based on the following considerations. First, year 2 students were chosen because they had accumulated some experience in writing academic essays and had more insights to share. Second, compared to science disciplines, such as science studies, environmental studies, and engineering, the discipline of psychology has a heavier demand for students’ academic writing skills. Third, as the research involved longitudinal monitoring, it was important that the case study students were committed so that they could complete the whole research cycle.

Participants’ english academic writing context

The participants were at the stage when they transitioned from an average of 15 years of Chinese-medium instruction at school to a university adopting English as the primary medium of instruction. Prior to university, these students had learnt English for about 10 years but experienced very limited writing. They were required to write descriptions and narratives in 60 words at the primary level, 100 words at the junior secondary level, and up to 250 words at the senior secondary level. They had no experience in writing academic essays up to 3000 words, which is the standard requirement of courses in their university. Because their school English lessons only provided them with knowledge of pronunciation, vocabulary, and grammar, they entered university with little understanding of the academic writing conventions that they were expected to follow. Upon entering university, the participants were required to complete about 30 courses in 4 years, about 4 to 5 courses each semester. Language enhancement courses and general education foundation courses are offered in years 1 and 2. Written assignments, either in English or in Chinese, constitute an integral part of the assessment activities for almost every course.

Instruments

Three instruments were employed in this research: a discourse rubric, a writer log, and a set of semi-structured interview questions. The detailed discourse rubric from Wang and Xie (2022) was adopted to evaluate the participants’ written texts. This rubric assesses the five components of DC in 10 textual features at five levels. The complete set of 10 acronyms and their referential connotations is shown in Appendix 1.

Writing logs and semi-structured interview questions were used to understand the participants’ composing processes. The design and development of the writer log and interview questions were based on the processing model of Stapleton (2010) and the academic writing knowledge categories in Zhao (2019). References were also made to Lee and Mak (2018).

Data collection and analysis

Data collected included student writing samples, writer logs, and multiple semi-structured interviews. Altogether, six essays were collected from these two participants, along with the subject teachers’ grading results. Two essays were written for Basics of Psychology (BP), two for General Education Foundation Course (GE), one for Intercultural Communication (IC), and one for Special Educational Needs (SEN). These essays were chosen because they fit our task selection criteria: academic essay rather than article review or reports, individual work rather than group work, and written in English but not in Chinese. Writing prompts for the four courses are presented in Appendix 2.

To address RQ1, each essay was double-coded using the rubric. Before applying the rubric to the target essays, the first researcher and a second coder trialled it with nine essays selected from a corpus of 372 essays produced by first-year L2/EFL writers. Disagreements were discussed, some descriptors were revised, and a consensus on quantitative criteria for some items was also reached. Then, the two raters independently assessed the essays from the two case study writers. Inter-rater reliability was found to be satisfactory (average Kappa = .649, p < .001, Neuendorf, 2016). The two raters’ disagreements were mostly between the levels of “Basic” (Level 2) and “Fair” (Level 3). These disagreements were further discussed and achieved consensus through further examination of related texts.

Participants’ writer logs were created in a Google doc and shared with the first author. Instructions and sample logs were given so that they could document their processing activities with sufficient detail for the purpose of the research. Participants were allowed to write in either English or Chinese but were required to keep the writing log from the time they received the writing task to the submission of the final paper. They were reminded three to five times every week via WeChat to write on the logs of their progress. Other information (such as their consultation details with their subject teachers) was also collected whenever necessary. Altogether, 17 entries of writer logs were collected, along with consultation documents and WeChat messages.

Semi-structured interviews were conducted with each participant four times. In a pre-writing interview, students were guided to share their general understanding of academic writing and writing strategies, as well as their perceptions about their own writing performance. The two while-writing interviews probed more details about how and why they adopted the strategies reported in their logs. The post-writing interview focused on an overall reflection on how successfully they believed they had solved the problems in their writing.

The interview recordings were first transcribed using the automatic transcribing software Xunfei and then verified and refined via human listening and editing. This generated an interview document of about 74,000 words in the language of the interview (Mandarin Chinese). Afterwards, the interview transcripts and writing log data followed a similar procedure, though with different focuses. Coding of the interview transcripts focused on the composing strategies and individual academic writing knowledge; coding of the writing logs focused on the six composing processes. They were coded in the Word document first before being transformed into the txt. format for further analysis in AntConc 3.5.8. The functions of Concordance and File View of AntConc were employed to synthesize and visualize the coding data (Anthony, 2019).

Before coding, a coding manual and a coded sample were generated, which included six typical processing activities. More codes/themes and sub-codes were generated in the continuing coding process (see Appendix 4 for details). Two researchers coded the first two interview data (about 20,000 words) independently, discussed the differences, made refinements and clarifications for each code, and finalized the coding framework. The final coding framework had a total of 85 codes (19 for research, 9 for planning, 8 for formulation, 3 for revision, 1 for evaluation, 8 for collaboration, 24 for academic writing knowledge, 3 for attitude/emotions, 2 for motivation, and 8 for other). The inter-coder reliability reached 82%. They went on coding the remaining essays independently and discussed their final codes to resolve the differences.

The following section reports the main findings regarding the discourse features of the essays written by the two cases, their processing strategies, and metacognitive knowledge.

Results

This section first reported the students’ overall discourse features and then reported findings from individual cases, respectively. Within each case, the relative strengths and weaknesses of the students’ written texts were first introduced, and their problematic discourse features and weaknesses were further examined in relation to individual processing strategies as uncovered in their composing processes and post-writing reflections/interviews. This three-layered presentation of the results corresponded to the three RQs introduced earlier in the literature review section.

Table 1 presents a summary of their discourse features. Both students demonstrated strong and consistent performance on three features (F8, F9, and F7) under the components of local coherence and logical connectives. Their performance on the other features is less strong and more fluctuated, particularly in body paragraphs (F5), controlling idea (F3), topic/focus (F1), and conclusion (F6). The following sections report the findings of discourse problems in relation to their processing strategies and metacognitive knowledge about writing.

Table 1 Results from essay coding

Discourse features and the composing process in Lucia’ case

The three essays collected from Lucia were written in year 1 (BP and GE) and year 2 (IC). Lucia demonstrated consistent fair/good performance on F8 (connective complexity) and F9 (connective accuracy) but also consistent basic/poor performance on F2 (thesis statement), F3 (controlling idea), F5 (body paragraphs), and F6 (conclusion). Her performance on the other aspects of topic building and global coherence remains unstable, indicating that Lucia may have problems with the two inter-related disciplinary fields. In her IC essay, 4 out of the 10 discourse features were diagnosed as “poor 1 or basic 2” (F2, thesis statement; F3, controlling idea; F5, body paragraphs; and F6, conclusion), and the other six as “fair 3” (F1, topic/focus; F4, introduction; F7, theme-rheme development; F9, connective accuracy; and F10, complexity of hedges and boosters).

Lucia’s essay for the IC course was written in the first semester of year 2. The assignment requires students to “critically evaluate the values of your own culture (i.e., Mainland China, Hong Kong, Nepal, Germany) in about 1500 words”. Between the release of the task and the submission date, Lucia had 14 days (see Table 2), of which she spent the largest chunk (D1–11) on researching, including working out an outline, designing and distributing a questionnaire, and analyzing data to generate evidence.

Table 2 Lucia’s processing strategies for the IC essay

Upon receiving the writing task, she began searching the Internet and checking lecture notes to review theories and identify a specific focus for her writing. After about 5 h of work, she devised an outline on day 3. After consulting her course instructor, Lucia designed a questionnaire and collected data online in the following days. On day 13, 1 day before submission, Lucia was still analyzing the data. She produced only a rough draft on the morning of the due date (D14) and had little time left for evaluating and revising it.

One of the problematic discourse features in Lucia’s essay is controlling idea (F3) under the component of topic building, which seems related to the research process, especially researching and interpreting the written instruction to identify the topic/focus. Lucia had an idea about focusing on filial piety when she saw the assignment task. Compared to other topics and concepts introduced in this course, she felt that she had knowledge of this cultural value and that it was also a topic of many people’s concern. She searched online forums for 1 h, collected information concerning filial piety, and made her decision to write her assignment on this specific topic/focus. Indeed, Lucia had a clear focus and conceptualization of her topic, which, unfortunately, was not revealed in her written work. While she had sufficient research skills to collect relevant information, she did not sufficiently present her topic knowledge at the formulating stage.

Another problematic feature in Lucia’s essay is body paragraphs (F5) under the component of global coherence. This feature is related to both research and planning strategies. A salient planning strategy she adopted was writing an outline, which she used both to help generate ideas before writing and organize ideas in the formulating stage.

Lucia: I like writing an outline before formal writing. When making the outline, I usually make a list of what I want to write, then I look for some literature to provide evidence and support for my opinion … . For the intercultural essay, I click on Zhihu* [a popular Q&A website in China], read the posts (on this topic) with [a] relatively higher number of “likes”, and add them in [my notes].

While making an outline, Lucia relied on her intuition and background knowledge of the topic, but not what she had learnt from her first-year EAP (English for Academic Purposes) writing course. When she commented on what she learnt from the EAP course, she mentioned some typical essay writing patterns/features, which she intuitively associated with IELTS writing tasks but not her disciplinary assignments. Lucia did not seem to apply the knowledge gained from the EAP course to her academic writing.

Furthermore, within global coherence, Lucia exhibited problems with the concluding section (F6). She was aware that she needed to summarize the main points to conclude but did not want to repeat the main ideas, and she did not have enough time to think of a better alternative. Using the remaining 5 min before submission, she wrote two sentences to end her essay, where she called for taking action, a rhetorical strategy often employed in the conclusion of essays in Chinese. Some excerpts of her IC essay are presented in Appendix 3.

Another problematic aspect of Lucia’s writing is local coherence (F7). Lucia liked bolding her keywords in sentences to emphasize her point, as in the following sentence taken from her essay: “Nowadays, more and more middle-class people are better educated, they see the differences of treatment between China and western countries of filial piety”. She said it was because she was not sure whether the teacher could understand her point. Clearly, Lucia needed to learn about different sentence structures that could be used to strengthen her points. Had she reserved sufficient time for polishing her writing, problems at this level might have been less severe.

A key problem in Lucia’s composing activities was her excessive commitment to research for idea generation at the expense of the other aspects of writing activities (especially formulation and revision). Although she was in her fourth semester at the university, she had not yet formed a clear understanding of the requirements of academic writing. When asked about her writing experience in the past semesters, she could not articulate the format or the style of the academic assignment that she had completed. Instead of reading the writing instructions closely, she tried to imitate the format of the reading materials provided in the course. For instance, she wrote an “abstract” at the beginning of her assignment simply because some articles she read had an abstract. As such, her understanding of different sections of an essay was superficial. When asked about the differences between an abstract and an introduction, she said an abstract was followed by keywords, but an introduction was not.

Discourse features and the composing process in Wei’s case

Similarly, Wei’s three essays were written in year 1 (BP and GE) and year 2 (SEN). Wei displayed unstable discourse features across essays. The majority of features were “basic” or “fair” in her year 1 essays but deteriorated in the year 2 essay, especially in terms of topic building (F1, topic/focus; F2, thesis statement; and F3, controlling idea). Her relative strength was in connective complexity (F8), followed by connective accuracy (F9) and complexity of hedges and boosters (F10).

There are multiple factors that explain the problematic features in Wei’s essays. First, the fluctuation in topic building seems to relate to the difficulty levels of disciplinary knowledge required by the written tasks. The first-year writing task in the GE course required less disciplinary knowledge than the second-year SEN course. Second, academic writing is a dynamic process involving six key interactive and iterative activities. Wei spent considerable time and energy on research and accumulated rich disciplinary knowledge. However, she failed to present such knowledge in her written work. This suggests that Wei’s formulation skills did not interact effectively with her research outcomes. Meanwhile, Wei herself attributed her essay problems to a decline in English language ability.

Wei’s SEN essay was written for her disciplinary course in the third semester. The writing task requires students to “write an essay of not less than 2200 words about a study of an issue or a case (empirical or non-empirical) related to teacher–parent relations or parent involvement in school education, making use of the theories, models, strategies, and skills learnt”. From receiving the assignment topic to the submission date, Wei had about 70 days to prepare. However, as shown in Table 3, it was only in the last 7 days that she began writing an outline and 3 days before the due date that she began writing an introduction. It is worth noting that she spent 6 h and only wrote 65 words. The next day, she spent another 6 h writing the first 700 words of her essay and did a more literature search. She completed her 2200-word essay in 9 h on the due date without leaving time for adequate proofreading or revision. After submitting her work online, she spent 1 day tidying up her references and editing the essay to submit a hard copy.

Table 3 Wei’s composing activities for the SEN essay

Wei’s essay, titled “Using Home-School Collaboration to Improve School-Based Services for Emotional and Behavioral Disorder in Current Hong Kong”, has particular weaknesses in topic identification. When asked how she decided on the focus, Wei said she identified her topic by synthesizing content from another course. Both were her core subject courses, with considerable new concepts and theories. Probably because of the complexity in integrating ideas from the two courses, Wei seemed lost in details of subject knowledge and was unable to convey the essay clearly. Her 2200-word essay consisted of five subsections, of which the first two (around 700 words) introduced the concept of emotional behavioral disorder and its situation in the Hong Kong context. The following two (another 700 words) explained the concept and importance of home–school collaboration. Only until the last section did she link the concepts together and make three suggestions. She did not present her overall intention of the essay clearly in the introduction, and it is difficult for readers to discern her intention. Her essay is organized in such a manner that each section answers one question, but different sections do not seem to connect to each other. There were no cohesive devices to link the sections to each other, nor clear semantic coherence or topical progression.

Similar to Lucia, Wei had a particular weakness regarding the conclusion. Worse than Lucia’s, her essay does not have a conclusion at all. Yet, her writing logs recorded that she spent 6 h working out an outline and brought it to consult her course instructor. Her outline covers typical organizational components required in an exposition essay, including a conclusion. She also specified the length for each section. When asked about the absence of a conclusion, Wei said she did not have time to write it.

Researcher: I noticed that you did not write the concluding paragraph. Where is your conclusion? Is that because you forget the conclusion part?

Wei: No, I didn’t forget. It is just that I didn’t want to continue writing. According to the writing instruction, I had already achieved the required length. Then I thought, if I continued to write, I could not submit on time.

Her words indicated that she seemed to consider the conclusion to be dispensable. Further evidence of her misconception was found in the other essays she produced in the first three semesters, some of which did not have a conclusion. When being pressed further, Wei said she observed that many articles she read come with both an abstract and a conclusion; these two seemed to be complementary to each other. Because she did not write an “abstract” at the beginning, she considered it unnecessary to have a “conclusion” at the end. This finding reveals how far misconceptions of genre features could go when students were left to learn by themselves from examples.

Disciplinary knowledge seemed to pose a special challenge in Wei’s essay writing, especially regarding topic building. Wei did not decide on her focus until 7 days before the due date, she said, because only after she had completed all the lectures could she accumulate enough knowledge to think about writing the essay. In the last 7 days during her writing, Wei also reviewed all her lecture notes and course slides and searched for other materials both in online blogs and in journal articles.

Discussion and implications

Discussion

Following the sequence of writing processes, our diagnosis found that novice EFL learners have difficulty presenting clear topic building. Often shown explicitly through the thesis statement in the opening paragraphs, topic knowledge is essential in guiding content staging and idea development. The lack of an effective thesis statement in students’ writing makes it difficult for readers to identify the main focus and causes confusion in understanding subsequent ideas. Bereiter and Scardamalia (2013) distinguished between knowledge-telling and knowledge-transforming models of novice and skilled writers. Similar to their observation of the knowledge-telling model in novice writing, we found that when engaged in disciplinary essays, novice EFL learners tended to be preoccupied with conceptual and factual knowledge. There is often an absence of linguistic cues or explicit organizational devices to inform readers of the overall intention and specific focus of the essay.

Our diagnosis also found that global coherence is a shared and often persistent problem. In both cases, there is a lack of adequate reader orientation in the opening paragraph and a lack of an appropriate conclusion at the end. They appeared to be ignorant of the rhetorical functions of the introduction and conclusion sections for academic writing. Because the academic assignments in this context often come with detailed guidelines for required content, our participants did not have many problems with staging their content. Their challenges lie in connecting and organising the contents coherently. Their surface problems associated with DC seem to be the result of their poor time management because neither student had enough time for formulation and revision.

Our findings corroborated the subject teacher’s comments on the student essay, which pointed out that the problems are in the language and essay organization and how students present their ideas. While the subject teachers may have a rich reservoir of academic writing knowledge and skills, they tend not to have a fine set of pedagogical tools or sufficient time in class to address these problems systematically. Thus, an effective division of labor and collaboration between the language/writing teachers and subject lectures is necessary.

In terms of processing strategies, we found that both writers were deadline fighters whose first draft was also the final one. A similar finding was noted by Ruan (2014). That is, student writers hardly revised their writing except for a quick check of lexical and syntactical accuracy. One reason may be that students lack revision knowledge and skills, particularly regarding the revision of structural features. In Kellogg’s (2008) term, novice writers focus more on “lexical substitution” than on “semantic changes”. Occasionally, they make revisions of organizational features, but inappropriately: Wei decided not to write a conclusion, though she planned one in her outline; Lucia added keywords after her introduction. Sommers (1980) analyzed the differences in revision strategies between student writers and experienced writers and identified two salient advantages of experienced writers over student writers regarding revision, i.e., a holistic perspective and the recursive nature of revision. These two salient features may serve as useful strategies in revision instruction in the L2 writing classroom context.

Another reason for the lack of revision may be related to lack of time. As identified in Qin and Zhang (2019), the hastily completed final product may suggest students’ lack of appropriate time allocation skills in planning their writing. Stapleton (2010) also observed short formulation time and insufficient revision in a master’s student’s course paper composing process. In our study, regardless of whether the assignment was released early or later in the course, both cases reserved little or no time for revising their first draft. This may suggest that a lack of revision tends to be a persistent problem in students’ course paper writing. Corroborating the findings of Roca de Larios et al. (2008) and Stapleton (2010), we observed imbalanced time distribution among all the writing activities these novice EFL learners engage in. In light of this finding, we suggest raising students’ awareness of the role of revision and providing instruction on the orchestration of the writing processes and time management.

Writing is a planning-intensive task where a detailed outline plays a major role for both idea generation and organization (Kellogg, 2008; Oh et al. 2015). According to Faigley et al. (1993), three kinds of knowledge should be involved in organizing ideas: knowledge of the subject matter, knowledge of potential readers’ expectations and beliefs, and knowledge of discourse structure. Our study found that students’ outline for idea generation and organization was largely based on their knowledge about the subject content, with little indication of reader expectations or target discourse structure. They tend to write information in the order that they retrieve the information, creating a writer-based essay. Neglecting readers’ expectations also causes considerable strain on the part of readers to follow the flow of ideas.

Our third tier of diagnosis probed writers’ academic writing knowledge status, including self-perception of individual strengths and weaknesses. The two cases had relatively high English ability and strong motivation to learn; both were confident in their ability to learn. Different from the student writers in many other studies, such as those with a disengaged attitude in Ruan (2014), they had a positive attitude towards English academic writing. Compared with their favorable affect and motivation, their genre knowledge was strikingly inadequate. Our study found that they lacked awareness of textual functions of academic writing; their knowledge about the writing task and genre was fragmental and based on personal interpretation of the disciplinary articles given to them as reading materials. Besides, in the composing processes, they did not give enough attention to the textual and linguistic aspects, and they also did not have a proper revision. Lack of systematic knowledge on the nature and functions of textual features of the writing task may frequently lead them to confusion and misconceptions in their writing. Insufficient revision further leaves these complementary strategies unattended.

Overall, our three-tiered diagnosis identified a pattern of complex interactions and interrelation among discourse features, composing activities, and related academic writing knowledge. Mastery of adequate academic writing knowledge, or the lack of it, could explain student writers’ composing strategies and the discourse features in their written products.

Implications for academic writing instruction in an EFL context

Academic writing is a recursive process that takes place over an extended period in multiple settings. Examining the process is essential for diagnostic assessment because when a writing teacher works on the products of writing, it is like a coach coming onto the playing field after the game is over (Blanton, 1994). Incorporating the knowledge base, the actual experience, and particular strategies as a whole, metacognition instruction enables teachers to empower students with adequate knowledge and strategy before and during the game; it could also facilitate reflection after the game is over.

Our research attempted to identify the links among symptoms in discourse features, composing activities, and writers’ metacognitive knowledge status and uncovered some interesting connections. This effort had both theoretical and pedagogical implications for researching academic writing in higher education. Theoretically, the three-tiered diagnostic design can be considered as one small-scale demonstration of the cognitive process–based approach to writing assessment proposed in Zhao (2022). This approach contributes to creating a dynamic and holistic profile of EFL learners’ writing development features. Pedagogically, three main stakeholders in EAP classroom learning are language/writing teachers, subject lecturers, and student writers. Thus, it is a shared responsibility at least between EAP writing teachers and the disciplinary lecturers to help students handle various struggles and difficulties in their academic writing. Close collaboration of these two parties can provide a solid scaffold for effective execution of the work.

On the part of EAP language/writing teachers, we suggest employing metacognitive instruction based on the principles of awareness, avoidance, and advance. Specifically, we suggest teachers adopt awareness-raising activities with students diagnosed at a poor level of a discourse feature. These could be knowledge-passing activities to build up and consolidate their knowledge of discourse features and corresponding strategies. For those diagnosed to be at the basic level, we suggest teachers provide a detailed checklist of typical problems (such as avoiding vague topic specifications or improper thesis locations) to remind the students to avoid these problems during the composing process. For those whose discourse features were considered good, teachers could encourage them to learn more sophisticated discourse features and processing strategies. Such a differentiated and principled procedure could be useful for writing instruction, especially for novice EFL writers in an undergraduate academic learning context.

Despite the strength of qualitative methods in eliciting rich and in-depth findings, one limitation of this approach concerns the transferability of the findings. In the present study, we examined only two cases; readers may rightly wonder whether or to what extent the findings may be extended to understand other EFL writers. To address similar queries, Merriam and Tisdell (2015) stated that the precise objective of a case study is to understand the particular in depth instead of finding out what is generally true of the many. Thus, the transferability of a qualitative study is not achieved in a statistical sense. Instead, the potential of transferability in a case study may lie more in the appliers who could benefit from the sufficient rich and thick descriptive data afforded by a case study (Lincoln and Guba, 1985). Merriam and Tisdell (2015) further suggest that the transferability of qualitative research may occur when the readers/appliers consider the results as a working hypothesis, which can be monitored and evaluated for a better decision by weighing the local context.

Conclusions

This study conducted a diagnostic assessment of two students’ academic writing by examining their writing samples and tracking their writing processes, strategies, and metacognitive knowledge status. Our results show that novice EFL student writers have problems with global coherence, particularly in terms of introduction and conclusion; shortage of adequate discourse organization knowledge of academic writing, and imbalanced time allocation across different writing processes are primary contributors to their difficulties. Based on the findings, we suggested that EAP writing teachers provide differentiated metacognitive instructions to students at different levels of DC.

Given the limitations of the case study, similar studies should be conducted in other contexts to explore the applicability of the results and enrich our understanding of the links among EFL writers’ writing products, processes, and strategy use. Another limitation of the present study is that it only investigated one type of academic writing and took the perspective of the writer. Future studies could investigate other types of academic writing tasks and take the perspective, for instance, of disciplinary teachers to provide a deeper and more comprehensive understanding of undergraduate students’ development of DCs in academic writing across curricula.

Appendices

Appendix 1

A brief of the discourse assessment rubrics based on Wang and Xie (2022)

Scale			5 Excellent	4 Good	3 Fair	2 Basic	1 Poor
Topic building	Topic/focus	F1	A thesis is established concisely and consistently. The language reads clearly.		A thesis is established, but it is not consistent or is somewhat general.		It is difficult to identify the focus or the key topic.
	Thesis statement	F2	The thesis statement is specific, appears in a proper place, and is clearly presented. The language reads clearly.		There is a thesis statement clearly stated; it appears in a proper place, but it is somewhat general.		No identifiable thesis statement.
	Controlling idea	F3	The controlling sentence shows the overall communicative intent explicitly and is concisely presented. The language reads clearly.		There is a controlling sentence, but it is not in a proper place, or it is not clearly stated.		There is a lack of a controlling sentence or idea.
Global coherence	Introduction	F4	The opening paragraph(s) provide(s) an effective background or context to orient the reader to the topic. The language is fluent and has sufficient complexity.		There is some background or context information to orient the reader, but the idea can only be sensed due to inadequate language.		The opening paragraph(s) provide(s) little or no background information to orient the reader.
	Body paragraphs	F5	The body paragraphs have clear, effective, and logically sequenced topic sentences to organize the text. The language is fluent with sufficient complexity.		The body paragraphs begin with an identifiable topic sentence. However, some topic sentences are not coherent.		Topic sentences are often missing, or they digress from the theme.
	Conclusion	F6	The conclusion rounds off the essay effectively. The language is fluent with sufficient complexity.		The conclusion provides a limited summary of the main points. Or, it lacks important moves. The language is not easy to understand.		The conclusion fails to (re)state the main points and does not create a sense of closure.
Local coherence	Theme-rheme development	F7	The theme-rheme development is competent. The language is fluent with sufficient complexity.		The local coherence within a paragraph is often perceived from an understanding of the writer’s mother tongue.		There are frequent unrelated thematic progressions or coherence breaks.
Logical connective	Connective complexity	F8	A variety of connectives are used accurately and carefully but not mechanically.		Some simple connectives are frequently used, but they may sound mechanical.		A very limited range of connectives are used, or connectives are rarely used.
	Connective accuracy	F9	The connectives are used precisely and concisely.		There are misused or overuse or underuse of certain connectives.		Connectives are often misused or absent.
Reader–writer interaction	Complexity of hedges and boosters	F10	There is a rich variety of hedges and boosters. Hedges and boosters are used in a proper balance overall, conveying a convincing and engaging stance.		There is frequent use of simple hedges/boosters, but occasionally with grammar mistakes. Alternatively, hedges or boosters sometimes do not fit the context.		Hedges and boosters are rarely used.

Note: For quick comprehension of particular discourse features, a set of acronyms were established by taking the initial (i.e. F) of “Feature” accompanied with an assigned Latin number, namely, F1–F10, in alignment with their respective ordering in the assessment rubrics

Appendix 2

Writing prompts

Year 1 Semester 1 Basics of Psychology (BP) course (Lucia and Wei): Identify one news report (commentaries and research articles are not allowed) within the 6 months preceding the submission deadline, and write a 1000-word (± 10%) essay analyzing the incident or phenomenon described in the news clip, applying psychological concepts/principles to explain human behaviour in educational settings.

Year 1 Semester 2 General Education (GE) course (Lucia): “Do you think religion plays a useful role in society nowadays?” Write 1500–2000 words, excluding references and other components of the ePortfolio.

Year 1 Semester 2 General Education (GE) course (Wei): “To what extent can the concept of Confucianism, which was discussed in Professor Dennis Cheng’s lectures, be applied to the people and society nowadays?” Write 1500–2000 words, excluding references and other components of the ePortfolio.

Year 2 Semester 1 Intercultural communication (IC) course (Lucia): Critically evaluate the values of your own culture (i.e., Mainland China, HK, Nepal, Germany). Write in 1500 words (± 150 words).

Year 2 Semester 1 Special Educational Needs (SEN) course essay (Wei): Write an essay of not less than 2200 words (in English) about a study of an issue or a case (empirical or non-empirical) related to teacher–parent relations or parent involvement in school education, making use of the theories, models, strategies, and skills learnt.

Appendix 3

Sample excerpts from Lucia’s IC essay

Appendix 4

Semi-structured interview questions and coding scheme

Some prepared interview questions:

• 1. What do you think about the similarities or differences in writing in English in secondary school and in university study? What kind of writing have you done in the past three semesters?

• 2. What do you think is good writing? What do good writers do when they write? Where did you get such an impression?

• 3. What have you learnt from your English writing courses?

• 4. How do you feel about your writing abilities? What do you think are your strengths and weaknesses in completing this essay?

• 5. Why do you decide to focus on this topic? How long did you spend on this essay? To date, what are your main efforts in preparing the essay? What took your most time?

• 6. How do you plan your organization? Do you write an outline before writing? How do you write the outline? Did you seek any help, for instance, from your teachers or classmates?

• 7. What sources have you referred to? How did you get to know these sources?

• 8. When did you begin writing? How do you know you can write now?

• 9. When did you feel like you had completed the first draft of the essay? Would you think about making any revisions after finishing the first draft? If yes, what aspects of the changes did you make? Why do you think there is a need for revision? If not, why?

• 10. Did you have any difficulties while planning, drafting, or writing this essay? What typical difficulties or problems do you encounter? How do you cope with them?

• 11. Reflecting on the whole writing process, what have you learnt from preparing this essay? Do you think this paper is as effective as it should be? If you had a second chance to prepare the essay, would you do the same or make any changes?

• 12. Would you have similar procedures for preparing essays for other courses?

The main codes for analyzing interview protocol and writer log

Code		Main features
1.1	[Research]	Formulate the problem, search sources, read, take careful notes, design, gather and analyze data, etc.
1.2	[Planning]	Set goals (long-term/short-term goals, major and sub-goals, writing quality, time schedule on typical writing activities), planning idea generation (thesis and main points), and planning idea organization
1.3	[Formulation]	First attempt to get ideas on the page
1.4	[Revision]	Tangible changes of reorganizing, deleting, adding, and substituting
1.5	[Evaluation]	Make judgement on the degree of success of one’s own pragmatic, textual, and linguistic decisions based on criteria and standards
1.6	[Collaboration]	Co-create, compromise, contribute, give feedback, receive feedback, and share
2	[Academic writing knowledge]	<linguistic>: accuracy, complexity, and appropriateness of sentence structure and vocabulary
		<discourse>: organization, clear presentation of ideas, and consistent/confident reader–writer interaction
		<disciplinary knowledge>: relevant and rich content
		<source integration>
		<mechanics>
		<systemic knowledge/skills of academic writing>
3	[Other]	[Other]<transferring skills>
		[Other]<attitude and emotions>: affective response to the writing activities and the written outcome. Some typical words include sadness, happiness, anger, fear, disgust, surprise, confusion, frustration, boredom, anxiety, and flow/engagement
		[Other]<motivation>: a reason or motive in writing, recognized in terms of “For …”
		[Other]<monitoring> awareness and modification of current thoughts and behaviours in the writing process, mainly engaging time management skills in monitoring the execution and cooperation between research, planning, formulation, and revision