Assessing peer review pattern and the effect of face-to-face and mobile-mediated modes on students’ academic writing development

Abstract

This study examines the distribution of peer review in face-to-face and mobile-mediated peer review groups and their effects on students’ revision skills and academic writing development. Seventy-two first-year English for academic purposes (EAP) students participated in an 18-session IELTS academic writing course in a Canadian university the mobile-mediated peer review group (MMPR) used Telegram to exchange peer comments synchronously, while the face-to-face peer review group (FFPR) did peer review in the classroom. An adapted analytic scheme (Journal of English for Academic Purposes, 2, 193–227, 2003) and the IELTS academic writing assessment criteria were used to conceptualize the peer comments in terms of frequency, area, type, nature, and IELTS assessment categories. Results indicated that the total number of comments, the percentage of revision-oriented comments and actual revisions made by the MMPR group were statistically more significant than those by the FFPR group. Furthermore, the MMPR group made more local revision-oriented comments than that of FFPR. However, the revision-oriented suggestion in local areas was the most distributed type of comment made by both groups. Regarding the IELTS assessment criteria, the FFPR group made more comments on task achievement and coherence and cohesion, whereas the comments made by the MMPR group targeted more lexical resources, and grammatical range and accuracy. In addition, the results showed that both MMPR and FFPR groups developed their IELTS academic writing skills while the MMPR mode of collaboration outperformed the FFPR.

Introduction

Peer review is a collaboration between students to provide constructive comments on each other’s oral and written output with the goal of development (Jurkowski, 2018; Liu & Hansen, 2005). Students would get the two roles of an assessor who provides comments and an assessee who receives comments (van den Bos & Tan, 2019). Peer review has been criticized by the second language (L2) teachers and learners over the past decade, if it could help learners to write better (Saeed & Ghazali, 2016; Shang, 2017). Neumann and Kopcha (2019) maintain that teachers might be concerned about the order and sequencing of peer comments received by students. Moreover, the challenge commonly lay on the grounds that learners might provide inaccurate, unclear, and simple comments. They sometimes become too critical while offering peer comments or too defensive to accept the comments of their classmates. This is more obvious in International English Language Testing System (IELTS) preparation courses, where most candidates might show uncertainty towards their peers’ comments and consider them unreliable or invalid due to the nature of this international test. Additionally, another concern in peer review research is if learners can utilize the peers’ comments and make revisions on their writing as some researchers take the positive (Lam, 2010; Min, 2006; Pham & Usaha, 2015; Shang, 2017) and some (Guardado & Shi, 2007; Liu & Sadler, 2003; Saito & Fujita, 2004) doubtful standpoint towards that. They argue that some factors such as the characteristics of peer comments, type of written tasks, peer review training, and modes of communication affect the incorporation rate of comments. Moreover, revision demands cognitive load for students as they compare their first drafts with the suggestions given in peer comments and decide to either accept or reject the comments (van den Bos & Tan, 2019). Different theories exist in the literature regarding the use of peer review in L2 settings such as process writing theory (Hayes & Flower, 1980), collaborative learning theory (Bruffee, 1984), sociocultural theory (SCT) (Vygotsky, 1978), and interactionist theory (Hyland & Hyland, 2006). These theories and perspectives emphasize the role of learners in creating knowledge and using various skills. Process writing theory considers writing as an ongoing, recursive process in which students engage in peer review to produce multiple drafts in a meaning-making activity (Hayes & Flower, 1980). Collaborative learning theory emphasizes that learning, and even knowledge itself, is constructed socially through communication with knowledgeable peers in a community. It is through collaboration among peers that some kinds of knowledge can be acquired (Bruffee, 1984; Liu & Hansen, 2005). According to SCT, learning is a social process that enhances interactions among learners in a cooperative context by relying on the concepts of zone of proximal development (ZPD) and scaffolding (Vygotsky, 1978). In other words, learners might use various supportive behaviors to assist each other for achieving higher level of regulation (De Guerrero & Villamil, 2000). Interactionist theory suggests that language learning would be enhanced when we create opportunities for learners to negotiate meaning in group work (Hyland & Hyland, 2006). Negotiation is the key factor in this theory that assists L2 development by creating a more comprehensible input, drawing students’ attention to their linguistic problems and errors, and highlighting the negative evidence (Gass, 2003).

Conventionally, teachers encourage students to conduct face-to-face peer review group (FFPR) during written in-class sessions by working in pairs or groups (Guardado & Shi, 2007; Hanjani & Li, 2014; Lam, 2010; Liu & Sadler, 2003). In this type of peer review, only one or two students can review their peer’s work due to time limitation, classroom seating, and class workload. The comments they share is mostly oral, and there is no systematic way to collect and restore them for future students. However, the present study attempts to address these problematic areas and compare the FFPR mode and explore differential effects of a new mode of peer review namely as mobile-mediated peer review group (MMPR) in order to give more insight into the field and shed more light on the peer review effectiveness.

With the advent of smartphones and ubiquitous access to the internet, users take advantage of various applications to stay connected and communicate outside the classroom and engage in collaborative learning (Andujar, 2016; Pham & Usaha, 2015; Shang, 2017; Tang & Hew, 2017). The field of education is no exception as mobile instant messaging (MIM) applications, such as WeChat, WhatsApp, and Telegram have facilitated the process of teaching and learning (Tang & Hew, 2019). MMPR involves both synchronous and real-time communication (Andujar, 2016; Shintani, 2015) and asynchronous or communication with no immediate time limitation (Saeed & Ghazali, 2016; Wu, Petit, & Chen, 2015) environment for conducting the peer review. This study uses the MMPR environment to create a more comfortable, accessible mode of peer review for the IELTS academic writing tasks. Besides, students experience a collaborative learning environment by having a larger number of audience, shaping a community of practice and having the ability to restore and retrieve their peers’ comments for future reflections. The review of literature reveals that the mode of peer review, as discussed below, has significant effect on learners’ commenting characteristics and revisions.

Face-to-face peer review (FFPR)

The FFPR occurs in the classroom where students are assigned to a pair or group to read each other’s first drafts and give comments while using both verbal and non-verbal forms of communication. Some benefits are associated with FFPR. Wooley, Was, Schunn, and Dalton (2008) demonstrate a link between the development of cognitive abilities among peer reviewers when they try to articulate explanations to their classmates. Pritchard and Morrow (2017) refer to social benefits of FFPR as students deal with non-threatening audience which has similar characteristics like them, they receive comments immediately in a direct way, the act of exchanging peer comments can be exciting because it reduces the writing apprehension, motivation for writing increases as students assume the process as something collaborative, and the interpersonal communication and sense of cooperation are developed. Some studies only examined the differential effects of FFPR (Cho & Cho, 2011; Hanjani & Li, 2014; Min, 2005; Vorobel & Kim, 2014), whereas some other studies have compared FFPR with computer-mediated peer review (CMPR) (Chang, 2012; Ho, 2015; Liu & Sadler, 2003; Rouhshad, Wigglesworth, & Storch, 2016; Yilmaz, 2012) and argued that the difference in the mode of interaction has affected the number, area, nature, and type of peer comments as well as students’ revision skills and writing performance.

In FFPR, students might be more focused when sharing comments because they initiate the commenting directly without greeting their peers or sharing unrelated words (Chang, 2012). Previous studies have reported that FFPR students generated revision-oriented comments by focusing on issues of writing at both the macro-level, such as idea development, organization, and purpose (Cho & Cho, 2011; Min, 2005; Vorobel & Kim, 2014) and micro-level issues such as grammar (Hanjani & Li, 2014). In addition, one study showed that students made non-revision-oriented comments such as welcoming, thanking, greeting, and praising only to create a positive environment of peer review (Hanjani & Li, 2014). Regarding the type of comments, several studies have documented explanation, confirmation, clarification, problem-identification, and error correction as functions of comments among FFPR groups (Hanjani & Li, 2014). With respect to revision and writing performance, Cho and Cho (2011) argued that macro-level comments improved students’ revision skills and writing quality. In addition, the collaborative FFPR creates some interaction patterns which affects the writing performance.

To date, several studies have combined/compared the mode of FFPR with CMPR Liu and Sadler (2003) investigated whether students from various modes of peer review shared different commenting features. The findings showed that the number of total comments in all forms was larger in CMPR class than the FFPR one. Yilmaz (2012) investigated the effects of feedback types in the two modes and found that feedback is more effective when delivered via a technology-mediated environment regardless of type and quality, Chang (2012) compared the online group with FFPR and realized that when students make comments in FFPR, and they generate more global revision-oriented comments by targeting the content, audience, and organization of the writing drafts. In contrast, Ho (2015) reported students in both groups offered more revision-oriented comments; however, when considering the area, students gave more local comments in CMPR. In a recent study, Rouhshad et al. (2016) argued that the FFPR students’ interaction patterns were collaborative and less dominant with numerous opportunities to negotiate meaning. Although the previous studies have used both synchronous and asynchronous modes of communication in ESL and EFL CMPR groups with general writing tasks, no previous study, to our knowledge, has focused on comparing FFPR with a synchronous mode of interaction in an ESL context to develop students IELTS academic skills by considering the standard IELTS assessment criteria.

Mobile-mediated peer review (MMPR)

Mobile learning has received significant attention in education recently by providing mobile MIM services (Aghajani & Zoghipour, 2018; Allagui, 2014; Andujar, 2016; Fattah, 2015; Miller, 2016; Tang & Hew, 2017). Unlike CMPR with asynchronous nature, MMPR provides both synchronous and asynchronous contexts for exchanging comments. Mobile learning research mainly employs learning theories with the focus on mediation such as situated-learning theory, collaborative learning, and self-paced learning (Chang & Hsu, 2011). The portability of content; versatile chat features such as negotiation, repair, comprehension, clarification, and confirmation checks; and convenient use of media sharing tools such as voice, text, image, and emoticons are associated with mobile-mediated writing environments (Andujar, 2016).

Previous studies have used mobile applications to develop writing. In a study by Allagui (2014), students used a mobile app (WhatsApp) in a request/question-and-response learning activity to share their messages on a particular writing topic and received comments from their peers. The results showed that majority of the students reported progress in their writing and grammar skills. Fattah (2015) created a WhatsApp group containing 15 college-level students in order to facilitate their writing practice. Students went through several stages of pre-writing, drafting, reviewing, and editing before posting their final drafts in the group. The results indicated that the WhatsApp group outperformed in writing various sentence structures and expressing their ideas. Miller (2016) also explored English for specific purposes course with the focus on process-based scriptwriting. Students worked in WhatsApp to write short scripts, share images and files, and offer comments. The findings showed that the students experienced a considerable amount of writing support, more negotiation, and participation and created a micro-community. Andujar (2016) designed a question-response activity and encouraged the students to take part daily by sharing typed responses and giving peer comments. Students writing development was examined in terms of accuracy and complexity. At the end of the course, the mobile group not only produced fewer grammatical, lexical, and mechanical errors but also showed active participation and negotiation. In a recent study by Aghajani and Zoghipour (2018), the authors used three common methods of writing correction such as self-correction, peer-correction, and teacher-correction by using Telegram in order to practice writing correction. Students wrote between 50 and 70 words on a prompt and posted them in their groups. The analysis showed that students obtained better results in their writing development in self- and peer-correction groups compared to the teacher-correction group.

Previous studies have undoubtedly given us insights about the use of MIM applications for language learning. However, they only emphasized on one or two aspects of MIM such as students attitude towards MIM or the interactional patterns with only a small number of participants. Very few studies have conducted a discourse analysis of peer comments in MIM environments by using standard analytical schemes for a high-stakes test like IELTS in a Canadian context. This study will address these limitations by relying on some theoretical foundations to examine the effectiveness of MIM in academic writing courses.

The study

This study examines not only if the differences between the two peer review modes result in different commenting distribution and actual revisions but also how they affect students’ IELTS academic writing skills. In order to classify peer comments, this study uses two analytical schemes. The first one is Liu and Sadler’s (2003) scheme which classifies the comments into area, nature, and type. Through this division, researchers would be able to conceptualize peer comments and identify if they are global or local (area), revision-oriented or non-revision-oriented (nature), or direct and indirect (type). The second scheme is the IELTS assessment criteria. It is used to categorize students’ comments on the IELTS academic task 1 tasks in the four categories of task achievement (i.e., paraphrasing topic, summarizing key points, writing an overview, supporting the graphs with data, making comparisons, writing a summary, and using limited certain number of words), coherence and cohesion (i.e., text organization, linking devices, sense of progression), lexical resources (i.e., using a range of appropriate vocabulary, collocations, prepositions, verbal phrases, academic terms, and vocabulary specific to the topic), and grammatical range and accuracy (i.e. using various grammar structures, sentence structures, and punctuation marks). A mark between 1 and 9 is given to both each of these four areas and the overall mark with intervals of 0.5 (University of Cambridge ESOL Examinations (UCLES), 2018).

Although many previous studies have examined and compared the effectiveness of CMPR with that of FFPR mode, the research in MMPR mode remains limited. As far as we know, no studies investigated the effectiveness of using Telegram and FFPR mode on students’ peer review practice, revision skills, and IELTS academic writing task 1. In other words, the benefits of MIM applications in developing a collaborative writing environment have remained under-investigated. Finally, this study is based on the necessity to extend research related to the use of mobile applications in L2 writing classrooms by exploring its pedagogical values.

Research questions

This study examines the following research questions:

• RQ1. Do the MMPR and FFPR commenting modes result in a different distribution of peer comments in terms of the frequency, area, type, nature, and the IELTS assessment criteria?

• RQ2. To what extent do students revise their IELTS task 1 samples based on comments made in the MMPR and FFPR groups?

• RQ3. Do the MMPR and FFPR modes improve the students’ IELTS academic task 1 band score?

Methods

Participants and setting

This study was conducted in 18 sessions (9 weeks), from February to April 2019. The population of this study was limited to non-native speakers of English language who studied English for academic purposes (EAP) courses at a private university in Vancouver, Canada. Through a flyer posted at the campus, 113 students volunteered to attend this program. All these students were studying at upper-intermediate level (course code 030) and had submitted their IELTS overall band score of 6 as part of their admission. To get assurance as the homogeneity of groups in terms of language proficiency level, all participants took an online DIALANG test and only students who achieved level B2 were invited to attend the program. Out of 91 students who achieved B2, only 72 students attended the peer review training workshop that was mandatory for entering the program. Therefore, the study sample comprised of 47 female and 25 male students, coming from various nationalities (Iranian, Indian, Chinese, Russian, Mexican, Indonesian, Sri Lankan, and Turkish) with ages ranged between 22 and 34 with an average of 26. The participants did not have any peer review training prior to the program. Students were purposefully divided into the MMPR (n = 36) and FFPR group (n = 36). To reflect the diversity of the population, both males and females, and students from various nationalities were placed in each group. The FFPR participants were divided into three classes with 12 students in each, and MMPR students were invited to 3 Telegram groups of 12 students. The participants met for two sessions per week and 90 min each. The researcher was the instructor in all the groups, running the face-to-face classes at the campus and the mobile ones on Telegram. The teacher used the same assignments, peer review guidance sheet, and logs for both groups. Table 1 presents the detailed demographic information of the participants.

Table 1 Demographic information of participants

Instruments

DIALANG test

DIALANG test was administered to measure the participants’ level of language proficiency. This is an online language assessment test which offers tests in several languages and follows the standards of Common European Framework of Reference (Alderson & Huhta, 2005). This test measures the general language proficiency based on the assessment scale ranging 6 levels from A1 to C2. Level A1 represents the lowest level and C2 the highest level.

IELTS academic writing task 1

The IELTS academic writing task 1 was used as pretest and posttest to assess the academic writing skills of the participants and how the program implementation affected their IELTS writing. The pretest was task 1 adopted from Cambridge book 13 (Cambridge, 2017a, 2017b) with the topic of “The chart below shows the percentage of households in owned and rented accommodation in England and Wales between 1918 and 2011.” The posttest adopted from Cambridge book 12 (Cambridge, 2017a, 2017b) with the topic of “The chart below shows how frequently people in the USA ate at fast food restaurants between 2003 and 2013.” These tests were the standard practice tests of the IELTS exam. The students’ academic writing was evaluated based on four assessment criteria of the IELTS academic writing task 1 (University of Cambridge ESOL Examinations (UCLES), 2018). This task 1 was employed because (1) the purpose of this study was only targeting describing graphs, (2) it requires students to write at least 150 words compared to task 2 which requires at least 250 words, and (3) it follows an objective style of writing which does not involve writers’ interpretation and opinion while describing the graphs.

Synchronous peer review tool (telegram)

Telegram application (version 1.5.12) was used in this study to exchange peer comments. Telegram is an instant messaging platform which provides the opportunity to create groups of users for interaction. This application has both asynchronous and synchronous modes of communication that connects to wireless networks and provides a facility for participation. In this study, students in the MMPR group attended the session in real time and posted their paragraphs in Telegram.

Data collection

As presented in Table 2, this study followed the design of the pretest, peer review training sessions, peer review comments made by the groups, and posttest which lasted for 18 sessions (9 weeks). Session 1 was the pretest, and sessions 2 to 5 (4 sessions) were devoted to peer review training workshop. Then, the intervention took place from session 6 to 17 (12 sessions). At the end, the posttest was administered in session 18.

Table 2 The design of the study

In session 1, IELTS academic writing task 1 was administered to both FFPR and MMPR group as a pretest. The duration of each IELTS writing task 1 test was 20 min for writing at least 150 words.

From session 2 to 5, the peer review training workshop adapted from previous studies (Lam, 2010; Liu & Sadler, 2003; Min, 2005) was conducted. This aimed to get participants familiar with some basic themes and procedures related to peer review such as purposes, benefits, and challenges, and Min’s (2005) four-step procedure for giving peer comments (clarification, identification, explanation, and suggestion) for peer review provision. The participants got familiar with the classification of comments based on the Liu and Sadler’s (2003) taxonomy and the IELTS academic tasks by using the designed peer review log and guidance sheet.

From session 6 to 17 (6 weeks), all participants received the intervention. They wrote three IELTS task 1 samples for the purpose of peer review and revision as follows:

Sessions 6 to 9 devoted to IELTS academic sample 1 which was adopted from Cambridge Academic IELTS 13 (Cambridge, 2017a, b) with the topic of “The bar chart below shows the top ten countries for the production and consumption of electricity in 2014.” IELTS academic sample 2 in sessions 10–13 was adopted from Cambridge Academic IELTS 11 (Cambridge, 2016) with the topic of “The graph below shows average carbon dioxide (CO2) emissions per person in the United Kingdom, Sweden, Italy, and Portugal between 1967 and 2007.” Sample 3 was used from sessions 14 to 17, and the topic was “The first chart below shows how energy is used in an average Australian household. The second chart shows the greenhouse gas emissions which result from this energy use.” The researchers selected this topic from Cambridge Academic IELTS 10 (Cambridge, 2015). Each of these three samples as an intervention took four sessions (12 sessions in total).

In intervention session 1, all participants did brainstorming on the topic in order to get ready for writing their first drafts. The FFPR group wrote their first drafts in the class and gave them to their instructor, whereas the MMPR group typed and submitted them on Google Classroom.

In intervention session 2 and 3, the teacher returned the first drafts of the FFPR group in the class and the ones for MMPR on Telegram. Both groups were required to give peer comments on their classmates’ drafts by using the guidance sheet (Additional file 1) and filling out their peer review logs (Additional file 2). In order to attain efficiency in the visibility of the content posted on Telegram to three created groups, the MMPR group posted their writing in two segments: (1) introduction, overview, and body1, and (2) body 2 and summary. They received comments from the other participants via using the electronic version of the guidance sheet and log. The participants were given 3 min in Telegram to read the segment and got ready to share comments. Each participant had to post at least 1 to 3 comments only in English. Participants were also allowed to share images, voice messages, and stickers. The teacher was a member of the group and facilitated the process by managing the turn-taking, timing the session, encouraging the members, solving the technical problems, and wrapping up the sessions. Figure 1 shows a sample of peer review exchange among participants in Telegram.

Fig. 1

A sample of peer review exchange in Telegram

In the FFPR group, the participants showed their entire first drafts to their classmates with a 10-min time slot to read and they had 10 min to fill their log and give comments. A sample of peer review log completed by a participant is presented in Additional file 2. Moreover, the teacher asked the students to work with a different partner each time to encourage the richness of peer comments.

In intervention session 4, both MMPR and FFPR group were asked to revise their draft based on their peer comments and complete the peer review log accordingly and return to the teacher. In session 18, the IELTS task 1 test with the same graph type (bar chart), level of difficulty, and instruction was administered to both groups as the posttest to check the participants’ academic writing skills.

Data analysis

As indicated in Table 3, the comments were analyzed and categorized based on 12 factors (8 from Liu and Sadler’s Taxonomy and 4 from IELTS rubric). The number of comments was counted in terms of areas, types, nature, IELTS assessment criteria. The frequency of actual revisions made and revisions suggested on all the three IELTS task 1 samples was also calculated.

Table 3 The analytical scheme for classifying peer comments

The short comments such as “Well done!” “Ok,” “Good,” and “Very nice” which did not target the peer review comments were excluded. Two official IELTS examiners were also invited to code 15% of the coding comments. The inter-rater reliability was determined as .91. Table 4 shows the scheme for peer review classification.

Table 4 Percentage of revision-oriented feedback leading to actual revision by group and by area

The pretest and posttest were scored based on the IELTS assessment criteria. A sample of 15% of the total data coming equally from the pretest and posttest was scored by a second rater. The inter-rater reliability for the pretest and posttest were .99 and .97, respectively.

Results

Distribution of comments in FFPR and MMPR

RQ1: Do the MMPR and FFPR commenting modes result in a different distribution of peer comments in terms of the number, area, type, nature of comments, and the IELTS assessment criteria?

As illustrated in Fig. 2, the most significant difference in the area of peer comments provided by both MMPR and FFPR group was the number of local comments. The local comments contributed 61% of all the comments in MMPR groups, and it accounted for 56% of the comments by the FFPR group. The interactive nature, easy-to-use commenting features, and a relaxed environment in Telegram encouraged the participants to collaborate more and direct their comments to more lexical and grammatical errors. The MMPR group made 5% more comments than the FFPR. However, the global types of comments were the smallest percentage of distributor for both groups involved in the study. The FFPR group provided 4% more global comments than the MMPR group.

Fig. 2

Feedback distribution percentage by group and by area

The second part of research question one was to examine the peer review distribution type, namely as evaluation, clarification, suggestion, and alteration. As revealed in Fig. 3, the evaluation type of comments was the largest made by the two groups; 50% of comments made by the FFPR, and 45% of the comments made by MMPR. The clarification and alternation types of comments were the minority in both groups. The higher percentage of the clarification comments in the MMPR group might refer to the synchronous environment of peer review that provides a context to ask questions instantly and receive the answers immediately. In the FFPR group, students shared slightly more alteration comments (15%) than the MMPR group (10%). The number of suggestion type of comments, on the other hand, was another large difference. As Fig. 3 illustrates, 33% of the comments made by the MMPR group were suggestive, and 27% of such comments made by the FFPR group.

Fig. 3

Feedback distribution percentage by group and by type

The nature of comments was another dimension of the investigation. It is necessary to mention that some types of feedback (e.g., alternation) are revision-oriented in essence in a way that it provides an alternative item to correct the problem (e.g., “You have made a spelling mistake in the last paragraph, summary not summary”). However, the evaluation comments could be either revision-oriented (e.g. “Sentences are not linked properly”) or non-revision-oriented (e.g., “Your overview paragraph is easy to understand”).

As displayed in Fig. 4, the comments which led to the revisions are dominant in number and percentage in both MMPR and FFPR group, as 76% of the comments provided by the FFPR group were revision-oriented, compared to 65% for the MMPR. As mentioned earlier, all alternation comments were revision-oriented; therefore, this percentage difference between the two groups is because of a more substantial number of alternation comments made by the FFPR group (15%) compared to the MMPR group (10%), as illustrated in Fig. 3. However, in order to obtain the real percentage of revision-oriented comments, the alternation comments were excluded from both groups. The total number of comments provided per group differs by 653 (1451 versus 798). Nevertheless, the percentage of revision-oriented feedback provided by the FFPR group is still 10% larger, as presented in Fig. 5.

Fig. 4

Feedback distribution percentage by group and by nature

Fig. 5

Feedback distribution percentage by group and by nature (after alternation feedback exclusion)

The research question number 1 also analyzes the peer review distribution through the IELTS assessment criteria. Students’ comments were classified based on task achievement (e.g., “There is no paraphrasing of the title in the introduction”), coherence and cohesion (e.g., “The sentences in paragraph two are not linked properly”), lexical resource (e.g., “More vocabulary can be used instead of rented and owned”), and grammatical range and accuracy (e.g., “Start the new sentences with capital letter”).

As demonstrated in Fig. 6, the task achievement was the most distributed type of comment given by both groups among the other types. The percentage of peer comments related to task achievement and coherence and cohesion type was larger in the FFPR group than that of the MMPR group. More specifically, the FFPR group provided 8% more comments related to the task achievement and 2% related to the coherence and cohesion than the MMPR group. The percentage of comments, on the other hand, pertinent to the grammatical range and accuracy (28% versus 22%) and lexical resources (27% versus 24%) was higher in the MMPR group than that of the FFPR group.

Fig. 6

Feedback distribution based on the IELTS rubric

Peer review adoption rate in FFPR and MMPR

• RQ2: To what extent do students revise their IELTS task 1 samples based on comments made in the MMPR and FFPR groups?

As discussed earlier, most comments provided by the MMPR and FFPR group were revision-oriented in nature. The second research question examines these revision-oriented comments which led to the actual revision by considering the nature and type of comments. As revealed in Fig. 7, the MMPR group indicated a larger percentage of revision-oriented comments than the FFPR group (82% versus 75%) with a higher percentage of actual revisions compared to the FFPR group (41% versus 33%). This result indicates that the students in Telegram incorporate more comments into their revisions.

Fig. 7

Comparing the total feedback, revision-oriented feedback, and actual revisions

By considering the area in which the groups made peer comments, it is evident from Table 4 that the number of peer comments in the local area was more than the global area for both MMPR (994 versus 627) and FFPR (535 versus 408). The MMPR group provided a larger number of comments based on the area and possessed a larger percentage of revision-oriented comments in both local (88%) and global areas (74%). The possible explanation is that more comments are being shared in the mobile groups than FFPR, and this abundance of comments directs the students towards more specific comments. While both groups made the same percentage of revisions in the global areas (40% versus 40%), examining the number of revision-oriented feedback leading to the actual revision, however, revealed that the MMPR group also maintained the trend of superiority compared to the FFPR group and made a greater percentage of revisions in the local (42%) areas. The findings by area give us a general standpoint towards the differences between the groups; however, a more detailed analysis of the peer review types, their revision-oriented comments, and the number of actual revisions in each type will reveal more their differences.

As Fig. 8 shows, the MMPR group gave 732 evaluation comments in total, and 520 (71%) of those were revision-oriented which 221 (30%) led to the actual revisions. In the FFPR group, of 472 evaluation comments, 268 (57%) were revision-oriented which 84 (31%) of those resulted in actual revisions. The MMPR group provided a larger number of evaluative comments and produced a slightly larger number of revision-oriented comments. Furthermore, the MMPR group produced 11% larger percentage of actual revisions (42% versus 31%). This indicates that the mobile-mediated peer review may have more influence on the writer, which resulted in more revisions on their drafts. The MMPR group also revealed a larger percentage of global (38% versus 36%) and local (33% versus 21%) evaluation comments compared to that of the FFPR group.

Fig. 8

Evaluation comments and actual revisions by group and by area

As illustrated in Fig. 9, both groups had the same number of globally and locally revision-oriented comments. The FFPR group (37%) acted 1% more on the feedback and resulted in greater percentage of revisions than the MMPR group (36%). The MMPR group made 35% of the revisions globally and 37% locally. While the FFPR group made 40% of the revisions globally and 35% locally. The low number of revision-oriented feedback both globally (25 versus 65) and locally (37 versus 117) for the FFPR group indicates that the participants have not much utilized peer comments on the paper.

Fig. 9

Clarification feedback and actual revisions by group and by area

Figure 10 depicts the distribution of suggestion feedback by which both groups provided more local than global peer comments. Nearly 87% of the suggestion comments in the MMPR and 90% in the FFPR were revision-oriented. Nevertheless, global suggestion comments (e.g., “Your overview part should stay as it is”) and local suggestion comments (e.g., “You should keep this collocation here”) can be non-revision too. The MMPR group acted 7% more upon revision-oriented comments and made actual revisions (47% versus 40%) comparing with the FFPR groups. While the MMPR group revised their draft locally more than the FFPR group (46% versus 36%), the FFPR group made more actual global revisions than the MMPR group (52% versus 48%).

Fig. 10

Suggestion feedback and actual revisions by group and by area

As shown in Fig. 11, all the alternation comments given by both groups in both global and local areas are revision-oriented. A larger number of alternation comments were locally distributed in both groups. Overall, the MMPR group produced 2% more actual revisions than the FFPR group (28% versus 26%). More specifically, the percentage of revision-oriented feedback in the local area in the MMPR group led to actual revisions was more than that of the FFPR group (28% versus 23%). In the global area, the FFPR group made more actual revisions than the FFPR group (41% versus 26%).

• RQ3: Do the MMPR and FFPR modes improve the students’ IELTS academic task 1 band score?

Fig. 11

Alteration feedback and actual revisions by group and by area

The pre and posttests were given to measure both the FFPR and MMPR students’ IELTS academic writing development. The results of the Shapiro-Wilk test indicated that the data significantly deviated from a normal distribution, p < .5. Therefore, a Mann-Whitney U test was run to examine the participants’ writing development after intervention in the MMPR and FFPR group. As presented in Table 5, the participants in the MMPR group (Mean Rank = 39.94, Mdn = 1.00) revealed more improvements in their writing than the FFPR group (Mean Rank = 33.06, Mdn = .50) in the posttest. However, the analysis indicated no significant difference between the two groups, U = 524, z = − 1.51, p = .131, and r = .17. Table 5 presents the results of the Mann-Whitney U test on the participants’ writing development in the MMPR and FFPR group.

Table 5 Results of the Mann-Whitney U test on participants’ writing development

Discussion

Overall, the findings of this study revealed that the total number of comments, the percentage of revision-oriented comments, and the percentage of actual revisions made by MMPR students were larger than FFPR students. The following section discusses the findings related to the three research questions followed by a discussion of the effectiveness of the two commenting modes.

The discussion of the first question which sought to determine the peer review distribution is divided into four characteristics of frequency, area, nature, and type as well as the IELTS assessment criteria. First, MMPR students generated more comments than FFPR; similarly, Chang (2012) and Ho (2015) found that not only CMPR groups made significantly more comments than FFPR, both their comments were more revision-oriented. Telegram created a more collaborative environment which encouraged more participation and communication among the members.

The second feature was the area of comments. Both FFPR and MMPR made more local comments in total. Unlike some previous studies which reported the role of online peer review in identifying the global areas of writing (Bradley, 2014; Guardado & Shi, 2007; Hewett, 2006; Magnifico, Woodard, & McCarthey, 2019; Saeed & Ghazali, 2016), or the one by Chang (2012) who reported a balanced number of local and global comments. The findings of this study corroborate those which reported higher percentage of local comments in CMPR groups (Chang, 2009; Hanjani & Li, 2014; Ho, 2015; Liu & Sadler, 2003). The reason for the locality of comments might be that students produce more elaborate comments when they gradually become used to ways of exchanging peer comments.

The third feature of the analysis referred to the nature of comments which is either revision-oriented or non-revision-oriented. The results revealed that the revision-oriented comments were more prevalent in both MMPR and FFPR groups. The existence of revision-oriented comments in this study shows that the participants remained on the task and aimed at various problems in their peers’ writings (Saeed, Ghazali, & Aljaberi, 2018). This is in line with other previous studies which reported that students in technology-enhanced environments take part more in making revision-oriented peer review (Bradley, 2014; Ho, 2015; Liou & Peng, 2009; Pham & Usaha, 2015). However, in a recent study by Magnifico et al. (2019), the number of students’ non-revision comments such as cheerleading and praising were more prevalent in online peer review sessions. Furthermore, the majority of revision-oriented comments were local in both MMPR and FFPR. The global revisions made by the two groups were similar; however, the local revisions made by the students varied widely. In other words, the two modes of peer review were successful at assisting students in identifying local writing issues, while the MMPR group made more local revisions than the FFPR group. The low percentage of both global and local actual revisions in the MMPR group corroborates with other similar studies which reported low revision rates in synchronous peer review environments (Liou & Peng, 2009; Liu & Sadler, 2003; Magnifico et al., 2019). Several factors might contribute to this low revision rate in this study: First, the nature of the IELTS as a high-stakes test make students rely on teacher’s comments rather than peers since an incorrect revision reduces their band score. Second, confusion in deciding turn-taking and long conversation maintenance influences the effectiveness of MMPR mode and results in lower revision rates (Ho, 2015). Third, students might only choose the comments they find helpful and neglect the others or use the comments which they know how to revise.

The fourth characteristic was the type of comments. Results indicated that in both MMPR and FFPR group, the most commonly used type of comment was the revision-oriented suggestion in local areas, whereas the least commonly used type was revision-oriented alteration in global feedback. The results are consistent with data obtained in some previous studies (Chang, 2012; Ho, 2015; Ho & Usaha, 2013) in which the number and percentage of suggestion comments were the highest, and alteration comments were among the lowest. However, the findings do not mirror those of Liu and Sadler’s (2003) which reported suggestion and alteration comments by 19% and 46%, respectively, and Ho and Usaha [2013] which showed 29% for suggestion and 27% for alteration. The high percentage of offering locally comments might be because both FFPR and MMPR modes required participants to negotiate with each other and offer suggestions based on their peers’ request. Moreover, the participants might consider the IELTS task 1 sample as a sophisticated way of objective academic writing which requires them to use less direct comment types such as evaluation and suggestion rather than alteration.

The results further revealed that students’ comments were slightly different on the scale of IELTS academic assessment criteria. Task achievement and lexical resources were among the commonly used comments the FFPR group shared in the classroom, whereas the MMPR group focused more on task achievement and grammatical range and accuracy. Surprisingly, the FFPR group made more comments on task achievement than the MMPR group. The possible reasons might be the FFPR group shared more global comments due to the face-to-face nature of communication, while the MMPR group tried to be more specific when commenting in a non-verbal situation. Although it is commonly believed that task achievement and coherence and cohesion target the global area of feedback, they might refer to local areas such as requiring the student to support their sentences with appropriate and accurate data or using linking devices and pronouns. In contrast, lexical resources and grammatical range and accuracy deal with local areas of feedback. In addition, the results showed that both groups improved at IELTS academic writing skills while the MMPR outperformed the FFPR group. Although the difference between the overall band score between the pretest and posttest was not significant, students increased their overall mark by average 0.5 band score which can be noticeable in this high-stakes test.

The discussion of the second research question refers to the incorporation rate of peer comments. Although the number of revisions in MMPR groups was more than FFPR groups, the adoption rate percentage of 41% is not satisfactory. The findings related to the adoption rate of comments seem to be consistent with other research which found similar adoption rate of 48% (Liou & Peng, 2009) and 47% (Liu & Sadler, 2003) and more promising than the one by Paulus’ (1999) study with only 13.9%. The poor adoption rate in Liu and Sadler’s (2003) study might be that the researchers used MOO for online chat, yet they required students to exchange their electronic drafts via email first; therefore, they could not see their peers’ drafts and their chat dialogs simultaneously. For Liou and Peng (2009), posting the comments on weblog which was an asynchronous mode of interaction lacked sense of immediacy (Ho, 2015). However, the results contradict with Min’s (2006) study who reported the high adoption rate of 90%.

There are some other reasons why such a low adoption rate occurred in both FFPR and MMPR groups. Firstly, students in this study might only select the peer comments that know the revision strategies for. Sometimes the comments students receive from their peers might not be informative and meaningful for some students due to lack of knowledge. Secondly, it seems that the peer review training workshop did not make the students more willing to accept and take action after receiving their peers’ comments. As mentioned above, in Min’s (2006) study the high adoption rate was after the peer review training workshop. Obviously, students need to receive more organized and elaborate training not only to improve peer review skills but to adopt more comments and lead them into actual revisions. Thirdly, students’ uncertainty about the quality of peer comments and interaction difficulties between the peers intended meanings might affect the peer comments’ implementation (Ho, 2015).

The third research question addressed the effect of peer review on improving students IELTS academic writing skills development. The results showed that both groups improved at IELTS academic writing skills while the MMPR outperformed the FFPR group. Although the difference between the overall band score between the pretest and posttest was not significant, students increased their overall mark by average 0.5 band score which can be noticeable in this high-stakes test. This finding is in accord with a recent study by Neumann and Kopcha (2019) which showed a lack of significant progress in students writing due to lack of attempts from students’ side. As writing improvement is measured based on three factors of fluency, accuracy, and complexity, students’ development in their IELTS band score only is linked with accuracy and complexity. The fluency is not affected as students are required to write between 150 and 170 words for IELTS task 1 writing. The improvement in students’ overall band score cannot be associated directly with the peer review sessions, other factors such as the intervention itself can affect the IELTS writing skills among the students. The results are in accord with previous studies indicating that peer review improves writing revision skills (Bikowski & Vithanage, 2016; Bradley, 2014; Ho, 2015; Wang, 2015) by encouraging a collaborative learning environment. The theory of collaborative learning provides a useful account of how peer review provides students with opportunities to facilitate the completion of a writing task through dialog and interaction with their peers (Hirvela, 1999). Moreover, the findings are in line with those of previous studies that supported the use of mobile applications for developing student’s writing skills in synchronous environments (Andujar, 2016; Tang & Hew, 2017).

Although both FFPR and MMPR group experienced synchronous modes for their peer review, the effectiveness of commenting varied between the two groups. Telegram provided a less-threatening synchronous environment; however, it created some challenges in the form of interruptions, distractions, maintenance, and timing throughout peer review sessions. The challenges are divided as follows: (1) Some students interrupted the lessons by asking sudden questions or taking part in mobile discussions with delay. (2) During the peer review session, some discussions went off the topic when students used stickers to show surprise or happiness, and all these things made it hard to get the attention of everyone back to the discussion of problematic sentences. (3) When the discussion of an issue continued, the students could not see the entire chat on their screens; they thus found it hard to trace the main discussion. (4) Although, the researcher discussed the mobile review sessions policy and procedures at the beginning of the course and used some signs to manage the time and turn-taking, the difficulty of determining the turn-taking among the students, a feeling of rush to type their comments, followed by losing the interest after several comments on a specific paragraph decreased the effectiveness of peer review in Telegram. In addition, students who did not have high typing speed fell behind the concurrency of mobile interaction. (5) The duration of peer review conversations was time-consuming. One possible reason might be the lack of non-verbal communication in Telegram. As students were from different nationalities, lack of non-verbal communication might create communication barriers and misunderstanding (Bradley, 2014). (6) The speed of receiving and frequency of peer comments was faster in the MMPR group because 8 students shared their comments one after another in Telegram, while only two peers shared theirs in the FFPR group. (7) Some participants who were more willing to take part in conversations initiated and led the discussion and did not let others express their ideas. Finally, the small size of the screens, phone storage space, and distractions coming from other apps was among the common technical difficulties.

The FFPR group reported valuable results too. FFPR group used their guidance sheet and peer review logs efficiently, except some comments were not eligible for their peers. The participants did not report any difficulty regarding the handouts as they all were photocopied and printed for them before the class begins. As for the FFPR group sharing comments on the printed logs and modifying the notes were convenient, turn-taking procedure was face-to-face and straightforward, exchanging peer comments was done with no rush, discussing the comments and revisions were more organized and focused. Nevertheless, the only challenge was at the beginning of the course when some students were unwilling to write all the negative or positive comments in their logs and preferred to say it verbally. The researcher encouraged them to write both the corrective and positive comments.

Limitations, implications, and suggestions for future research

Some limitations are associated with this study. First, the small sample size of 72 participants and a limited number of 3 writing tasks for peer review will affect the reliability of the results and are not meant to be generalized outside the domain of this study. Moreover, this study did not conduct the inter-group and inter-assignment analysis to find out the differences between the 3 groups of MMPR and 3 groups of FFPR and their performance on different tasks. Finally, this study did not use a qualitative measurement of analysis both to explore students’ attitudes and perceptions towards peer review in both groups and why some students did not incorporate some comments into their revisions.

Several pedagogical implications are drawn from the findings in spite of the limitations. First, mobile-mediated peer review in either synchronous or asynchronous modes may assist both teachers and students to experience a collaborative writing environment outside the classroom due to its various technological affordances. For instance, teachers can introduce these mobile apps for short and less formal discussions outside the classroom by encouraging one or two students to take a photo from their paragraphs and posting it in the group at a time. The current study’s findings might be helpful for education professionals who design peer review training workshops. Besides, curriculum designers might customize and design some supplementary materials to enhance the use of peer review in the classroom.

Further research is required to investigate a larger sample of participants to augment the generalizability of the findings. The results might be different if the same peer review practice is implemented in a larger population by considering various individual differences such as age, nationality, gender, and language background. Moreover, future research should use more qualitative approaches for a deeper understanding, and to use a variety of writing tasks to explore the effectiveness of peer review in a wider scope. Another interesting area of investigation will be training EFL/ESL instructors in order to implement mobile applications in their classrooms and use them mainly for developing their students’ writing skills. Finally, research can examine the use of mobile applications for other international exams such as TOEFL, SAT, or GRE.

Conclusion

This study investigated how different modes of peer review affect the commenting patterns and revisions among L2 writers. Some considerations should be made while comparing FFPR and MMPR modes: (1) the findings of the present study confirmed that the number of overall comments and the percentage of revision-oriented comments made by mobile-mediated peer review mode was larger, consequently, resulting in a larger number of revisions. Thus, it is suggested that making peer comments with the use of mobile applications is effective in developing academic writing skills. (2) Although the use of mobile applications resulted in a more efficient peer review practice, the FFPR has its own advantages. Approaching peer review from both modes will provide researchers with an overall understanding of group dynamics and expectations. The act of peer review will be more influential when combined with either face-to-face or asynchronous and either FFPR, CMPR, or MMPR modes of communication which encourages students to generate more focused, deliberate peer comments. (3) Language teachers should encourage interaction exchanges not only in learning environments but in social spaces, especially in CMPR and MMPR. Students should involve in social exchanges in peer review groups to resemble a real-life experience of learning. Moreover, this engagement motivates them to accept their peers’ criticism and integrate their comments into their revisions. Social interaction in peer review provides the opportunity for learners to use English as a means of socialization and communication. (4) Well-organized training sessions are required to not only familiarize students with the procedures of peer review, new technologies, and ICT skills. Moreover, explicit instructions are needed, especially for global-oriented comments to prepare the students on what and how to comment on their peers’ writings by considering both global and local issues. (5) Equally important, as MMPR affects the group dynamics, participation level, discourse patterns, classroom time, teacher and students’ role, classroom management, and assessment, teachers should be trained to be able to create collaborative opportunities for L2 writers by being aware of both modes of peer review.