SOLO taxonomy as EFL students’ disciplinary literacy evaluation tool in technology-enhanced integrated astronomy course

Abstract

Whilst the role of disciplinary literacy in science, technology, engineering, and mathematics (STEM) education is becoming more prominent, there appears to be little adaptation or allowance made for English as a foreign or second language (EFL/ESL) students learning science in English, as well as no clear understanding of what comprises disciplinary literacy in science. In this article, we define the construct of disciplinary literacy in science education and justify the use of the Structure of the Observed Learning Outcome (SOLO) taxonomy (Biggs and Collis, Teaching for quality learning at university 1982) as a tool for measuring EFL students’ disciplinary literacy development. This SOLO taxonomy was trialled in a STEM integrated astronomy course enhanced with a digital storytelling approach. Data were collected from adolescent EFL students’ written responses in their pre- and post-Astronomy Diagnostic Test. The results show a positive change in participants’ science disciplinary literacy development and contribute to the understanding of the EFL/ESL students’ science disciplinary literacy development and assessment.

Introduction

The role of literacy for science, technology, engineering, and mathematics (STEM) education is gaining international recognition (Bauerle et al. 2014; Dutton and Rushton 2018) that ignites a debate about how to define literacy within and across each of the STEM disciplines. Research recognises that “disciplinary literacy” is a unique and independent phenomenon, but does not provide a clear understanding of what it comprises as an empirically valid measurable construct that is generalisable within the educational context (Spires et al. 2018). Inconsistency in defining disciplinary literacy creates an assessment issue as the construct is unclear. Therefore, there is a need to establish a definition of the construct in science education in order to develop appropriate instructional and assessment approaches.

This article contributes to the definition of disciplinary literacy within science education and outlines the outcomes of a pilot integrated STEM course targeted at disciplinary literacy development in astronomy, one area of science curriculum content in many educational jurisdictions. This research also trialled a new approach to disciplinary literacy assessment in science education and argues that the Structure of the Observed Learning Outcome (SOLO) taxonomy (Biggs and Collis 1982) is a valid science disciplinary literacy assessment tool.

The significance of this study is underscored by the growing trends in international student populations around the globe (Langman and Fies 2010; Taylor 2014; Williams 2011), coupled with an increasing demand for English-speaking STEM professionals (Bauerle et al. 2014), a decline in students’ engagement in learning science (Goldspink and Foster 2013), and the poor performance in standardised science literacy tests of EFL/ESL students in comparison with the test results of the students who speak English as their first language (National Assessment Program 2015; The Nation’s Report Card 2015). Therefore, this study specifically targeted EFL students’ disciplinary literacy needs in learning science. This research used digital storytelling (DST) as a teaching approach to develop disciplinary literacy in EFL students enrolled in an astronomy course. This course integrated STEM areas within an inquiry process (Prairie and Buckleitner 2005) and created opportunities for authentic classroom interactions that scaffold EFL students’ shared knowledge construction (Van Aalst 2009).

Disciplinary literacy construct in science

The recent study by Spires et al. (2018) brings new insights on disciplinary literacy in STEM education and argues that literacy is a multidimensional construct that represents students’ competency in reading, writing, and thinking processes. Thus, reading and writing in science are the means by which various phenomena could be explored, described, and explained (Goldman et al. 2016), or that reading and writing reflect students’ cognitive processes.

Science has its own discourse conventions distinctive from the discourses of other disciplines in both style and genre (Shanahan and Shanahan 2008; Yore et al. 2003). Some of the scientific discourse functions are as follows: argumentation, claiming, informing, and aligning a claim to an argument (Yore et al. 2003). Scientific style is usually featured by clarity and brevity of the presented information (Holland et al. 1986).

According to Fang (2013), the major genres of science discourse are as follows: “procedure, procedural recount, description, report, explanation, and exposition” (p. 277). Spires et al. (2018) define disciplinary literacy in science as a combination of fluency in scientific terminology, an ability to synthesise multiple sources such as prose and various graphic representations, for example, graphs or diagrams (Goldman et al. 2016; Shanahan and Shanahan 2008), and to apply analytical thinking, for example, linking data to arguments and conclusions (Gillis 2014).

In their research, Spires et al. (2018) identify three core literacies that comprise the disciplinary literacy concept: source literacy—an ability to work with sources, for example, identification of an author’s point of view; analytic literacy—an ability for quantitative reasoning; and, expressive literacy—an ability for interpretation of figurative language and rhetorical devices. The authors argue that science disciplinary literacy is mainly a combination of source and analytic literacies, and claim that science teachers are mostly employing the analytic component of disciplinary literacy since an understanding of authors’ claims and data interpretation in science require learners to move between prose and visual representations whilst reading (Spires et al. 2018). Moje’s (2008) definition of science disciplinary literacy summarises all of the above-mentioned features and is therefore adopted for the purposes of this study. Hence, science disciplinary literacy is students’ ability “to communicate their learning, to synthesise ideas across texts and across groups of people, to express new ideas, and to question and challenge ideas held dear in the discipline and in broader spheres” (Moje 2008, p. 99).

Literacy assessment

The increasing complexity in defining “literacy” prompted the evolution of literacy assessment procedures and instruments. Biggs and Tang (2011) argue that the outcomes of students’ learning can be traced quantitatively through the amount of detail in students’ answers, and qualitatively through the structural patterns they use to organise their ideas. In other words, disciplinary literacy acquisition implies that students shift from memorisation and reproduction to the personalisation of knowledge (Campbell et al. 1998). The Structure of the Observed Learning Outcome (SOLO) taxonomy suggests that this shift in students’ knowledge is reflected in their cognitive processes (Biggs and Collis 1982). Therefore, students’ writing could serve as an indicator of the complexity of their disciplinary literacy development.

Hypothesising that the SOLO taxonomy accurately reflects a student’s level of thinking, Rembach and Dison (2016) piloted the use of the SOLO taxonomy as a strategy to develop pre-service teachers’ disciplinary literacy in the social sciences and in inclusive educational contexts. Familiarising students with the rubric based on the SOLO taxonomy enabled the researchers to gain an insight into students’ knowledge construction by encouraging higher order thinking and critical engagement with the discipline (Rembach and Dison 2016). They argue that the strength of the SOLO taxonomy lies in its approach to the complexity of human knowledge acquisition that embraces both content and context. However, this is only possible when the assessment tasks are not matched to the learning outcomes in a formulaic way. Otherwise, there is a risk of the mechanical use of the taxonomy to help them create their responses to the assessment task (Rembach and Dison 2016).

Adopting Moje’s (2008) definition of science disciplinary literacy assumes that knowledge transformation will be reflected in students’ level of thinking. Therefore, the response could be assessed against the different levels of the SOLO taxonomy (Biggs and Collis 1982). This will reflect the complexity of the student’s thinking processes in responding to the assessment question.

EFL/ESL students

The increase of foreign student populations in education causes a discrepancy in the academic science performance between the students who learn English as a foreign or second language (EFL/ESL) and monolingual English speakers (National Assessment Program 2015; National Center for Education Statistics 2017, 2018). Since literacy is multidimensional (Spires et al. 2018), literacy acquisition may not be a linear process (Shanahan and Shanahan 2008). The differences in the literacy development process for monolingual and non-monolingual speakers of English suggest that EFL/ESL students may experience more difficulties whilst learning school subjects. Therefore, teachers may need to provide some scaffolding strategies to narrow the literacy gap between their monolingual and EFL/ESL students (Fung and Yip 2014; Kasper 2000; Menken 2013; Ryoo 2015).

Technology in teaching science

Considering that the ultimate goal of literacy instruction in STEM education is to empower learners with the ability to critique and modify the already existing disciplinary knowledge (Office of the Chief Scientist 2014; NGSS Lead States 2013), students need to develop an understanding about a process of knowledge construction in a discipline (Lee 2007) rather than merely memorise information presented by a teacher. This disciplinary literacy acquisition goal could be achieved by integrating technology in the educational process to create a context for sharing, collaborating, and networking (Nicholas and Ng 2012). Technology can grant independence to students and a responsibility for the selection and processing of information (Anderson et al. 2018; Kasper 2000; Ng 2011; Peters 2000). This may enable students to conceptualise and analyse their understanding of science (Scull and Bianco 2008), and hence, may be particularly favourable for EFL/ESL students’ learning (Anderson et al. 2018).

Digital storytelling creates a context that provides students with control over their learning (Pirbhai-Illich 2010; Pirbhai-Illich et al. 2009), and teaches them to make informed choice among resources (Ashman and Elkins 2009; Laycock and Stephenson 1993; Sprague and Pixley 2008). Whilst working on their videos, EFL/ESL students become engaged in the authentic decision-making process. This type of students’ interactions creates opportunities for disciplinary literacy development, since each decision requires group negotiation, multiple trials, and drafting. In other words, students become responsible for the content of their course and communication of their learning outcomes to various audiences (Kasper 2000) that provides a scaffolding of disciplinary literacy acquisition in science (Moje 2008).

Authoring software, such as movie editors, is becoming increasingly accessible to all users (Godwin-Jones 2000). For example, becoming proficient in using the iMovie video-editing program does not require much time due to the automation of many tedious processes and the user-friendly and intuitive interface (Godwin-Jones 2000). Thus, it could easily be used to engage students in the process of learning without being too time-intensive (Gruba 2006). Nevertheless, despite the increase in interest towards technology-mediated instruction in education and growing accessibility of authoring software, there is little research on the integration of authoring software to enhance EFL students’ disciplinary literacy development. Specifically, an analysis of the existing research revealed that there are no studies that investigate the impact of engaging EFL school students in the process of digital storytelling about the concepts they learn within the context of STEM integrated astronomy course. Therefore, this study addresses the following research question: How does an integrated astronomy course enhanced with digital storytelling affect EFL students’ disciplinary literacy development in the astronomy context?

Methodology

Study design

This research is an extension of an established educational package in astronomy aligned with the Australian science curriculum in primary/elementary and secondary school science: A Journey through Space and Time (JTST) (McKinnon 2013). The research uses data from a pilot study (Chubko et al. 2019); digital storytelling teaching components were integrated with the original JTST resource package, so that EFL students could not only inquire into the astronomy concepts, but also share and discuss their understanding of these new concepts with their peers. It was expected that approaching the newly acquired astronomy concepts from multiple perspectives through the sequence of the video-making stages such as brainstorming, scriptwriting, storyboarding, shooting, and editing would facilitate EFL students’ acquisition of the disciplinary literacy in astronomy. Therefore, this research aligns itself with social constructivist theory (Vygotsky 1962).

The whole project (Chubko et al. 2019) lasted for 25 h over a 9-day period. It was offered as a bonus course to the EFL students who were graduating from the pre-intermediate level of general English course conducted in one of the extracurricular educational centres in Bishkek, Kyrgyzstan, during December 2017. Table 1 presents the outline of this pilot extracurricular astronomy course The Secrets of the Stars.

Table 1 The Secrets of the Stars course program overview

Participants

Participants were six Kyrgyzstani EFL students aged between 12 and 16 years. There were three males: Manas, Stas, and Talant; and three females: Diana, Elena, and Sayora. For ethical reasons, students’ real names have been substituted with pseudonyms. These students were the recent graduates of a pre-intermediate level English course and had similar entry levels of general English language proficiency. For all of the participants it was the first time they were learning science with English as the medium of instruction. Traditionally, their language of instruction was Russian. None of these students had received any instruction in astronomy prior to their participation in this course.

Ethics approval and consent

This research was granted approval from the University’s Human Research Ethics Committee (HREC) on March 8, 2017. All participants received an information letter about the project and because of the age of the students, an informed consent document to participate in this research was signed by students’ parents or legal guardians. Approval was also sought from the participating site.

Data collection tools

The data were collected through pre- and post-qualitative responses extracted from the Astronomy Diagnostic Test (ADT) included in the JTST package (McKinnon 2013) used in the pilot course (Chubko et al. 2019). Previous research about astronomy education used the ADT to explore students’ astronomy content knowledge development and students’ alternative conceptions about astronomy (Slater et al. 2018). However, as argued in the earlier section of this paper, the SOLO taxonomy (Biggs and Collis 1982) can also be applied to the written responses of the ADT in order to gain an insight into students’ astronomy disciplinary literacy development.

Data analysis

The Structure of the Observed Learning Outcome (SOLO) taxonomy (Biggs and Collis 1982) measures students’ reasoning development, but it does not measure the factual correctness of the answer (Fig. 1). In this research, students’ ADT written entries were assessed on a 0 to 5-point scale based on Biggs and Collis’ (1982) taxonomy guidelines:

• 0- no entry was made,

• 1- Prestructural level of thinking—the answer repeats a question or is not relevant to the topic,

• 2- Unistructural level of thinking—only one piece of relevant information is provided,

• 3- Multistructural level of thinking—the answer contains two or more pieces of relevant information often connected with “and”,

• 4- Relational level of thinking—the answer shows the relationships among the pieces of relevant information, and

• 5- Extended abstract level of thinking—the answer is connected to the bigger concept or theory.

Fig. 1

SOLO Taxonomy (Biggs 2003, p. 48)

As a 15-question version of the ADT (McKinnon 2013) was used, the maximum SOLO score attained by a student could be 75 points, and the minimum—0 points. The maximum score could be attained only if all the questions would be answered at the extended abstract level of thinking. A 93% inter-rater reliability was achieved using this approach for marking, with two researchers independently assessing students’ written entries. Quotations of students’ ADT written entries are given verbatim and therefore may contain original grammatical errors made by the students.

Results

Table 2 presents a summary of each participant’s overall SOLO rating (Biggs and Collis 1982) distribution. Analysis of these data shows that the participants improved their SOLO rating (Biggs and Collis 1982) between the pre- and post-test occasions, with a minimum shift of +2 and a maximum of +27 points. However, it is also clear that this increase in SOLO rating was mainly attained due to the students’ attempt to explain more answers in their post-test rather than they did in the pre-test (Table 2). Therefore, in this case, the gain score is not an accurate reflection of students’ disciplinary literacy enhancement.

Table 2 Overall SOLO taxonomy rating

To gain an understanding of EFL students’ disciplinary literacy development, all non-English and non-paired entries were removed from the sample. Out of the 90 possible paired entries, were left 33 matched entries. This new matched sample was exported to SPSS and analysed (Table 3). The Cohen’s d effect size was calculated from the difference between pre- and post-test scores, where d = 0.41. This means that there was a moderate change in students’ disciplinary literacy that might be attributed to involvement in the astronomy course enhanced with video-making. Overall, the results suggest that there is a moderate improvement in students’ disciplinary literacy development.

Table 3 Descriptive statistics

Even though the participants were allowed to use Russian, their L1, in cases where they struggled to find an appropriate way to express their answers to the ADT questions in English, it should be noted that there were no entries in Russian in the post-test and only three entries in Russian in the pre-test throughout the whole sample; one made by Diana, and two made by Manas (Table 2). These responses in students’ L1 allow to track the shift occurring in students’ disciplinary literacy as a result of engagement with this astronomy course enhanced with the DST teaching approach. One of the examples is Diana’s reasoning of her answer to the question 3. This question contained a diagram representing an observation of Moon Phases changing over a week. To answer this question, students had to explain the cause of such an observation. In her pre-test, Diana answered:

Так как тень от Земли падает на Луну под разными углами, мы не всегда видим луну полностью, а только её освещенную часть. Этот процесс (изменения видимой части) называют лунным циклом.

[Translation: Since the shadow from the Earth falls on the Moon under different angles, we do not always see the whole Moon, but only its lit part. This process (the process of changing of the visible part) is called the Moon cycle.]

Although this written response provided by Diana in Russian is obviously an alternative conception, it was rated as a five on the SOLO taxonomy (Biggs and Collis 1982). In her answer, Diana first establishes a shape to angles relation, and then she generalises her answer to the bigger idea, Moon cycle, which is indicative of Extended Abstract level of thinking. In her post-test, Diana answered the same question (question 3) in English and scored only four points on the SOLO taxonomy (Table 2):

She saw the Moon in different shapes, because the Moon has 8 faces, which changes in a certain order every day.

Even though in her post-test entry Diana abandons her alternative conception about the shadow of the Earth causing the phases of the Moon, her reasoning now only presents the facts and is not generalised to the overarching concept. Nevertheless, in her post-test written entry to question 7 (Table 2) on the scale model of the Earth-Moon system, Diana demonstrates that she actually has the ability to present an argument at the Extended Abstract level of thinking in English (Fig. 1) that did not occur earlier in her pre-test. In her answer to question 7, Diana uses the same structure as she used in her L1 answer to question 3:

I depend on calculating at the process of scailing (sic). For this case it’s calculation will be 9 m.

Diana takes her answer to the Extended Abstract level of thinking by referring to the generalised overarching concepts of Moon cycle in L1 answer to question 3 and scaling in her English answer to question 7.

During this pilot astronomy course, Diana did not participate in making a video about Moon phases even though she participated in the Moon phases inquiry activities along with all other students. However, Diana was engaged in the video-making about Scale Models of the Solar System (Table 1) and this possibly helped her to develop an understanding about the scale model of the Earth-Moon system and equipped her with the terminology to argue about this concept in her post-test entry in English.

During the course, Manas was engaged in making a video about Seasons (Table 1). Two of his pre-test entries were written in his L1; his answer to question 4:

Земля наклонена и та сторона которая наклонена к солнцу там лето, а та сторона которая отклонена от солнца там зима.

[The Earth is tilted and that side that is tilted to the Sun, there is summer, the side that is further from the Sun, there is winter.]

and his answer to question 8:

У нас есть времена года потомучто Земля наклонена, а если она не будет наклонена то времен года не будет.

[We have seasons because the Earth is tilted, but if it were not tilted we wouldn’t have seasons.]

Both of these answers were rated as four out of five. In his English post-test answer to question 4, Manas demonstrated the same level of thinking as in his L1 (Table 2):

Earth is tilted on 23^o and the side wich (sic) is facing the sun there is summer and the part that is away from sun there will be winter.

His answer to post-test question 8 scored one-point higher (Table 2):

because of the tilt Earth has seasons but with out (sic) this we would just have

day and night.

The comparison of Manas’ answers in his L1 and in English demonstrates that Manas was familiar with the concepts prior to the course, but did not have enough technical vocabulary to explain his understanding. Most probably Manas acquired the concept of a tilt during the process of making the video about Seasons that helped him to present his argument in English.

Discussion

The present study was designed to determine the effect of a digital storytelling approach on EFL students’ disciplinary literacy development in astronomy when taught in English. The overall results showed that engaging EFL students in making videos about their learning during the astronomy course had a positive effect on their disciplinary literacy development. All of the research participants had a positive gain between their pre- and post-ADT performance when coded against the SOLO taxonomy scale (Biggs and Collis 1982). This means that they attained a more advanced level of expressing their thinking, and therefore, enhanced their disciplinary literacy in a foreign language.

Very little was found in the literature on the question of what comprises disciplinary literacy in science (Moje 2008; Spires et al. 2018). Therefore, this finding broadly supports the existing research on the effect of technology on EFL students’ learning of science (Godwin-Jones 2000; Nicholas and Ng 2012; Ng 2011; Pirbhai-Illich 2010; Pirbhai-Illich et al. 2009; Scull and Bianco 2008) and contributes to our understanding of science disciplinary literacy as a measurable construct generalisable within the educational context. The positive outcome from this pilot study (Chubko et al. 2019) provides a stable foundation for the replication of the research with a larger sample. This will allow for an enhanced exploration of the true potential of this instructional approach towards disciplinary literacy development.

Another important finding from this research was that the EFL students enhanced their disciplinary literacy in a foreign language. In the pre-test, the participants left many questions unanswered, which might be a result of the students not understanding the questions being asked. During the astronomy course, the EFL students were engaged in making videos covering key astronomy concepts associated with the ADT test. Whilst the students made multiple edits on their videos, they participated in ongoing discussions about the astronomy concepts with their peers. As a result of the interactions, the students improved their post-ADT scores.

As mentioned in the literature review, there is a literacy gap between EFL/ESL and monolingual students (Fung and Yip 2014; Kasper 2000; Menken 2013; Ryoo 2015). Given that prior studies noted the importance of mediating teaching with some scaffolding strategies (Fung and Yip 2014; Kasper 2000; Menken 2013; Ryoo 2015), a possible explanation for the EFL students’ post-ADT test results was that students received appropriate scaffolding from the DST experience. The findings revealed an increase in students’ disciplinary knowledge enhanced by using a DST approach to teaching. Further research will focus on a greater understanding of the relationship between video-making and science disciplinary literacy development in a foreign language.

Limitations

In this study, it was impractical to attempt a control of diverse external factors (e.g. sickness, end of school year examinations, evening classes, and distance from home), which influenced the participation of students in the astronomy course. Additionally, the course might have encouraged some participants to read more widely beyond the confines of the classroom. Consistent with the small participant sample, generalisability does not apply, but the next research phase should engage a larger sample of EFL learners. Despite the small sample size used in the original research pilot, the outcomes provide a valuable indicator of the disciplinary literacy development in EFL students and a foundation for further research.

Implications

We propose to complement students’ standardised evaluations with the SOLO taxonomy approach, so that teachers can measure shifts in students’ level of thinking to gain a better understanding of their disciplinary literacy development. Using SOLO (Biggs and Collis 1982) or similar taxonomies may prevent teachers from confusing disciplinary literacy with content knowledge acquisition, where the former reflects students’ cognitive processes and the latter is limited to factual information acquired or memorised by the students. The development of disciplinary literacy will allow students to acquire the necessary information; whilst in the case of gaining the content knowledge, students may only be relying on their memory. The outcomes of this study suggest that incorporation of DST in teaching repertoire would be beneficial for EFL students. Even though this study is restricted to astronomy context, we believe that this strategy could be integrated into other disciplines to bridge EFL and non-EFL students’ learning outcomes.

Conclusion

The aim of this research was to explore how the integration of digital storytelling within the astronomy course would influence EFL students’ astronomy disciplinary literacy development in English. Overall, this pilot showed that engaging EFL students in a DST-enhanced course helps to scaffold their disciplinary literacy development in a foreign language. Changes observed in the participants’ pre- and post-test SOLO ratings indicate that they attained a more advanced level of expression in a foreign language; and, therefore enhanced their disciplinary literacy in a foreign language. The positive increase in students’ knowledge using the DST teaching approach creates an argument for further research into understanding the relationship between video-making and science disciplinary literacy development.