An enduring lens for a continuing problem: Close analysis of conceptually scored cloze items

Language Testing in Asia

Table 1 Mean Sample Difficulty on English Language Features

Student Sample	No. of pupils	Noun	At’cle	Verb	Prp’n	Tchlty	Word Stacks	Pas’ve Voice	Cohsv Device	Overall D’fclty
		%	%	%	%	%	%	%	%	%
		*wrong*	*wrong*	*wrong*	*wrong*	*wrong*	*wrong*	*wrong*	*wrong*	*wrong*
		n=18	n=5	n=12	n=5	n=10	n=6	n=3	n=15	n=50
U/G Chinese	114	53.17 (SD=13.73)	39.30 (SD=21.27)	51.90 (SD=21.12)	47.19 (SD=18.65)	47.81 (SD=20.51)	32.02 (SD=19.97)	42.98 (SD=28.65)	37.60(SD=17.87)	48.63 (SD=15.09)
U/G Chinese	*Rank* ^@	1	4	2	3	1	4	2	3
Secondary EALD ^#	*Rank*	2	4	3	1	3	4	2	1
Secondary EALD ^#	679	42.89 (SD=18.06)	28.19 (SD=26.93)	39.80 (SD=25.84)	48.22 (SD=23.80)	31.69 (SD=20.29)	18.14 (SD=18.63)	36.38 (SD=33.19)	46.55 (SD=19.03)	43.41 (SD=19.83)
Secondary English	127	47.55 (SD=23.18)	30.08 (SD=28.72)	40.16 (SD=29.55)	41.10 (SD=26.94)	39.29 (SD=26.49)	25.33 (SD=26.05)	34.12 (SD=34.75)	45.88 (SD=23.36)	44.38 (SD=24.34)
Secondary English	*Rank*	1	4	3	2	2	4	3	1

Notes:
^@ Rank = Rank within category, most difficult = 1.
^#EALD = English as an Additional Language or Dialect.
Table 1 is based on analysis of student responses on a single cloze test.
*Student data was recoded so that a clear error = 1 and acceptable replacement = 0.
*Cloze test deletions were classified by language category.
*Two SPSS routines were written (“dictionary categories” & “modern grammar categories”).
These did the following:
1 count number of items representing a particular language category (eg., nouns) coded as clear error.
2 divide that number by the number of items representing that category (eg., nouns) deleted (in this case, 18) to give the mean category (in this case, noun) difficulty.
3 multiply that mean by 100 to yield a percentage.
4 average that mean across the particular student sample (114 students for the undergraduate Chinese group: 53%).
The top row language descriptors are discussed previously, in Deletion categorization. The number at the base of each heading refers to the number of deletions categorized as reflecting that descriptor.