Testing

The information below can be used to improve attainment tests, but are primarily used with tests that assess a single ability or personality characteristic. Concepts of validity and reliability are used by both research paradigms but are particularly associated with the Scientific Paradigm.

Validity simply means that a test is measuring what it is supposed to. Whilst validity in research is crucial it is often under-recognised in discussing research outcomes. Achievement tests/subject examinations, for example, must relate to syllabus aims and objectives. Yet comparisons of achievement over time - or in different countries - are regularly done when the aims and objectives differ sometimes widely. It is for practical reasons impossible to compare the effectiveness of teaching, 30, 20 or even 10 years ago with that done now because even where the topic(s) are still taught, not only may the time to teach them differ because of the insertion/removal of other topics, but the cultural context will change the importance of learning the ideas to the children concerned - Why learn multiplication tables when there are calculators? And even if a Government's national curriculum insists that tables will be learned, will schools have the time and pupils the inclination to chant multiplication tables etc. every morning at assembly (the writer's experience). And even if the schools did insist on such chanting, children won't need to use them as part of their normal life (metric money and measurement systems don't require the mathematical skills of non-metric ones). Finally, all such comparisons must be limited because the only evidence available is that supplied by tests used at the earlier time - inevitably such tests cannot assess new skills and may even disadvantage modern children because of the context in which the topic is presented.

Content validity can be described statistically, but is normally assessed qualitatively. Construct validity is more difficult to assess but would be normally described using correlation or, say, 2-way chi-test scores.

Reliability is the assessment of the consistency of the evidence collected, the comparability of different observers/ markers/sources of information etc. and was traditionally assessed using correlation scores. Thus a test would be given twice and the similarity of scores obtained by the same pupils would be compared (test-retest reliability). Because of the difficulties incurred in arranging retesting, quantitative assessment of reliability is now commonly assessed using internal comparisons (split-half reliability).

Although, split-half reliability would originally be assessed by comparing two halves of the original test (say, odd questions marks compared with even question marks), such a design would be impractical in many attitude questionnaires. In such cases, the researcher needs to either identify parallel portions of the test/questionnaire attitude scale or at least check the correlation of particular pairs of questions. Cronbach's Alpha Reliability analysis allows the researcher to assess the reliability of measurement/atitude scales because the analysis compares each item with the whole group.

Triangulation is a term particularly used by researchers associated with the socio-anthropological paradigm. Whilst it is a method of assessing how reliable evidence is, it is surprisingly often often assumed to be checking validity

The original meaning of the term triangulation is that it is the method surveyors use to create accurate maps. A base-line has to be created (in England this was a measured mile on Salisbury plain). The two ends of the line are the initial places for observations, from these two points the position of any third point can be accurately determined. It is essential to recognise with triangulation that the findings are only relevant if what is being measured are valid

II. Tests of Facility and Discrimination. These calculations are carried out to check that each item is being used effectively Deale (1975) Assessment and Testing in the Secondary School. Schools Council Examination Bulletin 32. London: Evans/Methuen.

Discrimination ensures that the items are working in concert: a low discrimination score (less than 30%) means that the item would be reducing the effectiveness of the test, because it would be measuring something else. Too high a score and the item would only be answered correctly by the most able children and would not help identify those with some ability.

Facility measures the percentage of pupils who answer the item correctly: there is really little point in having questions that are so difficult/easy that few children can answer them correctly/incorrectly - note, with a choice of three answers, a facility of 33% simply reflects the effect of chance.

It is already well established that rhyme awareness in pre-readers is crucially important to the later development of basic literacy skills. It is therefore seems appropriate that children whose rhyming skills are weak despite appropriate experience, are identified for intervention before they start school. In the study described here 240 children participated in class-based language routines before being assessed on explicit rhyme awareness. Further intensive training followed for those identified as having a specific weakness in making rhyme judgements. Although the effects of training are not yet known observations on young children’s responses to phonological assessment are available and these are presented here for consideration.

The ability of pre-school children to make judgements about the sound structure of words is now recognised as a powerful predictor of progress in written language. Studies, notably those by Lynette Bradley and her colleagues, have demonstrated that young non-readers can, given the experience of nursery rhymes and jingles, spontaneously discover that syllables can rhyme (Bradley, 1988). Additionally, Maclean, Bryant and Bradley (1987) have shown that children who can demonstrate rhyming knowledge make better progress as beginning readers than their peers who are insensitive to the sound structure of language. Elsewhere, there is an abundance of evidence which suggests that weaknesses in the phonological domain, often revealed through an inability to make phonological judgements, is at the centre of severe written language difficulties (Olson, Wise, Conners, Rack and Fulker, 1989; Stanovich, 1990; Snowling, M., 1991)

Following the work of Rebecca Treiman (1985) and Usha Goswami (1994) it is now generally believed that phonological awareness, especially rhyme awareness, supports the development of alphabetic skills by indicating the segmental nature of spoken language, particularly signalling the division of words into onset - the ‘sh’ in ‘ship’, and rime - the ‘ip’ in ‘ship’. In the normal course of literacy learning readers will observe that patterns which they hear (rhymes) are often represented by letter patterns which they see. That is, they will use analogies to guide reading and spelling attempts before they have fully mastered the alphabetic code with its numerous rules for encoding individual sounds or phonemes.

The accumulated research evidence has given rise to a number of studies which aim to promote phonological awareness in young children, in the expectation that poor sensitivity could place the child at risk of reading difficulties and academic failure: Lundberg (1994) summarises the results of a large-scale longitudinal Danish study in which 235 six to seven year-old pre-schoolers were given intensive training, by pointing to the significant and sustained improvement on twelve measures of literacy performance of the experiment group over a control group; similarly by identifying an at-risk group, on the basis of very poor performance on a range of phonological measures, such as syllable segmentation, rhyme awareness, phoneme analysis and synthesis, Lundberg and his colleagues concluded that pre-school phonological training can prevent the emergence of phonologically-based literacy difficulties.

Similarly, as part of their longitudinal study Maclean, Bryant and Bradley (1987) assessed children from the age of 3 years 4 months on a range of phonological measures including rhyme detection which was demonstrated by their ability to judge which word in a set of three did not rhyme with the other two. While the children were still under 4 years old they also tested children’s knowledge of traditional nursery rhymes. Subsequent analyses showed a powerful connection between nursery rhyme knowledge and rhyme detection. In the same study Bradley et al were also able to confirm a causal relationship between early phonological awareness and reading and spelling success. This study incorporated a training programme for children with poor phonological awareness, which was conducted over two years. During this time the children were in mainstream schooling and were therefore also receiving classroom instruction in literacy.

However, given the abundance of evidence which confirms the importance of pre-reading phonological skills to the acquisition of alphabetic literacy and also demonstrates that phonological awareness at the onset-rime level is easily within the capacity of the 4 year-old pre-reader, it would seem that any measures to prevent or reduce the effects of poor phonological awareness should be taken before children begin to learn to read and spell. Accordingly it is essential to establish a means of discovering rhyme and alliteration awareness in young children and to discount the possibility that poor performance on such measures could be attributed to lack of experience, both with metalinguistic routines and with the terms associated with such routines.

A study carried out by the present authors was designed to identify children with poor awareness of rhyme and alliteration and to promote this awareness before the children entered mainstream school, that is, before they embarked on literacy instruction. Before assessment, a total of 240 children, all in nursery-classes attached to LEA schools, engaged in daily activities which focused on the metalinguistic aspects of words - that is, activities which require children to de-centre from the meaning of words, focusing instead on their structure. Games were designed to draw children’s attention to individual words, to the label, ‘word’; to attend to the syllabic structure of words, and finally, a variety of activities required the children to make rhyming and alliterative judgements. The programme of activities was devised for the study so that children would not perform poorly when they were assessed simply because they were unfamiliar with ‘word-games’. Nursery-class teachers were given some preparation in implementing the class-based activities so that they could understand the relevance of these procedures and that of phonological awareness to the acquisition of an alphabetic literacy.

Earlier investigations which set out to establish the phonological awareness of pre-school children, particularly that by Maclean, Bryant and Bradley, 1987, provided a model for the assessment procedure which was developed for our investigation. Maclean and her colleagues demonstrated that young, non-readers can judge rhyme and alliteration, and that there is a strong, specific correlation between nursery rhyme knowledge and rhyme detection. The assessment procedure for our programme followed the general format of this study but findings from the pilot study which preceded our investigation led us to make certain modifications: Firstly, it was found that young children cannot readily make judgements based on ‘negative exclusions’, that is, by isolating the ‘Odd-One-Out’. Therefore, the respondents were invited to judge whether words rhymed or not. Secondly, researchers designing this programme conceded that young children find it difficult to de-centre from the meaning of words, therefore a significantly innovative feature was the inclusion of the pre-assessment experience phase. After one term of class-based activities all the children were tested on three measures, starting with their knowledge of the five traditional nursery rhymes used in the study of Maclean et al., 1987. These are: Jack and Jill; Baa-baa Black Sheep; Hickory-Dickory Dock; Humpty-Dumpty; Twinkle-twinkle Little Star.

The children’s rhyme awareness and alliteration awareness was established by reference to their performance on tests developed for this study. These tapped children’s ability to judge which of a set of three words rhymed with/ had the same onset as a stimulus word. The main consideration directing the design of the tests was that every child tested should be able to complete them successfully. That is, it was intended that the effect of external factors which might divert the child’s attention away from demonstrating the skills in question should be reduced to the minimum. So both tests were carefully controlled to take account of the receptive vocabulary of four-year-olds; to exclude phonological complexities such as variable vowel lengths; to exclude semantic ambiguities and to reduce memory loading. With this latter consideration in mind it was decided that, although judgements on spoken words were to be the focus of the assessment, pictures should be presented as memory aids. These illustrations were simple and unambiguous.

For all tests each child was seen individually in the nursery classroom by a researcher. All tests were conducted during one session. The child was first asked to recite the five nursery rhymes and performance on this task was scored: 2 for a complete recitation; 1 for partial recitation, that is, beyond the first line or ‘title’ line; 0 for the first line or less. Thus the maximum score on this test was 10. As noted above in connection with the Maclean, Bryant and Bradley study (1987) nursery rhyme knowledge is known to be closely related to rhyme awareness so this task was included for validation purposes. However, it also served as an ‘icebreaker’ because the majority of children know at least some parts of the traditional rhymes and are keen to display their skill.

This was immediately followed by the Rhyme Detection test. The test is made up of ten items, following a two-item trial, and each test item comprises a target word and three alternatives, two of which rhyme with the target plus a distracter. Pictures illustrating the words were placed in front of the child after s/he had correctly repeated each one and s/he was then required to judge which word(s) rhymed with, or sounded the same as, the target word. The test was presented as a game, with a puppet which liked rhyming words but vigorously discarded the picture illustrating the non-rhyming distracter. A response was correct when the child accurately identified two rhyming words and one distracter. The words used in the test appear as Table 1.

The same format which was devised for the Rhyme Detection test was adopted for the Alliteration Detection test, but for this test the child had to judge which two items had the same onset as the stimulus word and which was different. The design of this test took account of the same factors which had guided the production of the rhyme test and, in addition, each target word was examined for the effects, on the onset, of co-articulation - whereby a phoneme is differentially realised according to adjacent phonemes. The words used in the test appear as Table 2.

The maximum score for the Rhyme test and for the Alliteration test was, in each case, 10.

When a child performed very poorly on measures of phonological awareness further testing followed in an attempt to establish whether the child’s apparent inability could, in fact, be due to a generalised difficulty ......with sustaining attention or responding to a one-to-one testing situation, for example. A measure requiring a non-verbal response, Raven’s Coloured Progressive Matrices (Raven, 1984), was used for this purpose.

240 children with a mean age of 4 years 5 months attending nursery-classes attached to 22 schools in the West Midlands were assessed using the tests described above. On the test of Nursery Rhyme knowledge the mean score was 7.388 (SD = 2.531); for the Rhyme Detection test the mean was 6.096 (SD = 3.830) and for Alliteration Detection the mean score was 4.291 (SD = 2.968). Table 3 below shows the correlation between the scores on the three tests.

All correlations are significant at, at least, 0.001. However, the observed correlation of Alliteration Detection with Nursery Rhyme knowledge is lower than that of Rhyme Detection. It was therefore decided to regard the Rhyme Detection test as the more valid indicator of phonological awareness and to examine the pattern of responses to individual items in the test, for facility and discrimination by using an Item Analysis test.

Facility:		F value preferred is between 30 & 70%
	Man	Chair	Gun	Door	Pot	Keys	Swing	Whale	Sock		Light
Correct=	146	139	135	138	142	153	161	139	141	152
F%=	61%	58%	56%	58%	59%	64%	67%	58%	59%	63%

Discrimination:			A score >=.5 is good
	Man	Chair	Gun	Door	Pot	Keys	Swing	Whale	Sock	Light
Low	6%	8%	16%	5%	11%	19%	21%	5%	6%	13%
Middle	78%	66%	54%	68%	66%	73%	80%	69%	71%	79%
High	99%	100%	99%	100%	100%	100%	100%	100%	99%	99%
D=	0.93	0.93	0.83	0.95	0.89	0.81	0.79	0.95	0.93	0.86

With Item Analysis, items requiree high Facility scores to demonstrate that they are neither too easy nor too difficult and they should distinguish clearly between the respondents who, overall, score high or low by yielding a high Discrimination score (Deale, 1975). It is generally recommended that the facility for each item (% correct) should be between 30% and 70%. In the Rhyme Detection test developed for this investigation the highest facility is 67% and the lowest, 56% (see Table 2 above). With regard to indications of discrimination a score of 0.5 is usually considered adequate. In our test the discrimination scores are high, the lowest being 0.79.

Once the internal consistency of the test was established further analyses were conducted to determine what factors might be responsible for certain score patterns.

From the description of the Rhyme Detection test above it is obvious that a high proportion of respondents could attain some score by guesswork alone. Closer examination of all the factors suggests that, in fact, 41% could achieve scores in the middle range (4,5 or 6) without any understanding of the demands of the test (It must be assumed that only a score of 7 or above on this test is an indication of rhyme detection ability).

Figure 1 shows that the proportion of children actually assessed at the levels in the middle range was much lower than that which could be achieved by chance and so we should consider what strategies were being employed.

Forty-one children (17%) scored 0, that is, they did not correctly identify any item. This must have been largely due to an inability, or an unwillingness, to make any choice: the majority of children failing to score judged either that all the alternatives rhymed with the stimulus or that none of them rhymed with the word, despite the format of the ‘game’ which required the child to select a non-rhyming word-card for the puppet to seize and discard.

When the pattern of correct responses of those children who scored poorly is examined (Table 3) it is clear that some factor other than pure chance was operating to direct children’s selection of a distracter (or, at least, a picture representing their idea of "a word which doesn’t sound the same"). Table 3 demonstrates that children tend not to select the word represented by the picture in the middle position. Indeed, those children who scored only 2 appeared to be guessing correctly when the distracter was in the final position. Only those children who obtained a score of 1 appeared to be making random guesses.

Table 5: Position of distracter chosen by poor scorers (those with 1,2 and 3)

The tests described here were devised to identify children who, despite having taken part in class-based activities for enhancing an awareness of the structural properties of language, remain insensitive to its phonological structure. In particular, they were intended to show whether the children in our population could make the rhyming and alliteration judgements which are believed to underpin the ability to segment words into onset and rime. In addition, testing on Raven’s Coloured Progressive Matrices revealed where generalised difficulties with responding to tests might be at the root of poor test performances so that it was possible to identify cases where the phonological deficit was unexpected and therefore specific. Subsequently, it was decided that the ability to detect rhyme was the more valid indicator of a pre-reader’s phonological awareness so the results of the Alliteration Detection test were disregarded in selecting children for further involvement in the study. Children thus identified were then allocated to an experimental group or to a control group, and those in the experimental group were given intensive training in the skills believed to underpin phonological awareness.

The predictive validity of the Rhyme Detection test cannot be established until the children in both experimental and control groups have had sufficient exposure to written language and written language instruction to give them the chance to develop and demonstrate alphabetic literacy skills. In order to allow the children ample opportunity to reach this level further testing will be deferred until they are approximately 7 years old. It is expected that the children whose intervention featured activities to promote the component skills of phonological awareness will, at the age of 7 years use and recognise the alphabetic principle to decode and encode new words. Conversely their peers whose intervention was based around categorising skills without any particular emphasis on linguistic concepts should, according to the theoretical premises of the investigation, be experiencing some difficulty with this aspect of literacy. Meanwhile it is appropriate to offer some general observations on the Rhyme Detection test developed for this programme.

While the test described here appears to demonstrate internal consistency such that children with good rhyme awareness are able to answer all the questions with equal facility the design and presentation of the test predispose some children whose phonological awareness is not yet explicit to adopt a strategy whereby their choice is dependent on the position of the pictures used as memory aids. This was supported by observations of the children while they were responding to the test. When it was obvious that some children did not recognise the significance of rhyming they still understood that they should make a choice of one word from three and so they opted for a strategy based on the picture supporting a memory of the spoken word. Such children would inevitably make a choice and accompany their actions with, "That one and that one" as if indicating the pictures rather than the words whereas successful rhymers would verbalise the rhyming words. The hypothesis that in some cases poor scorers are basing their selection on the position of the pictures illustrating the words, is supported by an analysis of the data. It is therefore recommended that when this format is adopted the majority of distracters (the pictures representing words which are discrepant with the target words) should appear in the middle, the position least favoured by this strategy.

An analysis of the results of the three cohorts comprising the total population of this study included a comparison of the performance of those children for whom the items were presented in an order determined by chance and of those for whom the items were presented in a fixed order. This showed that randomising the order of the items serves little or no useful purpose. However it is advised that the varying positions of the distracter should be evenly distributed throughout the test.

Inevitably, in a population with a mean age of 4 years 5 months, tiredness and distractibility are factors which may affect performance. Many children who were obviously not totally conversant with the notion of rhyming judgements appeared fatigued or bored as the test progressed. This is evident in respect of those children who scored at a ‘medium’ level, that is, between 4 and 7. When the pattern of incorrect responses was examined against the order in which items were presented it seems that the performance of this group deteriorated towards the end and a tendency to start guessing set in. This then suggests that 10 items is the upper limit for length as beyond this responses would increasingly represent guesswork.

The assessment of phonological awareness in pre-school children presents special problems for both the researcher and the practitioner. Young children are being asked to disregard the semantic element of language and attend to its sound structure, and this requires a radical shift of emphasis for most pre-readers. Tests must, then, be short and straightforward and yet this increases the possibility that many children can achieve correct responses by guessing or, in their eagerness to please, at least adopting a strategy other than that intended for the resolution of the test items. Therefore it should not be assumed that in a test such as that described here, scores of 4 or 5 indicate a higher ability than 0. Instead, given the proven internal consistency of this test, a child with established rhyme awareness should be able to score at least 7, with any failure or incorrect response attributable only to fatigue. Conversely the child who is unable to achieve this score despite good average intelligence and adequate opportunity to engage in rhyming activities and related word-play, in a group setting, should be considered at risk of encountering difficulties at the alphabetic stage of literacy development. Strategies which enable the child to develop the skills that underpin rhyming judgements should then be implemented on an individual basis at the pre-school level so that potential obstacles to independent literacy can be removed well before s/he starts to learn to read and spell.

The research described in this paper was funded jointly by the Department for Education and Employment and the Oak Foundation. It was conducted by the School of Education, University of Birmingham, in conjunction with Hereford and Worcester Dyslexia Association.

Lyn Layton is a Research Fellow, Karen Deeny is a Research Associate, Graham Tall is a Lecturer in Education and Graham Upton is Professor of Special Education and Educational Psychology and Pro-Vice-Chancellor at the University of Birmingham.

Content or Face Validity	Is the aspect being measured relevant to the topic being studied. Does the test/examination used meet the teaching aims and objectives. A key check must be that the test contains a fair sample of the work envisaged. A test/questionnaire etc. would be commonly checked by asking teachers/experts in the field for their opinion.
Construct Validity	Is the test measuring what it is claimed to. If tests are designed to assess, say, a child's reading age, their level of autism, their level of dyslexia or their ability in a particular type of employment etc. then the test must be effective in its assessment. In the latter case, for example, are those obtaining higher scores more effective in the job?

Trial items:
	Item	Choices
	frog	zip	dog
	car	jar	tie

Test items
Item Order	Item	Choices
1	man	pan	book	van
2	chair	bear	key	hair
3	gun	dog	sun	bun
4	door	saw	car	paw
5	pot	knot	dot	pig
6	keys	fork	knees	peas
7	swing	king	ring	mouth
8	whale	nail	man	tail
9	sock	bed	rock	clock
10	light	kite	fight	spoon

Trial items:
	face	cat	fan
	boot	bus	cake
Test items
Item Order	Word Presentation Order
1.	comb	cot	box	cup
2.	milk	dog	man	mat
3.	sun	leaf	sock	sand
4.	duck	doll	dice	bag
5.	fork	feet	moon	fish
6.	toast	pig	tap	tin
7.	bird	ball	bike	pan
8.	hen	house	sink	hat
9.	knife	nest	phone	net
10.	gun	ghost	gate	boat

	Nursery Rhyme	Rhyme	Alliteration
	Knowledge	Detection	Detection
NR Knowledge	1
Rhyme Detect.	.451	1
Allit. Detect.	.344	.581	1

Number of items correct	Distracter - first	Distracter-middle	Distracter- last	Total correct	Number of poor scorers
1	3	3	3	9	9
2	7	5	14	26	13
3	20	9	16	45	15
TOTAL	30	17	33	80	37