FAQs

Educational evaluation and testing services for Irish schools

CAT3 FAQs

  • What is CAT3?
  • Why are there three test batteries?
  • What makes CAT so reliable?
  • What are the benefits of using CAT in a school?
  • How will CAT benefit the child?
  • Can CAT be used to identify underachieving pupils?
  • Can I use CAT every year to monitor pupils' progress?
  • What change in scores over time represents a real improvement or decline in pupil ability?
  • When is the difference in a pupil's standard scores on the three batteries considered significant?
  • What conclusions can be drawn from patterns of test results for year groups (e.g., pupils tending on average to score better in one battery than in another)?
  • What is the correlation between CAT and IQ scores such as Wechsler Intelligence Scale for Children (WISC) used by educational psychologists?
  • What explanation could be given for differences in a pupil's scores on an IQ test and CAT?
  • To what extent is the reliability of CAT results affected by children with dyslexia, dyscalculia or specific learning difficulties in regard to following multiple instructions?
  • How reliant is CAT on the quality of the adult explaining and supervising the test?
  • One of my pupils has a lower score on the Non-Verbal test than on the other tests. Are there any learning strategies that could be employed that would help him?
  • If I read out the CAT Verbal Battery questions to pupils, will this invalidate the test?
  •  

    What is CAT3?

    CAT3 stands for “Cognitive Abilities Test, Third Edition”. The test as a whole assesses an individual's ability to manipulate and reason with three different types of symbols: words, quantities and spatial patterns, each of which plays an important part in human thought. A section, called a test battery, of the complete CAT test is devoted to each of these ways of reasoning, and each battery is further divided into three sorts of test item that test different aspects of that style of reasoning.

     

    Why are there three test batteries?

    Children are found to have different preferences among these three styles of reasoning. Some prefer verbal reasoning, some reasoning with numbers and some abstract reasoning with shapes and patters. By looking at a child's performances in each battery compared to the others, information can be gathered about which of these he or she maybe better at, and for which he or she may need additional support.

     

    What makes CAT so reliable?

    CAT items have been exhaustively trialled on large samples of children; any that exhibited odd characteristics or showed evidence of sex or ethnic bias were discarded. Then the assembled test was administered to a sample of 16,000 children to establish the norms of performance for children of different ages. It is from these norms that a child's standardised age score is derived.

    Full details of the standardisation process are to be found in the CAT3 Technical Manual, pages 7 to 9. The thoroughness of this process means that the standardised scores resulting from the tests are very reliable, and that, for instance, large differences in a pupil's scores between test batteries can be treated as having real meaning.

     

    What are the benefits of using CAT in a school?

    The great virtue of CAT results is that they are a lot less dependent upon a pupil's classroom and learning experiences than are tests of knowledge, such as the end-of-year school exams. Thus, CAT scores can provide a measure of a child's ability largely independent of which school he or she went to, and what experiences he or she had there.

    As a way of setting targets for individual children, for example, CAT scores are a lot fairer than attainment test results, such as end-of-year tests, would be. Also, being standardised, CAT scores can be compared over time, so that, for example, the overall ability of successive years' intake of a large comprehensive school can be compared. The case studies presented in the CAT area of this website have been collected over a number of years to illustrate good practice, and a range of ways in which CAT scores have been used effectively in schools.

     

    How will CAT benefit the child?

    Any hard information that enables teachers better to understand the abilities of their pupils should operate to those pupils' advantage. A pupil's three reasoning scores will enable more effective teaching, learning programmes and interventions to be designed. Children also benefit from CAT scores enabling teachers to set better targets containing the right degree of challenge. And anything that personalises teaching and learning is in keeping with the Government's intention that children should be prepared to take more responsibility for their own learning.

     

    I am looking at some CAT results with scores under three headings: 'Verbal', 'Quantitative' and 'Non-Verbal'. I am unclear as to the precise meanings of these headings and what skills each test includes?

    The Cognitive Abilities Test (CAT) is an assessment of reasoning skills. It consists of nine short subtests within three 'batteries' or sections. The batteries are labeled as verbal, quantitative and non-verbal reasoning. For examples of the sorts of questions used in each of these, please see the CAT Digital demo.

    The verbal reasoning battery assesses reasoning processes using the medium of words; the quantitative test battery assess the same processes but use numbers as the symbols. The non-verbal battery again tests reasoning processes but use shapes and figures. As the non-verbal subtests require no knowledge of the English language or the number system, they are particularly useful when assessing children with poor English language skills, or disaffected pupils who may have failed to achieve in academic work through low motivation. Also, they are very important for gaining a full assessment of the abilities of pupils who have a greater ability to think in images than in words or numbers, and who may prove to have particular aptitude for subjects like science, mathematics and design/technology.

     

    Is CAT a measure of innate ability? Are CAT results in any way affected by teaching?

    There is no such thing as a measure of innate ability. The quality of prior teaching, opportunities to learn, parental support, pupils' educational experience, and their emotional and physical well-being, including nutrition, will affect pupils' performance on all educational tests. However tests of the taught curriculum - reading, mathematics, spelling etc. - are likely to be influenced by these factors to a greater degree than reasoning tests.

    Attainment tests, such as National Curriculum tests or GCSE examinations, are designed to measure outcomes of specific learning and instruction, and the test content is drawn directly from the taught curriculum. In contrast, reasoning tests tap into a general set of prior experience by assessing the perception and manipulation of relationships, content that is not generally part of the taught curriculum. Non-Verbal Reasoning (NVR) tests, with their relatively low language demands, are least likely to be influenced by the nature and quality of teaching. This matter is dealt with in some detail in Chapter 8 of Getting the Best from CAT (Strand, 2003).

     

    Can CAT be used to identify underachieving pupils?

    CAT scores are less likely to be affected by school experience than attainment tests. Comparisons between pupils' CAT scores and their attainment in school subjects such as English and mathematics can therefore be helpful in identifying pupils whose reasoning ability is average or above average, but whose attainment in curriculum subjects is low. Such pupils may be characterised as underachieving, and may benefit from intervention. In a few cases, pupils may apparently 'overachieve', in that they do better in school subjects than their CAT scores would suggest was likely, based on national data. In these cases, it is important to consider possible explanations for the lower CAT scores, such as test anxiety or temporary distractions (e.g. difficulties at home, bullying). If the CAT scores do seem a genuine reflection of the pupils' reasoning abilities, then one must conclude they are unusually highly motivated to do well at school and/or the school is exceptionally effective in its teaching.

     

    Can I use CAT every year to monitor pupils' progress?

    Reasoning test scores tend to be more stable over time than attainment test scores. Annual testing is therefore more appropriate for reading, spelling or mathematics tests, for example, which assess how well children are performing in relation to the curriculum that has been taught. Examples of such tests are the Drumcondra Maths tests. These tests include 'progress scores' which specifically allow users to compare the progress made by pupils over the course of a year with the progress made over a corresponding period by a large and nationally representative sample of pupils.

    However, reasoning scores can and do change over time. For a minority of pupils, these changes may be quite substantial. The mean scores for a group of pupils or even a whole school can also change substantially, for example where there has been an intervention such as the National Literacy or Numeracy Strategy.

     

    What change in scores over time represents a real improvement or decline in pupil ability?

    For individuals pupils, you should remember that any test score is based on performance on one particular day and may be affected by a wide range of motivational or other influences. For instance, the pupil may have been distressed or upset by an incident at home earlier that day; he or she may feel disinclined to co-operate in the test for some reason or another. Also, being limited in length, the test itself can only ever be an approximate measure of pupils' reasoning abilities. It is important therefore that the score is placed within a 'confidence band' so that you do not over-interpret small changes in standard scores. As a rule of thumb for the whole CAT test, or for one of the three batteries, there would have to be a change of 10 or more standard score points before you would say a pupil had shown a real change in their CAT score, rather than a change that could be simply attributable to the sort of factors described above.

    What about change in the mean scores for groups of pupils? We should remember that when using standardised age scores, a consistent score over time is to be expected if pupils are progressing normally. For example if a group achieved a mean standard score of 102.5 in Year 7, and the same group achieved a mean score of 102.5 two years later in Year 9, then the group would have maintained their relative level of reasoning ability, in comparison with the national population. To interpret any differences, you would also need to place 'confidence bands' around the group mean scores, the size of which would depend upon the number of pupils in the group. Another rule of thumb: in order to signify a real change, the difference would have to be at least 2 standard score points for a group of 100 or more pupils. For a smaller group the difference would have to be larger to signify a real change rather than a change attributable to chance factors.

     

    When is the difference in a pupil's standard scores on the three batteries considered significant?

    In most cases the three standard age scores (verbal, quantitative and non-verbal) will be broadly in line with each other. Scores will rarely be exactly equal and there has to be a difference of 10 or more standard age score (SAS) points between a pupil's score on any two CAT batteries before the difference would be considered significant. The implications of any score differences will depend on the particular batteries between which the differences exist, and whether they indicate relative strengths or weaknesses. Chapter 3 of Getting the Best from CAT describes a detailed system for analysing CAT pupil profiles, their implications for teaching and learning and practical guidance on strategies. (From Autumn 2006, this system, known as “verbal-non-verbal profiling”, will be available as part of the scoring service.)

    It is rarely advisable to give advice based on test scores in isolation. Test scores are only a small part of the picture of a pupil's abilities and attainments and you need to know the whole pupil in order to interpret the results in context. Test scores should feed into a broader assessment, bringing to bear knowledge of the pupil's achievements in school subjects, their personal background and their attitudes, motivation and behaviour. For this reason a pupil's teacher will be best placed to judge the implications of the CAT scores.

     

    What conclusions can be drawn from patterns of test results for year groups (e.g., pupils tending on average to score better in one battery than in another)?

    First, you would have to determine whether any score differences are significant (see FAQ5). If they are, then there may be general implications. For example, where the mean VR score for a year group is lower than their mean NVR score, this may indicate a need for specific interventions to address low verbal skills. Getting the Best from CAT (Strand, 2003), pages 72-79, includes an Excel spreadsheet to allow you to evaluate the significance of score differences for groups of pupils. (From Autumn 2006, this system, known as “verbal - non-verbal profiling”, will be available as part of the scoring service.)

    Do remember, though, that any difference in the mean scores for the group will be a generalisation and will not necessarily apply to all individual pupils. You will need to look at each individual pupil's scores in order to identify those who might benefit most from any intervention.

     

    What is the correlation between CAT and IQ scores such as Wechsler Intelligence Scale for Children (WISC) used by educational psychologists?

    In general, we find a high correlation between the CAT Verbal and Non-Verbal batteries and the WISC Verbal and Performance IQ respectively. Also, experimental work carried out in collaboration with the University of Sheffield in 2004 made it possible to equate CAT scores on one test with scores on the WISC, so that in psychological or clinical assessment of children, either test could be used to estimate performance on the other one.

     

    What explanation could be given for differences in a pupil's scores on an IQ test and CAT?

    The CAT is a group administered test, while typical IQ test batteries are individually administered. This can affect the performance of some pupils, for example those with attention problems may lose focus during a group test but remain 'on-task' with one-to-one testing. The individually administered verbal tests involve the examiner reading the material to the pupil, while the CAT verbal tests require the pupil to read the materials, so this may be another factor. Additionally, IQ tests incorporate a wider range of cognitive tasks, so overall performance may be affected by specific abilities not assessed in CAT. For both CAT and IQ tests, scores can sometimes vary because of extrinsic factors such as tiredness, distraction, lack of motivation, incorrect administration etc.

    Where there are score differences between two tests you need to know something about the individual who is the focus of the assessment. One-to-one work with, or observation of a student over the course of a school day can sometimes be far more informative than a whole battery of test scores. Perhaps the important message is to use all the available evidence, from every source, when making any educational decisions.

     

    To what extent is the reliability of CAT results affected by children with dyslexia, dyscalculia or specific learning difficulties in regard to following multiple instructions?

    One of the great benefits of the CAT is in the diagnostic use that can be made of the pupil's profile of performance across the three batteries. For example, a specific language difficulty (such as “dyslexia”) might be manifested in a low score on the VR battery compared to the QR and NVR batteries. A specific arithmetic difficulty may show as an uneven profile with a low QR score relative to VR and NVR. In either case, it is appropriate to follow up the CAT results for such pupils with further one-to-one assessment. The CAT scores are likely to be the starting point for hypotheses and questions which will require further detailed investigation.

    However, it should be stressed that it is important to recognise the experience and skills that are presupposed in all children if the tests are to provide valid assessments. For example, The CAT Verbal battery presupposes that pupils have a 'normal' level of attainment in reading for their age. The actual reading demands are kept to a moderate level, so that all such pupils will cope easily with that aspect of the tests and the tests will therefore measure differences in reasoning abilities not in reading skills. Any child whose skill in reading is adversely affected by dyslexia cannot therefore be validly assessed for reasoning ability with the CAT or any other test that uses written words. In such cases, the Quantitative and Non-verbal batteries may provide valuable alternative evidence, though it is important to remember that pupils with dyscalculic difficulties or simply very low attainment in number skills will be unfairly disadvantaged on the Quantitative test.

     

    How reliant is CAT on the quality of the adult explaining and supervising the test?

    It is important that the administrator should follow the instructions given in the test manual. Each sub-test starts with demonstration and practice questions; the administrator must use these to ensure that pupils are familiar with the test layout and question formats before they start the test. Being a timed test, it is vital that a stop-watch or clock is used to ensure the correct amount of time is given for each sub-test: for example, ten minutes for the Number Series test means exactly 10 minutes, not 9.5 minutes nor 10.5 minutes. Providing the test manual is followed accurately and professionally by the administrator, the influence of the particular adult who administers the test should be minimal.

     

    One of my pupils has a lower score on the Non-Verbal test than on the other tests. Are there any learning strategies that could be employed that would help him?

    Suggestions for activities that schools can use and parents can undertake at home to support their children are given in the book Getting the Best from the CAT (Strand 2003).

     

    If I read out the CAT Verbal Battery questions to pupils, will this invalidate the test?

    You can and should read out to the pupils all the instructions, and the demonstration and practice questions, explaining these in detail and using community languages if appropriate. However if you read out the test questions in the Verbal Battery then you cannot use the standard age scores, because this was not how the test was administered to the standardisation sample.

    If you have concerns about a pupil who may find the Verbal Battery too challenging, then you should assess the pupil in two stages:

    First, administer all three CAT batteries in the standard fashion. If a pupil has specific language difficulties, then this should be revealed in the profile of his or her scores across the three batteries. We might expect the Verbal score to be significantly lower than either the Quantitative or Non-verbal score. (Significant in this context means a difference of 10 or more standard age score points.) If this is the case, then you have independent evidence to confirm the pupil's difficulties in reasoning with words as compared to numbers or spatial concepts. The book Getting the Best from the CAT (Strand 2003) describes detailed action with the pupil that might follow in terms of teaching and learning.

    Second, to isolate a specific problem with reading, you could read some of the verbal questions to the pupil. If their response to these questions is markedly better than on the first occasion this might isolate reading as the problem. This would suggest that the pupil's 'true' reasoning score is obscured by reading difficulties. If this is the case, then you should base any indicators, target setting or similar activity on the mean of the Quantitative and Non-verbal scores alone, ignoring the Verbal score. Of course, it is absolutely vital that the resources are also found to address the pupil's reading difficulty, otherwise his or her potential is unlikely to be fulfilled.