WHAT CAN PISA TELL US ABOUT NC Warwick B. Elley
Summary This article outlines some implications of the latest PISA surveys for New Zealand education. Firstly, it explains why we need to take note of the PISA scores, in the face of numerous criticisms. Next, it identifies a clear pattern in the results. All nations which have adopted a “high-stakes” standards-based assessment system, with league tables, have shown remarkable declines over time, declines rarely found in other countries. Finally, it explains why our Government should take note of these PISA findings, and review our NCEA and National Standards policies, before we follow the downward trends of these other nations.
and the tests measure important interpreting and problem solving skills in Mathematics, Reading and Science. Why then did the scores of New Zealand and several other nations slump by around 20 points in 2012? A close look shows an important pattern, with serious implications for New Zealand.
A Clear Pattern in the Declines The case of Sweden is most instructive. In the international surveys of 32 nations in 1990, Swedish students were in the top three countries in Reading Literacy at two age levels. The Swedes were delighted, and teachers were praised. However, in 1994, the government embarked on radical reforms. They Introduction sought to improve achievement levels even further by spelling The 2012 PISA survey of 15 year-olds in 65 nations raises urgent out clear standards for each grade level in basic subjects, just questions. Can we trust the results? What caused the slump in as New Zealand has. Teachers were required to assess their our scores – and those in three other high-achieving nations? students against these standards by internal assessment, as ours do. Schools were advised to get some guidance by using “lowWhat lessons can we learn from the trends in the scores? stakes” standardised tests, as ours do. Principals were required to The Validity of PISA Results: report the numbers of students reaching the standards, as ours Some critics have questioned the validity of the tests used, are. Media are able to inform public of the schools’ results, just or the comparative ages of the student samples, or the data as ours do. What happened? processing methods. Having Starting in 2000, the average been closely involved in past The mere fact that the PISA means are PISA scores of Sweden’s 15 yearinternational surveys, I have olds tumbled consistently, in more confidence in them. very consistent across time for most surveys every three years, in In the 50 years since such countries is also a further indication that every subject. In Reading they international studies began, dropped from 520 in the first researchers have learned the results can be trusted. survey to 483 last year. In Maths much about how to ensure the they fell from 510 to 478, and in fairness of the tests and procedures, and the analytic methods Science from 513 to 485. Educators are now “in shock”. used do have the blessing of most statisticians elsewhere. If Australia has had a similar experience. Starting in Queensland there are variations in promotion policies, they would have little in the late 1980s, educators set out to define the standards effect on changes over time, as all PISA surveys are based on age expected of all high school students in each senior grade and (15y3m–16y2m), not grade. In New Zealand, over 80 per cent subject. Teachers used internal assessment to assess students of the samples come from Yr 11, the first year of NCEA. Even against these standards. NSW and Victoria followed suit and the sceptics concede that comparisons of PISA mean scores over the movement spread to primary schools with “high-stakes” time, can be used to make fruitful comparisons. I believe there tests at several key grade levels. Federal authorities have built is a great deal more to learn. on these state projects to devise a set of nation-wide standards. The mere fact that the PISA means are very consistent across Schools are using common assessment procedures to generate time for most countries is also a further indication that the national league tables for their NAPLAN programme, a project results can be trusted. The rankings may seem to move around dear to Julia Gillard’s heart. Like Sweden, Australian students’ arbitrarily, because there is little difference between some PISA scores have fallen in nearly every case, from 529 to 504 in countries in their overall patterns, so a small change in mean Maths, with gentler declines in Reading and Science. No state scores could result in a large change in ranks. However, if we was exempt. Like Sweden, they are searching for answers to stop focus on mean scores, any rise or fall of 15 points or more, from their unexpected slides. one survey to the next, is rare. Changes over 20 are extraordinary. What about the two nations which started this business These PISA surveys are independent and technically sound, model of monitoring an education system? In England a 9-level
NCEA AND NATIONAL STANDARDS?
curriculum with clear standards was rolled out in 1989. Teachers they still out-performed all other European nations in both these were to report the numbers of students achieving these standards subjects. Apologists believe a new policy which saw a reduction by internal assessment, although external “high-stakes” tests were in time spent on Maths in the schools was partly to blame for imposed at three “Key Stages”. League tables were published and the drop in that subject. schools praised or shamed. The graph below shows the slump in PISA means in the five Likewise in USA, each state defined clear standards in basic nations that have embraced standards-based assessment (SBA) subjects, and successive presidents promoted their “No Child – NZ, Australia, Sweden, England and USA. The average decline Left Behind” policy, with regular testing and league tables, and in these five nations, from one survey to the next, is consistent, sanctions for the under-achieving schools. America, too, is now except in the case of Science, where there was a temporary small building on these ventures to develop nation-wide “Common rise. Only two other nations, Uruguay and Iceland, have declining Core Standards”. Have these two nations pointed the way to trends like these – for different reasons. Not only is there no rise higher achievement? On the contrary. in the achievement levels of countries which use high-stakes SBA. Their slumps started early. These two erstwhile world leaders They have shown a uniform decline. Our own large decrease are now engaged in serious soul-searching for their students’ was more recent, between 2009 and 2012, as NCEA, with its repeatedly poor showing. Back In 1990, in the IEA Survey of high stakes SBA did not begin until 2003, and the effects of our Reading Literacy, America’s Yr 9 students were ranked 8th, (out recent National Standards policy are, of course, not reflected in of 31 nations), with a score of 535, on a scale just like those of the 2012 survey. However, the message of these other nations PISA. In the TIMSS surveys of Maths and Science, in 1995, their cannot be ignored. The big declines for England, US and Sweden 14 year-olds had scores of 502 and 515 respectively. However, started sooner, as they were using SBA with league tables before after the PISA surveys of 2000, nearly every American sample the first PISA survey in 2000. of 15 year-olds scored below the global average (500) in all three Further confirmation of these findings comes from Canada. subjects. In 2006, US was close to the bottom of the OECD scale. Ontario Province, the largest in Canada, set out to follow the US England’s PISA record has also been consistently poor. In and English models of accountability, with SBA and league tables, Maths, their students slumped from 527 to 494, surely be neglecting those needs. As a nation, too, we are now heavily involved in a race to in Reading from 523 to 494, and in Science, from the bottom. 529 to 514. Neither country comes close to the excellence that Tony Blair and George Bush vowed would follow their reforms. Meanwhile many less well-resourced nations have leap-frogged over them. But what about that high-flier Finland? Its scores in all PISA subjects have hovered round the 540+ mark in all three subjects assessed. It too showed a significant decline in Maths – from 541 to 519. Finland is an interesting case. Its educators have always rejected the market-driven accountability reforms of England, USA, and Australia – and more recently, New Zealand. When Finnish educationist Pasi Sahlberg visited last year he warned us against these reform agendas. There is no formal highstakes testing of Finnish students till the end of secondary school, yet Finland has risen to the top in most international surveys, regardless of age, grade and subject assessed. For the first time the latest PISA results for Finland in Maths revealed a significant decline, with smaller drops in Reading and Science. However,
back in 1993. It reveals a consistent decline in Maths scores, by a total of 20 points, since the survey of 2003, when Maths was the main focus. Their means fell in Science and Reading by 10 and 5 points respectively. Meanwhile neighbouring province, Quebec, resisted such reforms, and has risen to the top of all Canadian provinces. Their assessment policies for school leavers sound very similar to ours before NCEA appeared. Some might question whether the slumps were exaggerated because the nations affected started from a high base. To check this possibility, I have compared each nation’s means with sets of five other non-SBA nations, starting from the same point in the baseline surveys of 2000, (or 2003 in the case of Maths), and find no reason to doubt our conclusions. The relevant tables are given below (Tables 1–3). Most of the non-SBA nations show improvements over time, or merely hold the line. I have also corresponded with assessment specialists in four of the nations affected, and all agree with my analysis. Lessons for our Government In view of these clear patterns in PISA scores, over 12 years, an outcome which many of us were predicting, the lessons for our Government and NZQA should be clear. Neither National Standards in primary schools nor NCEA in secondary schools will raise performance levels. As high-stakes events, they generate more testing, less creative teaching, obvious grade inflation, less “stretching” of brighter students, less cooperation between schools, lower achievement levels and larger gaps between high and low-performing schools – to name a few of the negative consequences. A “one size fits all” system which sets arbitrary and vaguely-worded standards, which are too easy for some, and too hard for others, ignores the natural individual differences that exist between students. It often fails to challenge the high-fliers, as they slow down once they have achieved enough credits – and the data do show the declines are greater at the top. Moreover, it does not solve the problem of the under-achieving tail. In fact as the league tables start to bite, there is a usually a drift away from low socio-economic schools, which lose their reputation, their best students and teachers, resulting in a loss of morale and a real problem in attracting experienced, committed teachers. The gap between schools only gets wider. In any new policy, there are some winners and some losers. Using our own figures, our Minister has created an illusion of more winners. In an international context, using assessments that search more broadly, we expose the gaps in our students’ knowledge and skill, and the upshot is that we are actually falling back. Many of these consequences have been documented overseas before, but our policy makers ignore them. Is there any other possible explanation for these falling scores? They are unlikely to reflect changes in curricula, or methods, as the declines cut across all three subjects, and what curriculum reforms there were in these countries had little in common. Nor has there been any sudden, nation-wide change in teacher training policies. There have been no common funding policy changes in these five nations. The kinds of moderation of school-based assessments vary from one country to another, from tight exam-based systems, to group consensus methods, to weak post-hoc moderator visits. None appears to halt the slide. The common denominator in all five of the selected nations is a policy which requires schools to report the numbers of students achieving in relation to a set of vaguely-worded standards, with or
without internal assessments, and the results are then publicised in the form of league tables, making it all high stakes. The impact of such a policy, changes the behaviour of teachers and students in such a way that the long-term effects on achievement levels is consistently negative. In a naive push for greater accountability, we are damaging the fabric of our hitherto world-renowned education system. No country that has adopted a policy of highstakes SBA has produced an improvement in achievement levels. Anyone can confirm these figures by checking the reports on the Internet. As professionals, teachers are charged to give highest priority to the needs of their students. If we persist with these ill-starred standards-based schemes, we will surely be neglecting those needs. As a nation, too, we are now heavily involved in a race to the bottom. TABLE 1 PISA Mathematics Means for 5 Nations Using High Stakes SBA 2003–2012 (Matched with Five Other Nations in 2003) SBA Nations
Non-SBA Nations
2003
2012 Change
2003 2012 Change
New Zealand
-23
Switzer land
+4
Australia
-20
Belgium
-14
Sweden
-31
Austria
England
-31
Macao
+11
USA
-2
Latvia
+8
Mean
513.2. 491.4
514.4 516.2
+1.8
-21.8 Mean
TABLE 2 PISA Reading Means for 5 Nations Using High Stakes SBA, 2000–2009 (Matched with 5 Other Nations in 2000) SBA Nations
Non-SBA Nations
2000
2009 Change
2000 2009 Change
New Zealand
-8
Canada
-7
Australia
-13
Korea
+13
Sweden
-22
Japan
+14
England
-29
Ireland
-3
USA
-4
France
+1
Mean
520.6 504.4
521.8 525.4
+3.6
-16.2 Mean
TABLE 3 PISA Science Means for 5 Nations Using High Stakes SBA, 2000–2012 (Matched with 5 Other Nations in 2000) SBA Nations
Non-SBA Nations
2000
2012 Change
New Zealand
Australia
Sweden
England
2000 2012 Change
-12
Austria
-16
-7
-27
Finland
+9
Czech R.
-3
-18
Canada
-4
USA
-3
France
-2
Mean
520.0 506.6
520.2 517.0
-3.2
-13.4 Mean
ADVERTORIAL
WHY MOBILE DISPLAY SOLUTIONS? JENNY BARRETT
NATIONAL SALES & MARKETING MANAGER, SITECH SYSTEMS NZ LTD
THE MAJOR OPPORTUNITY presented by switching from projector to panel is adding mobility to your learning space. We find Ewan Macintosh’s seven spaces a wonderful starting point for schools considering their display options. When he talks about group spaces he poses the challenge to leave the space as open as possible and to have the objects required on the side and students drag over what they need and create their space. That tallies with our ethos as a company too: make the display device flexible to ensure it can be used for as many learning ‘purposes’ as possible. So a mobile touchscreen could be rolled into a ‘group space,’ a nest where small teams can interact with a decent sized screen instead of crowding around an iPad. Then a teacher running a mini-lesson in a ‘check-in’ space might use it whilst the students relax on couches and have a discussion. Then you pull the whole class together and students share their findings again using the same display, perhaps showcasing four screens simultaneously using a wireless connectivity solution. It could also provide the ‘publishing space’. With digital signage software, news and notices can be broadcast as a screensaver in downtime or before and after class. A ‘participation space’ interests us too. We liken it to what Suzie Boss in Edutopia magazine refers to as a ‘Tinker station’ and use our science trolley. Visualisers linked to the panel in
these spaces allow for ‘visible thinking.’ ‘Watching spaces’ are being overlooked. Ewan Macintosh urges that schools ‘create genial spaces for lectures, spaces that thrill and delight and celebrate those occasional moments of lone insight that only a real, living, flesh and blood teacher or visitor or student can offer.’ You need awesome audio, HD display and the option to record video. At other times (two of Suzie’s other ideas) it can be the Skype On space or the Video Booth. Lastly, ‘data spaces’ where you share live raw information for interpretation. Try the data logging elements of the science trolley or fixing a weather station outside. Digital signage software can share this information or anything of interest to the current inquiry. Once you and your staff have had these discussions; we advise that you seek expert help. Feel free to contact our team of national sales consultants to discuss rolling over display devices. We do not charge for this service, all we ask is that we have an opportunity to quote.
http://edu.blogs.com/edublogs/2010/10/-cefpi-clicks-brickswhen-digital-learning-and-space-met.html
http://www.edutopia.org/blog/thinking-through-project-basedlearning-suzie-boss
Trade in your fixed Interactive Whiteboard for the new generation of MOBILE touchscreen displays
$500 discount Trade in and receive $500 off the Sitech Touch 60” Trolley or the Sitech Interactive Teaching Station 50” Term 1 only. Contact us for more details.
www.sitech.co.nz/IWB_tradein Free Ph: 0800 100 607 Email: sales@sitech.co.nz