Sunday, January 31, 2010

'My School' Website: A blunt and inadequate instrument

Declaring my conflicts of interest

I had hoped to avoid commenting on the 'My School' website released in Australia on Friday, because I figure that no matter what I say about it, lots of people reading the blog will disagree. But when one of the readers of the blog (Jo) asked for my opinion I thought that I should say something. As well, the publishing of league tables of the 'best' and 'worst' schools made me quite angry. My apologies to international readers of the blog for this more parochial topic, but I'm sure that some of you will have had similar models for national assessment in your countries

Let me first declare some background that some might see as conflicts of interest:
  • I was an academic adviser to the first comprehensive national literacy assessment in this country in the late 1990s (the National English Literacy Survey) that was ultimately foundational to the current testing that informs the site.
  • I spent 10 years as a primary school teacher and have a strong interest in assessment.
  • My two daughters went to the closest public schools to our homes and my only school-aged grandchildren are currently attending the closest public school to their home.
I am a strong supporter of public education but I respect the right of parents who choose to send their children to private schools. The comments that follow are not motivated by self-interest, nor am I captive to any interest group, I'm simply concerned about the site.

A few givens about schools and families

1. Teachers vary in ability and at times after 30+ years in the classroom the odd teacher wishes they were elsewhere (as in all occupations), BUT I've rarely met a teacher who wasn't doing their best to help children learn.

2. My own family literacy research over a period of 15 years (see my website for more details HERE) demonstrated many things, but the most self-evident of the findings was that in spite of the varied resources, education and life experience of parents, that it is difficult to find a parent who doesn't want the best for their children.

3. Australia has one of the finest school education systems in the world. The international data speaks for itself. The results from the OECD Programme for International Student Assessment (PISA) have shown over the last decade that Australia is consistently one of the highest achieving school systems in the world. Overall literacy achievements have remained in the top 6 in the world, mathematical ability has been in the top 12 and scientific knowledge in the top 3. While we have ongoing underachievement by Indigenous students, overall performance on international comparisons is outstanding.

4. Parents want to know how well their children are doing at school, not just in terms of their effort and ability, but how they compare with other children. This has led some parents and parent advocates to be critical at times of school assessment practices and the feedback given to parents.

5. Teachers know that basic skills tests can only ever sample a limited range of the things that are taught at school and that they fail to measure many other things that they see as fundamental to judging the success of schooling.

6. The majority of children no longer go to the closest school to their homes. Instead, they attend varied schools: near parent's place of work; where after school care is provided; that are private; that are selective and so on. Hence, parents today make choices about where they should send their precious children and feel the need for information that makes comparison possible.

What's right about the 'My School' website?

1. The website is an attempt to satisfy the demands of parents for data on how well their children's school performs relative to other schools.

2. It is simple and attempts to make comparisons between schools that serve comparable communities. The latter relies on a measure of socio-economic advantage based on census districts associated with schools, that considers 16 variables that may influence educational outcomes, including measures of income, the proportion of non-English speaking background, employment and educational attainment.

3. It is a response to a public call for greater accountability and transparency in school reporting on standards.

4. It offers measures across a broad range of academic areas.

5. It offers some basic comparative information on enrolments, student attendance and promises to eventually provide data on funding levels.

What's wrong with the site?

1. The implicit assumption that you can measure school performance, and hence teacher competence, using this blunt instrument, is incorrect and grossly unfair to teachers. The test data offer an assessment of school cohorts in a range of academic skills and competencies that can be tested using a mass assessment programme.

2. While the site is able to show which student groups perform best on the above skills and competencies, it is unable to identify fairly why this is occurring. For example, I looked at one school that seemed to have outstanding results for year 3 but whose results for year 5 were average. Does this drop off reflect the quality of the teachers in tear 4 and 5? Or some quirky difference in the demographic over time? Perhaps a new private school nearby had attracted some of the best students after year three? No doubt, teachers and parents in the community could suggest some reasons for this drop in achievement. I also considered one of the elite schools that the newspapers declared one of the stars. Interestingly, in spite of very high achievement, the growth from year 7 to year 9 was limited in reading. What does this show? No value-added by the year 8 teachers? The reality is that the data need to be interpreted with material extraneous to the site.

3. The site isn't able to measure what some parents find most important. What does the site tell a parent (or a politician) about the arts? What about creativity and the ability to solve problems? Leadership development? Sport and fitness? Programs for children with learning difficulties? The care and compassion of staff? Emotional well being? Are the children happy in these schools? How open is the school to parental involvement? How approachable are staff members? What is the school budget for professional development, and how does this compare to other schools? In the case of secondary schools, what is youth culture like in the school? What values are shaping my child and what is the school doing to promote the values that I see as important?

Unhelpful responses

There have many unhelpful comments based on the website comparisons, in fact too many to count. However, surely the newspapers that published the 'top of the class' and the 'bottom' in the State, deserve to be criticised strongly. I have spent extended time as a researcher in some of the so-called worst schools and always found children who were seeking to do their best and dedicated teachers. At times, these schools lack resources and struggle in communities where there are a myriad of social problems. In spite of the number of families who struggle to cope with the challenges of life, they were (on the whole) doing the best that they could to ensure that their children had the best education possible. Publishing the names of such schools on a public list as failures is an indictment upon those who are responsible; those who condone it; and those of us who stand by and say nothing to defend them. I applaud the teaching profession for asking serious questions about the way that the My School Website is being used.

However, the teaching profession cannot simply refuse to cooperate in providing public information about the performance of the children in every class. The fact that newspapers have published league tables and many people have bought them, suggests that parents do want to know more about the quality of education in their children's school. The question is, how can schools meet this expectation without using a blunt instrument of the type that we've seen demonstrated? Efforts to improve the quality of the feedback that schools offer to parents is critical. But a key starting point is that such reporting needs to be at the individual child, class and school level. While the aggregating of national and state data is relevant, the type of public league tables that we've seen are unsupportable.

Related Posts

Sydney Morning Herald article (here)

20 comments:

carly said...

I am yet to even bother to look at the website.

One of the blogs that I follow made a remark about the website, that you know that Mt Druitt would be down the bottom and that the Northern Beaches schools will be up the top.

When I made the comment that Mt Druitt school receives a lot of funding and has some great teachers. She laughed it off, and made a comment that I would never send my children there. I live in another suburb and if we are still in this area, my children will be going to the closest school. Probably the one at which I will be working.

I think there needs to be more assessment of the children, as you stated, what about the emotional maturity; if the children like to attend that school; the parental involvement; how are these being reflected my the website? They aren't.

Most parents will select schools based on areas alone, so I guess that this is where the website will come in handy. But I don't feel it places enough emphasis on the teachers and how they teach, etc. What if the year that was tested had always been below average for that school, from Kindy? Or the children were from NESB and had just moved to Australia?

I understand why parents want it, but I think it is being used incorrectly.

PlanningQueen said...

Trevor this is the most sensible commentary I have read yet on the My Schools Website. Thank you very much for sharing your thoughts on this contentious issue.

Keith said...

I too worked for a number of years for the NT education Dept before moving to NSW, and I totally agree.
Good post, well done.
Le Loup.

Trevor Cairney said...

Thanks for your comments Miss Carly, Planning Queen & Le Loup. This agenda has some way to run in view of the Prime Minister's comments yesterday. It's worrying how quickly people have begun to misuse the site.

Thanks everyone,

Trevor

Laetitia :-) said...

"It's worrying how quickly people have begun to misuse the site."

Worrying but not unexpected. Also interesting to note how quickly after its launch the PM said they'll "review" it.

Mr Paroxysm said...

Good write up. I'd also add that when looking at the data for the few school I access the My School website is laughably inaccurate. Some numbers being 40-50% off their reality.

Anonymous said...

Thanks for your fair commentary. I have been a teacher for more than 30 years, privileged to use my chosen career in Australia, Scotland and currently New York City, and I couldn't have said it better. My grandchildren are not school aged, but their parents are nonetheless interested in all things school related. The website and subsequent media attention have had them asking many questions. As you say they want the very best for their precious ones.

KylieM said...

This is fantastic commentary of what I believe to be a useless tool in the matter of choosing schools. My son started high school yesterday, and went to the local school. I did a lot of research, visited the school, had a number of meetings with the principal and was very happy with what they were going to offer my son. Published in the paper the school rates in the bottom, but I am not fazed. I know the hard work that has been done in the school over the last few years, and its small size is going to offer my son everything he needs. I am actually at the point of thinking about excluding my middle son from the Year 3 NAPLAN test this year, because he does not cope with the testing process so the results will not show what he is capable of (for example he tests at level 19 in reading, and yet at home reads novels). Thankyou for your sensible and valuable words!

Trevor Cairney said...

Thank you to everyone for the latest batch of comments (Laetitia, Mr Paroxysm, Anonymous & Kylie M.). It's wonderful to hear different perspectives on the topic. Great to hear from a parent Kylie. Trevor

Unknown said...

My Wife looked at the site and commented that the local school is below the national average, yet the word of mouth comments we receive rates it very highly. I said to my wife at the time that their numbers are purely based on tests and not whole school experiences. As mentioned arts, and social atmoshpere and pure communituy involvement are not included, and would be almost impossible to measure anyway.

A Fantastic commentary and I feel a very non biased and open appraisal, one I will be forwarding to all my firends.

Thank you

jenny said...

I've just found your blog, and I'm so glad I did because I enjoyed reading your balanced commentary. I totally agree that we need to look at ways for schools to improve transparency for parents as they are obviously eager to know more about their child's education.

As a teacher at a progressive school we have an open door policy and you aren't sidelined as a parent. With daily contact with the teachers, weekly class news via the newsletter complete with work samples and regular information evenings, education forums and parent / teacher conferences as a parent of the school I come away feeling that the teachers really know my child and I really know exactly where my child is at. I am aware of any problems as they arise, and I am aware and understand what the teachers are doing in the classroom and the reasoning behind it.

We can do this at such an individual level because we keep our class sizes small and I would love to see the effort and funds and debate generated by the My School Website better directed into looking at alternatives that will help parents understand what is happening in our schools and to take measures such as reducing class sizes.

Thanks again for your post - glad I found you :)

Megan said...

Like you I have been quite angry at both the lack of rigour in how these numbers have been presented on the website and at the way they have been used by those newspapers that published ranking, which compounded the problem. My eldest started school this year, and while we were keen to send him to a close public school, we had 2 within walking distance so we looked at both to ascertain which was best for our child (which is a different question from which has the best academic scores in a couple of subjects). As it happens, the one we chose ranked under the one we decided against. Both are good schools and we still think we chose well because of other factors. As you say, it is a very blunt, and I think also inaccurate instrument. I think it reveals the problems of making comparisons between schools - there are so many factors that come into play. I was particularly concerned with 2 issues: the first is that the formula used to calculate socio-economic advantage does not seem to work. I looked at many schools in areas I know and their scores on this did not reflect my own knowledge of those areas and schools. Also, as you mention, there hasn't been much attention paid to value adding - it is much more important what a school does with what they get then how they do compared to other schools. This would also be a difficult thing to assess, for instance, Year 5 results could be down because the top students have gone to the OC class for the district.Those schools with the OC class would then get a boost in results from Yr 5. I hope that parents do 2 things: pressure the government to put more work into making this a useful and valid instrument, and encourage the hard working teachers of their kids, many of whom may be feeling demoralised.

Jo said...

Yes, another thank you here Trevor, that list of positives and negatives was very helpful.
I really feel for those teachers at schools that have tested poorly, and are going to be judged unfairly.
As a parent though, I must say I felt very relieved when I saw the results for the local public school that my children attend, when really I know (deep down) that it is not an accurate measurement of all the school has to offer, a school is so much more than academic results.

Also, I didn't know that Australia ranked so highly in world comparisons, wow!

Trevor Cairney said...

Thank you to those who offered the latest group of comments.

I appreciate your comment John, you're right, there is certainly a lot more to assessing a school than scores on a limited number of tests.

Great summary Jenny of how teachers evaluate and give feedback every day. I also agree that we need to spend money on other things too (e.g. additional staff development in the area of assessment).

I also liked your summary Megan of how you went about choosing a school for your child. And I'm aware of the O.C. exodus factor having a possible impact on at least one school's year 5 performance.

Thanks also Jo - you of course inspired the post!

Regards,
Trevor

Anonymous said...

Hooray! Finally, a really good review of the pros and cons of the myschool website.
Personally, I've been amazed at the parents I know who have bagged our local school teachers (regarding our school's poor NAPLAN results) and yet those same parents cannot be bothered to hear their 7 yo child read for 5 minutes a day.

Anonymous said...

That is a very fair and balanced assessment of the webiste. There is so much more to choosing a school than a few scores on a website - for my son's school (chosen because it seemed to be the one that 'fit' him best) - looking at last years scores, the Y3s and Y5s are very average. The previous year were all brilliant - does this mean last years lot were a bunch of thickos? Or maybe the constant disruption of a school under renovations impacted on the kids ability to pass the tests?

Who knows... There is so much more to making a good school than a good score!

Marita said...

Thank you for this interesting commentary on the My Schools Website.

My daughters are attending different schools this year. Oldest daughter is in Grade One at the closest public school in walking distance. Younger daughter is attending Pre-Prep at a public school in the next suburb.

Whilst I would love for my girls to be at the same school - so much easier! I acknowledge that they have very different needs and the schools they are attending meet these needs.

But if I just went by the My Schools site then I would never send my younger daughter to the school she is going to.

Yet the website makes no mention that this school offers several base rooms for children with special needs. It doesn't mention the wonderful environment of the school or the amazing playground (seriously the best tree fort EVER).

I think you've summed up very well how it is more than just test results that make a school.

Leah said...

Good commentary :)

The highschool I went to (I graduated in 2004) is often said to be one of the best public schools in Townsville (North Queensland) - and often better than the private schools. It often produces more OP1 students than any other school in the city - or at least enough to rate it in the top 3 (OP1 is our highest tertiary entrance score that graduating grade 12s receive). It has also won many local, state, national and even two or three international music awards and is a strong sporting competitor (the girls' athletics team has won the local interschool athletics carnival 22 out of the last 23 years). It has a very strict bullying policy, and is considered *the* performing arts school in the region. This isn't just my bias - these are either facts, or reports I've heard even from parents and students who do not attend the school. In my personal experience, I have known several students to leave their old schools to attend this one, and none to have left this school to attend another (unless they were expelled or moved to another city).

However its "My School" score was quite average. In the 2009 table, it scored 'white' in all areas except one 'light green'. (The 2008 table is a bit better).

HOWEVER: In Queensland, highschool is grade 8 - 12; therefore, the ONLY tested grade is Grade 9. The results on the website omit the performance of 3/4 of the school (in grades 8, 10, 11 and 12).

I think it's particularly important for ratings of highschools to include the performance of senior years (gr 11 and 12) because at this point you've gotten rid of a lot of the kids who don't want to be at school (because they are allowed to drop out and get traineeships etc after gr 10). You get the statistics regarding kids who are actually trying and not just bumming around in class.

Personal biases aside, the report "My School" gives is very skewed.

Trevor Cairney said...

Thank you to the last group of people who have made comments. Thanks Corymbia for your comment. Thanks Harriet for reminding us that it takes "so much more to make a good school" than the teaching of the basic skills tested. Thank you also to Marita for your personal insight into your two children's experiences at different schools. Finally thanks Leah for your comments; I agree that senior high school performance would add another dimension to the 'My School' site. I appreciate your contributions to the discussion.

Trevor

Anonymous said...

What a great resource!