Improved results: More mirage than miracle

Summary - South Africa’s matric pass rate improved by almost 25 per cent over the past four years, to 73 per cent in 2003. However, there was more scepticism than celebration when the results were announced. Many people believe that the higher pass rate reflects lower standards rather than an improved education system. The fact is that most matriculants are unemployable and ill-equipped for tertiary study. Many big companies are spending up to 10 per cent of payroll costs on education and training, compared to the international norm of 4 per cent – 6 per cent. Moreover, the results do not square with comparative studies. In a Grade 8 maths and science study conducted in 38 countries in 1998, South African pupils’ mean scores were 275 and 243 respectively, far below the international mean scores of 487 and 488, and below two African countries, Morocco and Tunisia. Other studies show that pupils are performing two to three grades below the requirements of our own new curriculum. The department of education reacts defensively to such criticism. It contends that the higher matric pass rate is the result of a host of intervention programmes targeted at historically disadvantaged areas. An analyst at the Education Foundation agrees that interventions are working; she says that any attempt to manipulate the exam results would have been leaked to the press. But in a new book, Getting Schools Working, three top educationalists argue that the system has proved almost impervious to a decade of reform. A look at the numbers helps to sort through the welter of conflicting opinions. Firstly, 71 207 fewer pupils wrote the matric exam in 2003 than in 1998, so improvements measured in percentages are misleading. Also, almost 115 000 pupils (or 20 per cent) who entered Grade 12 in 2003 did not write the final exam. Had they all written and failed, the final pass rate would have been 58 per cent, not 73 per cent. Educationists speculate that schools may be discouraging poor students from writing in order to shore up their pass rates. This implies that standards are not dropping, and that improvements in the quality of schooling are not as significant as the department would like us to believe. Professor Jonathan Jansen, dean of education at the University of Pretoria, believes that until the government reveals how and to what degree it adjusts exam marks through the internal moderation process, the perception will remain that the process is vulnerable to political pressure. One easy way to allay these doubts would be for the department to commission an independent agency to compare a random sample of papers from last year with those of five to ten years ago. It could also accelerate plans to introduce standard national exams at Grades 6 and 9, which would provide useful benchmarks against which to compare matric results. However, Joe Muller, UCT’s deputy dean of research, humanities, believes that given the department’s inherent distaste for testing, it will probably continue to resist pressure for assessment reform.

The phenomenal improvement in South Africa's matric results - by almost 25 per cent over the past four years - should have earned education minister Kader Asmal a standing ovation. Instead, last year's 73 per cent pass rate was met more with scepticism than rejoicing.

Rather than seeing the improved matric pass rate as evidence of a remarkable recovery in the education system, many saw only a watering down of standards.

For employers and universities, the rosy matric results fail to square with the facts on the ground: that the majority of pupils are unemployable as they leave school, incompetent in maths and science, and barely equipped for further study.

Abacus Recruitment director Org Geldenhuys says a matric certificate already means very little in the job market. If there is an added perception that standards are dropping, any chance of matriculants finding employment in the formal sector will be utterly dashed.

"The long-term effect of releasing under-qualified students, from both schools and universities, into the economic mainstream means the corporate world faces the larger problem of having to spend ever-larger amounts of money and time on trying to impart true skills," he says.

Many large companies already spend up to 10 per cent of payroll costs on staff education, training and development, compared to the international norm of 4 to 6 per cent. It's a high price to pay, especially considering that South Africa spends 5,6 per cent of GDP on education, more than most other developing countries.

There is a significant and unexplained disjuncture between pupils' performance in matric and South Africa's abysmal performance in international comparative studies of pupil achievement, where it often ranks below other developing countries.

In 1998, the grade 8 maths and science study, TIMSS-R, was administered in 38 countries. South African pupils' mean scores of 275 for maths and 243 for science were well below the international means of 487 and 488 respectively, including below the mean scores of two African countries, Morocco and Tunisia.

Other studies have shown that South African pupils perform two to three grades below the requirements of our own new curriculum and have only a rudimentary grasp of reading and writing at the end of grade three, when they should be doing so fluently.

The message is that the school system is failing to get the basics right. How even 50 per cent of pupils then go on to pass matric is anyone's guess.

The department has reacted defensively to suggestions that not enough has changed in the education system to account for the dramatic recovery in the matric pass rate.

Head of communication Mathula Magubane argues that the improvement is a result of a host of targeted intervention programmes, particularly in historically disadvantaged areas. A few years ago, 1 042 schools achieved a matric pass rate below 20 per cent. Last year, only 154 (2,5 per cent) of schools performed that poorly.

She says there has also been a marked improvement in the delivery of learner support materials to schools. The majority now receive materials and conclude their admissions before the end of the year ensuring that teaching and learning takes place from the first day of the new school term.

The learner/teacher ratio is improving steadily, especially in disadvantaged areas, and the culture of teaching and learning has improved in the majority of schools. The 2003 report notes that all grade 12 pupils now write preparatory matric exams each year and many benefit from additional classes and winter and spring schools.

Furthermore, it argues that the filling of vacant posts, especially of principals, improvements in school governance, increased parental involvement, and the appointment of more subject advisors have all contributed to improve conditions in schools.

"With all the intervention programmes in place, the improvements in the matric pass rate do not come as a surprise to the department," Magubane says. "A lot of deliberate effort has been put in and the results are there for all to see, both in the form of matric results and in the visible changes in the schools themselves.

"To suggest that there is a lowering of standards is to condemn a large section of our society, particularly the historically disadvantaged, as being incapable of learning and achieving good results, no matter how good their learning environment may be."

Educationalists are divided over whether the matric results are legitimate.

Matriculation specialist Jennifer Shindler, an analyst at the Education Foundation, dismisses the possibility that either the department or the independent quality control body, Umalusi, deliberately manipulated the results.

"If there had been an instruction to increase the results by, say, 10 per cent I have no doubt that it would have leaked out," she says. "I think Umalusi would stand up to scrutiny in an investigation."

Shindler attributes the improvement in the matric results over the past five years to a series of interventions that have each added a few percentage points to the total pass rate. For instance, the decision in 2001 to increase the mark achieved in non-language subjects by 5 per cent for non-English and Afrikaans speaking pupils; the wholesale conversion of pupils from higher to standard grade; the weeding out of marginal performers in lower grades; upward mark adjustments for second-language speakers; and the fact that since 2001 continuous assessment has made up 25 per cent of the year mark.

The university of Cape Town's deputy dean of research, humanities, Joe Muller draws the opposite conclusion, saying that while all the explanations put forward for the improvement in the matric pass rate have some plausibility, not enough has changed in the system itself to account for such a remarkable recovery in the matric pass rate.

The core finding of a new book, Getting Schools Working by three of South Africa's top educationalists - Muller, Nick Taylor, CEO of Jet Education Services, and Penny Vinjevold, director of planning in the Western Cape Education Department - is that despite sustained budget increases and a myriad of initiatives, the school system has proved almost impervious to a decade of reform.

The picture they paint is very different to Magubane's official portrait.

They argue that principals are failing to ensure that teachers cover the curriculum, district officials are not supporting schools, provinces are under-spending and the national education department is failing to assure the quality of teachers' work.

The upshot, they claim, is that teachers are not teaching what they are supposed to teach and pupils are not learning what they are supposed to learn.

One way to penetrate the myriad of conflicting opinions is to look at the numbers.

The first striking feature is that 71 207 fewer pupils sat the matric exam in 2003 than five years ago when pass marks were at their lowest - so improvements measured in percentage terms can be misleading.

The average annual decrease in the number of candidates since the peak enrolment in 1997 has been 5,2 per cent with the greatest decrease (8,3 per cent) occurring between 2000 and 2001.

The Education Department has attributed the decline to its policy of excluding over-age learners and repeaters and the fact that schools have been preventing weak learners from progressing to grade 12 in order to shore up their pass rates.

In a snap survey in January 2003, South African Democratic Teachers Union (Sadtu) members attributed the decline to two factors: drop-outs due largely to poverty (with HIV/Aids, teenage pregnancies, drug abuse and lack of motivation acting as contributory factors); and the tendency for schools to discourage weak learners from sitting the exam.

In a statement, Sadtu urged the department to ponder the words of one respondent who said: "This happens because of the pressure exerted on schools to perform."

To prove there has not been widespread culling of pupils, the department has published flow-through rates. On average, 77 per cent of learners who enrolled in grade 11 in 2002 went onto grade 12 in 2003. This was a 5 per cent improvement on the 72 per cent flow-through rate between these grades in 2001 and 2002 and a 10 per cent improvement on the 67 per cent flow-through rate between 2000 and 2001.

The point, though, is not how many went through but whether these pupils actually sat the final matric exam. According to the department's report on the 2003 matric exam, 16 880 candidates or about 3,6 per cent of those who registered for the exam failed to write it.

However, what the report does not reveal is that almost 115 000 pupils (or 20 per cent) who entered grade 12 in 2003 failed to write the final exam. Had they all sat the exam and failed, the final pass rate would have been 58 per cent not 73 per cent and the debate about whether the exam is becoming easier would never have occurred.

Since 1996, the proportion of enrolled grade 12s who have failed to write the matric exam has hovered between 8 per cent and 11 per cent. In 1996 and 2001 it was 8 per cent, in 2002 it was 9 per cent. In 2003 it leapt to 20 per cent - a number totally out of sync with the preceding seven years. Educationalists speculate that either pupils are being discouraged from writing to shore up schools' pass rates or they are ghost pupils fictitiously entered by principals to increase their allocation of teacher posts, which are calculated on a school's enrolments.

Although the consequences for those pupils being forced out of the system could be dire, what these figures imply is that the standard of the matric exam has probably been maintained and although there has been some improvement in the quality of schooling, it is not as significant as Asmal would like us to believe.

Writing in the Sunday Times on 4 January 2004, professor Jonathan Jansen, dean of education at the university of Pretoria, argued that until government revealed how and to what degree it adjusted marks in specific exams through the internal moderation process, "there will always be the perception that the process is vulnerable to political pressure and manipulation".

According to an Umalusi information sheet, the statistical moderation of the matric results is done in a meeting between the examining body and Umalusi. The former brings to the meeting an analysis of the matric results that highlights any unexpected results and idiosyncrasies.

The standardisation is done partly by comparing the statistical distribution of raw marks in any subject with the expected distribution as determined by the raw marks achieved in the preceding five years.

If the actual distribution is close to the expected distribution (differing by not more than 2 per cent over the entire range of the middle 95 per cent of candidates) it can be assumed the examination was of the appropriate standard.

If not, marks are adjusted upwards or downwards to match the desired distribution, subject to the limitation that no upwards adjustment should exceed half of the actual raw mark or an improvement of 10 per cent. Downwards adjustments are subject to various conditions to soften their impact, and a cap of 8 per cent is recommended.

While Umalusi concedes that standardisation is "not a perfect solution" to variations in exam standards, it says it offers "at least a partial guarantee of comparability between successive examination standards, thus giving candidates equal opportunity over the years".

The department could lay the controversy to rest quite easily by commissioning an independent agency to compare a random sample of papers in different subjects, and the marks awarded, from last year with those of five or ten years ago.

Even better would be for Asmal to accelerate plans to introduce standard national exams at grade six and nine level. This would provide a better indication of the performance of the school system as a whole and an interesting benchmark against which to compare the matric results.

Writing in a new book, Changing Class: Education and Social Change in Post-Apartheid South Africa, Muller speculates that given the department's inherent distaste for testing, it will probably continue to resist public pressure for assessment reform.

Moves towards external assessment over the past five years have been marked by "foot-dragging and unaccountable delays," he says, despite the fact that only comprehensive systemic assessment can provide the kind of performance data on the school system that is desperately needed. Without it, the gaps and failures in the system are kept invisible and the system cannot be held to account.

So were the matric results manipulated or not? Are standards slipping? Is an A aggregate today worth less than it was 10 or 20 years ago?

The question appears to defy a straight answer. It is seems fair to conclude, though, that matric standards have been roughly maintained but that dramatic improvements in the matric results have more to do with declining enrolments and the careful screening out of candidates than a renaissance in the quality of schooling.

Human Sciences Research Council director Linda Chisholm put it well when she said: "When measured against the debates of the early 1990s and what was considered necessary to raise standards, standards are certainly high. But when measured against expectations for an education system in which poverty, inequality and injustice have been overcome, we still have a long way to go."