Agencies dispute over how to solve low literacy and numeracy scores

Documents obtained by RNZ show the Ministry of Education and Qualifications Authority have been at loggerheads over high failure rates in new literacy and numeracy tests.

The Education Ministry told the qualification authority (NZQA) in October that test design factors could be making them more difficult than they should be, the documents showed.

It also asked for more input into design, delivery and marking of assessments.

NZQA responded in November dismissing the ministry's requests and suggesting the ministry was not respecting its expertise in setting tests.

RNZ understands officials were worried by high failure rates in trial runs of the tests last year and in 2021.

A pilot run in the middle of last year with mostly Year 10 students had pass rates of 34 percent in writing, 56 percent in maths, and 64 percent in reading.

The tests will be become a compulsory part of NCEA in 2024 when students must pass all three assessments before they can be awarded any NCEA qualification.

Students would be able to attempt the tests multiple times and the assessments would be available twice a year.

The documents obtained by RNZ showed that in October last year the ministry requested changes including fewer questions, the use of simpler language, allowing students to use spell-checking software, and providing a paper-based option if students did not want to sit the tests online.

It also suggested avoiding complex words and situations that many students might not be familiar with such as tramping or taking food to a potluck dinner, and removing a section of the writing test that asked students to identify errors in a piece of a writing.

The ministry also wanted NZQA to provide dummy tests that students could attempt before sitting the real thing, and provide students who failed with a copy of their test so they could see where they had gone wrong.

NZQA's response in a memo said the ministry had made similar requests previously which it had refused.

It also said the ministry's document professed acknowledgement of the authority's expertise on assessment but did not reflect that, and the two organisations might need further discussion about the nature of partnership.

"NZQA also has concerns about some of the tone and language in the document. The document creates an impression that the assessments have serious flaws and that NZQA is not aware of these, nor taking appropriate action to address them. This is not an accurate reflection of the quality of the past assessment activities; the "fitness for purpose" of the activities is well supported by the statistical analysis undertaken by NZCER [the New Zealand Council for Educational Research] and by the review and observations of the marking panels."

'It shows worry there could be high failure rates' - Erica Stanford

National Party education spokesperson Erica Stanford obtained copies of the same documents as RNZ under the Official Information Act.

She said they showed the education ministry was worried by the low pass rates in the tests.

"I think the ministry are very worried that there are going to be high failure rates and the reason that they're worried is because they have no visibility prior to this test how kids are doing," she said.

Stanford said she agreed with the ministry's request that students be allowed to see their test papers, but she said NZQA was doing a good job of ensuring the tests were accurate and did not disadvantage students from particular backgrounds.

"Where we are going wrong is that we are not equipping kids, our young learners, in literacy and numeracy - so that by the time they get to this age they are able to sit these tests.

"So the massive failure rates we are seeing are not about these tests. They are about the fact that we are not equipping our kids," she said.

Stanford said another part of the problem was that pupils were not sitting tests often, so they were ill-prepared for high-stakes assessments like the literacy and numeracy tests.

Agencies 'are on same page' - Education Minister

Education Minister Jan Tinetti said she wanted to ensure young people were getting the best teaching possible which was why the writing, maths and literacy strategies were being introduced and the common practice model was being developed.

However, she was concerned some students were not getting the opportunities they should be and that would need to be looked into.

"The best of the best" academics and other education experts were working on the common practice model so the best possible teaching methods could be used in schools.

The planned model was about to go out to teachers for their feedback, Tinetti said.

"They're the experts who work in this area every day as well."

She said it was not about making the tests easier. Multiple ways were needed to establish what levels students had achieved because even some bright young people did not cope well with tests.

She believed the NZQA and the ministry were "on the same page" and were working together to fine-tune the tests before they were introduced next year.

What the Ministry of Education asked for:

The education ministry's brief in October said it set the standards and NZQA developed and administered the assessments.

It said the brief outlined revised recommendations to NZQA and it expected NZQA to respond explaining how it was incorporating the ministry's principles and expectations.

It also asked for direct input into the development of the assessments.

"Given the scale of the change, the high-stakes nature of the co-requisite assessments, and the fact that we are still in a pre-implementation phase of product development, the ministry requests more input in, and meaningful opportunities for our Learning Area and Wāhanga Ako leads (SMEs) to engage collaboratively with NZQA in decision-making regarding the design, development, delivery and marking of the Common Assessment Activities and portfolios, as well as in the determination of cut scores and pass thresholds."

The ministry recommended reducing the number of questions because schools were taking longer than 60 minutes to run the tests and to reduce the "digital cognitive load" on students.

"The additional cognitive load on text comprehension when reading digitally is well documented (Clinton, 2019, Delgado et al., 2018; Kong et al., 2018), and poses a significant challenge to less able readers," the brief said.

For the same reason, the brief recommended giving learners the option of doing the tests on paper rather than online.

It said paper-based tests would also make the assessments more equitable for students who were not familiar with using computers for school work.

"Based on the pilot assessments to date, all of which have taken a "digital first" approach, learners identified as Māori, Pacific, domiciled in Realm Nations, and/or from low-decile schools had significantly lower chances of achieving any of the new co-requisite standards than their Pākehā, Asian, and/or high-decile peers," the ministry said.

It said the tests should not assume "digital fluency" and should not require too much scrolling through information or graphics that were not helpful.

It said texts for the reading assessment should be at the expected level of reading, and not more or less difficult, and should be screened to ensure the vocabulary and readability was appropriate.

It said the test should assess foundational literacy, not academic literacy, and did not need to capture a range of performance such as from achieved to excellence.

The brief said two texts in last year's first pilot were about science and 22 percent of the words were low-frequency, topic-specific words such as ovipositor that learners might not be familiar with.

It said the literacy requirements of the numeracy tests should not be harder than the reading and writing assessments.

The Ministry of Education and NZQA respond to RNZ

Asked if NZQA had resolved the concerns the education ministry raised, the ministry told RNZ "good progress has been made".

"Both agencies are working together to strengthen and refine the assessment activities. Some of the actions are led by NZQA and others by the Ministry of Education," it said.

The ministry confirmed that it had confidence in NZQA's work.

Both the ministry and NZQA said they were confident the tests trialled this year would be fair, accurate and fit for purpose.

NZQA confirmed that it believed the ministry acknowledged its leadership and expertise in developing the assessments.

Asked which changes requested by the ministry the authority had made, it said the nature of a pilot meant changes were to be expected.

"For example, now we have had a full year of the trial, NZQA has been able to analyse our digital examination platform's data to better understand how long students took to complete the literacy and numeracy assessments.

"This will help us refine assessments so the majority of the 2023 student cohort will be able to complete them within 60 minutes, and there is no time limit - students should have as long as they need," it said.

"NZQA has involved ministry subject matter experts in the assessment design and development process, and has provided the ministry with early drafts of the activities for feedback."