The Inspector-General of Security and Intelligence has found "significant shortcomings" in how the Security Intelligence Services (SIS) carries out vetting for security clearances.
Problems with the vetting process were identified in Inspector-General Cheryl Gwyn' annual report last year, after her office received complaints from three people, employed in positions that required security clearances, who had lost their jobs as a result of SIS background checks.
A fourth complainant had an offer of employment withdrawn.
Ms Gwyn launched an inquiry, and has now released the first part of her report, which found "strengths but also some significant shortcomings in SIS practice that did not meet standard data protection requirements".
She has made a number of recommendations to improve the way the SIS stores and uses vetting information.
"For instance, I found electronic records for the largest category of clearance-holders and candidates were accessible at any time to 60 or so staff who carry out security clearances.
"Under standard data protection requirements, staff should have access only to files that they are working on and only while those files are active."
The report noted the investigations undertaken for security vetting were highly intrusive.
"The information collected in the course of vetting includes personal information relating to, among other matters, sexuality, social habits, physical and mental health, financial well-being, and religious and political affiliations.
"The consolidated records collected during the vetting process likely comprise the most sensitive repository of such personal information held by the New Zealand government.
"Any inappropriate use or unwarranted disclosure of that information could have serious implications for the subject of that information and others."
Ms Gywn said she acknowledged the "serious commitment to privacy" from the SIS staff who undertook this "difficult and sensitive" work.
"What is also needed, and what is required in any government agency that deals with personal information, are systemic safeguards to back up and verify that commitment," she said.
She has recommended tighter procedures so staff could only access personal information on a 'need to know' basis, and regular auditing of each record and file.
Ms Gwyn said she also found there were not adequate checks or assurances that records were never accessed for "unauthorised or improper" purposes.
There were some circumstances when that information could properly be used for other purposes, she said.
"In particular, it appears reasonable that, where the holder of a security clearance becomes the subject of a counter-intelligence investigation, some information from that person's vetting record may properly be used for that investigation.
"However, other use - for example, use of vetting information for general intelligence purposes - is likely to be unjustifiable."
Ms Gwyn has recommended that decisions about using information for other purposes should be made at a very senior level, and the circumstances would have to be serious enough to warrant it.
SIS Director of Security Rebecca Kitteridge declined to be interviewed.
In a statement, she said she welcomed the report and accepted all of the findings.
"NZSIS staff responsible for vetting are conscious of the sensitivity of vetting information and place emphasis on the importance of personal integrity and discretion."
Ms Kitteridge said the culture within the SIS of treating personal information with the "utmost sensitivity" needed to be backed up by more robust systems.
"The personal information obtained during the vetting process is very sensitive and it is important the appropriate safeguards are in place.
"Changes are already under way in the vetting area and implementing these recommendations will become part of a much larger piece of work to improve the vetting service," she said.
The second part of the Inspector-General's report, dealing with ICT systems, will be completed in the coming months.
Work under way to tighten procedures
The minister responsible for the SIS, Chris Finlayson, said work was being done to tighten up procedures.
"Well, vetting is an issue that was raised with me sometime last year by the [SIS] Director saying it was an area where she wanted to do an amount of work, because the vetting procedures were pretty out of date."
He said he welcomed the report and the work that she had been doing.
Mr Finlayson said suggestions that anyone could access that information were "hyperbolic", as there were protections.
"But there need to be greater constraints placed on who, within the organisation, can and under what circumstance, access the information.
"It's a constant, ongoing period of reform for the SIS, and I'm very happy that the report is out, but to a certain extent it's almost after the event because the SIS is working on this stuff anyway."
He said the complaints about people losing their jobs after being vetted were not addressed in the report, but were matters that were still being considered.
Green Party co-leader Metiria Turei said access to such deeply personal information should be much more tightly controlled.
"The Inspector-General has said it is not appropriate for so many people inside the SIS to have access to the very personal information of thousands of New Zealanders.
"She was very clear that not only is it deeply personal information about your private life, your extended family and all your friends, but that it is the largest body of such private information held by any government department."
Ms Turei also said the SIS did not have the proper controls or policy on how to manage access, and that it was allowing people to use it for purposes other than what it was handed over for.