Dear Electricity Authority,
A new wind turbine farm has been running for some months at Turitea, generating electricity and supplying the grid; further enlargement is proposed as well. Please supply the half-hourly data on generation (and consumption) from the start of its supply to the grid, as is established practice for other generators.

Yours faithfully,
Nicky McLean

Dear Ms. Ballinger,
Thank you for the 28’th of June omnibus reply from Ms. Gillies to my recent five requests for half-hourly electricity generation data at various power station sites: I shall distribute the appropriate parts to the individual requests, lest otherwise it seem that your organisation had not responded to them. Here I proceed with her response to the request for data on the new Turitea wind turbine “farm”.
I would have beeen happy to respond with just “Information received and assimilated; thanks.” - and I can imagine a possible universe in which this could happen – but this is not it.

As on previous occasions, your response has nominated not just a few data files containing information but collections of hundreds of data files, thus again occasioning a delay while the files are acquired and inspected. As you state, there is a web page offering a collection of revised files under https://www.emi.ea.govt.nz/Wholesale/Dat..., so, why not start with a spot check? The first-named sub-directory is Embedded_generation, and indeed it shows a number of monthly files with a new date stamp (of the 7’th June 2022) going back to January 2020. So, why not pick on January 2022: file 202201_Embedded_generation.csv? The first few lines (truncated here) are
POC,NWK_Code,Participant_Code,Loss_Code,Flow_Direction,Trading_Date,...
ASB0661,EASH,CNIR,H01,I,2022-01-02,5374.0260,5372.2522,5369.3906,...
ASB0661,EASH,CNIR,H01,I,2022-01-03,3504.0292,3506.3322,3491.9458,...
A glance at the second line shows that it is in error – or rather, in being for the second of January, it shows that the data for the first of January are missing. A straightforward search shows that the missing data for the first of January are not to be found somewhere else in the file.
The file has been downloaded multiple times over separate days, using four quite different computer systems: all copies are the same, so this omission is not due to some internet hiccough. Presumably you can verify this situation by more direct access to the file that does not involve the internet. But because this new file has replaced the previous file, you will not be able to see that the data were not omitted then – though given proper data management you should be able to consult an archived copy. Do you still claim to be merely passing on these files exactly as they had been (re-)supplied? If even they had been re-supplied. Your use of “regenerated” suggests that the new versions have been produced from an existing store.
This is yet another case of supplying data files with apparently random omissions. Given proper Data Management, nothing of the sort should be happening.
And there are the changes in format, away from the established usage. Some trivial, such as “Trading_date” becoming “Trading_Date”, others wasteful such as showing .0000 on the end of integer values (51,778 of them in this file), and not using the established date format. Not using proper line terminations also can cause trouble for the many users of windows systems.
When issuing a new version of a file, a basic Data Management procedure is to compare the old version against the new and thereby verify that the desired changes are present and also that undesired changes are absent. This simple procedure is blocked by the innumerable pointless format changes that swamp such a comparison with trivia. It is better to stick with the established format.
Enough. Rather than bloat this response with a schedule, I leave it to your staff to check for and correct errors in your other new files of the Metered_data collection, which I shall ignore until then.

You indicate a further collection of data files, https://www.emi.ea.govt.nz/Wholesale/Dat... as containing data for Turitea. A mere two hundred and ninety-eight files. Curiously, starting with August 1998 – where are the files for earlier months? (Not relevant for Turitea)
The date stamp on many of these files is for the 15’th October 2018, whereas an earlier manifestation of the collection has the 12’th May 2014. The two sets of files are identical until the file 199909_Generation_MD.csv, where the new version contains the same data (it seems: the files are the same size), except with some records in a different order. (Why?) This continues until file 200001_Generation_MD.csv where different data appear as well, notably so for Huntly, for instance on the first of January 2000 where a value of 102,840 KW becomes 136,740 KW in the new file. There are many other such large changes.
Unlike Schrödinger’s Cat which is both alive and dead in varying proportions until observed, the Huntly power station is real and comprehensively observed, specifically in the form of half-hourly data on its generation. How can your system generate different values from the historical record? The different values can’t both be right, and now that error has been observed, it is possible that both are wrong.
On the other hand, the new version of file 200107_Generation_MD.csv has data, the old version is empty of data; this is an improvement.
Then, starting with 2009, the file size expands by half again. Four decimal digits are no longer enough, instead there appear values such as 16517·869999999999 and 16559·130000000001. Seventeen digits of precision! Since these numbers are declared to be Kilowatt-hours, 0·000000000001 Kilowatt-hours is 0·000000001 Watt-hours, which is 0·0000036 Joules or 3·6 micro-Joules. Perhaps the energy of a drifting speck of pollen landing on a table.
For smaller power stations one or maybe two decimal digits could be reasonable, as for the Brooklyn wind turbine, but in the context of power stations producing hundreds of Megawatts, this is a waste of time and is just silly.
Also silly is the presence of data encoded as WTK.WTK0111.WATA.waitaki.Hydro.Hydro, for which some 114,818 values are provided, and all but eighteen of which are zero – the maximum is just 33. It’s true that the Marsden B power station was constructed and never run (other than for test purposes), but what is the point of this? Nor is it the only example.
Other curios: the main power station at Huntly is declared to be coal-fuelled (the recent usage), yet it has definitely run on gas as well during the times covered by these data files. It would be interesting to have a data series reporting the consumption of coal/gas (and fuel oil for gas “fireball stabilisation”), perhaps even half-hourly, but such a series would probably not come via Transpower. And what of September 2020? Perhaps the Huntly collective was poorly observed then as your file 202009_Unit_level_generation.csv (as revised on the 7’th June) contains no mention of Huntly, nor does file 202009_Generation_MD.csv. A peculiarity of the theory of Quantum Mechanics is that an observation is not real until it is recognised by a conscious being. (Copenhagen sect)
Similarly, New Plymouth’s power station was initially designed to run on fuel oil (as with Marsden B) but then was converted to gas. Its consumption of oil would be minor as well. And a name such as NPL.NPL1101.CTCT.new_plymouth.Gas&Oil.Thrml is troublesome due to the presence of a character more normally used as an operator in calculations. For this reason a properly managed information storage system would not use such symbols in a name.
It is not just Huntly’s data that have changed. I rather suspect there are other confusions within this secondary collection of data files: omissions, gaps, duplications, misinterpretations – but describing them would be tedious. Instead, the name Turitea is found, as LTN.LTN2201.MRPL.turitea.Wind.Wind. Aha!
Your reply offers a link https://forum.emi.ea.govt.nz/thread/turi... where Adam M Fuller declares “The monthly half-hourly files are great” Despite this praise Dr. Bishop rather dismissively mentions the Generation_MD files and some other possible sources that might be perused. But he seems not to know that amongst the routinely supplied files of Metered_data/Grid_import there appears a data series encoded as LTN2201,MRPL,GG,MRPL,kWh,I,F, which can now be taken as being for the Turitea wind farm.
Or, can it? There is a lot of activity attributed to the Linton substation as depicted in various data series using the code LTN, and this series being all zero for months suggested nothing useful to start with. If instead the code had been TRT for Turitea, there would have been a lead to follow. This could have been used for the wind farm’s substation at 40°26'11"S 175°42'3"E rather than the distant Linton substation ( 40°24'38"S 175°37'53"E) – so, I misguessed, and thus couldn’t find data that looked like it belonged to the Turitea wind farm.
However, because these data files are amongst those re-supplied with a date of the 7’th June, perhaps Dr. Bishop will not recommend them until they have been checked, as noted above.
I’m also guessing that the new 220KV transmission line between Linton and Turitea is deemed a part of the national grid (given the voltage), so the connection point should be called TRT or something other than LTN, because otherwise, the implication is that the generation is measured on arrival at Linton, as encoded by the use of LTN - which is not where the power enters the grid. This is supported by the GG encodement: Grid-connected Generation. Measurement should be “at the generator’s terminals”, though for a wind farm with scattered turbines it could be at the collection point, which would be TRT. Not LTN.
So, the remaining question is the location of the actual measurement of the generation due to the Turitea wind farm. And not some imputed location with a fiddle to the data to counteract the supposed transmission losses. You have mentioned https:/www.emi.ea.govt.nz/Wholesale/Datasets/_AdditionalInformation/ArchivedDatasets/201310_
CentralisedDatasetFinal and within the collection are data for the Hau Nui wind farm (S of Martinborough: 41°21'49"S 175°29'3"E) with a variety of names: HAU0331.GENE.EG. for example. But also, GYT0331.GENE.EG. Thus, the specific code for Hau Nui was discarded for the GYT used for Greytown, and, the values referred to Greytown are 7% higher than values at Hau Nui (where the data series overlap) – the idea seems to have been that a certain amount of power as supplied from Hau Nui would, were it to have come from Greytown, have had to be some 7% larger to offset transmission losses along the way. Though, a percentage is not a proper loss calculation. Whichever code name is used, the old data have been discarded in favour of your new Metered_data collection, and it stops offering data for Hau Nui with March 2008. There is no mention of Hau Nui at all in your Generation_MD collection.
There are other examples of this sort of adjustment, and I hope that the Turitea/Linton nexus is not another.

Otherwise, Turitea’s data are found. They were there all along, just under disguise.

As for the omitted data for Huntly, one imagines that September 2020 is not all that far off in history, and that the workings of the three power stations there that constitute the largest collective in the country would be of wide interest, both public and “sector-wide” and even to your own organisation yet it appears that no-one else has noticed. Perhaps they have not looked, or are not conscious observers. Collecting and publishing such data would be of greater value then omitting them.

Yours sincerely,
Nicky McLean