Standard deviation

Hi,

For old .ADP files viewed in ExploreP you could do a contour plot of standard deviation and use the range settings to quickly see where data exceeded a given threshold.  Is this possible in Storm (or Surge)?

(Rachel (Hjaltland) asked this in another thread... sorry if it was answered elsewhere)

Cheers

Alan

Dear Alan

The Nortek instruments that were developed from 1999 and onwards do not estimate the "standard deviation" as part the  data collection process.  This was a feature found in the older ADP/NDP systems that were discontinued in 2004.

The "standard deviation", which really was an estimate of the uncertainty in the mean velocity and not the standard deviation of the raw data ( i.e., sigma(V)/sqrt(N) and not sima(V) ) was replaced by raw data storage.  In the Aquadopp, we introduced "Diagnostic data", a time series of 1 Hz data that could be used to characterize the mooring motion.  It was soon also used for the estimating wave height and direction in shallow water so we introduced this same "wave burst" in the Aquadopp profiler (still the same thing -- 1 Hz data collection in between the mean current profiles).  Looking back, it is clear that we did not loose much by removing the "standard deviation" from the measurements (and we saved a lot of memory space) since it had not clear cut use.  Originally, it was intended as a quality parameter but in the end it tended to be dominated by the wave motion, especially in the cells close to the surface.  For this reason it never really caught on as a parameter that people  used in their work.  Of course, there are exceptions - I have seen examples where is used in conjunction with the amplitude data to identify fish - but it still lost out in the evaluation we made at the time.

Best regards, Atle Lohrmann

Atle, hi.

I'm afraid I'm about to hijack this post to bring the debate that we have been having by email into the public domain, because I am interested to know what quality control measures are used across the Nortek current profiling user-community and what tools they use to extract them from the raw data files.  Despite your maligning of the 'sample standard deviation' as not being of the same value as the 'single ping SD', it is still better than nothing, which appears to be what is offered with the current range of Aquadopp/AWAC profilers.

In our email exchanges you have suggested that you favour a 'spectral analysis' approach to establishing data quality; however, this does not appear to be a feature of your Storm/Surge analysis software.  Furthermore,  I struggle to see how reliable this might be for data that do not have strong periodic components.  Perhaps you could expand on this, as I'm not familiar with these analyses?

You have also made reference to the service that Nortek provides to its customers by undertaking to examine data and provide 'expert' advice as regards quality - I believe that you are not providing your customers with the tools to be able to undertake this analysis themselves, and make decisions about whether the returned data are of sufficient quality for whatever their particular end-purpose might be.

Although this is an old thread, even in 2008 data storage was cheap and compact; storage limitations are not a valid excuse for not recording quality/diagnostic data.  An Aquadopp recording data at 1minute intervals for a year, in all 128 cells, will use less than 600MB of memory, which is trifling by today's standards - and you'll have changed the battery more than 100 times to achieve this!  I suggest that there is no hardware constraint on recording both single-ping and sample SD.

Open water instrument deployments are expensive and often constrained by boat availability and weather; it is imperative that there are quantitative measures of data quality that allow users to determine whether they have obtained valid data immediately upon recovery, i.e. the tools to examine this should be incorporated into your proprietary analysis software; if the data are not valid, users need to be able to diagnose what can be done to improve the data quality prior to redeployment, and to make that diagnosis quickly and in the field without recourse to remote 'experts'.

best wishes, Jeremy

Dear Jeremy

"The standard deviation" (i.e. a recursive filter estimating the uncertainty in the average velocity) was introduced as a QA parameter in 1994 and we continued to use it up to about 2003.  As stated in  2008, we never heard or saw people use the parameter much.  This does not mean that it is useless, it just means that we did not see a need for it when we introduced our line of Aquadopp profilers.

Maybe you would care to post a few examples on how it can be used to detect QA issues that other more common methods (amplitude, vertical velocity, etc.) fail to detect?

Best regards, Atle Lohrmann