Post by thread:[PIC]: The tintinnabulation of the noise

{Quote hidden}>From: Lawrence Lile <.....llileKILLspam@spam@TOASTMASTER.COM> >Reply-To: Lawrence Lile <llileKILLspamtoastmaster.com> >To: .....PICLISTKILLspam.....MITVMA.MIT.EDU >Subject: [PIC]: The tintinnabulation of the noise >Date: Tue, 7 Nov 2000 09:21:00 -0600 > >(above quote is from Poe) >(yes, tintinnabulation is a word) > > >I'm sampling a DC voltage with lots O noise. I've set up a routine that >takes a sample, waits a few milliseconds, then takes another sample, etc. >When 15 samples are taken, it takes the median of the samples as the >result. Still get false alarms due to noise pulses every once in a while. > >Now, I am not real sure if I will have 60 hz noise (picked up from the air) >or 120 hz noise (from a full wave rectified power supply) as the major >noise component. It may actually be both, at various times. > >Does Old Nyquist rule in this situation, should I sample at twice 120 hz or >more to cancel out the effects of noise? I'm not interested in the >frequency of the signal at all, just in an accurate DC level. The sampling frequency should be based on 60Hz period, when the noise comes from. Sampling input signal with higher frequency, for example 5 * period, and averaging during a whole period of 60Hz works in this case like digital filter. Niquist said about the theoretical minimum sampling frequency, but in reality, it is much higher. Andrej _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. Share information about yourself, create your own public profile at http://profiles.msn.com. -- http://www.piclist.com hint: To leave the PICList EraseMEpiclist-unsubscribe-requestspam_OUTTakeThisOuTmitvma.mit.edu

On Tue, 7 Nov 2000 17:09:56 CET, you wrote: {Quote hidden}>>From: Lawrence Lile <llilespam_OUTTOASTMASTER.COM> >>Reply-To: Lawrence Lile <@spam@llileKILLspamtoastmaster.com> >>To: KILLspamPICLISTKILLspamMITVMA.MIT.EDU >>Subject: [PIC]: The tintinnabulation of the noise >>Date: Tue, 7 Nov 2000 09:21:00 -0600 >> >>(above quote is from Poe) >>(yes, tintinnabulation is a word) >> >> >>I'm sampling a DC voltage with lots O noise. I've set up a routine that >>takes a sample, waits a few milliseconds, then takes another sample, etc. >>When 15 samples are taken, it takes the median of the samples as the >>result. Still get false alarms due to noise pulses every once in a while. >> >>Now, I am not real sure if I will have 60 hz noise (picked up from the air) >>or 120 hz noise (from a full wave rectified power supply) as the major >>noise component. It may actually be both, at various times. >> >>Does Old Nyquist rule in this situation, should I sample at twice 120 hz or >>more to cancel out the effects of noise? I'm not interested in the >>frequency of the signal at all, just in an accurate DC level. > > > >The sampling frequency should be based on 60Hz period, when the noise comes >from. Sampling input signal with higher frequency, for example 5 * period, >and averaging during a whole period of 60Hz works in this case like digital >filter. Niquist said about the theoretical minimum sampling frequency, but >in reality, it is much higher. If you are trying to measure DC with 60Hz noise, 60Hz is exactly what you DON'T want to sample at as this is almost guaranteed to add a DC error. Avaraging samples at 120Hz ( or any multiple of 2 x the noise frequency) would be a lot better as the noise half-cycles would cancel each other out. -- http://www.piclist.com hint: To leave the PICList RemoveMEpiclist-unsubscribe-requestTakeThisOuTmitvma.mit.edu

> >The sampling frequency should be based on 60Hz period, when the noise >comes > >from. Sampling input signal with higher frequency, for example 5 * >period, > >and averaging during a whole period of 60Hz works in this case like >digital > >filter. Niquist said about the theoretical minimum sampling frequency, >but > >in reality, it is much higher. >If you are trying to measure DC with 60Hz noise, 60Hz is exactly what >you DON'T want to sample at as this is almost guaranteed to add a DC >error. Avaraging samples at 120Hz ( or any multiple of 2 x the noise >frequency) would be a lot better as the noise half-cycles would >cancel each other out. Mike, what I did mean, sample with n*60Hz over 5 or more periods. The samples in buffer should have exactly period of 2pi or multiples of period. Then calculate the average. Important are sampling frequency and exact periods. Andrej _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. Share information about yourself, create your own public profile at http://profiles.msn.com. -- http://www.piclist.com hint: PICList Posts must start with ONE topic: "[PIC]:","[SX]:","[AVR]:" =uP ONLY! "[EE]:","[OT]:" =Other "[BUY]:","[AD]:" =Ads

>This could easily ignite the Average vs Median debate that took up so much bandwidth about two years ago. if you are not averse to trying some simple and short code, then you could at least have a go with the exceedingly code and memory space efficient 256 point average from the piclist code archive. I have cut and pasted it below to make the point: www.piclist.com/techref/default.asp?from=/techref/piclist/../microchip/math/&url=../dsp/av-256w-aw.htm Written by Andrew Warren [fastfwd at ix.netcom.com]. ------------ AVEHI EQU [ANY FILE REGISTER] ;Holds the average [0-255]. AVELO EQU [ANY FILE REGISTER] ;Holds the fractional part ;of the average. NEW EQU [ANY FILE REGISTER] ;Holds the new sample. FILTER256: MOVF AVEHI,W SUBWF NEW,W SKPC DECF AVEHI ADDWF AVELO SKPNC INCF AVEHI 7 words, 7 instruction cycles, "NEW" is unchanged. ------------ it works fine. if you want to worry about sampling integer multiples of the noise waveform then fine, but if you are sampling sufficiently fast then the errors introduced by your window will be insignificant. feel free to ignore this code again, and not even bother to try it. Regards, Simon -- http://www.piclist.com hint: PICList Posts must start with ONE topic: "[PIC]:","[SX]:","[AVR]:" =uP ONLY! "[EE]:","[OT]:" =Other "[BUY]:","[AD]:" =Ads

On Wed, 8 Nov 2000, Simon Nield wrote: {Quote hidden}> >This could easily ignite the Average vs Median debate that took up so much bandwidth about two > years ago. > > if you are not averse to trying some simple and short code, then you could at least have a go with > the exceedingly code and memory space efficient 256 point average from the piclist code archive. I > have cut and pasted it below to make the point: > > www.piclist.com/techref/default.asp?from=/techref/piclist/../microchip/math/&url=../dsp/av-256w-aw.htm > Written by Andrew Warren [fastfwd at ix.netcom.com]. > ------------ > AVEHI EQU [ANY FILE REGISTER] ;Holds the average [0-255]. > AVELO EQU [ANY FILE REGISTER] ;Holds the fractional part > ;of the average. > NEW EQU [ANY FILE REGISTER] ;Holds the new sample. > > FILTER256: > > MOVF AVEHI,W > SUBWF NEW,W > SKPC > DECF AVEHI > ADDWF AVELO > SKPNC > INCF AVEHI > > 7 words, 7 instruction cycles, "NEW" is unchanged. > ------------ If you wish to get a weighted average different than this one, then you can try twist.asm. It was written for mixing two sinewaves for creating a DTMF tone, but it turns out that it works well as a low-pass filter. It's similar to what Olin was saying, but this routine has been optimized somewhat. Here are the comments from the header: ;;; -------------------------------------------------- ;;; twist ;;; ;;; The purpose of this routine is to evaluate: ;;; ;;; A*x + B*y ;;; ------------ ;;; A + B ;;; ;;; Such that A+B = 2^N ;;; ;;; This formula is useful for several applications. For example, ;;; DTMF generartors sometimes need to weight the individual sine ;;; waves differently. The term used to describe the difference in ;;; amplitudes is 'twist'. It's the ratio of the amplitudes in dB. ;;; Another application of this formulation is for low-pass filtering. ;;; If 'x' is the current filtered value, and 'y' is the latest ;;; sample, then this equation will be a recursive low-pass filter. ;;; z-transforms may be used to calculate the frequency response, ;;; but that's beyond the scope of this simple description. An intui- ;;; tive way to look at it is as weighted averages. If A is made ;;; large relative to B, then more emphasis is placed on the average. ;;; If B is large then the latest samples are given more weight. The whole thing may be found here: http://www.dattalo.com/technical/software/pic/twist.asm It comes nowhere near Andy's blazing 7-cycles, but it does have the flexibility of shaping the filter. Also Lawrence, as we said a couple of years ago, Median filters deal best with `spikey' noise for otherwise smooth data while linear filters work best for removing noise that is in the sampling bandwidth of the data aquisition subsystem. If there's continuous `high frequency' noise, then you're going to need an analog `anti-aliasing' filter to remove it. Without the analog filter, there's no way to reliably sample low frequency data. The only exception is that you could obtain the DC component. But even this assumes that there's no noise correlated to the sampling frequency (I imagine you could try dithering the sampling frequency to mitigate this problem - but I've never tried it). So imo, the best compromise is to combine the median, linear digital, and analog filters. The analog filter is an anti-aliasing filter. The median filter will remove anamolies that aren't easily filtered (like when a power switch switches). The digital filter removes the unwanted noise in the BW of your sampling. There are always tricks, however. If you're trying to get rid of 60hz noise, then you can use an integrating A/D convertor. What's that? Well, you could implement it several ways. For example you could take 32,64, or 2^N samples for 1/60'th of a second from an A/D convertor, add them togther and then divide the result by 2^N (i.e. shift right N with a possible rounding based on the N+1'th bit). Another method is to use a voltage-to-frequency converter. Essentially, you can run the V/F output into a counter, and control the interval of time for the counting. If the interval is 1/60'th of a second, you'll knock out 60Hz. I've used this technique to achieve over 100 dB of rejection of 60Hz! If you need to process the signal for frequencies other than DC, then you're almost forced to employ a digital filter. As an example, for a Power Quality instrument that I was once working on, we needed to measure 3 phase 60 Hz. I won't go into the details of the design, but I ended up with a 7'th order analog bessel filter and an 8'th order digital FIR filter for each channel. The analog filter knocked out the frequencies beyond the sampling frequency. The Bessel filter, as you may know, has a linear-phase characteristic (e.g. if you send a pulse into a bessel filter the output doesn't ring at all, but otoh, Bessel filter are slow). This was necessary to maintain the signal integrity. However, since a bessel filter has a very slow cutoff, it was necessary to use a digital filter to remove the signals that were beyond the frequencies of interest (approximately 3000 hz - the 50th harmonic of 60Hz). This required about a 10x over-sampling. With the two of these, I was able to get a very accurate representation of the 60Hz signal. (Too bad the company fizzled out before this became a product...) Scott -- http://www.piclist.com hint: PICList Posts must start with ONE topic: "[PIC]:","[SX]:","[AVR]:" =uP ONLY! "[EE]:","[OT]:" =Other "[BUY]:","[AD]:" =Ads

Interesting ideas, Scott! I did a little excercise just now. I decided to race median, average, 1 pole butterworth and 2 pole butterworth filters on some reandomly generated data. My data was made with a spreadsheet, that took the number 3.0, and added a random number between -.125 and +.125, and then added a sinewave between -.125 and +.125. i.e. noisy DC with 60 hz. So I have a nice model of real world noisy data, which we all "know" should measuure 3.0 in an ideal perfect filter. I then recalc'd the spreadsheet so that it generated different random data for a number of iterations. I did 56 runs, racing the methods to see which scored closest to 3.0: Median: was closest 21 times Average was closest 15 times 1 pole butterworth was dead last, losing 56 out of 56 times 2 pole butterworth was closest 20 times. If you add 3 pole butterworth filter to the race it is almost always closest. So for this type of noise, a median filter or a two pole butterworth are about even, and a 3 pole butterworth was best. I know my real world data is also likely to be spikey, which also leads me to conclude a median filter or 2 pole butterworth filter would work best. I'm still trying to wrap my brain around the 7 statement averaging code you posted below.. -- Lawrence {Original Message removed}