'Re: Averaging Algorithm'
Paul H. Dietz
| Lawrence -
Median and related nonlinear filters really are the right way
to deal with long-tailed distribution noise problems. But, as with
everything, you really need to understand your signal. If a spike
lasts for more than N samples in a 2N+1 point median filter, it will
not remove it. I presume from your comments that the spikes are
longer than this. If you can decrease the sampling frequency to
the point where you meet this criteria, it will solve your problem.
I would imagine that a 5 point median filter would work beautifully
with the right timing. Even a three point one (very easy to code)
might well be sufficient. The point here is that you really have
to look at your data to know what's going on.
Another post mentioned dropping min and max values, and then
averaging the rest. This is actually closely related to median
filtering. There is an amazing amount of (somewhat irrelevant)
theory behind these filters that comes out of robust estimation
theory. If your curious, check out the Kluwer book by Pitas and
Yet another cheap and dirty way to track the median is to
build a delta modulator. At each moment in time you have an
estimate of the value (this is the filter output) which you
compare to the input. If the input is greater, you add delta
to your estimate. If it is smaller, you subtract delta. The tradeoff
is to make delta big enough to track changes reasonably quickly,
yet make it small enough to effectively filter your noise. Why does
this track the median? Because the estimate will tend to go to the
point where 50% of the time it is going up, and 50% of the time
it is going down. That is the median. If it the estimate was
some place else, say at a point where you're going up 80% of the
time, it would tend to move up. Of course, this is not a true
median filter, but it is VERY easy to code and may work well
for your application. Note that there is no reason why delta
can't be smaller than the resolution of your input. This
might help you choose a good compromise delta...
Anyhow, I hope this helps. It sure beats going postal...
--- Paul Dietz
______________________________ Reply Separator _________________________________
Subject: Re: Averaging Algorithm
Author: MITVMA.MIT.EDU at WDI-INTERNET PICLIST
Date: 9/25/97 2:47 PM
Paul Dietz Wrote:
> There is a much better way! It is called a median filter.
> Basically, this is just like a moving average (or more
> generically, an FIR) filter, except that you take the median of
> the data instead of the average. You do this by sorting the
> data in the window by value, and then outputing the value which
> ends up in the middle.
Alas! I have spent a couple days messing around with median filters.
The first one I wrote pushed large values off the end of the sort
stack, into lala land. It would never climb up. The next one pushed
them off the bottom of the sort stack. Would never climb down.
Finally I got one that read in 32 analog values, sorted them, and
took the median as the setpoint. Code works. BUT the Relay fires
whenever I hit the lamp on my bench. Noise immunity is still out of
reach after a lot of hard work. Oh well, I know how to sort now. I
fear the noise immunity is harder to buy than this.
I see three avenues:
1. Go postal.
2. Try a longer time delay between measurements, until I get around
3. give up on median filters as being too complex, and going back to
Another day, another $0.50 US.
|> Lawrence -
> Median and related nonlinear filters really are the right
> to deal with long-tailed distribution noise problems. But, as
> with everything, you really need to understand your signal. If
> a spike lasts for more than N samples in a 2N+1 point median
> filter, it will not remove it. I presume from your comments
> that the spikes are longer than this. If you can decrease the
> sampling frequency to the point where you meet this criteria,
> it will solve your problem.
So true. I am aware that my first code probably sampled at a couple
of kilohertz, reeeee-diculously high for a signal that changes every
few seconds. Knowing your noise is the first rule of noise immunity,
and for this project anything faster than 1 hz is probably noise.
> I would imagine that a 5 point
> median filter would work beautifully with the right timing.
> Even a three point one (very easy to code) might well be
> sufficient. The point here is that you really have to look at
> your data to know what's going on.
Really! I was assuming that a lot of data points would be required
to make any sense out of a noisy signal. Friday I had given up in
disgust. Now after a weekendsd R&R, I'm figuring on sampling at
about 1/2 HZ, collecting 5 samples, and maybe putting a bit of
hysteresis in there as well.
I stumbled onto this as an effective noise canceling algorithm by
sheer luck, and it works great! I used a lookup table to find a
"weight" for each of 18 possible input states, then added the
wieght to the previous setpoint. The weights were chosen so that 100
or so good readings were required to change the output. Positive
weights were gioven to numbers that should drive the output toward
"on" and negative wqeights toward "off" . Instead of a purely
mathematical relationship between input and output, I was able to
give a weight of zero to a few signals at the extreme end of my range
that were always associated with noise, and zero for a few values
that were always ambiguous in the middle.
More... (looser matching)
- Last day of these posts
- In 1997
, 1998 only
- New search...