> Lawrence -
> Median and related nonlinear filters really are the right
> way
> to deal with long-tailed distribution noise problems. But, as
> with everything, you really need to understand your signal. If
> a spike lasts for more than N samples in a 2N+1 point median
> filter, it will not remove it. I presume from your comments
> that the spikes are longer than this. If you can decrease the
> sampling frequency to the point where you meet this criteria,
> it will solve your problem.
So true. I am aware that my first code probably sampled at a couple
of kilohertz, reeeee-diculously high for a signal that changes every
few seconds. Knowing your noise is the first rule of noise immunity,
and for this project anything faster than 1 hz is probably noise.
> I would imagine that a 5 point
> median filter would work beautifully with the right timing.
> Even a three point one (very easy to code) might well be
> sufficient. The point here is that you really have to look at
> your data to know what's going on.
Really! I was assuming that a lot of data points would be required
to make any sense out of a noisy signal. Friday I had given up in
disgust. Now after a weekendsd R&R, I'm figuring on sampling at
about 1/2 HZ, collecting 5 samples, and maybe putting a bit of
hysteresis in there as well.
{Quote hidden}> Yet another cheap and dirty way to track the median is to
> build a delta modulator. At each moment in time you have an
> estimate of the value (this is the filter output) which you
> compare to the input. If the input is greater, you add delta to
> your estimate. If it is smaller, you subtract delta. The
> tradeoff is to make delta big enough to track changes
> reasonably quickly, yet make it small enough to effectively
> filter your noise. Why does this track the median? Because the
> estimate will tend to go to the point where 50% of the time it
> is going up, and 50% of the time it is going down. That is the
> median. If it the estimate was some place else, say at a point
> where you're going up 80% of the time, it would tend to move
> up. Of course, this is not a true median filter, but it is VERY
> easy to code and may work well for your application.
I stumbled onto this as an effective noise canceling algorithm by
sheer luck, and it works great! I used a lookup table to find a
"weight" for each of 18 possible input states, then added the
wieght to the previous setpoint. The weights were chosen so that 100
or so good readings were required to change the output. Positive
weights were gioven to numbers that should drive the output toward
"on" and negative wqeights toward "off" . Instead of a purely
mathematical relationship between input and output, I was able to
give a weight of zero to a few signals at the extreme end of my range
that were always associated with noise, and zero for a few values
that were always ambiguous in the middle.
Best Regards,
Lawrence Lile