Searching \ for '[PIC] Compressing data from data logger' in subject line. ()
Make payments with PayPal - it's fast, free and secure! Help us get a faster server
FAQ page: www.piclist.com/techref/microchip/memory.htm?key=data
Search entire site for: 'Compressing data from data logger'.

Exact match. Not showing close matches.
PICList Thread
'[PIC] Compressing data from data logger'
2007\12\28@150630 by Rob Susmilch

picon face
Hello everyone,

First some background. I'm going to be building a simple stand alone data
logger for personal use (and the fun of creating it of course) since most stand
alone loggers are hundreds of dollars. I will most likely be using a pic18f4553
IIRC, and either interfacing to a ADC chip or using the on-board A/D. I also am
going to be interfacing to SD card since from what I've read they are easy to
do.

My question is, and I've searched high and low, how to implement compression on
this little bugger. Granted I suppose getting a jumbo SD card could negate any
compression done, but I may perhaps go to a simple serial EEPROM instead. All
the algorithms I've seen on-line seem VERY computationally expensive (which I'm
not sure if the PIC COULD do, or if it could would kill battery life never
sleeping) or memory hogs. I think I saw somewhere a person mentioned LZW
compression may work, with the difference from the current signal and last
signal. I have seen a few patents where they mention some adaptive data
compression using in audio but not much else. I understand that any voltage
signal is a wave form so an audio compression would probably be best? I would
like to go in a loss-less form. I would be measuring mostly things like temp,
humidity, sunlight, etc. A weather station would be it's primary focus, but I'd
like the option of using it for other things (or building a separate one more
tailored.). Does anyone know of a compression scheme that would work for such
things that it is worth the trouble to implement and extract some saving in
space? Maybe I don't need to worry at all? Also I am going to give
Sourceboost's C a try so C code references are OK. Sorry for the lengthy post.

Rob

2007\12\28@152130 by Bob Axtell

face picon face
Rob Susmilch wrote:
{Quote hidden}

Every compression algorithm I saw needed some scratch memory, but aside
from that the PIC can perform
compression. I'd use RLE if possible, it doesn't need much scratch area.
RLE means Run Length Encoding;
while not as powerful as LZW, it certainly is easy to use and is sparing
of scratch area. RLE is lossless as well.

--Bob A

For a scratch pad, I'd use an I2C RAMTRON at 32K x 8; there are NO write
waits.

2007\12\28@152610 by David VanHorn

picon face
Run length encoding might work, if the data is not changing a lot.
Basically, two bytes per entry, the first says "this many" and the
second says "of this value".
Commonly used to compress black and white images.

Compression costs time though, always.

2007\12\28@165151 by Nicola Perotto

picon face
Rob,
1) the result of data compression depends from algorithm and DATA
2) you are converting a STREAM of data (any buffering  valid for a non
streaming data compression algorithm is a big problem for a PIC: size,
time required to fill, etc)
3) lossless algorithm  are computing intensive (ram buffer, cpu clocks)
4) SD cards are cheap and relatively simpler!

           Nic


Rob Susmilch wrote:
{Quote hidden}

2007\12\28@165233 by Rob Susmilch

picon face
Well, some scratch area is fine, I am not opposed to so memory used for
compression. However it seems most algorythms want LOTS of memory, and I've
seen some that say they will only work on 16 or 32 bit architectures. The only
problem I see with RLE is perhaps the signal slowly changes, but just enough
that there aren't any runs to encode... perhaps then I'd need better filtering
if it's due to noise... but I haven't actually gotten to the hardware yet,
still on another short project. Just doing my research. Does anyone have
experiance with data logging? I couldn't find any raw data examples to play
with. I was looking into FLAC which is supposedly VERY good at lossless audio
encoding, but the code is a nightmare. I suppose with enough memory and time
maybe it could be done, but I bet the PIC would be awake and drawing current
the whole time killing battery life. Any other ideas besides RLE and LZW?

Rob

(PS thanks for the scratch pad suggestion)
{Quote hidden}

2007\12\28@170650 by David VanHorn

picon face
You can use RLE intermittently.
Take a chunk of data, RLE compress. If Len(result) > len(origina),
then don't compress.
Flag the block as compressed or not as appropriate, and move on.

2007\12\28@171246 by Jinx

face picon face
Hi Rob

> A weather station would be its primary focus

A recent Silicon Chip basic weather-monitoring project

http://www.siliconchip.com.au/cms/A_109187/article.html

uses an F88 and two 24C256 EEPROMs

The problem with any run of (probably) unique ADC readings
is that an attempt at compression will likely increase the data
size. Although this depends on the ADC resolution you choose
to use and the sampling interval. For example, if you were
stripping off the temperature decimal, you might have a run of
10x 20C for an hour. That would RLE fairly well. At other times,
eg dawn, you might find that each reading for an hour or two is
different. Compression would not work well to reduce data for
that period

As you've chosen a large PIC, you could go parallel (SD, NV,
static etc) rather than I2C or serial. The 4553 also has 32k of
16-bit memory (or 64k x 8 if you choose) and that's a good start

> I've seen on-line seem VERY computationally expensive (which
> I'm not sure if the PIC COULD do, or if it could would kill
> battery life never sleeping)

The thing to do is run the 4553 at high speed and have it woken
by a low-current clock source. That way the 4553 can get any
computations over very quickly. A slow clock, eg 32kHz, is not
always the answer to saving battery power. Tasks done at 32MHz
will be over in 1/1000th of the time at 32kHz, but 32kHz does
not use anywhere near 1/1000th of the power

You'll have to crunch the numbers based on sampling interval
and resolution

2007\12\28@172626 by Steve Wormley

flavicon
face
If you're not looking for true 'compression' you could just look into
encoding your data differently. One thing that might make sense for
something like temperature is just to record the differences. Then,
say, use 4 bits signed for these values instead of a full 8 bits, make
some value represent 'out of range' as a marker that the full reading
follows.You end up doing a fair amount of bit shifting but it can save
a fair bit of space. Obviously, this taken to the extreme is a
compression algorithm. Also make sure you're not wasting space on
storing 'useless' data, if the range of your wind direction is just
the 4 cardinal directions, don't waste 8 bits(or more) to store it.

-Steve

2007\12\28@173442 by Nicola Perotto

picon face
LZW needs to build a dictionary, you can decide how big but you need
some KB of RAM.
RLE can works for 8 bit but to be effective need highly correlated data,
if you have 4 channel probabily they are different one to other.
Otherside you can save the first sample and than the delta. In this case
maybe the data is well correlated. Of course the ADC must be well filtered!
With RLE you need a small buffer, large enough to accomodate n samples
where n is the maximum run_length count.

But... before wasting time thinking about data compression have you a
clear idea on sampling frequency, number of channels, required time
between data downloading, etc???
         Nic


Rob Susmilch wrote:
{Quote hidden}

2007\12\28@175354 by David VanHorn

picon face
Sort of a variant on RLE,  a header that says "the following N bytes
all are within the range of XXXyyy, where Xis are the MSB, unchanging
for the block, and Y are the variable part, then encode the Y part as
nybbles or whatever makes sense.  You could get almost 2/1 compression
if it's 8 bit data where the top 4 bits changes infrequently.  adjust
to suit your system of course.

2007\12\28@181715 by Jinx

face picon face
> something like temperature is just to record the differences.

> the 4 cardinal directions, don't waste 8 bits(or more) to store it

Thinking in bits rather than bytes is probably a better scheme (cf
CVSD) in this application. You aren't likely to experience a huge
difference, eg temperature, between one reading and the next. It
could happen, perhaps make those events special cases

I'd question whether 12-bits is useable or useful for weather. Say
you had a select range, maybe -10C to 40C and 10mV / degree.
50C range -> 500mV, x10 amp -> 5V. Difference of 2C -> 200mV
-> 0x28 (approx, for a 10-bit ADC) change in ADC result, which
fits in a nybble. V could be scaled to accomodate 4C change in a
nybble, or similarly for a 12-bit throw away a couple of LSb. I
think you could afford to without noticeably degrading final results

2007\12\28@193328 by wouter van ooijen

face picon face
> But I
> haven't actually gotten to the hardware yet, still on another
> short project. Just doing my research.

Then your first research should be what kind of data you will have to
store! You can't senisbly think about compression when you don't know
the characteristics of your data. Compression always relies on
redundancy in the data, so you need to know what redundancy is there.
Removing noise is always a good idea, because noise is not redundant but
also not usefull.

Wouter van Ooijen

-- -------------------------------------------
Van Ooijen Technische Informatica: http://www.voti.nl
consultancy, development, PICmicro products
docent Hogeschool van Utrecht: http://www.voti.nl/hvu



2007\12\28@193432 by Xiaofan Chen

face picon face
On Dec 29, 2007 4:06 AM, Rob Susmilch <spam_OUTtwosouls2getherTakeThisOuTspamyahoo.com> wrote:
> Hello everyone,
>
> First some background. I'm going to be building a simple stand alone data
> logger for personal use (and the fun of creating it of course) since most stand
> alone loggers are hundreds of dollars. I will most likely be using a pic18f4553
> IIRC, and either interfacing to a ADC chip or using the on-board A/D. I also am
> going to be interfacing to SD card since from what I've read they are easy to
> do.
>
> My question is, and I've searched high and low, how to implement compression on
> this little bugger.

I know this is not what you want, but it might be interesting to you and others.

WinZip for PIC18:
www.htsoft.com/forum/all/showflat.php/Cat/0/Number/19042/an/0/page/0#19042
www.htsoft.com/forum/all/showflat.php/Cat/0/Number/19027/Main/19027/#Post19027
http://forum.microchip.com/tm.asp?m=115148

Xiaofan

2007\12\28@193540 by Xiaofan Chen

face picon face
On Dec 29, 2007 8:34 AM, Xiaofan Chen <.....xiaofancKILLspamspam@spam@gmail.com> wrote:
> I know this is not what you want, but it might be interesting to you and others.
>
> WinZip for PIC18:
> www.htsoft.com/forum/all/showflat.php/Cat/0/Number/19042/an/0/page/0#19042
> www.htsoft.com/forum/all/showflat.php/Cat/0/Number/19027/Main/19027/#Post19027
> http://forum.microchip.com/tm.asp?m=115148
>

The last link should be:
http://forum.microchip.com/tm.aspx?m=115148.

2007\12\28@195743 by Jinx

face picon face
I wrote

> 0x28 (approx, for a 10-bit ADC) change in ADC result, which
> fits in a nybble

Oh no it doesn't (unless you count 1/2 a 16-bit byte as a nybble)

But you hopefully got my point. Organise the data so you save
enough, preferably the minimum, information needed to preserve
it, whether that's raw readings or the differences

2007\12\28@213954 by Herbert Graf

flavicon
face

On Fri, 2007-12-28 at 12:06 -0800, Rob Susmilch wrote:
> Hello everyone,
>
> First some background. I'm going to be building a simple stand alone data
> logger for personal use (and the fun of creating it of course) since most stand
> alone loggers are hundreds of dollars. I will most likely be using a pic18f4553
> IIRC, and either interfacing to a ADC chip or using the on-board A/D. I also am
> going to be interfacing to SD card since from what I've read they are easy to
> do.
>
> My question is, and I've searched high and low, how to implement compression on
> this little bugger. Granted I suppose getting a jumbo SD card could negate any
> compression done, but I may perhaps go to a simple serial EEPROM instead. All

Frankly, once you get the init done, an SD card in SPI mode is not
really much more complicated to use then a serial EEPROM. I would forget
about going the compression route and just use an SD card, they are so
cheap these days.

TTYL

2007\12\28@215646 by Jim Korman

flavicon
face
Rob Susmilch wrote:
> Well, some scratch area is fine, I am not opposed to so memory used for
> compression. However it seems most algorythms want LOTS of memory, and I've
> seen some that say they will only work on 16 or 32 bit architectures. The only
> problem I see with RLE is perhaps the signal slowly changes, but just enough
> that there aren't any runs to encode... perhaps then I'd need better filtering
> if it's due to noise... but I haven't actually gotten to the hardware yet,
> still on another short project. Just doing my research. Does anyone have
> experiance with data logging? I couldn't find any raw data examples to play
> with. I was looking into FLAC which is supposedly VERY good at lossless audio
> encoding, but the code is a nightmare. I suppose with enough memory and time
> maybe it could be done, but I bet the PIC would be awake and drawing current
> the whole time killing battery life. Any other ideas besides RLE and LZW?
>
> Rob
>
>  
Rob, One of the guys in the local area has his weather data online here

Graphical display
http://members.cox.net/mcsittel/currwx.htm

and as text (2 days worth)
http://members.cox.net/mcsittel/downld02.txt

Also a lot of weather data are at about 1 minute samples.
Wind direction and speed (from Davis devices for example) are
(10 sec samples) time averaged over various time ranges because surface
winds are fairly noisy.

Many data  (i.e. pressure) can have large constants subtracted out
before storage.

As others have mentioned, check the ranges

-40 to 40 C to the nearest 0.1C takes 10 bits, but 0.5C is about 1F degree
and you can encode that range in 8 bits.

Jim

2007\12\28@234709 by Vasile Surducan

face picon face
On 12/28/07, Herbert Graf <mailinglist4spamKILLspamfarcite.net> wrote:
>
> On Fri, 2007-12-28 at 12:06 -0800, Rob Susmilch wrote:
> > Hello everyone,
> >
> > First some background. I'm going to be building a simple stand alone data
> > logger for personal use (and the fun of creating it of course) since most stand
> > alone loggers are hundreds of dollars. I will most likely be using a pic18f4553
> > IIRC, and either interfacing to a ADC chip or using the on-board A/D. I also am
> > going to be interfacing to SD card since from what I've read they are easy to
> > do.
> >
> > My question is, and I've searched high and low, how to implement compression on
> > this little bugger. Granted I suppose getting a jumbo SD card could negate any
> > compression done, but I may perhaps go to a simple serial EEPROM instead. All
>
> Frankly, once you get the init done, an SD card in SPI mode is not
> really much more complicated to use then a serial EEPROM.

If you don't implement the FAT. Else...

I would forget
> about going the compression route and just use an SD card, they are so
> cheap these days.

I'm totally agree with this.

2007\12\29@030512 by Apptech

face
flavicon
face
>> Well, some scratch area is fine, I am not opposed to so
>> memory used for
>> compression. However it seems most algorithms want LOTS
>> of memory, and I've
>> seen some that say they will only work on 16 or 32 bit
>> architectures. The only
>> problem I see with RLE is perhaps the signal slowly
>> changes, but just enough
>> that there aren't any runs to encode... perhaps then I'd
>> need better filtering
>> if it's due to noise...

1.    Some sort of Hamming coding perhaps.
If it's usually different but slow moving then it must be
"hunting" over a limited range and there will be patterns to
it.
eg if 0 is down 0.1 degree and 1 is up 0.21 degree then an
eg 010010100111101 pattern may well be typical.
If you CARE about such fine changes then you could have a
lookup table of sequences.

2.    If changes are usually only by a few units you could
compress the step into a few bits.
eg nibble coding.
0000    Use extended code
0001    +0.1C
1001    -0.1C
...
0110    +0.6C
0111    +0.7C
1110    -0.6C
1111    No change.

Giving a +0.6C to -0.7C range in half a byte at the cost of
4 more bits when the step is greater than this value.

Or, if most steps are under about 0.3C you could use 3 bit
coding and not align at byte boundaries. Fun will ensue :-).

3.    If there are long periods without change then logging
the time when changes occur may be more data economical.

4.    ADPCM

5.    SD memory is getting so very very very cheap that
using it if possible sounds very attractive.


       Russell


2007\12\29@154606 by Herbert Graf

flavicon
face

On Fri, 2007-12-28 at 20:10 -0800, Vasile Surducan wrote:
{Quote hidden}

You don't have to do any FAT stuff. For example, my carmon project
doesn't even use the FAT:

http://repatch.dyndns.org:8383/pic_stuff/carmon

Once you have the actual sector that the file you want to write to
starts, you don't have to worry anymore about FS stuff if you set things
up correctly.

In my case, I simply write a file to the flash card (in my case a CF
card, I've done the same with an SD card, I can post that code if anyone
is interested) with a computer that fills the whole card. The file
contains all zeros.

The PIC simply scans the file, finds the first sector that doesn't have
zeros, and continues writing.

On the PC side I simply scan the file for all non zero data.

The only limitation to this method is you can't write zero to the file,
usually not an issue (I just write all ASCII).

The "init" portion consists of reading the partition table (one sector,
fixed location), reading the boot sector of the first partition (to find
the location of the root directory) and then scanning the root directory
for your file.

It all sounds more complicated then it really is. The actually "writing"
code simply keeps track of how close to the end of the card I am, and
what sector I want to write to next, barely more complicated then
writing to an EEPROM.

TTYL

2007\12\29@161815 by Spehro Pefhany

picon face
At 03:45 PM 12/29/2007, you wrote:

{Quote hidden}

Hi, Herbert:-

I'd be interested in seeing the SD card code. Will it work with the
newer cards with larger sector sizes?

Best regards,

Spehro Pefhany --"it's the network..."            "The Journey is the reward"
speffspamspam_OUTinterlog.com             Info for manufacturers: http://www.trexon.com
Embedded software/hardware/analog  Info for designers:  http://www.speff.com



2007\12\29@172540 by William \Chops\ Westfield

face picon face

On Dec 28, 2007, at 8:10 PM, Vasile Surducan wrote:

>> an SD card in SPI mode is not
>> really much more complicated to use then a serial EEPROM.
>
> If you don't implement the FAT. Else...

So it's been mentioned a couple times that a "simple"
solution is to first create a large empty file on a
FAT-capable system, and then just overwrite the file
on your micro (avoiding some of the complexity of FAT.)

Is there any published micro-side code that does this?
I've seen full FAT implementations; but I'm looking for
the simplified version that finds an existing file,
gets its length, and either checks for contiguity or
handles it appropriately...

(It's a bit mind-boggling that a 2G micro-SD card isn't
much bigger than a 256byte dip8 eeprom, and I've bought
a number of 64M microSD cards via eBay for about $1 each.)

BillW

2007\12\29@173656 by Herbert Graf

flavicon
face
part 1 1063 bytes content-type:text/plain (decoded 7bit)


On Sat, 2007-12-29 at 16:03 -0500, Spehro Pefhany wrote:
> Hi, Herbert:-
>
> I'd be interested in seeing the SD card code.

I've attached the main source code. Note this was written for the
Luminary micro, so you can't just drop it into a PIC directly, that
said, most of it is pretty high level so adapting it to pretty much any
platform would be very simple.

> Will it work with the
> newer cards with larger sector sizes?

Is that what they are doing with those HC cards?

This code assumes a sector size of 512bytes, I tested it on 512MB and
1GB cards (a 512MB card would last me around 8 months, so I never worked
much on using bigger cards).

There are several sections of code that assume 512byte sectors, so those
would have to be modified with the data from the CSD (I assume that's
where the card tells the host it's sector size?). I read several
parameters from the CSD, so adding sector size would be trivial there,
modifying the other portions of code should also be rather simple to do.

TTYL


part 2 5157 bytes content-type:application/zip; name=carmon.c.zip (decode)

part 3 35 bytes content-type:text/plain; charset="us-ascii"
(decoded 7bit)

More... (looser matching)
- Last day of these posts
- In 2007 , 2008 only
- Today
- New search...