Though I agree with the old technology, new market name thought…the mainframes could run multiple OSes back in the 60s…now it’s vmware. Sun E10K had dozens of CPUs way back when, now there are multi core multi processors from Intel/AMD (implemented differently). Databases wrote logs for lazy writes, now this is journaling file system (and concept behind many storage appliances to improve write performance)…and so on.

However, all these technologies have a better implementation which allows them to become so popular in the mainstream computer market.

I know, from past experience, that MP3 format audio just plain doesn’t compress well. I seriously doubt we would have seen “as advertised” compression ratios.

Again – DeDuplication is simply new-fangled marketing-speak for compression. The basic mechanism for compressing data isn’t anything that PKZIP hasn’t been doing for 20 years.

I’m hoping no-one is trying to push this as something to be done to production data? I wouldn’t run any kind of compression against production data for the same reason that I wouldn’t rely on disk-compression. It takes processing cycles/time, and puts you further at risk for corruption.

De-duplication hardware is expensive, it’s another working part to break, and it’s something else an already beleaguered SAN admin has to master.

This works well for us because we have a lot of spare cpu cycles on our VMware boxes at night when we run backups. Once we move to production we are planning of doing away with tape completely for VMWare and simply replicating the data to a 2nd Avamar appliance at our remote site.

i also understand that the DeDupe appliance uses variable block-sizes, whereas i believe the older compression engines look at a fixed block-size, say 2114bytes, and look for redundancies within.

Now add that to a key-based “de-duplication” engine and you have space-savings and encryption in the same barrel. Because once the data is de-duplicated it’s mostly encrypted as it is.

Ok you have my attention.

People are getting significant storage savings from this. At my previous employer we were doing backup to disk and running out of room. Adding a dedupe appliance allowed us to store 20 to 30x as much data in the same space.

I’ve seen it done before – but as with any incremental backup the longer you go between full stores the more tedious and time-consuming the restore/recovery procedure is, right?

I mean in ‘backup-class’ we learned that in the perfect world you’d have the time and storage to do full backups every day. You start from the ideal and work your way backwards.

What you’re describing as deduplication is EXACTLY how compression alghorhythms works. It’s almost akin to the difference between bitmap graphics and JPEG compressed graphics. With a bitmap you’re storing information on every pixel. With JPEG compression you’re storing infomration on one pixel, then counting how many similar pixels are around. The difference in the resulting picture quality is soly measured by how the compression algorhythm measures “similar.”

I wonder if we’re truly getting into that “nothing new under the sun” state of technology. Everyone’s new breakthrough seems to be a re-branding / re-marketing of the old technology. I don’t think you can put a faster processor and more cache / RAM in a computer and call it a new computer. You can call it an upgrade certainly, but not new.