Comments on research!rsc: Backups, heal thyself

I understand the birthday problem, but it could be...

2008-04-15T00:20:00.000-04:00

I understand the birthday problem, but it could be easily solved by moving from sha1 to sha256. This would be less space efficient when you were storing the trees, but that's not going to be the majority of your storage cost anyway. Doing this would make the chance of a collision insignificant for even the largest datasets.

As far as the overhead of using cryptographically strong hashes is concerned - I'm a lot more concerned about data integrity than a slightly slow backup, and it's always possible to move the hash calculation into hardware if it becomes advantageous to do so.

EMC's Centera uses the same concept, though it use...

2008-04-14T21:51:00.000-04:00

EMC's Centera uses the same concept, though it used MD5 hashes.

the 2^-160 is a bit misleading. as the first pape...

2008-04-14T17:50:00.000-04:00

the 2^-160 is a bit misleading. as the first paper points out (search for "birthday") you're going to start getting collisions when you have around 2^80 blocks. that's still a lot of blocks, but *significantly* less than the 2^160 you might infer from what you wrote.

The Git version control system also stores content...

2008-04-14T15:34:00.000-04:00

The Git version control system also stores content as blobs, indexed by SHA-1 hashes.

That's also how the StarTeam Native-II file reposi...

2008-04-14T14:16:00.000-04:00

That's also how the StarTeam Native-II file repository works. Stores full images of each version, compressed and indexed with MD5 hashes. So no reverse-delta penalty for diffing back versions of files.