All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Forensic Image Hash
PostPosted: June 25th, 2018, 21:39 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
I have had some inquiries about adding an option to HDDSuperClone for forensic imaging with a hash. My question is about the hash. Do the sectors need to be read and hashed sequentially, or can they be hashed out of order and still come up with the same hash?

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 25th, 2018, 21:44 
Offline

Joined: October 16th, 2013, 13:21
Posts: 713
Location: Brazil
It depends on the method for hashing/calculating the crc you use.


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 25th, 2018, 22:31 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
rogfanther wrote:
It depends on the method for hashing/calculating the crc you use.

How about the standard MD5 and SHA1?

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 25th, 2018, 22:34 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Spildit wrote:
rogfanther wrote:
It depends on the method for hashing/calculating the crc you use.


It shouldn't "depend" ...

If you have a MD5 hash of a binary file, let's say a jpg, that MD5 hash for the exact same file should be exactly the same no matter what tool you do use to calculate the MD5 hash, has long as the "hash" is MD5 and not any other like SHA-1, CRC-32, etc ...

If you compute one MD5 (or any other sort of hash) for your "image" file the produced "hash" has to be the same for ANY tool that does compute the same sort of hash for the same file !!!

Try it out.

I would say you will have to do a sector by sector implementation but you can test it out !!!

Grab a very small HDD like 1 or 2 GB and image it. Now create one hash using your own method. Now get something on the net that can calculate "hash". Use that software and compare the result to the hash that you have generated.

They will have to be the same.

I think you missed my question. I am asking if it is possible to hash data out of order and get the same hash. It is obvious that any program should come up with the same hash (same hash type) for any given data file.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 26th, 2018, 9:02 
Offline

Joined: October 16th, 2013, 13:21
Posts: 713
Location: Brazil
maximus wrote:
rogfanther wrote:
It depends on the method for hashing/calculating the crc you use.

How about the standard MD5 and SHA1?



I do not believe these will result in the same codes for out-of-order groups of bytes.
After all, if you change the position of two 512-byte groups ( even if you change position of just two bytes ) , the files will be different, so by design their MD5, SHA, etc, should be different also.

I may be wrong here, but from what I have read and seen, for forensic work you would need a way of checksumming that is strong and easy to explain to people. Think about trying to explain to a room with a bunch of lawyers that only knows hard disks as "those things that go inside a computer and get destroyed by viruses" about out-of-order checksums, polinomials and the like.

I don´t know how Encase/FTK does it, but I would venture that a checksum is calculed after the image is taken, and appended to it, so that it travels with the file.


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 26th, 2018, 17:21 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
+1 to calculating the hash after imaging, as that would be the ideal way. But the inquiries were about calculating the hash on the fly. I have never written any code to do any popular hashes, so I had to ask. But my instinct was that for any valuable hash, it needed to be done in perfect sequence.

I am not even sure if I will even add that feature, too many things about it that I don't like. I am just exploring it because it was asked by about three different people within a few months or so.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 26th, 2018, 19:16 
Offline

Joined: October 16th, 2013, 13:21
Posts: 713
Location: Brazil
My modest opinion is that the people who asked for it are misplacing the function with the tool. Do not want to offend anyone, more so if they are from this forum, but there are tools for that kind of thing, and they do not need to mix the functions of hddsuperclone in it.

About doing it on the fly. How would we define "on the fly" ? Hash any sector read ? Hash all of the sectors read until now ? And, before the cloning finishes, for what purpose ? What good would hashing/crc´ing something do in the middle of the cloning , and how would that be verifiable ?

The idea for CRCs/Hashes would for to assure the image has not been tampered with after being acquired. For that there are write blockers and Trusted labs/places that do the forensic imaging. After the imaging, a hash of the image can be generated and sent to the involved, so anyone that goes to work in that image later can verify if it has been changed or not. And even so, try to explain to people who do not understand, or worse, people that know how to put some tricky questions, how the process worked and if was possible to such and such evidence having been removed/inserted....


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 26th, 2018, 19:36 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
rogfanther wrote:
My modest opinion is that the people who asked for it are misplacing the function with the tool. Do not want to offend anyone, more so if they are from this forum, but there are tools for that kind of thing, and they do not need to mix the functions of hddsuperclone in it.

About doing it on the fly. How would we define "on the fly" ? Hash any sector read ? Hash all of the sectors read until now ? And, before the cloning finishes, for what purpose ? What good would hashing/crc´ing something do in the middle of the cloning , and how would that be verifiable ?

The idea for CRCs/Hashes would for to assure the image has not been tampered with after being acquired. For that there are write blockers and Trusted labs/places that do the forensic imaging. After the imaging, a hash of the image can be generated and sent to the involved, so anyone that goes to work in that image later can verify if it has been changed or not. And even so, try to explain to people who do not understand, or worse, people that know how to put some tricky questions, how the process worked and if was possible to such and such evidence having been removed/inserted....

That about sums up how I feel about it. I am not even sure why it was a requested feature. Your response helps me think that I just won't do it (which was my initial thought). It is not at all what the software is intended for. But I figured I would at least run the idea by others before condemning it.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 26th, 2018, 22:09 
Offline
User avatar

Joined: February 9th, 2009, 16:13
Posts: 2520
Location: Ontario, Canada
DeepSpar calculates on the fly, to the best of my knowledge. That is likely the reason for the question.

_________________
Luke
Recovery Force Data Recovery


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 10:25 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
lcoughey wrote:
DeepSpar calculates on the fly, to the best of my knowledge. That is likely the reason for the question.

Does anyone have any real information on this (like if they have used it), other than what can be found on DeepSpar's website? http://www.deepspar.com/forensics-ds-disk-imager.html

My thought would be that to hash on the fly, it would need to be done in a linear motion with no going back. It would require its own algorithm that was not the best for data recovery. Read in larger chunks for speed, when an error is encountered read the chunk sector by sector. Bad sectors are marked as such, and are never tried again. If the errors were too extensive, the forensic mode would need to be aborted and then a data recovery algorithm used, and hashed when completed.

It's not like I couldn't do it, it is a matter if I want to, and if it would even be that much of a selling point. But then we come back to the fact that three separate people asked me about it as a pro feature they would like to see...

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 11:56 
Offline

Joined: October 16th, 2013, 13:21
Posts: 713
Location: Brazil
Well, maybe you could get this people to explain exactly why/how they imagine this feature to be implemented/work. Maybe they are ok with the very long extra time it will take, or the uncomplete cloning. Or maybe they are thinking about something very different from what we are all thinking...


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 13:47 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
rogfanther wrote:
Well, maybe you could get this people to explain exactly why/how they imagine this feature to be implemented/work. Maybe they are ok with the very long extra time it will take, or the uncomplete cloning. Or maybe they are thinking about something very different from what we are all thinking...

I think the point of hashing on the fly is that it would be much faster. I think the CPU usage for hashing is nothing compared to the time it takes to read the data, and doing it on the fly would not add much time to the cloning. Anyone can hash it afterwards using simple Linux commands, but the whole image must be read, and it takes time to read a large amount of data, considering you just took the time to process that data in the first place when imaging.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 14:20 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Spildit wrote:
The problem is that you wouldn't be able to skip sectors or hash on reverse imaging, etc ...

Yes, but I think that when someone wants to forensically image a drive, they are most likely assuming it is in good shape. It would not be hard to handle a few bad sectors. But in the event that imaging starts to go bad, it would need to be stopped and switched to a recovery mode, which would no longer provide a running hash. The plus side would be that at least it would resumable and not have to start over. But it would need to be hashed after it is done.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 15:11 
Offline

Joined: October 16th, 2013, 13:21
Posts: 713
Location: Brazil
But ... even if on the fly it is faster, hashing what ? One sector a time ? One megabyte ? And do what with the hash(es) ? Store them alongside or just present to the user ?

Okay, you can keep running the calculations for the hash and keep feeding the bytes read to the polynomial calculation of your specific hash method. It would probably avoid some time at the end of the cloning to read the image and calculate the hash. But, would it enough to justify the increased complexity ?


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 15:25 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
rogfanther wrote:
Okay, you can keep running the calculations for the hash and keep feeding the bytes read to the polynomial calculation of your specific hash method. It would probably avoid some time at the end of the cloning to read the image and calculate the hash. But, would it enough to justify the increased complexity ?

THAT is the magic question :mrgreen:

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 15:31 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Let me rephrase the magic question:
Would adding this feature to the pro version in any way help to sell HDDSuperClone? THAT is the real magic question that will drive whether or not I do it. Would the return be worth the effort? :?

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 15:45 
Offline

Joined: October 16th, 2013, 13:21
Posts: 713
Location: Brazil
That would be better to the three people who asked for it to answer.

Just an opinion, but pc3k doesn´t have it. Maybe they considered and then decided to focus on the main purpose of fixing hard drives.

Maybe add an option to calculate a md5sum of the created image after done cloning ?


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 16:07 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
rogfanther wrote:
Maybe add an option to calculate a md5sum of the created image after done cloning ?

Yeah, something like:
Type one of the following commands to hash the image on your own using built in Linux commands so I don't have to deal with it:
md5sum -b imagefilename
sha1sum -b imagefilename
:mrgreen:

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: June 27th, 2018, 16:12 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
rogfanther wrote:
That would be better to the three people who asked for it to answer.

Maybe I will get ambitious enough to go back through my emails and find them, and ask a few questions. Maybe...

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: Forensic Image Hash
PostPosted: May 10th, 2019, 18:17 
Offline

Joined: November 22nd, 2017, 21:47
Posts: 309
Location: France
Perhaps an option to calculate the checksum for each copied / recovered sector and store it in a text file would be more relevant to the original purpose of HDDSuperClone. With a complementary feature which would load the text file and verify each sector against the image / clone, making it easier to pinpoint the exact locations of modified sectors, if any. This should be a lot easier to implement than calculating a global checksum, as was requested. But the resulting text files would be huge : with one MD5 checksum per line and no other data, that would still make a 232GB file, even more with a more secure method... :shock: A cluster level calculation would make it a tad more manageable, but the program would need to determine the cluster size, which can be different for each partition on the same device. Or there could be some kind of on-the-fly compression.
As for getting requests outside the original scope of your "flagship" tool, I read somewhere that any software being developped for a sufficient amount of time always ends up being able to send e-mail messages, so be prepared ! :)


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group