you've seen this.. yes?

http://forum.soft-center.ru/viewtopic.php?t=3507

not sure what all the softcenter stuff means
but might have some clues...

_________________
Expert in data recovery, RAID, VMware, Linux, Tape, Forensics, Cleanroom...
http://www.alandatarecovery.com

This simply means that there is a layout (combination of pre-defined sector structure and ECC mostly) which incorporates the XOR pattern automatically.

The other stuff relates to removal of mixes and assembly parameters.

Isn't it possible to determine the XOR pattern by looking at the encrypted data alone?

If it is just output = data XOR pattern, it should not really be that bad?
Given that most common character in data is zero, I suppose a sample about 10,000 times length of the pattern should be large enough.

Or is the scheme significantly more complicated, or is that already a common knowledge?

It's not encrypted, it's XORed !

[data on the chip] = [user data] XOR [XOR pattern]
[data on the chip] XOR [XOR pattern] = [user data]
[data on the chip] XOR [user data] = [XOR pattern]

You have to know two of them to find the last one...

_________________
Lemmy

Generally, XOR is a variant of encryption.

If it is that simple like

[user data] XOR [pattern] = [data on chip]

and if the length of [pattern] is small and known, then it is trivial to determine the [pattern] from [data on chip] alone, without knowing [user data].
If the length is not known, the task becomes more difficult, but probably something can still be done.

Let's take for example the [data on chip] ABCDEFABC and the length of pattern is 3.
You split data in blocks of three and write it like

ABC
DEF
ABC

then the most frequent character in each column gives you the respecitve character in [pattern], in this example it would be

ABC

This uses the fact that the most frequent character on a hard drive (assuming the data is not encrypted) is zero.

To give you an idea, I have attached the first 512 bytes from the XOR pattern of a PS2251 Phison controller (not the same XOR that is present on the OPs case)

The total XOR file is 1,097,728 bytes long (1MB) and is periodically XOR'ed against the data.

The XOR is a simple pattern against the data. The problem is two fold

1 - work out the pattern
2 - apply the correct pattern to the data

By knowing the disk format it has been possible to determine a basic pattern for the 0x400 long data strings. Each 0x2000 byte block uses the same string.

However, different blocks apply the string 2, 4, or 6 etc bytes from the start. I have determined a XOR string length of about 0x470 bytes, and an inverted value of the same.

I have not yet found a way to apply the XOR to sectors with unknown values, only known values. Thus, I can find the correct XOR string for the start of a photo, as they always start 0xff 0xd8 0xff and 0x2000 bytes of data are then correct. I have not yet determined if the service information also has the same XOR pattern applied

hddguy,

To derive an 1MB long pattern, you should be fine having about 1 GB data from the chip. This will most likely require some additional work, though. 10GB should be enough to get straight to the result. You should look at the data, not at the pattern.

That still assumes the data is continuous stream and the pattern is applied repeatedly with no interruptions.

The XOR pattern will be applied to all data, even where there is '00' in the data, this makes it extremely difficult to even attempt to find the pattern.

Most Flash media is factory-formatted to FAT(32) which uses a sequential 'counter' within the partition table, as this sequence consists of lots of consistency maybe there is potential to find pattern, but without knowing how the data looks before and after XOR identification of the pattern is near impossible.

Besides the XOR for most devices now is already known at Soft Center, Acelab and NRTEAM. As one of these commercial products is required for assembly anyway, I see no reason to try to find pattern yourself.

The trouble I have is PC-3000 flash does not seem to know the pattern on my chip

Sequences of '00' infact show you the pattern - you just need to recognise them!

If someone will ever need the algorithm to find the pattern from raw data, for a new controller or something, I have already put it up there. It is quite simple, actually, and will work if the task is within the listed constraints. As there is no need for that at the moment, I suppose the case is closed.

I have PC3000 Flash also, it is very limited in comparison to Flash Extractor. Limited XOR support, cannot implement a 4 way Join by Byte, some cases cant identify ECC correct, extremely difficult to read from LGA memory. Other cases require specific mixes removed, like Cross Bank on Silicon Motion, and Block rotate on some Phison etc, again PC3000 Flash cannot do this.

I have attached one 512-byte sector that has been XORed using a randomly made up 512-byte pattern. I challenge you to solve it and produce the original data / XOR pattern.

_________________
You don't have to backup all of your files, just the ones you want to keep.

hddguy, I suppose you have read that one needs a sample size from 1,000 to 10,000 times a pattern length?

ah, err, it was @ drc.

What if this is not available? If you recieve a 4GB Phison Flash drive with a XOR pattern of 1MB and only has 4MB of data on the stick how can you then decipher the XOR? Does this case then become unrecoverable?

Realistically we do not get samples and practice runs from clients, just broken media.

I stand by what I said earlier in the thread:



If you have a method of finding the XOR pattern, that is impressive, but in my opinion all you have to work with is a load of HEX which, without any reference to what the actual data or XOR pattern should be, effectively garbage.

4GB flash, of which I assume you have a 4GB dump.
It has 4 MB data on it, and the rest I assume are zeros?
That should be trivial, because

[zeros] XOR [pattern] = [pattern].

if you run the same statsitical computation as described above, it will give you the [pattern] perfectly.

it would be more difficult if you have 4GB flash filled to the brim with data. Still, that gives us 4K times pattern length, so should not be a problem.

The problem actually becomes worse if the length of the pattern is not known in advance. Not knowing the pattern length is the most significant difficulty I see.

If you have a new flash chip, I think sectors may read as zeros through the USB but will infact be unwritten locations. Often, these on the flash memory itself will be 0xFF. Only when the sector is actually written (with zeros) will the sector be allocated and the XOR value added.

This it is necessary to look for the zero based sectors from patterns - not easy or an exact science - but possible. Knowledge of FAT32 can help a lot.

Very well, the same 512-byte XOR pattern has now been applied to 5120000 bytes of data.

_________________
You don't have to backup all of your files, just the ones you want to keep.

mscotgrove,

Then, just discard anything that looks like a sector filled with 0xFF when analysing the dump. Something to which pattern was never applied is not helpful in searching for the pattern.

In which case, if only 4MB was ever written to the flash before it failed, you cannot derive the pattern from that - too small a sample to work with.