Problem with cuting bad bits in new VNR version 7.x and 8.x

[Gregory]

Can you bring back a previous version bad bit removal method from VNR 6.x? In versions 7.x and 8.x it has been changed and it works terribly.

If there is a single bad bit in a sector, the new version still works, but when I have a column with 3 bad bits, everything falls apart. The new version is based on a previously defined structure and VNR has problems with it. The cut bits go to the end of the sector, causing the structure to disintegrate.

In the old version, the cut bits go to the end of the page and I define which bits are to be cut at the end of the sector (structure does not have to be defined) and everything works fine in VNR 6.x.

Unless I'm doing something wrong and it doesn't work properly for me. Maybe someone can tell me what their experience is with cutting out bad bits in the new and old versions?

[arvika]

I have good expirience with new version. It was changed so need new habits. Probably you make some mistake or case is problematic.
But if possible restore old version as second option it won't be bad.

[Gregory]

But if possible restore old version as second option it won't be bad.

Yes, I do it using VNR 6.2 without any problem and I did it several times faster even with cutting bits at the end of the sector than when trying to do it with the new version. As I wrote earlier, I did cases using the new version but with single bits in sectors, so 1 bad bit cut work.

Problematic case have 3 bad bits in 1 byte (bits are in column), after cut it with new version, end of previous sector and begining of next sector falls apart.

The new version should be easier and faster, but it is problematic, making it more difficult and slower.

Maybe the bad bits in FC2279 work differently than in FC1178/79 or maybe I should treat this column with 3 bits as a bad byte and cut out the whole byte - that's an idea, I'll have to check it tomorrow

By the way, the controller is FC2279, I saw, Michał, that you made this controller using FE but you didn't have any bad bits.

ps. Rusolut Youtube tutorial how to cut bad bits is outdated - explanation concerns the old version.

[arvika]

I think explanation of new version was on conference. Ask Boguslaw, as I remember he was there.
Short explanation: define different sectors size according to BB/Bb, remove Bb, than remove BB if need.

[pepe]

btw, instead of defining sector sizes, one could define a mask for data for the whole page and an option wether to cut the bits/bytes from the xor. This could handle xored/non xored SA cases as well.

[Gregory]

Short explanation: define different sectors size according to BB/Bb, remove Bb, than remove BB if need.

I try did this, and it is not possible, sectors that have bad bits have a length in bits, it is impossible to define sectors in bits.

[arvika]

sectors that have bad bits have a length in bits

It is impossible.

[Igor_Loskutov]

Hello Gregory,
you can use approach from version 6.x IF there are additional bits written by controller at the end of codeword which you can cut too in order to align the beginning on next codeword, that was shifted when you removed bad bits before. For example, if there are 3 bad bits in a codeword, and controller wrote 5 additional bits at the end of codeword, you can remove 3 bad bits + 5 additional bits and everything will work fine.

But there are cases when controller does not write additional bits. Cases where there are empty bits at the end of ECC. If there will be some number of bad bits (smaller that number of those empty bits at the end of ECC), controller will not write any additional bits, but will use those empty ones. For example, if there are 4 empty bits at the end of ECC and there are 3 bad bits somewhere before, then in that codeword you will see only one empty bit at the end of ecc, and three previous will be used for the data shifted by existing bad bits.
And in this case, old approach will not work. If you remove 3 bad bits and you have nothing to remove at end of codeword, then what can you do...

Actually we bypassed it by inserting one artificial empty byte and then removing bits from that one, but it was to complicated, so the new method was done.

And now about the new method. In fact the only not very easy moment there is to find the beginnings of each data area (of each codeword) and set according to that the custom page structure. Each codeword (each data area) will begin from integer byte number, so you do not have to set it in bits. After that VNR will place all removed bad bits right before next codeword, and later if the size of codeword is bigger than it should be, you can remove additional empty bytes by bad column remover.

Please check the PDF with presentation on our website, Flash Data Recovery and Digital Forensics summit 2023 section.
https://rusolut.com/conference-2023/

[Gregory]

Hello Gregory,
you can use approach from version 6.x IF there are additional bits written by controller at the end of codeword which you can cut too in order to align the beginning on next codeword, that was shifted when you removed bad bits before. For example, if there are 3 bad bits in a codeword, and controller wrote 5 additional bits at the end of codeword, you can remove 3 bad bits + 5 additional bits and everything will work fine.

But there are cases when controller does not write additional bits. Cases where there are empty bits at the end of ECC. If there will be some number of bad bits (smaller that number of those empty bits at the end of ECC), controller will not write any additional bits, but will use those empty ones. For example, if there are 4 empty bits at the end of ECC and there are 3 bad bits somewhere before, then in that codeword you will see only one empty bit at the end of ecc, and three previous will be used for the data shifted by existing bad bits.
And in this case, old approach will not work. If you remove 3 bad bits and you have nothing to remove at end of codeword, then what can you do...

Actually we bypassed it by inserting one artificial empty byte and then removing bits from that one, but it was to complicated, so the new method was done.

And now about the new method. In fact the only not very easy moment there is to find the beginnings of each data area (of each codeword) and set according to that the custom page structure. Each codeword (each data area) will begin from integer byte number, so you do not have to set it in bits. After that VNR will place all removed bad bits right before next codeword, and later if the size of codeword is bigger than it should be, you can remove additional empty bytes by bad column remover.

Please check the PDF with presentation on our website, Flash Data Recovery and Digital Forensics summit 2023 section.
https://rusolut.com/conference-2023/

I know what you mean, I've encountered both types of bad bits, in the old version I manually inserted empty bytes at the end of sector using offset and cut BB if necessary and it also worked so was not that complicated like new method for me

I understand that the new method does it automatically, but in my case it does it wrong, I see that is insert bytes in wrong position.

I already sent it to support so that you could cut it out for me and we'll see if I'm doing something wrong or if there's something wrong with the automatic insertion of bytes at the end of the sector.

Thanks for answer, I check this pdf.

[pepe]

how about my idea with the mask?
that would be very flexible, perhaps not the fastest.

[Gregory]

I figured it out, it's a bit different: first BCR, then custom structure, then bBCR then again BCR and everything worked out nicely. ECC works. Now new version work.

[Igor_Loskutov]

I figured it out, it's a bit different: first BCR, then custom structure, then bBCR then again BCR and everything worked out nicely. ECC works. Now new version work.

Yes, kinda same like Arvika described. However, I'm not sure why do you need first BCR.

[Igor_Loskutov]

I figured it out, it's a bit different: first BCR, then custom structure, then bBCR then again BCR and everything worked out nicely. ECC works. Now new version work.

Hi Peter
Please explain your idea.

[Gregory]

Yes, kinda same like Arvika described. However, I'm not sure why do you need first BCR.

I just tried different ways, and it worked for me, maybe it's enough to cut it out at the end, but you have to cut out 2 BBs, the one that is in the dump from the very beginning and the one that was created after cutting out the bad bits. I'll check it on Monday.

[Igor_Loskutov]

Yes, kinda same like Arvika described. However, I'm not sure why do you need first BCR.

I just tried different ways, and it worked for me, maybe it's enough to cut it out at the end, but you have to cut out 2 BBs, the one that is in the dump from the very beginning and the one that was created after cutting out the bad bits. I'll check it on Monday.

well yes, if the dump has bad bytes and bad bits too, then it might be convenient to remove bad bytes firstly, then bad bits, and then those newly created additional bytes at the end of codewords.

[pepe]

Please explain your idea.

my idea is that sw should have library of FULL xor patterns, ie SA + ecc areas not zeroed out, and adding a way to define a mask for the xor. Size of mask would be equal to page size. (Current xor lib has separate patterns for xored SA+ecc and non-xored ones, these would be unnecessary by using the mask.
Result = source XOR(Pattern AND Mask), this way it is easy to leave any areas unmodified.
I would add an option to shift the pattern where mask is 0 (prior aplying the mask of course), providing a way to deal with the problem that started this discussion.
I would probably make it so that one could apply 2 masks, one with shifting one with not, so all braindead configs could be handled.
But i am doing too few flash cases to have the widest perspective of algos being used, so this idea should probably have more options to fit all cases.

[Gregory]

I checked and you can cut out 2 bad bytes at the end and it will also work. All in all, it turns out that it wasn't that difficult - there is simply no good example of how to do it, the documentation is theoretical, not practical. I consider the topic closed. Thank you all for your help.