All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 15 posts ] 
Author Message
 Post subject: 23x Samsung 860 EVO dead after poweroff
PostPosted: May 17th, 2022, 17:26 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
Hi, i have a FreeNAS storege with 24x Samsung 860 EVO. 23 are configured in the array and 1 as hot spare.

Have shutdown tha NAS for maintenance ad at power on I will see only the hotspare.

on dmesg I see a lot of error as:

(probe14:mpr0:0:10:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 275 terminated ioc 804b loginfo 31111000 scsi 0 state c(probe12:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
xfer 0
(probe14:mpr0:0:10:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe10:mpr0:0:8:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe14:mpr0:0:10:0): CAM status: CCB request completed with an error
(probe14:mpr0:0:10:0): Retrying command

sas controller, expanders and cable seems to be ok, mooving the working disk in other slot works without any problem.

Do you have any ideas about this problem?

thanks.


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: May 18th, 2022, 14:51 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15461
Location: Australia
Do you see the same error when the slots are empty?

What kind of maintenance did you do on the NAS?

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: May 18th, 2022, 14:54 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
fzabkar wrote:
Do you see the same error when the slots are empty?

no, with all slot empty, any error.
Update: i have tried with new disks and works fine in the same slots.


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: May 18th, 2022, 15:31 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
fzabkar wrote:
What kind of maintenance did you do on the NAS?

Any maintenance directly on NAS.
Removed from a rack and put in another one without neither open the chassis...


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: May 18th, 2022, 16:57 
Offline

Joined: October 3rd, 2005, 0:40
Posts: 4311
Location: Hungary
I suspect some fw corruption due to some firmware bug but could not find any references for 860EVO.
850EVO did have one which i started researching a while ago, but could not find any reference to that either :s

pepe

_________________
Adatmentés - Data recovery


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: June 6th, 2022, 4:03 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
I have sent all disks to RMA and seems to be changed on warranty, we are waiting for return shipment...


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 28th, 2023, 13:30 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
other 22 disk affected from the problem.

After a routine shutdown, 8 of them don't start.

changed on warranty from samsung. after reinstalled on the system, other 4 from the pool died.

I'm waiting for other 10... now are working but... I think that they will die in some days....


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 28th, 2023, 17:13 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15461
Location: Australia
Could we see a SMART report for the oldest working SSDs? Perhaps there is a firmware bug which is triggered after a certain number of power-on hours???

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 29th, 2023, 1:47 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 860 EVO 1TB
Serial Number: S4X6NJ0N418455E
LU WWN Device Id: 5 002538 e30414c28
Firmware Version: RVT04B6Q
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 28 10:28:50 2020 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 85) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 24214
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 45
177 Wear_Leveling_Count 0x0013 100 100 000 Pre-fail Always - 0
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0
181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0
183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0
187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0032 070 065 000 Old_age Always - 30
195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always - 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 19
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 138978

SMART Error Log Version: 1
No Errors Logged


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 29th, 2023, 14:16 
Offline

Joined: November 7th, 2020, 5:31
Posts: 1084
Location: The_UK
What type of data are you storing / processing on the nas?

_________________
Data Recovery Services in the UK.
https://www.usbrecovery.co.uk/


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 29th, 2023, 14:18 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15461
Location: Australia
My 1TB 860 Evo (fw RVT04B6Q) has done 28416 hours. I'm now afraid to power it down.

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 29th, 2023, 14:21 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
It’s a freenas with Iscsi storage for vsphere


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 29th, 2023, 14:44 
Offline

Joined: November 7th, 2020, 5:31
Posts: 1084
Location: The_UK
Are the VM's standard workstations or application servers ? Large SQL databases (or anything transactional) or exchange servers anywhere on there?

_________________
Data Recovery Services in the UK.
https://www.usbrecovery.co.uk/


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 29th, 2023, 15:04 
Offline

Joined: May 17th, 2022, 17:10
Posts: 8
Location: Italy
The first storage died was a DR with 40/50 vm synced every 4 hours.
The second one was a backup repository with a veeam VM with some backup repository and some synced vm


Top
 Profile  
 
 Post subject: Re: 23x Samsung 860 EVO dead after poweroff
PostPosted: August 30th, 2023, 3:32 
Offline

Joined: November 7th, 2020, 5:31
Posts: 1084
Location: The_UK
I have a feeling you're just killing these with enterprise level loads on commercial grade nand. If you're going to stick with retail flash then increase redundancy, stagger drive deployment times within pools, add more hot spares and mix up the brands.

If it were me I'd look at a lower number of higher capacity enterprise drives.

_________________
Data Recovery Services in the UK.
https://www.usbrecovery.co.uk/


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 15 posts ] 

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 29 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group