Hi, i have a FreeNAS storege with 24x Samsung 860 EVO. 23 are configured in the array and 1 as hot spare.

Have shutdown tha NAS for maintenance ad at power on I will see only the hotspare.

on dmesg I see a lot of error as:

(probe14:mpr0:0:10:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 275 terminated ioc 804b loginfo 31111000 scsi 0 state c(probe12:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
xfer 0
(probe14:mpr0:0:10:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe10:mpr0:0:8:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe14:mpr0:0:10:0): CAM status: CCB request completed with an error
(probe14:mpr0:0:10:0): Retrying command

sas controller, expanders and cable seems to be ok, mooving the working disk in other slot works without any problem.

Do you have any ideas about this problem?

thanks.

Do you see the same error when the slots are empty?

What kind of maintenance did you do on the NAS?

_________________
A backup a day keeps DR away.

no, with all slot empty, any error.
Update: i have tried with new disks and works fine in the same slots.

Any maintenance directly on NAS.
Removed from a rack and put in another one without neither open the chassis...

I suspect some fw corruption due to some firmware bug but could not find any references for 860EVO.
850EVO did have one which i started researching a while ago, but could not find any reference to that either :s

pepe

_________________
Adatmentés - Data recovery

I have sent all disks to RMA and seems to be changed on warranty, we are waiting for return shipment...

other 22 disk affected from the problem.

After a routine shutdown, 8 of them don't start.

changed on warranty from samsung. after reinstalled on the system, other 4 from the pool died.

I'm waiting for other 10... now are working but... I think that they will die in some days....

Could we see a SMART report for the oldest working SSDs? Perhaps there is a firmware bug which is triggered after a certain number of power-on hours???

_________________
A backup a day keeps DR away.

smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 860 EVO 1TB
Serial Number: S4X6NJ0N418455E
LU WWN Device Id: 5 002538 e30414c28
Firmware Version: RVT04B6Q
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 28 10:28:50 2020 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 85) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 24214
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 45
177 Wear_Leveling_Count 0x0013 100 100 000 Pre-fail Always - 0
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0
181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0
183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0
187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0032 070 065 000 Old_age Always - 30
195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always - 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 19
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 138978

SMART Error Log Version: 1
No Errors Logged

What type of data are you storing / processing on the nas?

_________________
Data Recovery Services in the UK.
https://www.usbrecovery.co.uk/

My 1TB 860 Evo (fw RVT04B6Q) has done 28416 hours. I'm now afraid to power it down.

_________________
A backup a day keeps DR away.

It’s a freenas with Iscsi storage for vsphere

Are the VM's standard workstations or application servers ? Large SQL databases (or anything transactional) or exchange servers anywhere on there?

_________________
Data Recovery Services in the UK.
https://www.usbrecovery.co.uk/

The first storage died was a DR with 40/50 vm synced every 4 hours.
The second one was a backup repository with a veeam VM with some backup repository and some synced vm

I have a feeling you're just killing these with enterprise level loads on commercial grade nand. If you're going to stick with retail flash then increase redundancy, stagger drive deployment times within pools, add more hot spares and mix up the brands.

If it were me I'd look at a lower number of higher capacity enterprise drives.

_________________
Data Recovery Services in the UK.
https://www.usbrecovery.co.uk/