I recently inherited a laptop that was broken by pouring some hot coffee on it. When I dissected it, it was pretty clear that most of it was unrecoverable: the CPU was completely fried, and its thermal paste splashed everywhere on the motherboard. (I wish I took a picture of it that I could share.) There were however a few pieces that looked in a good state. One of those components was a NVMe Solid State Drive (SSD). I decided to take this SSD and recycle it in my own laptop, maybe to join my LVM pool.
When I plugged it in my laptop however the SSD I tried to navigate the filesystem, and it appeared to be working quite slowly. Opening certain files sometimes would hang indefinitely. Upon inspection of the SMART data and the kernel logs, it was clear that the drive was returning plenty of read errors.
Here is a sample of the kernel logs:
$ dmesg
...
[ 860.465707] ata2.00: exception Emask 0x0 SAct 0x8 SErr 0x0 action 0x0
[ 860.465726] ata2.00: irq_stat 0x40000008
[ 860.465733] ata2.00: failed command: READ FPDMA QUEUED
[ 860.465737] ata2.00: cmd 60/08:18:58:c5:28/00:00:00:00:00/40 tag 3 ncq dma 4096 in
[ 860.465737] res 41/40:08:58:c5:28/00:00:00:00:00/00 Emask 0x409 (media error) <F>
[ 860.465750] ata2.00: status: { DRDY ERR }
[ 860.465754] ata2.00: error: { UNC }
[ 860.467010] ata2.00: configured for UDMA/133
[ 860.467046] sd 1:0:0:0: [sda] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[ 860.467054] sd 1:0:0:0: [sda] tag#3 Sense Key : Medium Error [current]
[ 860.467060] sd 1:0:0:0: [sda] tag#3 Add. Sense: Unrecovered read error - auto reallocate failed
[ 860.467066] sd 1:0:0:0: [sda] tag#3 CDB: Read(10) 28 00 00 28 c5 58 00 00 08 00
[ 860.467069] I/O error, dev sda, sector 2671960 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
...
[ 1057.914608] ata2: softreset failed (device not ready)
[ 1057.914623] ata2: hard resetting link
[ 1063.230631] ata2: found unknown device (class 0)
[ 1067.934891] ata2: softreset failed (device not ready)
[ 1067.934911] ata2: hard resetting link
[ 1073.270826] ata2: found unknown device (class 0)
[ 1078.486604] ata2: link is slow to respond, please be patient (ready=0)
[ 1102.970841] ata2: softreset failed (device not ready)
[ 1102.970860] ata2: limiting SATA link speed to 1.5 Gbps
[ 1102.970865] ata2: hard resetting link
[ 1108.034602] ata2: found unknown device (class 0)
[ 1108.194622] ata2: softreset failed (device not ready)
[ 1108.194638] ata2: reset failed, giving up
[ 1108.194642] ata2.00: disable device
[ 1108.194677] ata2: EH complete
[ 1108.194726] sd 1:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=232s
[ 1108.194740] sd 1:0:0:0: [sda] tag#6 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[ 1108.194748] I/O error, dev sda, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
...
These logs show that the SSD was returning errors (exceptions) to the operating system, and also that the SSD would sometimes become so slow to respond that the kernel would attempt to reset it (which didn’t really work, I can tell you).
Here is an excerpt of the SMART data:
$ smartctl -a /dev/sda
...
SMART Attributes Data Structure revision number: 0
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 166 001 006 Pre-fail Always In_the_past 0
5 Retired_Block_Count 0x0032 100 100 036 Old_age Always - 76
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1740
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 2247
100 Total_Erase_Count 0x0032 100 100 000 Old_age Always - 7654272
168 Min_Erase_Count 0x0032 253 096 000 Old_age Always - 0
169 Max_Erase_Count 0x0032 083 083 000 Old_age Always - 181
171 Program_Fail_Count 0x0032 253 253 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 253 253 000 Old_age Always - 0
174 Unexpect_Power_Loss_Ct 0x0030 100 100 000 Old_age Offline - 14
175 Program_Fail_Count_Chip 0x0032 253 253 000 Old_age Always - 0
176 Unused_Rsvd_Blk_Cnt_Tot 0x0032 253 253 000 Old_age Always - 0
177 Wear_Leveling_Count 0x0032 090 090 000 Old_age Always - 116
178 Used_Rsvd_Blk_Cnt_Chip 0x0032 100 100 000 Old_age Always - 399
179 Used_Rsvd_Blk_Cnt_Tot 0x0032 100 100 000 Old_age Always - 2460
180 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 2980
184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 9919
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 10051
188 Command_Timeout 0x0032 253 253 000 Old_age Always - 0
194 Temperature_Celsius 0x0002 038 000 000 Old_age Always - 38 (Min/Max 16/48)
195 Hardware_ECC_Recovered 0x0032 100 085 000 Old_age Always - 715203
196 Reallocated_Event_Count 0x0032 100 100 036 Old_age Always - 76
198 Offline_Uncorrectable 0x0032 253 253 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x0032 253 253 000 Old_age Always - 0
204 Soft_ECC_Correction 0x000e 100 001 000 Old_age Always - 13
212 Phy_Error_Count 0x0032 253 253 000 Old_age Always - 0
234 Unknown_SK_hynix_Attrib 0x0032 100 100 000 Old_age Always - 32297
241 Total_Writes_GB 0x0032 100 100 000 Old_age Always - 3715
242 Total_Reads_GB 0x0032 100 100 000 Old_age Always - 3680
250 Read_Retry_Count 0x0032 096 096 000 Old_age Always - 176835377
...
This table show various attributes for the operational status of the SSD. The
meaning of the numeric values is pretty much vendor-specific, so trying to
understand those number exactly is quite a challenge, but what matters is that
the numbers under the VALUE column are higher than the THRESH (threshold)
column. The WORST column indicates the lowest VALUE that has ever been
observed.
To my surprise, despite all the errors and hangs that the SSD was experiencing,
the SMART values looked pretty good. Sure, there’s a very low WORST value for
Raw_Read_Error_Rate (001, much lower than the threshold 001), and there is
also and indication that this attribute failed in the past, but besides that
everything looked acceptable enough.
Of course the SMART log was recording the read errors as well. Here’s another excerpt from the output:
$ smartctl -a /dev/sda
...
SMART Error Log Version: 1
ATA Error Count: 1875 (device log contains only the most recent five errors)
...
Error 1875 occurred at disk power-on lifetime: 1737 hours (72 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 41 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 08 70 98 31 af 40 40 00:02:32.920 READ FPDMA QUEUED
47 00 01 30 08 00 a0 a0 00:02:32.920 READ LOG DMA EXT
47 00 01 30 00 00 a0 a0 00:02:32.920 READ LOG DMA EXT
47 00 01 00 00 00 a0 a0 00:02:32.920 READ LOG DMA EXT
ef 10 02 00 00 00 a0 a0 00:02:32.920 SET FEATURES [Enable SATA feature]
...
Give the lack of concrete signs of old age or extended damage to the SSD, I wondered if it could be a link problem: maybe I did not insert the drive correctly, or maybe a pin was dirty. But no: upon inspection I did not find any issue, and after carefully reseating the drive, the problem was persisting.
I proceeded to run a SMART self test, here are the results (from most recent to oldest):
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short captive Completed: read failure 90% 1736 5712
# 2 Short offline Completed: read failure 90% 1736 5712
# 3 Extended offline Completed: read failure 90% 1733 50117792
# 4 Extended captive Interrupted (host reset) 90% 1730 -
# 5 Short captive Interrupted (host reset) 90% 1730 -
The first two tests were interrupted by Linux, which tried to reset the device while the tests were running. A self-test (as the name suggests) is completely self contained and does not involve sharing of data between the SSD and the operating system in the process. The fact that the self-test was failing due to bad blocks was therefore a sign that this was not a link error, but that the blocks were really damaged.
I decided therefore to give up on trying to fix the SSD, but I still wanted to
use it. After all, it was working for the most part: as long as you didn’t
access the bad blocks, the SSD would behave fine. So here is my plan: I would
format the SSD and create an ext4 filesystem on it, using mkfs.ext4 -c, which
would scan for and exclude bad blocks so that they wouldn’t be used. The
resulting filesystem would have less storage available than the advertised
capacity of the SSD, but that was an acceptable trade-off for me.
And here is the most interesting part: mkfs.ext4 -c discarded all blocks
before creating the filesystem. After that, it scanned for bad blocks and,
shockingly, it found none!
SMART self-tests also did not report any error:
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 1740 -
# 2 Short offline Completed without error 00% 1738 -
All the read errors, exceptions and the hanging problem that kept appearing before disappeared!
I’m not fully sure how to explain how this happened, but I did some research and the general consensus is that discarding bad blocks won’t recover them. My theory is that, when the coffee was poured on the laptop, a spike of voltage led to incorrect values to be written to a few blocks that were in use at that time, causing uncorrectable discrepancies between the data and the error-correcting-codes of the SSD. Discarding the blocks reset both the data cells and the ECC cells, removing all the inconsistencies.
Do you have a better explanation? Let me know in the comments!
Comments