New reason for Disk Management ("Shrink") not working

[Jim]

I was experimenting with a hard drive the other day,
a 4TB one, and making the C: drive really large.
This was to counter a complaint from some software
"insufficient space to..." type error.
So anyway, I could see a green stripe of material about
half way out on the disk. I figured, no problem,
if I want to shrink the disk, that material will move
out of the way for me.
It didn't.
I got an error, that said the material could not be moved,
and it was different than the usual material problem. It was
traceable to $BADCLUS, a cluster marked off by the file
system driver. Apparently, the SMART log reported a
series of "UDMA CRC errors". These are errors on the
packets on the SATA cable, causing the packets to be
retransmitted. That's not a conventional data CRC as such.
Yet, the software decided that this constituted a bad cluster,
so the area on the disk was marked off.
There is a option in CHKDSK, to verify bad clusters, but this
in fact, scans the entire partition again, as if doing a /r .
This would check for errors, and bad clusters.
    chkdsk /f /r  C:     # Fix structural errors, read-scan all
clusters to verify they are working
                         # Mark off new bad clusters. Do not verify
any existing bad clusters.
    chkdsk /b     C:     # This seemingly does the same thing, but can
turn bad clusters into good clusters.
                         # If a cluster was "flagged by mistake", this
can undo it.
On a large disk, this can take hours to complete (either command).
When the /b run completed, it reported
    "Removing 1 clusters from the Bad Clusters File."
and that is what I was hoping would happen on the /b run.
I went back to Disk Management, and I still could not shrink the volume.
It was still reporting there was a problem with the same issue as
previously.
On a hunch, I booted a Windows 7 disk, cabled up the affected drive,
and did a regular CHKDSK on the partition in question. I know at this
point, there is nothing wrong with any clusters, so there is no need to
do an hours-long scan yet again.
    chkdsk /f K:         # Windows 7 CHKDSK (of the second disk drive
affected partition)
and finally, after this, the Shrink menu started working again in
Disk Management. Presumably this is related to Windows 7
correcting $BITMAP-type issues. (W10/W11 don't handle $BITMAP
properly on partitions any more, leaving data at rest in an
indeterminate state.) Windows 7 cares about the $BITMAP, and
fixed it up for me.
Disk is now back to normal again. The shrink completed (2TB partition
down
to 200GB) with no trouble at all. When a partition starts small, you make
it huge, then there won't be a problem to make it as small as it was
at first.
Except if there is a bad cluster in there (a cluster flagged as bad
while the file system was running).
            _________________________________________________
           /                                                 \    4096
byte
          X   512   512   512   512   512   512   512   512   X
cluster on
           \              XXX                                /    OS
C: partition
            ------------------------------------------------
            If any sector is bad, the cluster is marked bad.
            The "cluster" is a unit of storage in NTFS.
    Paul

More reason to keep a spare Win7 machine /disk around huh.

What is Linux version of disk-check? In windows it takes hours as stated
above but what about Ubuntu?? What command to use to check this?

I have a 2 TB disk that is slow and showing signs of wear & tear so I
tried to clone it on a new SSD 1TB disk. Acronis states that it is
possible to clone on to a smaller disk as long as the actual data is not
more than the size of the target disk. The source has about 350GB of
data (out of 2TB disk size) but this can't be cloned because the disk
has bad clusters according to Acronis. So how do I correct this in Linux
ubuntu?

x-posted to ubuntu as well just in case they can respond quickly.

[Carlos E.R.]

...

Post by Jim

More reason to keep a spare Win7 machine /disk around huh.

What is Linux version of disk-check? In windows it takes hours as stated
above but what about Ubuntu?? What command to use to check this?

fsck. You also have badblocks.

Post by Jim
I have a 2 TB disk that is slow and showing signs of wear & tear so I
tried to clone it on a new SSD 1TB disk. Acronis states that it is
possible to clone on to a smaller disk as long as the actual data is not
more than the size of the target disk. The source has about 350GB of
data (out of 2TB disk size) but this can't be cloned because the disk
has bad clusters according to Acronis. So how do I correct this in Linux
ubuntu?

Write to the bad sectors to force them to remap.

Post by Jim
x-posted to ubuntu as well just in case they can respond quickly.

Better start a new thread in any of the linux groups, the Windows folks
will not be very happy.

[Paul]

Post by Jim

More reason to keep a spare Win7 machine /disk around huh.

What is Linux version of disk-check? In windows it takes hours as stated
above but what about Ubuntu?? What command to use to check this?
I have a 2 TB disk that is slow and showing signs of wear & tear so I
tried to clone it on a new SSD 1TB disk. Acronis states that it is
possible to clone on to a smaller disk as long as the actual data is not
more than the size of the target disk. The source has about 350GB of
data (out of 2TB disk size) but this can't be cloned because the disk
has bad clusters according to Acronis. So how do I correct this in Linux
ubuntu?
x-posted to ubuntu as well just in case they can respond quickly.

Unfortunately, you CAN actually clone $BADCLUS, assuming the underlying
sectors do not blow CRC errors on a read. This is one of the problems with
the $BADCLUS concept, is it marks off storage areas, yet during cloning,
few if any utilities handle $BADCLUS properly. Marking off areas of the
new 1TB disk as unusable (when there is nothing wrong with them), is
what would happen if cloning under any normal conditions.

You need *some* kind of utility, that maps what you know about the bad clusters,
to the existing NTFS $MFT file entries, to figure out what got damaged
when the $BADCLUS were mapped out in the first place. You're cloning a
disk with (virtual and physical) damage to one or more files on the disk.
You have to determine which files got hit.

Previously (earlier posting some time ago), I had a disk with four CRC
errors but no $BADCLUS. The file system had not discovered anything
about the files at that point. I was careful to not let Windows see what
was going on, and part of my sequences there were done from the Linux side.
Of the four CRC errors on the hard drive, two errors were in OS files on the disk,
two errors were in white space (so their handling is less of a problem later on).
Zero errors were in my Documents folder.

My first step there, was to acquire replacement files for the two OS ones.
And write those to the disk (in a different location).

The four errors were "reallocated" by the disk drive, by attempting writes
to the sectors. The disk uses a spare, and the disk keeps a table of
which sector is mapped out, and which spare sector is currently taking its
place. The cache DRAM chip on the hard drive, keeps that map while the
disk drive is running. That's because they don't want the disk constantly
rattling while looking up the map off the platter, for each sector that
needs mapping info.

Your disk, it sounds like you have run out of spare sectors on the
disk at the physical level. Using "smartctl" from "smartmontools" package,
can give you some information on physical disk health.

You can use "ddrescue" from "gddrescue" package, to copy a defective 2TB
disk drive, to a new 2TB drive. That will recover all the data which is
readable. Using the "rescue.txt" file from the repeated copy attempts,
gives you a final summary of what sectors could not be copied. Then
you have to do your best on the "new disk" to manage the file content
there. If the errors happen to fall in $BADCLUS, then you won't be removing
$BADCLUS until you've done a CHKDSK /b K: from Windows. And even a Windows 7
installer DVD is sufficient to do that. Maybe you could do that from a
Hirens disc, but I'm not a Hirens user so I don't know what is on that disc.
Some home-made discs that are based on WinPE (Windows Preinstall Environment)
are available out there, which allow some Windows utilities commands to
run, without "having a Windows PC" to do them. They can be run from the
live media supported by the WinPE files. That's why a Win7 installer DVD
works for this -- it's based on WinPE, there is a cmd.exe shell onboard
and CHKDSK is there to be used.

Letting your disk degrade to the point that stuff ends up in #BADCLUS,
that's an avoidable "own-goal". If you're to operate a computer, check
your hardware once in a while, and that will help prevent "a very
complicated repair/restoration recipe" from being needed. At the very least,
you'll be weighing the value of the different paths available to you.
Try and repair a flat tire, by drilling more holes in the flat tire
(sick disks are not going to "work with you" for a successful conclusion).
Additional errors will crop up, if you keep writing to the sick disk.

sudo smartctl -a /dev/sda # What is my disk health ? "Tell me a story"

As a result of that, I would not use Acronis (and it says it's not
going to accept the challenge anyway). I would start with a new 2TB
HDD in hand, and use the gddrescue package to move the data over.
Then, you will do the maths to determine what file(s) are damaged.
Since the new disk will have no CRC errors, every flat tire patch
applied to the new disk is going to work. You could have lost
some user data files (they could have a $BADCLUS hole punched in them).
If some OS files are damaged on a Windows OS partition, there is
DISM and SFC as an option.

sudo apt install gddrescue # perhaps this installs ddrescue in /sbin or /usr/bin ???

sudo ddrescue -f -n /dev/sda /dev/sdb /root/rescue.log # Live media, first pass copy, old to new
# Keep the rescue.log on a USB stick. Don't lose it!

# Examine the LOG file for details. A large log file means
# there are many many CRC errors.

gedit /root/rescue.log # text file, terse format, techno-babble

# Now, the second pass reads the log, and concentrates only on the
# not-yet-captured sectors. After a couple of these runs, there will
# be no further progress and the process stops. "Content with damage" now on /dev/sdb .

sudo ddrescue -d -f -r3 /dev/sda /dev/sdb /root/rescue.log

But this is not a trivial exercise, and I doubt I would have
the stamina to finish one involving this much damage. As it was,
I probably sat looking at a screen for 20 hours, just tuning
the fucking sequence to make this one problem go away :-)
That's what I mean by stamina. You have to be a very determined
individual to finish one of these.

This is a brief overview of the sequence. Have in hand, your
2TB bad drive, a brand new 2TB good drive. I generally recommend
to people they have two empty drives handy for recovery work, as one
drive may contain your "golden" recovered copy, while a second
recovery disk contains the "refined" content based on the "golden" one.
You can attempt to do certain repair commands, in-place on the golden
drive, and do everything on the one drive, but then imagine what
a "do-over" will entail, and what steps you will have to do over again!

/--- No CRC errors on this disk! --\

old ==> "golden" (no longer degrading) ==> "refined output" disk drive

1) ddrescue (gddrescue package) copy bad drive to good drive.
2) chkdsk /b K: on the good disk K: , to remove $BADCLUS
3) chkdsk /f K: to correct $BITMAP on the good disk partition.
4) Somewhere between step 1 and step 4, do the maths to determine
which files got hit. Rather than me write up all sorts of nonsense,
there are already threads for the purpose.

https://superuser.com/questions/247419/how-to-quickly-get-a-list-of-files-that-have-bad-sectors-blocks-clusters-whate

5) Replace the files that got hit, and that you have worked out what
they are. Since you did not mention having a backup, you may not
have copies of all the files in your Documents folder. OS files
can be replaced after a fashion (DISM SFC).

General purpose NTFS recovery software, works on "Data" disks where
only user files are stored and no OSes are present. Dealing with
an OS drive is another matter, and some Windows stuff won't
respond to a Repair Install unless the OS partition is still bootable.
Recovery of such items is a lot harder to do -- it's not impossible,
but it requires gobs of stamina for repeated disk runs. This is one
of the reasons that gddrescue is the first step above, it puts your
goods on a "solid footing", so that any additional commands don't
need to be quite as aggressive (or as obscure).

Paul