Disk Recovery now complete – vmfs mount on ubuntu

freebsd-iconSince I (re)started on the 24th of November 2012, the progress of recovering my drive went from fast to slow. More on my recovery.

What happend with my Maxtor drive was that it stopped working when it tried accessing defective blocks. A restart of the drive was required to continue the work.

Using recoverdisk on FreeBSD eventually saved the data.

Though for some reason, which I don’t know, I had multiple entries telling the software to copy the same block multiple times, a quick sample of 3 bytes showed 2 of them was going to be copied approx 500 times each, and the last one was going to be copied approx 1200 times, I don’t doubt there were plenty of these entries. The log file that recoverdisk used was > 400 megs in size. I used the…

cat logfile | sort -u > newlogfile

…command to sort all the unique entries, which ended up in giving me a 3.3 meg log file.

This was more manageable, and after 2 hours of intensive work, I finished my recovery lacking only 17kB of a 1TB disk.

When recoverdisk was finished, I could use the image file to write back to disk. I didn’t want to use the original image in case something went wrong. Recoverdisk image output can be used with “dd”.

The easiest for me was to write the image to disk, rather than making a copy of the file to a disk with enough space. That was just in case mounting the vmfs filesystem failed, and I would have to install the cloned disk into the original hardware, and boot from there.

Since the original disk had vmware ESXi installed I needed to access the vmfs filesystem.

For this I mounted it on Ubuntu, after installing vmfs-tools

sudo apt-get install vmfs-tools

The disk was formatted with default settings during the ESXi installation, and Ubuntu could see the many partitions. Most of the partitions are FAT formated, but not the one that contains the virtual machine. The disk was seen as “sdd” and after some struggling finding the correct partition I found it was sdd3.

sudo vmfs-fuse /dev/sdd3 /mnt

Accessing the mountpoint had to be done by root, but copying the virtual machine which was stored in it’s named dir in the root of the mountpoint was quite easy.

In general it was an easy task but took a very long time.

And since I had plenty of articles I forgot about, I was quite happy getting everything back.

Now you can see that I’ve gotten my articles back that to when I installed WP in 2011.

 

While we’re waiting. Recovering a disk.

As some of you might know, my old server harddrive ran into a series of problems. And no I don’t have a backup.

It all started with that sometimes the server, which is a small ITX board with Atom processor running VMWare ESXi, just stopped responding. So a hard reboot was required. I found out that accessing a few places on the guest servers OS, made this happen (hence why my backups failed).

I disregarded it until it died completely, I couldn’t boot the guest OS.

I should mention that the harddrive is a Seagate 1TB SATA2 drive.

I found out that the harddrive has bad blocks. And when these bad blocks were accessed, the harddrive would just stop responding, a power cycle (unplug hdd power and reinsert) would bring it back again. I tried doing a unix dd, which is a cloning of the harddrive, Windows users might relate to Ghost or ImageCast. The dd command didn’t work of course as the drive would stop responding when accessing the bad blocks.

I mentioned this to some of my associates at my weekly retro evening, which PHK and UJ both said.. hey, you should try recoverdisk from FreeBSD. So I did.

recoverdisk is similar to dd-rescue, basically a resumeable dd command.

3 weeks after starting, and unplugging/replugging power to the drive, restarting the command stopping the script, repeat. I was close to an end.. I got to 100% woohoo !! or not I found out at one time I’ve rebooted the computer and forgot to mount the target disk. There were two things to do, start over or try and grab the data that was separated, it was stored on the local disk anyway. The later seemed like a tedious task, so I went with a do-over. At the time of writing I’m around 87% done. Luckily for me it’s much faster this time as I’m using the drives on a SATA controller, where before it was on a USB to SATA adapter.

 

Back to the superb tool, which is part of the FreeBSD operating system: recoverdisk.

Recoverdisk is a command line utility and is quite simple to use actually.

What recoverdisk do, is that given the right parameters dumps the disk and keeps a logfile of what it’s done. It takes all the good blocks first, if it hits a bad block, it tries to copy that. If it’s unsuccessful it skips the bad until next pass. Where it adjusts the block size into smaller chunks, and repeats until it finishes, or until stopped.

In my scenario, which might be a rare one. When it hit the bad block, and skipped it, the disk would be rendered unresponsive. I had to stop the command, unplug power and reinsert the power again, restart the command to continue. I had to do that a lot, and by every pass it would want smaller block sizes, making the dumping even slower. This mostly happend because I left it working over night, now I keep an eye on it and stop it when I no longer can keep it under surveilance.

 

How to use

Part of the man pages

SYNOPSIS
     recoverdisk [-b bigsize] [-r readlist] [-s interval] [-w writelist]
                 source [destination]

DESCRIPTION
     The recoverdisk utility reads data from the source file until all blocks
     could be successfully read.  If destination was specified all data is
     being written to that file.  It starts reading in multiples of the sector
     size.  Whenever a block fails, it is put to the end of the working queue
     and will be read again, possibly with a smaller read size.

     By default it uses block sizes of roughly 1 MB, 32kB, and the native sec-
     tor size (usually 512 bytes).  These figures are adjusted slightly, for
     devices whose sectorsize is not a power of 2, e.g., audio CDs with a sec-
     tor size of 2352 bytes.

     The options are as follows:

     -b bigsize
             The size of reads attempted first.  The middle pass is roughly
             the logarithmic average of the bigsize and the sectorsize.

     -r readlist
             Read the list of blocks and block sizes to read from the speci-
             fied file.

     -s interval
             How often we should update the writelist file while things go OK.
             The default is 60 and the unit is "progress messages" so if
             things go well, this is the same as once per minute.

     -w writelist
             Write the list of remaining blocks to read to the specified file
             if recoverdisk is aborted via SIGINT.

     The -r and -w options can be specified together.  Especially, they can
     point to the same file, which will be updated on abort.

In it’s most simple form it’s like dd, read disk device and write to an image file mounted to the system (usually my system drives aren’t as big as my data drives).

recoverdisk /dev/da0 /my/mounted/disk/image.dd

This command is like using dd.

For the inital run you should write to a log file

recoverdisk -w /home/tomse/recoverlist /dev/da0 /my/mounted/disk/image.dd

This will write the log list into the specified dir. And for large drives that can take a long time to recover, my suggestion is to break out of it right away, this way we can set it to read the log file when it passes it’s first pass.

recoverdisk -r /home/tomse/recoverlist -w /home/tomse/recoverlist /dev/da0 /my/mounted/disk/image.dd

-r = read, -w = write. quite logical and very easy to use.

this is is.. let the drive finish until it reaches 100% you can break it at any time, and run the command again (with the -r/-w parameters) to resume it’s work.

When using the USB adapter in the first run I got a message they I should change the blocksize to 131072 using the -b parameter, fortunately for me I haven’t been needing to do so yet on the current setup.

recoverdisk -b 131072 -r /home/tomse/recoverlist -w /home/tomse/recoverlist /dev/da0 /my/mounted/disk/image.dd

By default, the block size is 1MB

So a big thank you to Poul-Henning Kamp for implementing this into FreeBSD, to  Ulrich Sporlein for doing some fixes and most importantly writing the man pages.

And to my associates at our weekly retro gathering.

Keeping your FreeBSD install up-to-date

Until very recently, I thought FreeBSD lacked the ability to upgrade binary packages, the same way as the debian repository. I was wrong of course, a friend of mine told me to use pkg_upgrade that comes with bsdadminscripts.

I’ll show you how to upgrade not only packages but also the system, and keeping it up to date.
The tools I’ll use are already installed, else I’ll go through the installation.

Ports

If you aren’t familiar with ports, I’d suggest you do so. Though I’ll make a very short description of it here.
Ports contains information of where source code can be found, dependencies and BSD patches if needed.
to install a port simply go to it’s directory and enter
# make install clean

i.e. we’ll need bsdadminscripts later
# cd /usr/ports/sysutils/bsdadminscripts && make install clean

To keep ports up to date there are a couple of methods, like cvsup, manual download and portsnap, there probably are a few more, but we’ll be focusing on portsnap.

If you make a fresh install, I can recommend not to install ports, but wait till later, I’ll tell you why in a few.

If you don’t have ports installed, and you need a fresh ports tree:
# portsnap fetch extract
This will download the latest database and extract it to it’s default location; /usr/ports

using portsnap

To update your already installed ports, you can run
# portsnap fetch update

this will download the diff from your existing ports tree and the newest, and merge them in your local ports tree.
Now, if you haven’t done so in a long time, or if you are updating your tree that was installed from the CD/DVD it can take a very long time, the download itself doesn’t take that long, depending on your connection of course, the most recent ports tree in copressed format is around 60-65mb in size. What takes a long time is the extraction of alot of small files. So as I said ealier, it’s better not to install ports from the installation media and just start with a fresh download after installation.

Compiled from ports

If you have an application compiled from ports and want to update it, make sure your ports tree is up to date before continuing.

using portmaster

Install portmaster by doing pkg_add -r portmaster

as an example I’ll upgrade sit which is a nice web based incident tracker, assuming the requirements for this package has been met on a previous install, all you need to do is.
# portmaster www/sit

This will fetch the new version, compile it if required, deinstall the old package and install the new version. Since this is a web app it might require to update the database schema, this is described on the packages homepage.

to recompile all packages installed
# portmaster -a

Will do that for you, it runs through all the packages and prompt you for config settings if required, before compiling the first package, this way you don’t need to accept configs all the time through the upgrade, as you normally would do with portupgrade.

Packages

You might know about the package system now, precompiled binary packages, ready to download and install.
# pkg_add -r bash

Using -r will download the package from one of the mirrors available, rather than take the input as a file located on the local harddrive.
Unless you actively do anything, you’ll always download the release packages, after some time these can get old, bugs and security issues found etc, so it’s recommended you upgrade these to the newest release.

Since I’m using C-Shell for my root user, I’ll modify /root/.cshrc, I’ll add the link to where the stable (newest) repository is stored.

So edit your /root/.cshrc with your favorite editor, and add:
setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/

Preferably with the rest setenv lines.
If you use another major version make sure you change the 8 to the major version you use. also change i386 to amd64 if you are running 64bit, or other supported architectures.

A full list of mirrors can be found here so pick on closest to you : http://www.freebsd.org/doc/handbook/mirrors-ftp.html

If you have problems using the FTP protocol, try using HTTP instead.

Save your .cshrc file, and reload it
# source ~/.cshrc

Installing a package as root now, will get the latest binary.

To uprade using the latest binaries you’ll need bsdadminscripts which can be found in ports in
/usr/ports/sysutils/bsdadminscripts (cd to the dir and type in “make install clean”)
or use pkg_add
pkg_add -r bsdadminscripts

This will give you some nice tools, amongst pkg_upgrade which we are going to use in a few.

Upgrading an application with latest precompiled package we can now use
pkg_upgrade mc

This will upgrade midnight commander, a norton commander clone, to latest binary, using the precompiled packages leaves out custom installs, so make not of that if you’ve previously compiled a package, i.e. apache with mod_dav.

pkg_upgrade -nav
This will give a list of all (-a) packages, not (-n) upgrading it, with verbose (-v) information.
removing the -n will upgrade all packages while giving you a nice verbose output on screen.

System

Updating your system is pretty easy to do, again FreeBSD delivers some nice tools out of the box to help you.

Using freebsd-update

freebsd-update fetch install
This will update your current version with latest patches.

If you want to upgrade from an old version i.e. 7.4 to 8.2 (which at current time of writing is the latest version).
freebsd-update -r 8.2-RELEASE upgrade

This will upgrade with binaries to the chosen version with latest patches. Though you might need to answer some questions about config files, for example if you edited /etc/hosts and others too. It’s usually not a big problem and you probably know what to do if you have an advanced setup.

When upgrading, it installs the kernel first, later it’ll need world, so it prompts you to reboot after the first step, and asks you to run freebsd-update install after a successful reboot.

Read the man pages for more info, when you run freebsd-update and portsnap without parameters you’ll get a small output of what you can do, if you ever are in doubt.

I suggest you try this first before doing it in a production environment. Don’t blame me if your system crashes or otherwise gives you problems, thats what the manual is for.

Good luck and happy updating.

Credits to Uffe Jakobsen

A feature I love with the csh / tcsh

The foreach command which I never remember how to do, is a very powerful tool in the csh, which is missed in bash.

For those who don’t know it, foreach can be used in the command line interface (cli) in csh, unlike some other shells.

With foreach you can do some very simple directory work without having to first make a shell script.

Below is a very simple way of using the command.

First lets prepare something to work with, create a test dir that can be easily be removed afterwards

# mkdir test && cd test

Prepare some directories. these can also be files if you want to work with this.

# mkdir 1 2 3 4 5 6 7 8 9

The few commands to populate each directory from 1-9 with a file called test

# foreach file (*)
# cd $file && touch test && cd ..
# end

Now go and take a look in each directory, and you’ll see a file called test.