USB Datastore (cont.)

Finally! I got my second SATA drive up and running and partitioned and visible to ESXi. I created a 15G datastore on the unused partition of the disk. It’s 15G because I have a 16G USB stick which formats to just over 15G. The cunning plan is to dd the datastore to the USB stick to fool the hypervisor.

But, first things first. At the moment I have two USB sticks plugged in: the one used to boot ESXi and one newly FAT32 formatted. From ESXi’s busybox I can see the one I boot from, but not the other.

An extract from fdisk -l shows my boot “disk”:

Disk /dev/disks/mpx.vmhba33:C0:T0:L0: 1988 MB, 1988100096 bytes
64 heads, 32 sectors/track, 1896 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot      Start         End      Blocks  Id System
/dev/disks/mpx.vmhba33:C0:T0:L0p1             5       900    917504    5  Extended
/dev/disks/mpx.vmhba33:C0:T0:L0p4   *         1         4      4080    4  FAT16 <32M
/dev/disks/mpx.vmhba33:C0:T0:L0p5             5       254    255984    6  FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p6           255       504    255984    6  FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p7           505       614    112624   fc  VMKcore
/dev/disks/mpx.vmhba33:C0:T0:L0p8           615       900    292848    6  FAT16

Partition table entries are not in disk order

But I should also be able to see mpx.vmhba32.

This is certainly recognised by ESXi when booting as there are multiple references to it in /var/log/messages. There’s a lot of interesting looking usb messages in the log file but the one I see which is causing my current problem I believe is the following:

Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu3:5590)<6>usb passthrough enabled; all eligible devices will be unclaimed by kernel drivers except for ESXi boot device vmhba33
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)<6>usb-storage 2-1.2:1.0: unclaiming vmhba32
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)usb storage warning (0 throttled) on unknown (SCSI cmd unknown): usb_stor_stop_transport called
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)WARNING: LinScsiLLD: scsi_remove_host: Removing Host Adapter vmhba32
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)ScsiPath: 4100: DeletePath : adapter=vmhba32, channel=0, target=0, lun=0
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)WARNING: NMP: nmpUnclaimPath: Physical path "vmhba32:C0:T0:L0" is the last path to NMP device "Unregistered Device". The device has been unregistered.

So I guess my next mission is to find out how to disable the unclaiming. In the meantime I might use Linux to copy the datastore to the USB stick and see what happens then.

I have a cunning plan

But I need to get a local disk attached to my ESXi server first which will require a purchase. Also, I need to make some time. The plan is as follows, so if anyone reads this they might be able to try it out. First of all, get a disk, the smaller the better, and partition it to the same size as a USB key, the bigger the better. Or a LUN. Not having created a VMFS filesystem before, I’m not sure if it will work on a partition or whether it swallows the whole disk. Anyway, ideally the USB key should be the same size as the disk.

Create a datastore on the locally attached disk. Use “dd” to copy that disk to a file and then dd the file onto the USB stick. That *should* (might) create a VMFS filesystem on the USB key. Basically that is how ESXi gets installed on the USB key albeit with VFAT filesystems and a partition table which gives you Hypervisor1-4.

Once that’s done, ESXi may or may not be able to recognise it. And if it does, will we be able to do anything with it?

That’s my theory (and some experiments to be done).

Datastore on USB stick

As promised I have been looking at this a bit more. I think I was a bit naive before…perhaps I should have believed everyone, including VMware 🙂

I have built a 4.1 ESXi hypervisor on a second USB stick and have been looking more closely at it. Logging on to the console a running “mount” shows the following:
~ # mount
none on /dev type devfs (defaults,rw,suid,dev,exec,async,atime,diratime,loud)
none on /proc type procfs (defaults)
none on /vmfs/volumes type vcfs (defaults)
none on /tmp type visorfs (2,192,01777,tmp)
updatestg on /tmp/stage type visorfs (0,750,01777,updatestg)
visorfs on /var/lib/vmware/hostd/stats type visorfs (63,63,0755,hostdstats)
~ #

Looking at it another way with “df” reveals:

~ # df -h
Filesystem                Size      Used Available Use% Mounted on
visorfs                   1.3G    257.8M      1.0G  20% /
vfat                    249.7M     87.8M    161.9M  35% /vmfs/volumes/ff060de6-cecc88e5-4d14-8726d7ed0132
vfat                    249.7M      4.0k    249.7M   0% /vmfs/volumes/65092bef-de8a06b5-22db-2bbbc32dc3d2
vfat                    285.9M    135.5M    150.4M  47% /vmfs/volumes/3c3693e8-f77a642a-1910-5c6bdcb26d3a
~ #

And finally, a thid way:

~ # fdisk -l /dev/disks/mpx.vmhba33\:C0\:T0\:L0

Disk /dev/disks/mpx.vmhba33:C0:T0:L0: 16.0 GB, 16062087168 bytes
64 heads, 32 sectors/track, 15318 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/disks/mpx.vmhba33:C0:T0:L0p1 5 900 917504 5 Extended
/dev/disks/mpx.vmhba33:C0:T0:L0p4 * 1 4 4080 4 FAT16 <32M
/dev/disks/mpx.vmhba33:C0:T0:L0p5 5 254 255984 6 FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p6 255 504 255984 6 FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p7 505 614 112624 fc VMKcore
/dev/disks/mpx.vmhba33:C0:T0:L0p8 615 900 292848 6 FAT16

Partition table entries are not in disk order
~ #
Clearly the filesystems are really FAT16 with the exception of one VMKcore filesystem. The fact that they are mounted on something called /vmfs is neither here nor there.

The thing to try is to run vmkfstools on another USB device and try to create a vmfs3 datastore. This has always failed in the past with some specious error. Unfortunately I can’t try it now because I have run out of USB devices. (I have a 1G stick but ESXi won’t recognise it).

The other thing to try is to create a datastore on another local disk (don’t have one) and see what changes. Or there’s the QNAP so I might try NFS.

I don’t hold out much hope at the moment. Even if a vmfs filesystem could be created on a USB stick, you’d still need to fool ESXi into thinking it was another type of storage. Tricky. If it can’t be tricked, a kernel module would likely be necessary, or an alteration to an existing kernel module which is vmware’s territory and they seem to have already made the decision not to do this.

Progress, sort of

The best progress is that my QNAP is now working, fixed by a firmware upgrade at QNAP’s behest. This surprised me as I was convinced the problem was in MacOS, as I believe was everyone else. It is now running on 3.3.1 Build 0720T.

The upgrade itself was a bit hairy. The first time it bombed out with an error at 28%. The second time it got “stuck” at 10% for several minutes, then proceeded into the 20s and announced it was finished. The subsequent reboot took ages to finish and I was beginning to think it was broken. A reminder that patience is required in this game. Anyway, Time Machine is now backing up to it which is good.

Also managed to resurrect my Centos build on my lab machine. It’s actually a Xen build with Centos as Dom-0. For some reason I described it before as Centos with KVM. This is what happens, I reckon if you work too late. Of course, that means I still have to try the KVM hypervisor, which people have been telling me is the more popular of the two.

At least that’s two things off the immediate list so I can get back to concentrating (I use the term loosely) on seeing if I can get a VMFS datastore installed on a USB key to use with ESXi.

To Do List

This is just a variant on “Where was I” (below). To which someone replied “something about virtualisation”. Ha Ha. BTW, does everyone’s blog get as much spam as mine? Guess so.

So, the list is getting bigger, not smaller:

  • Fix the QNAP. QNAP suggest downloading latest firmware but I’m sure the bug is with Apple.
  • Fix my Centos install. Can’t boot it (again!)
  • Try out the Xen hypervisor I compiled on Ubuntu.
  • Install OpenSolaris on a free partition.
  • Install more guests in KVM on Centos.
  • Get USB datastores to work.

I’m going to make the latter the priority as that is arguably the most useful. If I get it to work before the 17th when I am at a Vmware round table I get to brag about it to my peers. 🙂

Sidetracked by Time Machine

Excuse me while I make some notes on trying to get the QNAP to work with Time Machine (Leopard 10.5.8).

I have followed the QNAP instructions to enable Time Machine, including entering a password for the TimeMachine user. When I select the QNAP volume in Time Machine Preferences, after preparing for the backup, it comes back with:

Time Machine Error

The network backup volume could not be mounted
because there was a problem with the network
username or password.

Open System Preferences and choose Time Machine
then re-select the Time Machine backup destimation
to enter the correct username and password.

Well, maybe I typed the password wrong so I tried choosing the disk again. This time getting the error:

Keychain error -25299 occurred while creating a System Keychain entry for the username “TimeMachine” and URL “afp://TimeMachine@HOME-QNAS001.local/TMBackup”.

Note that this has to be fixed by removing the TimeMachine entry from the login keychain in the following file:

/Library/Keychains/System.keychain

which takes some finding. You have to add this as a keychain to the default user keychain that Keychain Access opens. Once that is removed I am back to square one error.

I’ve just been round the loop again using a basic password for TimeMachine on the QNAP. I can feel a support call coming on.

Update:

Sent an email to QNAP but thought of something else to try.

First of all turned on connection logging on the QNAP. After one attempt I noticed AFP connection logging is not enabled by default so have to turn that on explicitly.

I can connect to the TMBackup volume using the finder by entering the username and password manually. I can see the disk on my Mac, connected as TimeMachine and a successful AFP login in the QNAP log.

BUT using the same password in Time Machine fails with the connection error and gives an AFP login fail in the QNAP log. Hence Time Machine is mangling the password I give it somehow.