Obscenities

Posted on by
Reply

My brain is screaming obscenities because I have spend the last 2 hours trying to delete a datastore from ESXi4. It’s a small 16G one (the one I was using to try and copy to a USB stick) which resides on a 500G internal SATA drive. I have been trying from the vSphere client and from the command line with more and more severe methods culminating in “Reset System Configuration” from the console. Even that didn’t work which only leaves me the option of re-burning my ESXi on USB stick.

The error is as follows:

Error from Delete Datastore

Don’t bother to ask me to try something. I’ve tried it all.

LOSUG September

Posted on by
Reply

I attended the Oracle LOSUG meeting on September 15th to hear a talk from Phil Kirk on Zones and Crossbow.

I also took the opportunity to meet Alasdair Lumsden (who has set up openindiana).

I scribbled down a few notes to help jog my memory.

HISTORY

  • Zones were never meant to be like VMs. They were designed as a process container.
  • Zones have a shared I/P stack and routing.
  • There is (typically) a separate I/P alias per zone.
  • IPMP works.
  • Config is done from the global zone.
  • IPfilter works (v4).
  • DHCP, IPsec, raw sockets don’t work.

Some problems with zones:

  • Non-global routing is affected by global routing table. (Some examples).
  • Using a null route is often used to add a gateway entry but this is where global routing table changes can break zones.
  • Default routes are selected round-robin.
  • defrouter option in the zone config just does a route add (nothing clever in the kernel).
  • inter-zone traffic can be forced to go over the wire. Normally it would go via loopback for efficiency but some sites require audit/logging of traffic.

NOW

  • Each zone gets its own I/P stack.
  • Config is done in the zone.
  • Lots of zones need lots of NICs.
  • Can mix shared and exclusive stacks.

CROSSBOW:

  • Virtualisation at the data (mac addr) level. vNICs.
  • vNIC gives b/w resource management (dladm).
  • vlans are supported in Crossbow.

    • P.S. What happened to my complimentary UKOUG membership?

Poking about in ESXi4.1

Posted on by
Reply

Just looking about, waiting for inspiration, slightly guided by the impossible aim of making a USB stick into a datastore. Here’s one snippet from the log file:

Sep  9 17:24:40 vmkernel: 0:00:00:09.737 cpu0:4497)FSS: 3924: No FS driver claimed device
 'mpx.vmhba32:C0:T0:L0:1': Not supported

And here’s another useful command I ran across which lists useful information about all your storage:

/var/log # esxcli corestorage device list
t10.ATA_____ST3500418AS_________________________________________5VM89S46
Display Name: Local ATA Disk (t10.ATA_____ST3500418AS_________________________________________5VM89S46)
Size: 476940
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/t10.ATA_____ST3500418AS_________________________________________5VM89S46
Vendor: ATA
Model: ST3500418AS
Revision: CC38
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: false
Attached Filters:
VAAI Status: unknown
Other UIDs: vml.010000000020202020202020202020202035564d3839533436535433353030

mpx.vmhba33:C0:T0:L0
Display Name: Local USB Direct-Access (mpx.vmhba33:C0:T0:L0)
Size: 1896
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0
Vendor: Generic
Model: USB Flash Disk
Revision: 8.07
SCSI Level: 2
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: true
Attached Filters:
VAAI Status: unknown
Other UIDs: vml.0000000000766d68626133333a303a30

mpx.vmhba32:C0:T0:L0
Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
Size: 15318
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
Vendor: Single
Model: Flash Reader
Revision: 1.00
SCSI Level: 2
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: true
Attached Filters:
VAAI Status: unknown
Other UIDs: vml.0000000000766d68626133323a303a30

t10.ATA_____Hitachi_HDS721010CLA332_______________________JP2911HQ0MK9TA
Display Name: Local ATA Disk (t10.ATA_____Hitachi_HDS721010CLA332_______________________JP2911HQ0MK9TA)
Size: 953869
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/t10.ATA_____Hitachi_HDS721010CLA332_______________________JP2911HQ0MK9TA
Vendor: ATA
Model: Hitachi HDS72101
Revision: JP4O
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: false
Attached Filters:
VAAI Status: unknown
Other UIDs: vml.01000000002020202020204a50323931314851304d4b395441486974616368

Actually, that looks like 

esxcfg-scsidevs -l

in another format.

I did try again to create a filesystem on my USB device with the usual result:

/sbin # vmkfstools -C vmfs3 /dev/disks/mpx.vmhba32:C0:T0:L0
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs3 file system on "mpx.vmhba32:C0:T0:L0" with blockSize 1048576 and volume label "none".
/dev/disks/mpx.vmhba32:C0:T0:L0: Permission denied.  (Have you set the partition type to 0xfb?)
Error: Permission denied
/sbin #

There doesn’t seem to be a way to make other filesystem types from the command line, but then, why would there be?

Moving swiftly on, there are plenty of other commands to play with, e.g.

 /sbin # esxcfg-info

prints hundreds of lines of config.

By my count there are 157 commands to explore in /bin and /sbin. I got that number by the following command: 

/bin # find /bin /sbin -type f -perm +100 -exec ls -l {} \; | wc -l

Disabled ESXi 4.1 USB passthrough

Posted on by
Reply

I figured out a way to see both my USB sticks in the ESXi busybox shell. If you read my previous post you will know that the system enables passthrough of the non-boot USB devices for use by guests by default.

Whether or not there is a “right” way to do this I don’t know (vmware kb or google don’t turn up much of use).

So, going back to that mine of information, the messages file, I notice that /sbin/chkconfig is run immediately before it states that usb passthrough is enabled. Running /sbin/chkconfig –list shows:

~ # /sbin/chkconfig --list
DCUI on
TSM-SSH on
TSM on
usbarbitrator on
lbtd on
storageRM on
sensord on
vprobed on
vobd on
wsman on
slpd on
sfcbd-watchdog on
sfcbd off
ntpd on
hostd on
iked off
lwiod off
netlogond off
lsassd off
~ #

So as a guess I changed usbarbitrator to off with the following command:

~ # /sbin/chkconfig usbarbitrator off

Rebooting the system and looking again, I can now see both the devices: mpx.vmhba33 (the boot USB) and mpx.vmhba32 (my target datastore). The later appears as a FAT partition, despite my trying to dd my vmfs3 filesystem over the top of it.

I will try more dd experiments at the next free slot.

At least I made some progress!

USB Datastore (cont.)

Posted on by
Reply

Finally! I got my second SATA drive up and running and partitioned and visible to ESXi. I created a 15G datastore on the unused partition of the disk. It’s 15G because I have a 16G USB stick which formats to just over 15G. The cunning plan is to dd the datastore to the USB stick to fool the hypervisor.

But, first things first. At the moment I have two USB sticks plugged in: the one used to boot ESXi and one newly FAT32 formatted. From ESXi’s busybox I can see the one I boot from, but not the other.

An extract from fdisk -l shows my boot “disk”:

Disk /dev/disks/mpx.vmhba33:C0:T0:L0: 1988 MB, 1988100096 bytes
64 heads, 32 sectors/track, 1896 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot      Start         End      Blocks  Id System
/dev/disks/mpx.vmhba33:C0:T0:L0p1             5       900    917504    5  Extended
/dev/disks/mpx.vmhba33:C0:T0:L0p4   *         1         4      4080    4  FAT16 <32M
/dev/disks/mpx.vmhba33:C0:T0:L0p5             5       254    255984    6  FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p6           255       504    255984    6  FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p7           505       614    112624   fc  VMKcore
/dev/disks/mpx.vmhba33:C0:T0:L0p8           615       900    292848    6  FAT16

Partition table entries are not in disk order

But I should also be able to see mpx.vmhba32.

This is certainly recognised by ESXi when booting as there are multiple references to it in /var/log/messages. There’s a lot of interesting looking usb messages in the log file but the one I see which is causing my current problem I believe is the following:

Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu3:5590)<6>usb passthrough enabled; all eligible devices will be unclaimed by kernel drivers except for ESXi boot device vmhba33
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)<6>usb-storage 2-1.2:1.0: unclaiming vmhba32
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)usb storage warning (0 throttled) on unknown (SCSI cmd unknown): usb_stor_stop_transport called
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)WARNING: LinScsiLLD: scsi_remove_host: Removing Host Adapter vmhba32
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)ScsiPath: 4100: DeletePath : adapter=vmhba32, channel=0, target=0, lun=0
Aug 29 20:57:05 vmkernel: 0:00:00:31.460 cpu1:4370)WARNING: NMP: nmpUnclaimPath: Physical path "vmhba32:C0:T0:L0" is the last path to NMP device "Unregistered Device". The device has been unregistered.

So I guess my next mission is to find out how to disable the unclaiming. In the meantime I might use Linux to copy the datastore to the USB stick and see what happens then.

I have a cunning plan

Posted on by
Reply

But I need to get a local disk attached to my ESXi server first which will require a purchase. Also, I need to make some time. The plan is as follows, so if anyone reads this they might be able to try it out. First of all, get a disk, the smaller the better, and partition it to the same size as a USB key, the bigger the better. Or a LUN. Not having created a VMFS filesystem before, I’m not sure if it will work on a partition or whether it swallows the whole disk. Anyway, ideally the USB key should be the same size as the disk.

Create a datastore on the locally attached disk. Use “dd” to copy that disk to a file and then dd the file onto the USB stick. That *should* (might) create a VMFS filesystem on the USB key. Basically that is how ESXi gets installed on the USB key albeit with VFAT filesystems and a partition table which gives you Hypervisor1-4.

Once that’s done, ESXi may or may not be able to recognise it. And if it does, will we be able to do anything with it?

That’s my theory (and some experiments to be done).

Datastore on USB stick

Posted on by
Reply

As promised I have been looking at this a bit more. I think I was a bit naive before…perhaps I should have believed everyone, including VMware 🙂

I have built a 4.1 ESXi hypervisor on a second USB stick and have been looking more closely at it. Logging on to the console a running “mount” shows the following:
~ # mount
none on /dev type devfs (defaults,rw,suid,dev,exec,async,atime,diratime,loud)
none on /proc type procfs (defaults)
none on /vmfs/volumes type vcfs (defaults)
none on /tmp type visorfs (2,192,01777,tmp)
updatestg on /tmp/stage type visorfs (0,750,01777,updatestg)
visorfs on /var/lib/vmware/hostd/stats type visorfs (63,63,0755,hostdstats)
~ #

Looking at it another way with “df” reveals:

~ # df -h
Filesystem                Size      Used Available Use% Mounted on
visorfs                   1.3G    257.8M      1.0G  20% /
vfat                    249.7M     87.8M    161.9M  35% /vmfs/volumes/ff060de6-cecc88e5-4d14-8726d7ed0132
vfat                    249.7M      4.0k    249.7M   0% /vmfs/volumes/65092bef-de8a06b5-22db-2bbbc32dc3d2
vfat                    285.9M    135.5M    150.4M  47% /vmfs/volumes/3c3693e8-f77a642a-1910-5c6bdcb26d3a
~ #

And finally, a thid way:

~ # fdisk -l /dev/disks/mpx.vmhba33\:C0\:T0\:L0

Disk /dev/disks/mpx.vmhba33:C0:T0:L0: 16.0 GB, 16062087168 bytes
64 heads, 32 sectors/track, 15318 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/disks/mpx.vmhba33:C0:T0:L0p1 5 900 917504 5 Extended
/dev/disks/mpx.vmhba33:C0:T0:L0p4 * 1 4 4080 4 FAT16 <32M
/dev/disks/mpx.vmhba33:C0:T0:L0p5 5 254 255984 6 FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p6 255 504 255984 6 FAT16
/dev/disks/mpx.vmhba33:C0:T0:L0p7 505 614 112624 fc VMKcore
/dev/disks/mpx.vmhba33:C0:T0:L0p8 615 900 292848 6 FAT16

Partition table entries are not in disk order
~ #
Clearly the filesystems are really FAT16 with the exception of one VMKcore filesystem. The fact that they are mounted on something called /vmfs is neither here nor there.

The thing to try is to run vmkfstools on another USB device and try to create a vmfs3 datastore. This has always failed in the past with some specious error. Unfortunately I can’t try it now because I have run out of USB devices. (I have a 1G stick but ESXi won’t recognise it).

The other thing to try is to create a datastore on another local disk (don’t have one) and see what changes. Or there’s the QNAP so I might try NFS.

I don’t hold out much hope at the moment. Even if a vmfs filesystem could be created on a USB stick, you’d still need to fool ESXi into thinking it was another type of storage. Tricky. If it can’t be tricked, a kernel module would likely be necessary, or an alteration to an existing kernel module which is vmware’s territory and they seem to have already made the decision not to do this.

Progress, sort of

Posted on by
Reply

The best progress is that my QNAP is now working, fixed by a firmware upgrade at QNAP’s behest. This surprised me as I was convinced the problem was in MacOS, as I believe was everyone else. It is now running on 3.3.1 Build 0720T.

The upgrade itself was a bit hairy. The first time it bombed out with an error at 28%. The second time it got “stuck” at 10% for several minutes, then proceeded into the 20s and announced it was finished. The subsequent reboot took ages to finish and I was beginning to think it was broken. A reminder that patience is required in this game. Anyway, Time Machine is now backing up to it which is good.

Also managed to resurrect my Centos build on my lab machine. It’s actually a Xen build with Centos as Dom-0. For some reason I described it before as Centos with KVM. This is what happens, I reckon if you work too late. Of course, that means I still have to try the KVM hypervisor, which people have been telling me is the more popular of the two.

At least that’s two things off the immediate list so I can get back to concentrating (I use the term loosely) on seeing if I can get a VMFS datastore installed on a USB key to use with ESXi.