We have been having a number of problems with our backup software and its vStorage backups locking disks and thereby creating multiple phantom snapshots. We are having this problem so often I thought I better post our quick fix to it, which involved some PowerCli goodness.
- Determine the locked files. Go to the directory, and touch all the files in the directory to determine which are locked
touch *
- Discover which host locked the vmdk disk.
Run:
vmkfstools -D servername.vmdk
Indicates not locked or locked by service console process (may post later on how to solve this one)
[root@phovmwareXX vm]# vmkfstools -D vm-flat.vmdk Lock [type 10c00001 offset 80693248 v 66, hb offset 3219456 gen 11609, mode 2, owner 00000000-00000000-0000-000000000000 mtime 1100232] Addr <4, 135, 1>, gen 3, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 2144047674368, nb 255581 tbz 0, cow 0, zla 3, bs 8388608 Indicates host with MAC 3c:4a:92:78:0c:2a [root@phovmwareXX vm]# vmkfstools -D vm-ctk.vmdk Lock [type 10c00001 offset 67147776 v 35, hb offset 3219456 gen 11611, mode 1, owner 4d921417-f871d30a-27df-3c4a92780c2a mtime 743957] Addr <4, 102, 3>, gen 4, links 1, type reg, flags 0, uid 0, gid 0, mode 600 len 8179408, nb 1 tbz 0, cow 0, zla 1, bs 8388608
- Discover which host has the specific MAC address. I created a simple little PowerCli script to that for me.
#Enter which cluster the locked file is within
$cluster = Read-Host "Enter Cluster"
#Enter MAC Address of host (00:00:00:00:00:00 format)
$MAC = Read-Host "Enter Desired MAC (00:00:00:00:00:00 format)"
get-cluster cluster4 | get-vmhost | Get-VMHostNetworkAdapter | where {$_.Mac -match "$MAC" } | select-object VMHost, Name, MAC, IP, BitRatePerSec | ft -autosize
This should tell you immediately which Host has locked it open. You can PuTTy to the host and attempt to kill any process with it locked (lsof or ps -auwwwx | grep filename ). Alternatively, you can put the host into maintenance mode and reboot.
This post first appeared on Bowulf: Virtualization Engineer And Atkineer, please read the originial post: here