FatMin

Mapping nic hardware address to PCI address, Part II

This post is a quasi update to a post that I originally made just over a month ago. The original post is below...

Mapping nic hardware address to PCI address

Well I have found another useful bit of information hidden deep within the bowels of the ESX configuration. Stumbled across /proc/vmware/pci while trying to track down a flaky hba using the command below.

cat /proc/vmware/pci | grep vmhba

Well change this to...

cat /proc/vmware/pci | grep vmnic

... and you get a nice clean list of your network adapters, minus hardware addreseses.

003:00.0 14e4:164c 103c:7038 Ethernet Broadcom  5/ 16/0x79 A V bnx2     vmnic1
005:00.0 14e4:164c 103c:7038 Ethernet Broadcom  7/ 17/0x81 A V bnx2     vmnic2
013:00.0 8086:10bc 103c:704b Ethernet Intel    10/ 19/0x91 B V e1000    vmnic0
013:00.1 8086:10bc 103c:704b Ethernet Intel    10/ 18/0x89 A V e1000    vmnic3
014:00.0 8086:10bc 103c:704b Ethernet Intel     7/ 17/0x81 B V e1000    vmnic4
014:00.1 8086:10bc 103c:704b Ethernet Intel     5/ 16/0x79 A V e1000    vmnic5
017:00.0 8086:10bc 103c:704b Ethernet Intel     5/ 16/0x79 B V e1000    vmnic6
017:00.1 8086:10bc 103c:704b Ethernet Intel    10/ 19/0x91 A V e1000    vmnic7
018:00.0 8086:10bc 103c:704b Ethernet Intel    10/ 18/0x89 B V e1000    vmnic8
018:00.1 8086:10bc 103c:704b Ethernet Intel     7/ 17/0x81 A V e1000    vmnic9

Looking at the list above I can quickly see which two nics are my onboard Broadcoms (on my HP DL380g5). Now I can isolate vmnic1 from vmnic2 using the ethtool command to make the lights on each port blink on and off. In the example below I will make vmnic1 blink for 20 seconds.

ethtool -p vmnic1 20

Note that I have not found it particularly easy to identify which port is blinking, due to the fact that all my ports are binking in one way or another. I recommend getting a good look at the ports before you run ethtool, and then run it and look again. The comparison makes it easier to identify said port.

New Vmware Course - VMware vSphere: Troubleshooting

Vmware has recently made a few changes to their training line up, the most important of which (in my opinion) is the addition of a vSphere Troubleshooting course. I plan to attend sometime in Q1 2010, hopefully.

Also worth noting that the ESX 3.5 class will go the way of the Dodo bird at the end of December, so if you have no plans to upgrade to 4.0 and want to get your VCP, your time is running out.

Also note that Vmware is also changing their VCP exam retake policy; after 2009 they will no longer offer free exam retakes. Considering that the passing rate of the VCP is hovering around 50%, this is not good news for those (like me) planning to get their certification in 2010.

VMware vSphere: Troubleshooting Course

My 2 Rules of a Two Host ESX Cluster

Unfortunately for me I had the misfortune of learning that a two node ESX cluster is quite a fickle beast.

First off let me start by saying that this particular cluster is non-production, as my main production cluster has 8 nodes, and my secondary production cluster has 3 nodes. Less nodes often means less maintenance, which is why, I guess, that I have not run across these quirks before. So here are my new rules for a two node cluster...

• Rule #1. No DRS Rules. Having a DRS rule to keep virtual machines separate is stupid if you have only two nodes in your cluster. Try putting one of the nodes in maintenance mode with this rule in place. It will not work. Plus it will not tell you why it is not working. Plus if you loose one of your ESX hosts, HA will not be able to start these two guests on the single node.
• Rule #2. If you are going to put a node in maintenance mode, disable HA across your cluster first. Otherwise you are going to break HA on your other node (resulting in an HA error on your active node) and you are not going to be able to bring your node out of maintenance mode.

How to Cancel a Stuck Task in ESX

Today I was in the process of trying to put one of my ESX 4 hosts into maintenance mode. However after about 30 minutes of waiting the task was only 2% complete. Right-Clicking on the task and clicking cancel did nothing to help. So I ran the following commands to kill off each and every task that was stuck or in progress and brought sanity back to my cluster.

service mgmt-vmware restart
service vmware-vpxa restart

Mapping nic hardware address to PCI address

When logged into an ESX host the following command will list out all your nics with their PCI address. As you can see I have quite a few active ports on this box. Specifically I have two onboard ports and two quad port cards.

esxcfg-nics -l

vmnic0  0d:00.00 e1000       Up   100Mbps  Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic5  0e:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic8  12:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic2  05:00.00 bnx2        Up   1000Mbps Full   1500   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic4  0e:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic1  03:00.00 bnx2        Up   1000Mbps Full   1500   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic3  0d:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic6  11:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic7  11:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic9  12:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller

While this information is helpful, it does not tell me which ports are on the same card. However the following command gives me the mac addresses of each port, so I can figure out which ports are on the same quad port nic by comparing similar macs, I have highlighed the ports that are on the same card as vmnic9

grep "/net/pnic" /etc/vmware/esx.conf

root@rhost11 log]# grep "/net/pnic" /etc/vmware/esx.conf
/net/pnic/child[0000]/mac = "00:1c:c4:ec:65:58"
/net/pnic/child[0000]/name = "vmnic1"
/net/pnic/child[0000]/virtualMac = "00:50:56:5c:65:58"
/net/pnic/child[0001]/mac = "00:1c:c4:ec:65:56"
/net/pnic/child[0001]/name = "vmnic2"
/net/pnic/child[0001]/virtualMac = "00:50:56:5c:65:56"
/net/pnic/child[0002]/mac = "00:18:fe:2e:1b:21"
/net/pnic/child[0002]/name = "vmnic0"
/net/pnic/child[0002]/virtualMac = "00:50:56:5e:1b:21"
/net/pnic/child[0003]/mac = "00:18:fe:2e:1b:20"
/net/pnic/child[0003]/name = "vmnic3"
/net/pnic/child[0003]/virtualMac = "00:50:56:5e:1b:20"
/net/pnic/child[0004]/mac = "00:18:fe:2e:1b:23"
/net/pnic/child[0004]/name = "vmnic4"
/net/pnic/child[0004]/virtualMac = "00:50:56:5e:1b:23"
/net/pnic/child[0005]/mac = "00:18:fe:2e:1b:22"
/net/pnic/child[0005]/name = "vmnic5"
/net/pnic/child[0005]/virtualMac = "00:50:56:5e:1b:22"
/net/pnic/child[0006]/mac = "00:18:fe:2e:b3:91"
/net/pnic/child[0006]/name = "vmnic6"
/net/pnic/child[0006]/virtualMac = "00:50:56:5e:b3:91"
/net/pnic/child[0007]/mac = "00:18:fe:2e:b3:90"
/net/pnic/child[0007]/name = "vmnic7"
/net/pnic/child[0007]/virtualMac = "00:50:56:5e:b3:90"
/net/pnic/child[0008]/mac = "00:18:fe:2e:b3:93"
/net/pnic/child[0008]/name = "vmnic8"
/net/pnic/child[0008]/virtualMac = "00:50:56:5e:b3:93"
/net/pnic/child[0009]/mac = "00:18:fe:2e:b3:92"
/net/pnic/child[0009]/name = "vmnic9"
/net/pnic/child[0009]/virtualMac = "00:50:56:5e:b3:92"

Kill and VM from the Command Line

If you need to kill a guest VM via the command line, use the vmware-cmd command.

"vmware-cmd /<path to VM directory>/server.vmx stop"
"vmware-cmd -l" will get you your virtual machines names and config file locations

If that does not work you can take more drastic measures and kill the VM process itself.
ps -ef | grep <virtualmachinename>
kill -9 <pid>

Enabling Web Access in ESX 4

For some reason, Vmware has decided to disable the Web Access service in ESSX 4 by default. You will find this out if you attempt to click on "Log into Web Access" on your Vmware ESX 4 Welcome Page on a newly build ESX 4.0 server. You will get the following error "503 Service Unavaliable". Do not panic

To resolve this you should SSH into the ESX box and type the following

service vmware-webAccess start, to start the service

chkconfig –level 345 vmware-webAccess on, to set the service to start at boot.

vSphere 4 Configuration Maximums Document

When you select and configure your virtual and physical equipment, you must stay at or below the maximums

supported by vSphere 4.0. The limits presented in the attached document represent tested, recommended limits,

and they are fully supported by VMware.

 

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

VMware vSphere Editions Comparison

The Vmware Marketing team is at it again, as they have introduced even more ESX and ESXi version in VSphere 4. This link is from Vmware.com and helps clarify the difference between versions.  

VMware vSphere Editions Comparison:

http://www.vmware.com/products/vsphere/buy/editions_comparison.html

VMK load_Mod panic failed to load module VMFS2

This past weekend I was doing a few memory upgrades when one of the boxes that I was working on decided not to boot up properly. This was a bummer considering that it was 2am.

Networking did not start properly so I was unable to ssh to the box, however I found the above error messages in the vmware kernel log (/var/log/vmkernel), the hostd log (/var/log/vmware/hostd.log) and on the console.

First I needed to verify the modules that started using the esxcfg-module command with the -l switch. Below is the output of this command, as it should appear, however in this case as we found that the vmklinux and the vmfs3 were loaded up fine but there was no vmfs2 module.

esxcfg-module -l
Device Driver Modules
Module         Enabled Loaded
vmklinux       true    true
bnx2           true    true
cciss          true    true
e1000          true    true
lpfc_740       true    true
lvmdriver      true    true
vmfs3          true    true
etherswitch    true    true
shaper         true    true
tcpip          true    true
cosShadow      true    true
migration      true    true
nfsclient      true    true
deltadisk      true    true

vmfs2          true    true

The we checked whether vmfs2 is not enabled by using the "-g" switch to the same command and found it was not set to load at boot. We then set the boot option to true by using the "-s" switch for the "vmfs2". The performed an esxcfg-boot -b and then rebooted the ESX and it booted up fine.