FatMin

Mapping nic hardware address to PCI address, Part II

This post is a quasi update to a post that I originally made just over a month ago. The original post is below...

Mapping nic hardware address to PCI address

Well I have found another useful bit of information hidden deep within the bowels of the ESX configuration. Stumbled across /proc/vmware/pci while trying to track down a flaky hba using the command below.

cat /proc/vmware/pci | grep vmhba

Well change this to...

cat /proc/vmware/pci | grep vmnic

... and you get a nice clean list of your network adapters, minus hardware addreseses.

003:00.0 14e4:164c 103c:7038 Ethernet Broadcom  5/ 16/0x79 A V bnx2     vmnic1
005:00.0 14e4:164c 103c:7038 Ethernet Broadcom  7/ 17/0x81 A V bnx2     vmnic2
013:00.0 8086:10bc 103c:704b Ethernet Intel    10/ 19/0x91 B V e1000    vmnic0
013:00.1 8086:10bc 103c:704b Ethernet Intel    10/ 18/0x89 A V e1000    vmnic3
014:00.0 8086:10bc 103c:704b Ethernet Intel     7/ 17/0x81 B V e1000    vmnic4
014:00.1 8086:10bc 103c:704b Ethernet Intel     5/ 16/0x79 A V e1000    vmnic5
017:00.0 8086:10bc 103c:704b Ethernet Intel     5/ 16/0x79 B V e1000    vmnic6
017:00.1 8086:10bc 103c:704b Ethernet Intel    10/ 19/0x91 A V e1000    vmnic7
018:00.0 8086:10bc 103c:704b Ethernet Intel    10/ 18/0x89 B V e1000    vmnic8
018:00.1 8086:10bc 103c:704b Ethernet Intel     7/ 17/0x81 A V e1000    vmnic9

Looking at the list above I can quickly see which two nics are my onboard Broadcoms (on my HP DL380g5). Now I can isolate vmnic1 from vmnic2 using the ethtool command to make the lights on each port blink on and off. In the example below I will make vmnic1 blink for 20 seconds.

ethtool -p vmnic1 20

Note that I have not found it particularly easy to identify which port is blinking, due to the fact that all my ports are binking in one way or another. I recommend getting a good look at the ports before you run ethtool, and then run it and look again. The comparison makes it easier to identify said port.

New Vmware Course - VMware vSphere: Troubleshooting

Vmware has recently made a few changes to their training line up, the most important of which (in my opinion) is the addition of a vSphere Troubleshooting course. I plan to attend sometime in Q1 2010, hopefully.

Also worth noting that the ESX 3.5 class will go the way of the Dodo bird at the end of December, so if you have no plans to upgrade to 4.0 and want to get your VCP, your time is running out.

Also note that Vmware is also changing their VCP exam retake policy; after 2009 they will no longer offer free exam retakes. Considering that the passing rate of the VCP is hovering around 50%, this is not good news for those (like me) planning to get their certification in 2010.

VMware vSphere: Troubleshooting Course

Top 10 Free Vmware Administration Tools

In an effort to more fully immerse myself into the world of Vmware and Virtualization, I have been spending a good bit of time, as of late, poking around the virtual blogosphere looking for blogs that are "must reads". There is no shortage of great information out there to soak in.

Stumbled across the post below today and figured I would post it here so that I can be sure to find it in the future, when i actually get the time to read the blogs that I am finding.

Top 10 Free Tools that must be in every VMware Administrators toolkit

My 2 Rules of a Two Host ESX Cluster

Unfortunately for me I had the misfortune of learning that a two node ESX cluster is quite a fickle beast.

First off let me start by saying that this particular cluster is non-production, as my main production cluster has 8 nodes, and my secondary production cluster has 3 nodes. Less nodes often means less maintenance, which is why, I guess, that I have not run across these quirks before. So here are my new rules for a two node cluster...

• Rule #1. No DRS Rules. Having a DRS rule to keep virtual machines separate is stupid if you have only two nodes in your cluster. Try putting one of the nodes in maintenance mode with this rule in place. It will not work. Plus it will not tell you why it is not working. Plus if you loose one of your ESX hosts, HA will not be able to start these two guests on the single node.
• Rule #2. If you are going to put a node in maintenance mode, disable HA across your cluster first. Otherwise you are going to break HA on your other node (resulting in an HA error on your active node) and you are not going to be able to bring your node out of maintenance mode.

How to Cancel a Stuck Task in ESX

Today I was in the process of trying to put one of my ESX 4 hosts into maintenance mode. However after about 30 minutes of waiting the task was only 2% complete. Right-Clicking on the task and clicking cancel did nothing to help. So I ran the following commands to kill off each and every task that was stuck or in progress and brought sanity back to my cluster.

service mgmt-vmware restart
service vmware-vpxa restart

Multiple Bonding Modes in RHEL5

In RHEL 4, if you had only one bond, you would define your bonding mode in the /etc/modprobe.conf. Note that this configuration only lets you create one bonding device. If you wanted two bonds with different bonding options you had to do something similar to what you see below.

Instead of this in your modprobe.conf...

alias bond0 bonding
options bond0 miimon=100 mode=1
alias bond1 bonding
options bond1 miimon=100 mode=1

You would have this...

install bond0 /sbin/modprobe bonding -o bond0 miimon=100 mode=1
install bond1 /sbin/modprobe bonding -o bond1 miimon=100 mode=0

If you didn't and you tried to define both bond configs in your modprobe, you will only have one module loaded called bonding and bond1 will get the options of bond0. Not exactly what you want.

Note that you may need to modify your modprobe.com and add the following line... max_bonds=2.

In RHEL 5 you actually define the bonding options in your bond config file (/etc/sysconfig/network-scripts/ifcfg-bondo for example), with the BONDING_OPTS variable: BONDING_OPTS="mode=1 miimon=100". You will need to do something similar in both ifcfg-bond* files.

Mapping nic hardware address to PCI address

When logged into an ESX host the following command will list out all your nics with their PCI address. As you can see I have quite a few active ports on this box. Specifically I have two onboard ports and two quad port cards.

esxcfg-nics -l

vmnic0  0d:00.00 e1000       Up   100Mbps  Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic5  0e:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic8  12:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic2  05:00.00 bnx2        Up   1000Mbps Full   1500   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic4  0e:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic1  03:00.00 bnx2        Up   1000Mbps Full   1500   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic3  0d:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic6  11:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic7  11:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller
vmnic9  12:00.01 e1000       Up   1000Mbps Full   1500   Intel Corporation Intel PRO/1000 PT Low Profile Quad Port PCIe Gigabit Ethernet Controller

While this information is helpful, it does not tell me which ports are on the same card. However the following command gives me the mac addresses of each port, so I can figure out which ports are on the same quad port nic by comparing similar macs, I have highlighed the ports that are on the same card as vmnic9

grep "/net/pnic" /etc/vmware/esx.conf

root@rhost11 log]# grep "/net/pnic" /etc/vmware/esx.conf
/net/pnic/child[0000]/mac = "00:1c:c4:ec:65:58"
/net/pnic/child[0000]/name = "vmnic1"
/net/pnic/child[0000]/virtualMac = "00:50:56:5c:65:58"
/net/pnic/child[0001]/mac = "00:1c:c4:ec:65:56"
/net/pnic/child[0001]/name = "vmnic2"
/net/pnic/child[0001]/virtualMac = "00:50:56:5c:65:56"
/net/pnic/child[0002]/mac = "00:18:fe:2e:1b:21"
/net/pnic/child[0002]/name = "vmnic0"
/net/pnic/child[0002]/virtualMac = "00:50:56:5e:1b:21"
/net/pnic/child[0003]/mac = "00:18:fe:2e:1b:20"
/net/pnic/child[0003]/name = "vmnic3"
/net/pnic/child[0003]/virtualMac = "00:50:56:5e:1b:20"
/net/pnic/child[0004]/mac = "00:18:fe:2e:1b:23"
/net/pnic/child[0004]/name = "vmnic4"
/net/pnic/child[0004]/virtualMac = "00:50:56:5e:1b:23"
/net/pnic/child[0005]/mac = "00:18:fe:2e:1b:22"
/net/pnic/child[0005]/name = "vmnic5"
/net/pnic/child[0005]/virtualMac = "00:50:56:5e:1b:22"
/net/pnic/child[0006]/mac = "00:18:fe:2e:b3:91"
/net/pnic/child[0006]/name = "vmnic6"
/net/pnic/child[0006]/virtualMac = "00:50:56:5e:b3:91"
/net/pnic/child[0007]/mac = "00:18:fe:2e:b3:90"
/net/pnic/child[0007]/name = "vmnic7"
/net/pnic/child[0007]/virtualMac = "00:50:56:5e:b3:90"
/net/pnic/child[0008]/mac = "00:18:fe:2e:b3:93"
/net/pnic/child[0008]/name = "vmnic8"
/net/pnic/child[0008]/virtualMac = "00:50:56:5e:b3:93"
/net/pnic/child[0009]/mac = "00:18:fe:2e:b3:92"
/net/pnic/child[0009]/name = "vmnic9"
/net/pnic/child[0009]/virtualMac = "00:50:56:5e:b3:92"

Kill and VM from the Command Line

If you need to kill a guest VM via the command line, use the vmware-cmd command.

"vmware-cmd /<path to VM directory>/server.vmx stop"
"vmware-cmd -l" will get you your virtual machines names and config file locations

If that does not work you can take more drastic measures and kill the VM process itself.
ps -ef | grep <virtualmachinename>
kill -9 <pid>

Esxcfg-vswitch Commands

 
For us Unix guys, the idea of doing everything in ESX via a GUI is very hard to swallow. Especially when it comes to repetitive tasks like setting up vSwitches and Port Groups. Sure you can spend and arm and a leg on Enterprise Advanced licenses so that you can have distributed switches, but why when you can just script your commands and run that script on each box in your cluster. Below are a few examples of how to use the esxcfg-vswitch command.

Delete a Port Group:

esxcfg-vswitch -D "Service Console" vSwitch1

Add a nic (vmnic2) to a vswitch (vswitch1):

esxcfg-vswitch -L vmnic2 vswitch1 

Remove a pnic (vmnic3) from a vswitch (vswitch0):

esxcfg-vswitch -U vmnic3 vswitch0

Create a portgroup (VM Network3) on a vswitch (vswitch1):

esxcfg-vswitch -A "VM Network 3" vSwitch1

Assign a VLAN ID (3) to a portgroup (VM Network 3) on a vswitch (vswitch1):

esxcfg-vswitch -v 3 -p "VM Network 3" vSwitch1

Change your Service Console (vswif0) IP and Subnet Mask:

esxcfg-vswif -i 172.20.20.5 -n 255.255.255.0 vswif0

Add a Service Console (vswif0):
esxcfg-vswif -a vswif0 -p "Service Console" -i 172.20.20.40 -n 255.255.255.0

Add Vlan to a Port Group:

esxcfg-vswitch "vSwitch name" -p "portgroup name" -v "vlan ID" 

List the existing config so you can get your vswitch and portgroup name:

esxcfg-vswitch -l

Enabling Web Access in ESX 4

For some reason, Vmware has decided to disable the Web Access service in ESSX 4 by default. You will find this out if you attempt to click on "Log into Web Access" on your Vmware ESX 4 Welcome Page on a newly build ESX 4.0 server. You will get the following error "503 Service Unavaliable". Do not panic

To resolve this you should SSH into the ESX box and type the following

service vmware-webAccess start, to start the service

chkconfig –level 345 vmware-webAccess on, to set the service to start at boot.