FatMin

How to Use Hpacucli in ESXi

Wow just stumbled into this one today as I was dreading having to reboot my ESXi 4.1 boxes just to see a few new local disks.

Apparently you can install hpacucli in ESXi very easily, and whats even better is that you can install it directly on a datastore and use the install on any host that has access to that datastore. No need to reinstall on each and every host in your cluster. No need to reboot to configure your disks.

Anyway here is the link to the how to, its obviously written by a Windows Guy, but it works easy enough.

http://v-front.blogspot.com/2012/03/how-to-run-hp-online-acu-cli-for-linux.html

 

Update: Apparently HP now provides an official version of hpacucli for ESXi 5 as part of HP ESXi Utilities Offline Bundle for VMware ESXi 5.0.

Also note that I found that my HP servers with the P400 card did not detect the new logical drives, even after a rescan. Apparently a reboot is required. However my HP boxes with the P410 card did detect the new logical drives after a rescan and no reboot was required.

 

Cisco UCS Failover Via 6248 CLI

Need to fail over Cisco UCS Manager from one 6248 to another. Oh did I mention that you are in a hurry because you are going to have to sift through pages and pages of pdfs to get the information. I found this out the hard way. Anyway, out of the kindness of my heart I have documented this process below.

First log into one of your 6248s and figure out which one is the primary by running the command below.

ucs01-A# show cluster state
Cluster Id: 0x4b05e6042b6111e1-0x8c9e547fee4bbf24

A: UP, SUBORDINATE
B: UP, PRIMARY

Note that UCS01-B is our primary. So log into UCS-B and issue the command below.

ucs01-B# connect local-mgmt

Then run the command to make UCS-A our primary.

ucs01-B(local-mgmt)# cluster lead a
Cluster Id: 0x4b05e6042b6111e1-0x8c9e547fee4bbf24

Note that you must issue the failover command on the node that is the primary, otherwise this happens.

ucs01-A(local-mgmt)# cluster lead a
Cluster Id: 0x4b05e6042b6111e1-0x8c9e547fee4bbf24
request failed: local node is subordinate

During the failover process you will see the output below when checking the cluster state.

ucs01-B(local-mgmt)# show cluster state

Cluster Id: 0x4b05e6042b6111e1-0x8c9e547fee4bbf24 B: UP, PRIMARY, (Management services: SWITCHOVER IN PROGRESS) A: UP, SUBORDINATE, (Management services: SWITCHOVER IN PROGRESS) HA NOT READY Management services: switchover in progress on local Fabric Interconnect

ESXi 4 Quick and Easy Find and Remove All Snapshots Via CLI

I needed a way to find and remove all snapshots quickly and easily from the ESXi 4 command line. Note that I am not a Windows user and would rather punch myself square in the face than use PowerShell.

So here it is... and let me warn you its really nasty

 

# for vm in `vim-cmd vmsvc/getallvms | awk '{ print $1 }' | grep -v Vmid`
> do                                   
> vim-cmd vmsvc/snapshot.removeall $vm
> done

Note that this could actually take a really long time to run depending on the size of your snapshots.

Basically it finds the vmid of each and every vm running on the box and attempts to remove any snapshots (if they exist). Note that you will need to do this on every node in your cluster.  I plan to write a better version of this script at somepoint that will search the datastores for snapshots and the derive the vmid from that, but not today.

How to Reset the Cisco CIMC via the Command Line

Long ago I wrote a post on how to reset the ILO on a HP server via the command line as this was often necessary to get a slow responding ILO responding again.

Cisco has included similar functionality in the CIMC CLI, instructions below.

esx01# scope cimc
esx01 /cimc # reboot
This operation will reboot the CIMC.
Continue?[y|N]y

You use the scope command to attach a specific module. The two links below have quite a good bit of information on the CLI

Link1

Link2

Broadcom (bnx2) Network Adapters Dropping Recieved Packets Under Linux

So a few weeks ago some of our Centos 5.4 and OEL 5.5 servers started exibiting strange connectivity problems. Monitoring started alerting that hosts were down when they weren't; some boxes could ping target hosts and some couldn't; some boxes became unresponsive when interfaces were failed over, and the strangest of all is that some of the boxes would magically "repair" themselves. Like I said, strange.

Over the next week or so we ran into the issue a few more times and were able to see a pattern emerge. All the affected servers were running Centos 5.4 or Oracle Linux 5.5 and had broadcom (bnx2) adapters that were on the recieving end of some pretty decent traffic. Most importantly, all had a good number of dropped recieved packets that was continuously, albeit slowly, increasing.

A bit of google research led us to this bugzilla, which suggested changing the adapter's coalescense settings.... so a bit on coalescense.

Coalescense

In your network adapter, coalescence is all about interupts. Traditionally interupt coalescense (or IC) is used to reduce the number of interupts generated by the system by delaying the generating of an interrupt by a very short period of time...think less then a milisecond. In turn more traffic will be recieved by the host and the next interupt generated will be larger in size. You can find out more than you would ever want to know about coalesence here

The Fix

So apparently the Broadcom IC settings were not aggressive enough. Packets would come in, fill up the receive queue, and get dropped before they could be sent off for processing via an interrupt. This takes us back to the bugzilla above and the suggested settings below which you set with the ethtool command

 ethtool -C ethX rx-usecs 8 rx-usecs-irq 8 rx-frames 0 rx-frames-irq 0

Note that this was not an issue on any in Centos 5.6, any server with Intel adapters, or any server with 10g adapters. As a matter of fact, those servers had IC settings even more agressive then those above. See the Intel 82599EB 10-gigabit settings below

rx-usecs: 1
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

Final Configuration

So now that we know the fix we need to make it permanent, which is not as easy as editing a config file for the device as the coalescence config is set at boot and it part of the installed driver for the device. Rather than muck around with trying to modify the driver itself, we decided to set and configure our devices at boottime with a rc script that checks the checks the each network interface on the box and modifys their IC settings if they are using the bnx2 (Broadcom) driver.  We dropped the script below into /etc/rc.d and created a symbolic link to it in /etc/rc3.d.

#!/bin/bash

case "$1" in
start)

IFACE=$(ls /etc/sysconfig/network-scripts/ifcfg-eth*| grep -v bak | cut -d - -f 3)

for ETH in $IFACE
        do
                if ( ethtool -i $ETH | grep -qw bnx2 )
                then
                        echo "$Changing Settings for $ETH"
                        ethtool -C $ETH rx-usecs 8 rx-usecs-irq 8 rx-frames 0 rx-frames-irq 0
                else
                        echo "$ETH is not a broadcom"

                fi
        done
exit 1
;;

stop)

echo " hammer time"
;;

*)
    echo "usage: $0 (start|stop)"
;;
esac

Very High snmpd CPU Usage and 1000s of IP Addresses

For a while now we have been experiencing SNMP timeouts from our central monitoring server to a set of new and recently deployed servers. We at first attributed the issue to the network driver being a bit different than what we have used in the past (latest greatest hardware) and then to the fact that we are completely gutting our production network in situ. Normalization of the driver across this application farm made no difference and the issue began to get worse, spreading to just about all servers in this one specific service tier. As it was not impacting any other servers it wasn't surprising that we could not find an issue at the network layer so we had to keep digging.

In our environment this tier is unique for two reasons:

1. _{Very high IO per server. On the order of 10,000 write ops to disk per second at peak load.}
2. _{Thousands of private IPs per system (added as ip rules - not virtual addresses on interface devices).}

The first issue was in fact an issue, just not for SNMP. The snmpd process was getting lost in the noise generated by the high system load the disk IO was causing (lots or concurrent processes in this app). The team quickly addressed this issue and scaled the IO subsystem substantially to meet the growing workload. After this change the system load dropped from over 1000 to under 10 (1 minute average) during peak utilization on these servers. Unfortunately snmpd was still timing out. However, now with the noise of all those blocked processes out of the way, running a top showed the snmpd process stuck at 100 percent cpu usage for very long period of times. In fact, periods of less than 100 percent cpu were noted for their rarity.

That led us to difference number two. Recently we had been normalizing our system configurations which resulted in the whole tier of afflicted servers having even more IPs assigned. Ok great the team thought - we know where that lives in net-snmp - the IP-MIB. Our assumption was that net-snmp should only refresh this MIB based on client requests and for some reason our monitoring solution was requesting all the IPs from the IP-MIB portion. That would certainly explain the timeouts we thought. Step one was try the latest net-snmp just in case it was bug on our old default 5.4 install. No change. So the team's next step to trouble shooting this was to essentially block access to the ip-mibIP-MIB part of the OID tree.

We tried various versions of:

view systemview excluded ip.ipAddressIfIndex.ipv4
view systemview excluded ip.ipAddressType.ipv4
view systemview excluded .1.3.6.1.2.1.4.20.1


and so on to no effect. Next we did the obvious test - block all client access. No change - snmpd was still consuming 100% of a cpu all the time. Now what?

Unfortunately we were not on Solaris so no DTrace. However, strace did provide the clues needed to make progress. The strace of the snmpd process was almost entirely ioctl calls to the interface devices to fetch the IP addresses associated with them. These calls themselves were not so much the issue as the response time of the ioctl calls were in microseconds. A continuous snmpget was executed against a system while the strace was generated and the strace output compared to client experience. Client time outs lined up exactly with when the ioctl "storm" would start. As the system time consumed was tiny by snmpd during this, the issue had to be in the user-land component of net-snmp.

One thing the strace showed was snmpd was almost alway determining all the IPs on each interface in what looked like a very tight loop. There was perhaps two to three seconds between each storm. In addition, each subsequent ioctl call involved more and more IPs per call - almost displaying an O(n^2) behavior as the list of IPs in the previous call were then being listed again with only the addition of another IP. So now we knew snmpd was, for some reason, always rebuilding the .1.3.6.1.2.1.4 tree and doing it a nasty way.

The first thing that jumped to mind was somewhere the caching we assumed Net-SNMP would do on the ip table was misconfigured. We attacked the nsCacheTable and its ilk from all directions. No luck and no change in behavior. At this point a more thorough web search found two similar postings describing high snmpd cpu usage. One for system with a very large BGP routing table and another for a system with thousands of VLANs. Neither indicated a solution though the BGP postings indicated that sorting within snmpd was not very efficient. At this point your author started to change some of the cache constants within the net-snmp source code to see if the polling cycle on the interfaces would change. It did not.

Time to turn on debug :
The snmpd daemon that comes with Net-SNMP has a pretty thorough debug mode. So thorough in fact we went to that step last as in our environment the full debug mode generates about 75MB of log data every five seconds (remember all those ip addresses). We had tried various documented methods to only have it run debug on certain sub-modules like the IP-MIB, but we could not find an actual working command line option. The documented examples on the net-snmp wiki pages did not work at all on our build for some reason. So we were forced to deal with the fire hose of logging everything.

With full debug on (-DALL) snmpd was started. Five seconds or so was all that was needed to generate a trace that would help us track down the issue. The net-snmpd agent code is full of logging code like below:

DEBUGMSGTL(("access:ipaddress:container", "processing %d interfaces\n", interfaces));


which will generate a message like:

trace: _netsnmp_ioctl_ipaddress_container_load_v4(): ip-mib/data_access/ipaddress_ioctl.c, 171:
access:ipaddress:container: processing 4 interfaces


This makes it very easy to find where in the code the message was generated and simplifies tracing the application (relatively - there are 100s of modules). So at this point we did the tedious task of simply walking the log file in parallel with walking the code, focusing obviously on the interface and IP portions. Along the way numerous little hooks and debug messages were dropped into the code and snmpd recompiled. This helped us along the way to at least realizing the real problem was much larger than what we wanted to tackle. Essentially, the table access and cache components just will not work when one has 1000s of IP addresses. Best we could tell, snmpd is resorting the IP oid tree after each and every IP address it comes across. If you follow the snippets of debug out put below you will see how it does a table compare after each and every IP on a small test system. On our production box with over 7000 IPs snmpd is spending all its time doing index compares. The full debug snippet is available if anyone wants it.

Realizing there was no way for us to quickly and safely modify either the cache code which honestly, we couldn't figure out why the IP table was called as often as it was. Even interface stats (.1.2.1.3.6.1.2.1.2.2) were being refreshed at a much higher refresh rate than nsCacheTable would imply] or the table and index code we decided to cut the monster off at the head - the call into the "access:ipaddress:container" portion of the code. Through a bit of debugging your author determined the head of the monster was the _netsnmp_ioctl_ipaddress_container_load_v4 function in the ip-mib/ipAddressTable/ipAddressTable_interface.c source file. This function seemed to lead to the discovery of all the IP addresses associated with an interface and hence all the sorting, indexing, and other madness we were experiencing.

Solution
The solution is neither pretty nor elegant and is a bit of the nuclear fly-swatter type. The function identified above loops over all the physical interfaces with IP addresses assigned. In our case that is the loopback and a set of bonded interfaces on various VLANs. Lucky for us, the 1000s of IPs we have on each server are on a set of dedicated bonds on their own VLANs. These are completely separate from the base server VLAN and other "user experienced" addresses. In addition, there is nothing related to these addresses in SNMP that we have ever used or need to use via snmp. As we determined, our monitoring software didn't even know these IPs existed nor really ever hit anything in the IP-MIB table. All it ever knew or cared about was the base IP assigned to each bonded interface.

So in our case, the obvious solution was to just not even process these bonded interfaces to determine what IP addresses were on them. We found some references in the code to essentially leverage ablack list but we could not figure out how to use it. As such we did the next best thing - modified the code of the loop in the _netsnmp_ioctl_ipaddress_container_load_v4 to just skip a set of known bonds - essentially do something like:

if (strcmp(ifrp->ifr_name,"bond1.16") == 0) {
DEBUGMSGTL(("access:ipaddress:skipping",
" interface %d, %s\n", i, ifrp->ifr_name));
continue;
}


The full diff is below. Note the code was written so it would be easy to generate a patch based on our automation tools (i.e. if new bonds come and go) and to be obvious to folks without much C experience to know what to change. Not super efficient but good enough. After making this change snmpd CPU usage went from a continuous 100 percent to under 0.5 percent. Better yet - no more timeouts! All we lost was access to the portion go the IP table for the excluded interfaces. Something we didn't use anyways.

We still don't know why snmpd was determined to rebuild the ip-mib continuously. As our use case is probably pretty rare it is not surprising there are so few reports of this behavior.

Code Diff

--- ipaddress_ioctl.c.new
+++ ipaddress_ioctl.c.orig
@@ -165,11 +165,6 @@
DEBUGMSGTL(("access:ipaddress:container",
" interface %d, %s\n", i, ifrp->ifr_name));

- /* Ops was here */
-
- DEBUGMSGTL(("access:ipaddress:containerSPNew",
- " interface %d, %s\n", i, ifrp->ifr_name));
-
if (AF_INET != ifrp->ifr_addr.sa_family) {
DEBUGMSGTL(("access:ipaddress:container",
" skipping %s; non AF_INET family %d\n",
@@ -177,36 +172,6 @@
continue;
}

- /* Working around issue with 6000 IPs on a host
- * So we exclude known problem interfaces
- */
-
- if (strcmp(ifrp->ifr_name,"bond1.16") == 0) {
- DEBUGMSGTL(("a ccess:ipaddress:skipping",
- " interface %d, %s\n", i, ifrp->ifr_name));
- continue;
- }
-
- if (strcmp(ifrp->ifr_name,"bond1.20") == 0) {
- DEBUGMSGTL(("access:ipaddress:skipping",
- " interface %d, %s\n", i, ifrp->ifr_name));
- continue;
- }
-
- if (strcmp(ifrp->ifr_name,"bond1.24") == 0) {
- DEBUGMSGTL(("access:ipaddress:skipping",
- " interface %d, %s\n", i, ifrp->ifr_name));
- continue;
- }
-
- if (strcmp(ifrp->ifr_name,"bond1.28") == 0) {
- DEBUGMSGTL(("access:ipaddress:skipping",
- " interface %d, %s\n", i, ifrp->ifr_name));
- continue;
- }
-
- /* End Ops Hack */
-
/*
*/
entry = netsnmp_access_ipaddress_entry_create();



Below are some snippets of the snmpd debug data for the offended component.


mib_init: initializing: ipAddressTable
trace: init_ipAddressTable(): ip-mib/ipAddressTable/ipAddressTable.c, 52:
verbose:ipAddressTable:init_ipAddressTable: called
trace: should_init(): snmp_vars.c, 454:
mib_init: initializing: ipAddressTable
….
**looking at interfaces
**found lo
verbose:access:interface:ioctl: ioctl 35123 for lo
access:ipaddress:container: if 0: addr len 4, index 0x1
access:ipaddress:container: address 0x10100007f
access:ipaddress:container: flags 0x49
access:ipaddress:container: interface is up
access:ipaddress:container: is a loopback net
access:ipaddress:container: resources allocated
trace: _netsnmp_ioctl_ipaddress_container_load_v4(): ip-mib/data_access/ipaddress_ioctl.c, 166:
access:ipaddress:container: interface 1, bond0.175
trace: netsnmp_data_list_add_node(): data_list.c, 92:
data_list: adding key 'ioctl_extras'
trace: netsnmp_access_interface_ioctl_ifindex_get(): if-mib/data_access/interface_ioctl.c, 408:
access:interface:ioctl: ifindex_get
trace: _ioctl_get(): if-mib/data_access/interface_ioctl.c, 52:
**found another one
verbose:access:interface:ioctl: ioctl 35123 for bond0.175
…..
**setting indexes
compare:index: compare .1 to .2
compare:index: result was -1
…..
**found another one
access:ipaddress:container: interface 2, bond0.41
**rebuild the index again??
compare:index: compare .1 to .1
compare:index: result was 0
compare:index: compare .1 to .2
compare:index: result was -1
compare:index: compare .2 to .3
compare:index: result was -1
……
**found another one
verbose:access:interface:ioctl: ioctl 35123 for bond0.51
…..
**wow lots to compare with only 4 interfaces
compare:index: compare .1 to .2
compare:index: result was -1
compare:index: compare .2 to .2
compare:index: result was 0
compare:index: compare .2 to .3
compare:index: result was -1
compare:index: compare .2 to .2
compare:index: result was 0
compare:index: compare .2 to .4
compare:index: result was -1
compare:index: compare .3 to .4
compare:index: result was -1
compare:index: compare .1 to .2
compare:index: result was -1
compare:index: compare .2 to .2
compare:index: result was 0
compare:index: compare .2 to .4
compare:index: result was -1
compare:index: compare .2 to .3
compare:index: result was -1
compare:index: compare .2 to .2
compare:index: result was 0
compare:index: compare .3 to .3
compare:index: result was 0
compare:index: compare .3 to .4
compare:index: result was -1
**generate the OIDs
trace: _add_new_entry(): ip-mib/ipAddressTable/ipAddressTable_data_access.c, 253:
…..
build_oid_noalloc: generated: iso.4.10.175.175.29
compare:index: compare .1.4.127.0.0.1 to .1.4.10.175.175.29
compare:index: result was 1
….
build_oid_noalloc: generated: iso.4.10.160.41.64
compare:index: compare .1.4.127.0.0.1 to .1.4.127.0.0.1
compare:index: result was 0
compare:index: compare .1.4.127.0.0.1 to .1.4.10.175.175.29
compare:index: result was 1
compare:index: compare .1.4.127.0.0.1 to .1.4.10.160.41.64
compare:index: result was 1
compare:index: compare .1.4.10.175.175.29 to .1.4.10.160.41.64
compare:index: result was 1
trace: _add_new_entry(): ip-mib/ipAddressTable/ipAddressTable_data_access.c, 253:
…..
build_oid_noalloc: generated: iso.4.10.160.51.64
compare:index: compare .1.4.10.175.175.29 to .1.4.127.0.0.1
compare:index: result was -1
compare:index: compare .1.4.127.0.0.1 to .1.4.127.0.0.1
compare:index: result was 0
compare:index: compare .1.4.127.0.0.1 to .1.4.10.160.41.64
compare:index: result was 1
compare:index: compare .1.4.10.175.175.29 to .1.4.10.175.175.29
compare:index: result was 0
compare:index: compare .1.4.10.175.175.29 to .1.4.10.160.41.64
compare:index: result was 1
compare:index: compare .1.4.10.175.175.29 to .1.4.10.160.51.64
compare:index: result was 1
compare:index: compare .1.4.10.160.41.64 to .1.4.10.160.51.64
compare:index: result was -1
compare:index: compare .1.4.10.175.175.29 to .1.4.10.160.51.64
compare:index: result was 1
**and more and more index compares…
trace: netsnmp_access_ipaddress_container_free(): ip-mib/data_access/ipaddress_common.c, 138:
access:ipaddress:container: free
verbose:ipAddressTable:ipAddressTable_cache_load: 4 records
compare:index: compare .1.4.10.160.41.64 to .1.4.10.175.175.29
compare:index: result was -1
compare:index: compare .1.4.10.175.175.29 to .1.4.10.175.175.29
compare:index: result was 0
compare:index: compare .1.4.10.175.175.29 to .1.4.10.160.51.64
compare:index: result was 1
compare:index: compare .1.4.127.0.0.1 to .1.4.10.175.175.29
compare:index: result was 1
compare:index: compare .1.4.10.175.175.29 to .1.4.127.0.0.1
compare:index: result was -1
compare:index: compare .1.4.10.175.175.29 to .1.4.10.160.51.64
compare:index: result was 1
compare:index: compare .1.4.10.160.41.64 to .1.4.10.160.41.64
compare:index: result was 0
compare:index: compare .1.4.10.160.41.64 to .1.4.10.160.51.64
compare:index: result was -1
compare:index: compare .1.4.127.0.0.1 to .1.4.127.0.0.1
compare:index: result was 0
compare:index: compare .1.4.127.0.0.1 to .1.4.10.175.175.29
compare:index: result was 1
…..
**SNIP noise)

ipAddressPrefixTable...


Poor Man's eSATA Drive Hot Swap without AHCI or Hotplug Support Under Linux

 

 

Unfortunately, hot-swapping an eSATA drive is a bit more complicated than hot swapping a USB drive.

First off,  your BIOS needs to support AHCI (click here for more info on AHCI), and your SATA controller also needs to support it as well. Secondly your OS, needs to specifically support hot plug, and in the case of Windows 7, it wont boot if you change to AHCI after the OS has been installed.

So, In my case I need to update firmware on lots of SATA SSDs and want to do so without rebooting, and without worrying about changing bios settings. So in order to keep things simple, I followed the procedure below.

First, you need to detect your drive. So watch dmesg to see what drive letter is assigned to your new disk upon initial connection.

#dmesg

[86527.985994]  sdd: unknown partition table
[86528.012820] sd 8:0:0:0: [sdd] Assuming drive cache: write through
[86528.012823] sd 8:0:0:0: [sdd] Attached SCSI disk
[86528.456281] device label btrfs devid 1 transid 11 /dev/sdd

Then, when its time to remove the disk device to the following. Subsitute your disk device letters.

# echo 1 > /sys/block/sdd/device/delete

Now you are free to swap your disk. No reboot, no bios changes, required.

Configuring Remote Syslog in ESXi 4.1 via the Linux RCLI

First thing you need to know about ESXi is that it rolls its log after a reboot, meaning if your ESXi server crashes there will be no logs to review and no way of knowing what when to hell and where.

For this reason it is imperative that you setup a remote syslog server and send your logs there. Vmware support will tell you this and shame you if you have not setup remote syslogging.

Fortunately the first hit on google for "ESX syslog how to" will take you directly to the VMware KB article. You can find it here. However this page does not contain directions on how to do this via the Linux rcli, it only contains directions on how to do this via the VMA (management appliance) and from Windows PowerCLI. Really Vmware?

Follow the directions below if you run a real operating system and are not a Windows Admin.

First configure your remote ESXi host to forward syslog to your syslog server.

esxcfg-syslog --server esx04 --setserver vsyslog02 --setport 514

Then verify your settings

esxcfg-syslog --server esx04 --show

Output will appear as follows if everything worked right.

Current remote syslog server settings:
Server name : (vsyslog02.atlis1)
Server port : 514