FATMIN

November 09, 2014

Puppet: How Not To Generate a Certificate with Your Correct Hostname

So, I've been hacking away in my homelab as of late, building out a CentOS kickstart server, a Git server, and a puppet server. Right now, I am working on how to roll my puppet agent installs into my kickstart process. I just started on this, so I have yet to nail it down.

So currently, when kicking a VM, I am not yet setting my new CentOS node's hostname before the install process. Sadly I am setting it manually as I am still building my kickstarts, and they are no where near where I want them to be.

Well, this whole hostname mumbo-jumbo just creates all sorts of issues for puppet... the hostname is one thing initially, then puppet installs as part of the post, and the hostname is set manually to finalize the install. Well this is no good, as you are are not going to be able to add your new node properly until you step in and provide a bit of manual persuasion.

Now while its not hard to find documentation on how to troubleshoot puppet node and master certificate issues -- see here and here for example -- none of it was written to help troubleshoot the mess that I had created.

Here was my specfic error.

Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
Certificate fingerprint: BE:B6:B6:5E:AC:B8: ..truncated

And here verbatim, is the output that you get in response to the error above.

To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

On the master:
puppet cert clean localhost.localdomain

On the agent:
rm -f /etc/puppetlabs/puppet/ssl/certs/localhost.localdomain.pem
puppet agent -t

So we try that and it doesn't work. The next cert I generate identifies my node as localhost again.

So heres how to fix the issue.

# rm -rf /etc/puppetlabs/puppet/ssl

Now before we generate another certificate for our node, lets test what hostname a new cert would have using the command below.

#puppet agent --verbose --configprint certname

If the command above does not spit out the correct hostname, then you my friend, are in luck. Edit the file below

# vi /etc/puppetlabs/puppet/puppet.conf

Now change the entry below by removing the localhost.localdomain, and replacing that mess with the correct hostname

certname = correcthostname.localdomain

Now kickoff a puppet run on the node

#puppet agent -t

Log into the UI, or ssh into the puppet master, and accept the new node request.

Kick off another puppet run after you have accepted the request to seal the deal and update the new node properly.

Nov 9, 2014 6:35:35 PM

Reblog It 0

October 31, 2014

Getting Started with Git: Creating a Git Repo

So first off let me start by saying that I know that there is a ton of information out there on how to get started with Git. Heck, when you create your repo in GitLab it spits these instructions right out in front of your nose. However, what I have found is that most instructions tell you what to do to get started with git, however they do not tell you exactly what you are doing. You end up running a few command and then sit back and try to figure out what you actually just did.

That being said, getting started with Git has been the hardest part of the process, as most of us traditional grey-beard sysadmins are not that familiar with code management. When I was first getting started in technology, there were developers and there were sysadmins, and those two worlds were extremely separate. Now, we as we enter the age of DEVOPS and automation these two worlds are once again colliding (like they did in the beginning, but more on that another day).

In my lab I decided to build a stand alone vm for Git and Puppet. After doing some research – and asking others-- on how and what to do to get Git up and running with a nice front end web interface, I decided to get started by installing the GitLab Omnibus package for Centos 6. This process was quick, easy, and painless.

Once I had a working webUI up and running it was time to create my first repo. Instead of trying to accomplish this from my Fedora workstaion, I just clicked on the "New Project" button on the GitLab dashboard and created an empty repo called "General_Scripts".

Back on my Fedora workstation I created a new directory in my home dir called git, and inside that directory I created a directory called "General_Scripts" as I had done in the webUI.

Now it was time to use Git.

First off you need to configure a few global Git options. This only needs to be done once.

#git config --global user.name "Fatmin"

#git config --global user.email "[email protected]"

Once these global configs are set you can then you can move on to seeding your repo. Here you see my change directories to the Global_Scripts directory and tell Git to initialize this directory as a repo.

#cd Global_Scripts

#git init

Now lets create a file and add it to the repo. Here were are going to create a simple README containing a description of my new repo. The instructions do not tell you that you have to put anything in this initial file, however what good is an empty README anyway

#vi README.md

Now tell git that this file needs to be added to the "General_Scripts" repo that we created a few steps ago.

#git add README.md

Now lets commit that file with a nice little comment. Commit comments should describe what the file we added or what we changed in an existing file

#git commit -m "Added README.md"

Now we need to tell our local git command what remote repo we are going to sync to. Note my git repo url is puppet.lab.localdomain, fatmin is my user's namespace, and General_Scripts is my repo.

#git remote add origin [email protected]:/fatmin/General_Scripts.git

Now we need to actually push the local files to the remote repo (origin) in the master branch.

#git push -u origin master

Now wait a bit and go check out the webUI. You should now see the README.md file in your new remote repo.

Oct 31, 2014 8:05:54 AM | Devops, Git

Reblog It 0

October 16, 2014

Enabling Remote Command Execution in Redhat Satellite

As many of you know, a very useful feature of Redhat Satellite is the ability to execute remote commands on a set of servers. You pick a set of servers and plug in your command and schedule the job. Most of the time these remote commands run just fine, however they can error out if the server is not configured to allow remote command execution. Let's discuss how to fix this.

If remote command execution is not allowed, you will probably see an error similar to those shown below,

Local permission not set for action type script.run

Invalid function call attempted

Note that these two error messages usually mean the same thing. You need to enable remote command execution on the remote server.

First off you need to make sure that the command rhn-actions-control is installed. If it is not then we first figure out which rpm provides the command. You can do so with the command below

# yum whatprovides */rhn-actions-control

Now that we know what to install, lets install it.

# yum -y install rhncfg-actions

Now we enable remote command execution

# rhn-actions-control --enable-all

Now you should be able to reschedule your job against the server via the webUI.

Oct 16, 2014 7:17:55 AM | Redhat

Reblog It 0

September 27, 2014

Vmware CapacityIQ Unregister a Vcenter Server via the CLI

Ok two in a row, now thats strange. Anyway.. I was working on a rebuild and re-ip of my Vcenter Server as I was in the process of changing over to the Vcenter appliance, when i realized that I was going to have to lay hands on a lot of tools and change them to point/connect to my new Vcenter Server instance. Anyway. I started of with VCOPS, and I was surprisingly sucessful. So I figured I would move on to Capacity IQ.

Again, being the Linux guy that I am, I hit up the cli and stated fiddling around. I stumbled across the ciq-admin command and was able to find the following

[ciqadmin@ciq ~]$ ciq-admin register status
vCenter Server Registration Status = Registered
vCenter Server address = https://vc00.lab.localdomain/sdk
vCenter Server user = LAB\user

Hey ok, that looks good so far. Lets see if we can unregister.

[ciqadmin@ciq ~]$ ciq-admin unregister
Connection to VC failed. Disconnected CIQ, but could not remove CIQ extension from VC.

Well crap, you will never be able to connect to my original Vcenter Server... it's dead and its not coming back. Luckily it appears that we can force the registration of a new Vcenter Server

[ciqadmin@ciq ~]$ ciq-admin register --vc-server vc01.lab.localdomain --force --user root --password mypasswd
Stopping CIQ : [ OK ]
Starting CIQ : [ OK ]
ciq-watchdog is now enabled

Now head back on over and get a gander at the webUI. Still shows the old Vcenter Server? Just logout and log back in again and life should be good.

Sep 27, 2014 8:16:38 PM | Vmware

Reblog It 0

Vmware Vcenter Operations Manager Unregister a Vcenter Server via the CLI

So my Windows based Vmware Vcenter Server went belly up again. Something to do with the SSO database not starting. Not being a lover of Windows I decided to give the Vcenter Server Appliance a shot. Install was great and I am kicking myself a bit as to why I spent so much time fighting with Windows. My new Vcenter Server, which has a different IP address then the original Windows box (might make a great Veeam server) was not registered with Vcenter Operations Manager. I was not prepared to reinstall that thing again. So I needed to figure out how to manually unregister a Vcenter instance and add register another one in its place.

Since VCOPs runs Linux, I decided to ssh into the server and see if I could figure it out. First thing I found that I needed to do was figure out the registered Vcenter Server name and Vcenter Name (whatever that's supposed to be) I was able to do this using the vcops-admin command.

admin@vcops:~> vcops-admin summary

This command output a bunch of stuff, but the important bits for this task are below.

Registration Details
--------------------
vCenter Server address = https://vc00.lab.localdomain/sdk
vCenter Server name = Lab Vcenter

So now we need to unregister the sucker above. Note that this command takes a bit to unregister

admin@vcops:~> vcops-admin unregister --vc-name Lab\ Vcenter --vc-server https://vc00.lab.localdomain/sdk --user LAB\\userid --password mypassword--force

vCenter Server unregister = success

So flip on over to your browser and log in. The unregister process will cause the webUI to reload, so if you were already logged in you will find that you still need to log in again.

Now you can register your new Vcenter Server via the webUI.

Sep 27, 2014 7:37:37 PM | Vmware

Reblog It 0

September 24, 2014

HomeLab Adventures: Freenas Volume 1

So I am not going to lie, I am a very sick man, but I am also not afraid to admit it. I have a terrible, terrible addiction which is my homelab.

It all started out so innocently... An old Sun Ultra 5 to learn Sparc Solaris at home.. A couple of desktops converted over to rack mount cases and racked in a cheap telecom rack in my unfinished basement. This was very early in my career when I had a lot to learn and plenty of free time to study. However that was many moons ago.

I measure the time that has past since then by the amount of gray that has crept into my beard. As I moved from one role to the next, I found that I had the pick of the litter when it came to retired equipment. Previously I would have been lucky to land an old Xeon (without virtualization support) to take home, something chock full of PCI-X cards (or worse, SCSI) that were useless to me in a desktop. However now I was landing quad core Nehalems (perfect for virtualization) with handfuls of memory and sexy pci-e SAS/Sata raid controllers. Oh and tons of SSDs that were considered too small not 6 months after they were unboxed. Lets not even get into my networking setup... as that is a tale for a different day.

Once I had a deployed a couple of very nice and fully loaded ESX servers, I came to find that the performance bottleneck in my lab was storage. Sure I had terabytes of SAS and SATA disk, but it was all local. I had nothing that allowed me to fail over between host. Thus began a quest.. a quest for a ages

Knowing myself as I do, I knew that I was not going to be satisfied by throwing a cheap NAS together out of a couple or SATA disk. No, desktop performance was not going to cut it. I needed 15k SAS, a raid controller with battery backup, a handful of spindles, and a beefy tower to allow for plenty of expansion (yes, all my machines were converted to towers). I also knew I was going to need to use LACP or some other network bonding to cable my creation into my network. Heck, I even dared check out the cost of a cheap 10Gb small business class switch (yup too expensive... lets wait a year or so).

Which brings us to today. The day I fired up my first freenas box.

My rough specs are as follows.

Gigabyte Z97-HD3
Intel Core i3 3.8Ghz
5x600gb 15K SAS -Raid-Z1
1x32gGB SSD
2x4tb 7k SATA - Raid 1
16GB Memory
LSI 9260 8i

So now what - move some VMS onto it and call it a day. Well that's no fun. Lets see what kind of performance we can push through this baby. I mean after all, we are not using 15k SAS drives for nothing.

Side note, its not exactly plug and play when it comes to using SAS drives in a standard tower. Even if you have a SAS capable controller, you are going to need a backplane of some sort to provide power and i/o connectivity. Finding something that will fit the bill, without having to use a cheap one-off backplane is a challange to say the least. For my lab I picked up a couple of these. 99% of what you see in the box stores will not support SAS drives, and its not always obvious at first glance... you have to check the specs on the side of the box. Also don't walk into Fry's thinking you will find one... I have tried. Microcenter seems to be the only large chain that stocks an internal SAS enclosure.

For testing I am have ssh'd into a linux desktop that is on the same network as the freenas box. The desktop has only 1gb network interface. Both systems a cabled northbound to a Cisco 3560g.

First lets mount up our RaidZ-1 volume by sticking this in our /etc/fstab and running mount /mnt.

freenas:/mnt/freenas-vol-1 /mnt nfs rsize=8192,wsize=8192,timeo=14,intr

Boom, there it is our new fancy mount. Now to run the tests. However that will come in part 2 as I plan not to rush through this. As far as I understand, there can be a bit of tuning in Freenas, so it might take me a bit to get everything dialed in.

Sep 24, 2014 8:34:58 AM | Homelab, NAS, SAN, Storage

Reblog It 0

September 23, 2014

How to Add and Delete Persistant Routes in AIX

Wow, AIX does not like to make anything easy. Nor do they like to make things intuitive. Need to remove a route from AIX, well get ready to have one command to temporariliy remove a route and another command to remove the route for good. Same goes with adding a route. "Quit your bitching, and use Smitty", you say? Well smitty does not make anything any easier, especially since the UI likes to show you fields that you does not necessarily need you to use. Oh, plus they let you type in them. Asking me for a netmask when adding a static route does not seem like a crazy request to me? But jokes on you, you were not supposed to type anything there.

Anyway, I was tasked with cleaning up a few bad routes that were added to a handful of servers. Note that I was not interested in adding the routes temporarily before I added them as persistent routes. These were simple one liners that I felt very comfortable adding to the ODM right out the gate.

So first we need to check the ODM for the routes that we need to remove. In this example we want to delete the route fro 10.11.1.56, so lets find just that route. Note all of these are host routes.

# lsattr -El inet0 | grep 10.11.1.56
route host,-hopcount,0,,-if,en1,,,,-static,10.11.1.56,10.22.13.1 Route True

Now lets delete the route above

# chdev -l inet0 -a delroute=10.11.1.56,10.22.13.1

We have a second route to delete so lets kill that one too

# chdev -l inet0 -a delroute=10.11.1.57,10.22.13.1

Now lets add the correct routes - the format is as shown below

chdev -l inet0 -a route=host,-hopcount,1,-netmask,netmask,network,gateway

Now on a few of the servers I was working on I had to remove network routes as well as host routes. You need to know that the syntax is anoyingly different adding network routes. In the example below I first need to remove the network route.

chdev -l inet0 -a delroute="net,-hopcount,0,,,,,,-static,10.11.1.56,10.11.1.254"

Now lets add our new and correct route

chdev -l inet0 -a route="host,-hopcount,0,,,,,,-static,10.11.1.56,172.30.150.190"

Sep 23, 2014 3:57:38 PM | AIX

Reblog It 0

August 24, 2014

Vmware Esxtop Displaying Incorrectly In ESXi 5.5 Over SSH From Linux

So I ran into this odd issue last night working in the old homelab. I was working on troubleshooting an issue with datastore performance on one of my Supermicro ESXi servers. I have two and they are awesome beasts that have been migrated into fancy Cooler Master cases, however that is a tale for another day. Anyway, I digress... the datastore. So yeah, it's a local raid 5 datastore attached to an LSI 9260-8i controller. This particular datastore is 3 600gb 15k SAS drives, and for some reason its slow as all get-out.

So being the command line guy that I am, I ssh from my Linux (Fedora 19, specifically) into my troublesome ESXi server and attempt to run esxtop to see if I can get a read on the disks. Much to my chagrin, the output of esxtop is a garble of text Unreadable to say the least.

an example of the horror i saw that night

Turns out that this is an issue related to terminal emulation type. Thanks to this bro, who ran into same issue on his Mac, I was able to get back in the game and determine that yes, you should not configure raid 5 with a dead BBU. Anyway, while his fix was not exactly help for me, it let me know what I needed to do to fix this issue.

The gist of the issue is this. In Linux (Fedora at the very least) the default terminal emulation type is xterm-256color. Set your terminal type to this, ssh to and ESXi 5.5 box, and experience this formatting debacle for your self.

Thankfully the fix is simple, just change your terminal emulation type to "xterm" and you will be good to go. I accomplished this with the command below.