Thursday, September 24, 2009

unnecessary netstat freakout

So the command is "netstat -pnl". I run this command regularly to see what programs are listening on my servers and if there is any funny business going on. One day I started noticing the program name and PID were missing and all that was there was a hyphen "-" on two processes. One of them is actually listening on a port up in the 50000 range. I attempt to telnet to it and as soon as I enter one keystroke I'm kicked out. I run "lsof -i :(port number)" and nothing shows up.

I called my linux guru buddy and asked him about it. According to him, if I am root, then I should see all program names and PIDs with that command. This must be a process being run by the kernel that a hacker planted at the kernel level. Naturally I freak out.

I install another virtual host and run the command immediately and don't see it. Now I'm convinced. I start looking at differences between the two servers and can't find any after almost a full day.

Convinced that my new host is clean, I move a few virtual machines on it and head home. The following day they are there again! Again two processes and one of them is listening on a high port range. Now I'm convinced that someone has something on my network that is infecting other machines.

I spend another 6 hours on it with the last half hour being the most lucrative. After finally finding the right Google search I run across a mailing list where someone has the right information for me. "rpcinfo -p" is the command. I see my high ranged port number next to a process called nlockmgr. A quick search online tells me that this is the rpc process that handles locked files over NFS (my virtual machine images). WHEW!!!!

Turns out my linux guru buddy was in the middle of some heavy thought and just threw an answer at me to get me off the phone. I can't blame him, I've done the same myself many times.

Wednesday, September 23, 2009

KVM 85-1 in CentOS 5.3 on Liquid IQ

This howto is specifically for a Liquid IQ solution with my configuration. If you have no idea what that means then check out Liquid Computing. I address specific pieces of hardware during the setup that will not be available to most other computing solutions.

This howto covers a base install only of CentOS with no gui to administer. All administration is done remotely using virt-manager through SSH for security reasons. DO NOT USE X11 FORWARDING!!! I was running X11 forwarding over SSH to pull up virt-manager on each machine and found serious vulnerabilities available to hackers on compromised systems (basically they can configure X11 forwarding to open tunnels and take their hacking scripts along for a ride with the X11 forwarding session).

CentOS 5.3 media has a buggy kernel on the installer that will result in a kernel panic after initial install. Download and either burn to DVD or mount through the Liquid KVM solution a CentOS 5.2 installation media.

Boot to CentOS 5.2 media
Hit Enter at the linux # prompt
Tab to the "Skip" button and press Enter at the Media Check box
Click "Next"
Select your language and click "Next" (English used in this howto)
Select your keyboard and click "Next"
Click "Advanced storage configuration"
Click "Add Drive"
Select "eth4 - Intel Corporation 82571EB Gigabit Ethernet Controller"
Uncheck "Use dynamic IP configuration (DHCP)"
In "IPv4 Address:" /
Fill in target information
Click "OK"
Check "Review and modify partitioning layout"
Click "Next"
Click "Yes" on the Warning box that pops up
Select "VolGroup00" in the device list and click "Edit"
In "Volume Group Name" rename to vg_ls# (where # is logical server number)
Select LogVol1 in "Logical Volumes" and click "Edit"
In "Logical Volume Name" rename to lv_swap
Select LogVol0 in "Logical Volumes" and click "Edit"
In "Logical Volume Name" rename to lv_root

These renaming conventions are useful for recovering guest instances within the host. If volume groups and logical volumes are named the same as the instance you are trying to recover them from there is confusion in LVM.

Final outcome should look like the below image (I'm installing ls4 in this instance)

Click "Next"
Click "Yes" to the Partitioning Warnings box that pops up.
Click "Next"
In the Network Devices list uncheck eth0
Check eth2 and click "Edit"
Configure IPv4 settings for your Management network
Uncheck "Enable IPv6 support"
Click "OK"
Check eth4 and click "Edit"
Configure IPv4 settings for your Boot network
Uncheck "Enable IPv6 support"
Click "OK"
Manually assign the hostname if desired
Fill in Gateway and DNS Name Servers
Click "Next"
Select Time Zone and click "Next"
Fill in the Root Password and Confirm
Click "Next"
Uncheck all packages and repositories
Select "Customize now"
Click "Next"
Scroll through all items in the left list box and clear all check marks from the list on the right for each item.

At the time of this writing the items are:

  • Editors
  • Text-based Internet
Base System

  • Base
  • Dialup Networking Support

Click "Next"

Pull up in your browser and find a good station to listen to.

When the "Reboot" screen appears remove all media and click "Reboot"

Log in as root


Select Disabled under SELinux
Click "OK"


From a client computer on the Management network (In this howto it's Fedora 11 workstation setup):

ssh root@<management ip address>
mkdir .ssh
scp ~/.ssh/ root@:~/.ssh/ws
ssh root@<management ip address>
cat ~/.ssh/ws &gt; .ssh/authorized_keys
rm -f ~/.ssh/ws
vi /etc/ssh/sshd_config
:s/PasswordAuthentication\ yes/PasswordAuthentication\ no/g
service sshd restart
vi /etc/yum.repos.d/CentOS-Base.repo
vi /etc/yum.repos.d/lfarkas.repo
name=LFarkas Repository
yum -y update yum
yum -y update

Adjust your slacker radio station if the music isn't good

ssh root@<management ip address>
yum -y install vconfig kmod-kvm kvm libvirt.x86_64 python-virtinst

This reboot is done to initiate the kvm kernel modification

ssh root@<management ip address>
vi /etc/sysconfig/network-scripts/ifcfg-eth2

add the line at the bottom: "BRIDGE=br2"
change BOOTPROTO=none

vi /etc/sysconfig/network-scripts/ifcfg-br2
BROADCAST=<first 3 octets of management ip range>.255
IPADDR=<management ip address>
NETMASK=<management subnet mask>
<first 3 octets of management ip range>.0
service network restart
cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Now we have a physically bridged network connection that we can use to get our first VM on the management vlan. The last step turns off filtering for our guests to allow all traffic over bridged devices.

Now we just need a VM

qemu-img -f qcow2 /path/to/images/myfirstvm.img 10G

virt-install -n myfirstvm -r 1024 --vcpus=1 --os-type=linux --os-variant=rhel5 -c /path/to/isos/CentOS-5.3.iso --disk path=/path/to/images/myfirstvm.img --network=bridge:br2 --vnc --noacpi

This will get you back to your local machine.

yum install libvirt virt-manager

Run virt-manager and create a new connection to the newly built KVM node using SSH.

This is the most secure way I have found to manage my KVM hosts.