So the command is "netstat -pnl". I run this command regularly to see what programs are listening on my servers and if there is any funny business going on. One day I started noticing the program name and PID were missing and all that was there was a hyphen "-" on two processes. One of them is actually listening on a port up in the 50000 range. I attempt to telnet to it and as soon as I enter one keystroke I'm kicked out. I run "lsof -i :(port number)" and nothing shows up.
I called my linux guru buddy and asked him about it. According to him, if I am root, then I should see all program names and PIDs with that command. This must be a process being run by the kernel that a hacker planted at the kernel level. Naturally I freak out.
I install another virtual host and run the command immediately and don't see it. Now I'm convinced. I start looking at differences between the two servers and can't find any after almost a full day.
Convinced that my new host is clean, I move a few virtual machines on it and head home. The following day they are there again! Again two processes and one of them is listening on a high port range. Now I'm convinced that someone has something on my network that is infecting other machines.
I spend another 6 hours on it with the last half hour being the most lucrative. After finally finding the right Google search I run across a mailing list where someone has the right information for me. "rpcinfo -p" is the command. I see my high ranged port number next to a process called nlockmgr. A quick search online tells me that this is the rpc process that handles locked files over NFS (my virtual machine images). WHEW!!!!
Turns out my linux guru buddy was in the middle of some heavy thought and just threw an answer at me to get me off the phone. I can't blame him, I've done the same myself many times.