CouchDB on Android

Lately, I've taken on a new project developing an application for Android mobile devices.


Synopsis:
The company I'm doing this for gets assigned a building(s) to provide better cellular coverage for one or multiple carriers. There are extremely expensive devices that are specialized to take over-the-air (OTA) readings and map them out onto floor plans. The problem with that solution is not only the expense, but that the devices are perfectly tuned and calibrated unlike a standard handset. Enter Android.


Design Goals:

1. Gather as much signal strength and location information from the handset as possible.
2. Provide a way to load up a floor plan of a building and designate problem areas.

After discussing with my genius buddy Sean Summers. We decided it would be best to just keep gathering information from the phone when the app gets loaded and never stop. If a sufficient GPS reading is available, we would have data to discover the best available towers to get our donor signal from. With a little training on the app (ie: User must start application for 1 minute before entering building), this would give us the necessary information.

After creating the logging mechanism, I realized that we were recording over 1,000 records per minute! This isn't too big of a deal except that I need to get this data to my server and the user may or may not have internet coverage. We are going into a building with known coverage issues after all. Also, we are counting on each job having about 4-8 different models of Android phones (to detect anomalies among them) and each job could last up to 2 hours. So per job we were looking at upwards of 960,000 records.

Instead of focusing on how to create my own synchronization mechanism with checks and balances, I started looking into what Android had to offer. Enter CouchDB http://couchdb.apache.org/.

I installed the CouchDB app from the market and manually configured it to sync to my home server. I also had to remove the timeout as the application will close itself after so long. I changed my data I was gathering into JSON and it worked beautifully.

Of course, I couldn't go through and manually install CouchDB on every device and configure them. That was when I ran into libcouch-android (https://github.com/couchone/libcouch-android). At first I got a little frustrated because I thought documentation was lacking and I was having some issues that I eventually worked out. Libcouch installs a very stripped down version of CouchDB and when started randomly generates and checks for a port to run on. I figured out it was sending me this port information in a call back that has since been documented. My other main issue was strictly PEBCAK. Too embarrassing to mention here :). On a related note, through my other genius friend Matt Turland, I discovered Rubberducking which I will religiously use from now on. Except I'm thinking of getting a Tux doll instead.

Nonetheless, my application is up and running and syncing data.

Advantages of Hypervisor Based Cloud Computing

We're creating content for our website... Every. Techie's. Worst. Nightmare. The question came up: "What sets Abacus Data Exchange apart from Rackspace and Amazon's EC2?". Ugh.

We offer a fully customizable virtual machine option. Try getting a 1 CPU, 8 GB RAM, 1 TB persistent storage option with those other two. Most cloud services limit you to specific RAM/Storage sizes. Each of our images come with pre-installed operating systems ready to go on statically sized disk images which don't have much room left on them. At any point, even during creation, you can add additional persistent disk images to your virtual machine (you could try that too, but you won't be able to). Unlike the other two where you can change CPU/RAM/Storage "on-the-fly", we require a reboot of the virtual machine for the changes to take effect.

We also offer hardware based VLAN capabilities. While they cannot be provisioned on demand, a simple request to tech support will get one created free of charge. Combined with a virtual appliance installed with Vyatta, a customer can completely secure a set of virtual machines from the outside world. That's how we run our services. Even our router for our internet service is a virtual appliance.

I believe our biggest advantage is to the local market in Lafayette, LA. Lafayette is home to a successful Fiber-to-the-Home network that is municipally owned. Any home/business on the fiber network can get 100 mbps connectivity to any other peer. Soak that in. If you live in Lafayette, you could create a virtual machine and get 100 mbps connection to that virtual machine. How is that not amazing?!?

Of course, we didn't do this on our own. We have to give props to the awesome team over at Enomaly. Other than that, most of our infrastructure is open sourced. So here's a shoutout to: CentOS, Xen, Fedora, Ubuntu and FAN. Don't forget the ones I already mentioned!

Cloud Computing

Wow. What a whirlwind the last few months have been. The company I'm consulting for at Abacus Data Exchange, is now running a cloud computing solution for end users. We've always offered virtual machines. I've always had redundancy and fail-over on those host machines. The only real difference now is that instead of having to call me to create that virtual machine, you can: create an account, enter a credit card, create a virtual machine and have remote access to that virtual machine within minutes. Our smallest image (router software) completes in about 5 minutes. Our largest (Windows Server 2008) takes roughly 40-50 minutes.

Here's a quick outline of how it went down:

Our only initial goal was to create a customer interface to at least power cycle a virtual machine. I created the outline in my head of the python driven site to allow for this. Before I could even get it on paper, I began to realize how much it encompassed. I thought of frameworks to use such as Tornado. Still seeing the amount of development considering security and user management, I realized I should do what I always do, find the open source solution and go from there.

I came up with Eucalyptus, Enomaly, and OpenNebula. Eucalyptus showed lots of promise, but it was extremely immature and clunky. I've since heard it's made great strides even in the last few months and has become a real contender. Enomaly looked like it was exactly what I wanted hands down. The features it did lack that we wanted were available in the paid version (which came with support which never hurts). OpenNebula gave us a great option on the control side. Clients could manage their virtual machines exactly how I wanted them to.... from the command line. I spoke with a few interface developers and got an idea on cost to build the front end. I called Enomaly and got pricing. Easily, Enomaly was the choice.

The JQueryUI interface is slick, easy to use and full of features. The admin interface, which I didn't even expect to get when this all started, is easy and full featured as well. The development team over there has taken full advantage of hypervisor technology and keep adding features monthly. Just about every feature that I first felt were lacking have all been added in features that have since been released. The only one I still haven't seen is the ability for a customer to boot an existing virtual machine to an iso image, but I hear it's coming soon.

There was only one problem still, I had to create new user accounts as admin. So I grabbed my favorite developer, and we went to town on a php driven user signup/verification process (cause php is so damn quick and easy). We verified credit cards and, using Enomaly's API, gave ability to manipulate virtual machines after a sign-up process.

So with much less cost, we only had to develop roughly 1/10th of what we would have had to develop to begin with. Plus we got all kinds of features we didn't expect!

I guess basically all I'm really sayin is .... Enomaly... FTW!!!!

Droid vs. iPhone

I've been running the Verizon Droid phone for about 2 weeks now and figured I would write up a little comparison. I switched because AT&T was a nightmare to deal with on multiple occasions when it came to repair, replacement and finally stolen iPhones. Having only been with Verizon for a couple of weeks, I've not had the chance to compare customer service, so I'll be leaving that out.

Usability is the iPhone's biggest seller. On the iPhone, a simple click on an icon and your app is open. You want to change the way the application acts in some manner. Close the app, open your settings, locate the app and make your changes. Re-open your app. On the Droid, a simple click and your app is open. You want to change the functionality of the app, you tap the menu button and the better written apps have the options right there. NICE!
On the iPhone, you scroll from screen to screen searching for your app (or use the never used search feature that might help). If you're an organized freak, then you have certain screens for certain types of apps (which I had until the fourth wipe out of the iPhone and gave up). With the Droid, you only have 3 screens to place icons, shortcuts or widgets by default (It's my understanding there are a few apps that give you more screens or desktops, but I haven't played with them yet). The widgets take up too much screen room, so I generally use shortcuts. However, you can create folders. VERY NICE!


Multi-tasking was always my biggest complaint on the iPhone. I wanted to listen to Pandora while collecting my road treasures on Waze. Not Possible. While this is possible on the Droid, I see the setback. A majority of apps written for the Android platform don't seem to come with a Quit option. You must use a free app that kills any running services. I pull this app up close to 5 or 6 times a day. I do use the Quit function in an app when it's available, but it is my belief that this should almost be a standard for these applications.


Multimedia seems to work equally well with a much better sound quality coming out of the Droid. Videos look phenomenal on both devices, probably better on the iPhone with it's slightly larger screen. Sound sounds way better coming out of the Droid due to the larger speaker on the back that is roughly 1/4" high and spans the entire width of the phone. I always hated holding the bottom of my iPhone to my ear to hear something low coming out of that tiny speaker down there. I rarely ever use the built-in speakers anyway as I've mostly plugged my devices into my car stereo when I listen to music.


Application Quality is the biggest let-down for the Android platform. The Facebook application doesn't work nearly as well as the iPhone's version and most of the games are more arcade like than iPhone's. I've not had any real issues with finding applications that suit my needs. I can still SSH into my servers, listen to my favorite internet radio stations and play car racing games. I can still track my mileage, record my expenses and check the weather complete with radar. I can ALSO see how many satellites have me for GPS, view the location of nearby wifi hotspots and their quality and BEST OF ALL buy MP3s from Amazon with NO DRM W00T!!!
I say it's a let-down because the apps just don't function the same and have the same usability. Perhaps I'm just used to the iPhone interface and need some adjusting, but they just seem a little "clunkier".


Service, keep in mind I didn't say customer service, is the biggest improvement I've seen. Sure this isn't a comparison of iPhone to Droid as much as it is a comparison of AT&T to Verizon, but I have to mention it. I have not dropped a single call yet (2 or 3 times a day on the iPhone). My calls actually ring on my phone instead of a voicemail popping up 15 minutes later when I walk outside. The call quality is outstanding.


Summary
iPhone - End user toy. It just works. If you're happy with the way it works, then it's great for you. A simple clean interface that just about anyone (even my 3 year old) can pick up and learn how to use.
Droid - A techie toy. It works and it works well. If you don't like the way it works, you have options to change it up. You have applications that actually alter the user experience. My kinda gadget!

Restore address book from iTunes backup

I've recently had my last iPhone stolen and went ahead and made the switch to Verizon with the Droooiiiiiid (I love that ringtone). My problem was that I had no way of retrieving contacts since I didn't sync the contacts with any mail program or contacts because I strictly use gmail and their web interface for mail. Perhaps there was a way to sync those contacts with gmail, but I was unable to find the process.

So now, the only chance I have of retrieving my address book is through the backup files made by iTunes. I was using Windows 7 Home Premium at home (only because I had the opportunity to buy an advanced copy for $50). iTunes stored the backups in C:\Users\<username>\AppData\Roaming\Apple Computer\MobileSync\Backup. I found a very useful tool called iphonebackupextractor. I ran it from the windows machine and was able to export the Address book sqlite database with it. After that, it was clear sailing. I suppose I should mention that the only reason this tool worked was because I never told iTunes to encrypt my backups. If you do have them encrypted, you will have more issues.

I copied the sqlite database file to my linux computer and went at it from the CLI that i'm comfortable with.

From here on out, I'll be posting actual commands with notes:

sqlite3 /path/to/sqlite/file
.headers on
.mode csv
.output addressbook.csv
select ABPerson.first as 'First Name',ABPerson.last as 'Last Name',ABMultiValue.value from ABPerson,ABMultiValue where ABMultiValue.record_id=ABPerson.ROWID;
.quit

Now I had addressbook.csv on my drive and imported that into my Google contacts . I had to do quite a bit of cleanup after this. With a little more research and some tricky SQL queries, I could have fixed most of the issues.

Basically the ABMultiValue.value field is a lot of things such as: Notes, Email addresses, Phone numbers... all related to the contact. I could have identified each value type and labeled them as such, but 90% of my contacts only had phone numbers. Google contacts gave me a quick and easy way to merge duplicates and such so it wasn't too bad dealing with my 200+ contacts.

My brush with DMCA

Last week I went to Youtube to watch one of my new favorite videos:

Israel Kamakawiwo'Ole 'IZ' 'Somewhere Over The Rainbow'

To my surprise, it had a notification that WMG music issued a DMCA takedown notice for the lyrics in it. They still had the video there, but were forced to remove the sound. I did some research and found that WMG wasn't even his label. They simply owned the rights to the lyrics of "Over the Rainbow". I was disgusted with the thought process of music executives and went here:

http://www.wmg.com/contact

and issued this complaint:

Taking down the audio from Israel Kamakawiwo'ole's video of Over the Rainbow only shows how greedy and selfish music production companies such as yourself are. He performed a beautiful rendition of the song and all your company cares about is profiting from his memoirs. I've never boycotted anything in my life, but from now on (no matter how much I want the music) if there is a WMG label on it, I will not purchase it.

with the title "Complaint". I posted it on Facebook and as far as I know only my girlfriend went and did the same thing.

This past weekend, I decided to hit up my iPhone and did a search for him again. I found the exact same video WITH sound! I'm not saying that my girlfriend's and my complaints got WMG to allow the sound back onto Youtube, as I'm sure there were many more people complaining about the matter since he had so many followers that loved his music. I do however, have to say that I am happy that WMG could take these sorts of complaints and consider how utterly useless their actions were.

My close friends know that I'm agnostic and generally don't believe much of anything unless there is proof of it somewhere. Since the most proven facts (that I've looked up) come from physics and one of the most basic facts of physics is "for every action, there is an equal and opposite reaction", I decided to go back and send WMG this:

Last week I submitted a complaint with a threat to boycott WMG for their DMCA takedown notice to Youtube for Israel Kamakawiwo'Ole 'IZ' 'Somewhere Over The Rainbow'. I have recently gone back to Youtube and found that the sound has been restored. I felt the need to write back to your company and thank you for listening to all of your potential customers. I have a new found respect for WMG and will proudly purchase any music produced by your company.

with the subject "Thank you".

I'm sure we've all heard the saying "the squeaky wheel gets the grease". To be honest, it felt good to know that I took the time to write back to them in a favourable way instead of just to complain.

unnecessary netstat freakout

So the command is "netstat -pnl". I run this command regularly to see what programs are listening on my servers and if there is any funny business going on. One day I started noticing the program name and PID were missing and all that was there was a hyphen "-" on two processes. One of them is actually listening on a port up in the 50000 range. I attempt to telnet to it and as soon as I enter one keystroke I'm kicked out. I run "lsof -i :(port number)" and nothing shows up.


I called my linux guru buddy and asked him about it. According to him, if I am root, then I should see all program names and PIDs with that command. This must be a process being run by the kernel that a hacker planted at the kernel level. Naturally I freak out.

I install another virtual host and run the command immediately and don't see it. Now I'm convinced. I start looking at differences between the two servers and can't find any after almost a full day.

Convinced that my new host is clean, I move a few virtual machines on it and head home. The following day they are there again! Again two processes and one of them is listening on a high port range. Now I'm convinced that someone has something on my network that is infecting other machines.

I spend another 6 hours on it with the last half hour being the most lucrative. After finally finding the right Google search I run across a mailing list where someone has the right information for me. "rpcinfo -p" is the command. I see my high ranged port number next to a process called nlockmgr. A quick search online tells me that this is the rpc process that handles locked files over NFS (my virtual machine images). WHEW!!!!

Turns out my linux guru buddy was in the middle of some heavy thought and just threw an answer at me to get me off the phone. I can't blame him, I've done the same myself many times.