50Micron.com

To say I’m overwhelmed is an understatement.  Hip deep in a major Tech-Refresh for a (non-EMC) vendor that is sucking my life dry, I sometimes forget that this blog is ever here. (As evidenced by the lack of activity)

On the blog/webhosting front things have been interesting.  CATBytes is hosting about 50 users or so.  Mostly informally, just bloggers and the like looking for a cheap place to park their wordpress sites.

I guess the part of it I forgot about was security.  I am *NOT* a big security wonk, and I’m learning this stuff as I go.  One of my users used a simple password and allowed their site to be hacked, and while that SHOULDN’T have been a big deal, it allowed some user to start sending out Denial-Of-Service attacks using one of my webservers.

For about a month.

And It didn’t occur to me because I wasn’t getting any complaints about bandwidth, speed, etc. (my equipment is good, my internet uplink is good, so it was hardly noticible.

Until the bill came.  See I pay $38/MB for a 10MB commit, but it’s a 100MBit pipe.  They don’t bill me the extra bandwidth so long as I don’t exceed my 10MB for more than like 5%.  And normally I don’t, by a long-shot.

Except for this month.  And since the hack managed to straddle two billing cycles, It double-hit me.

Now my provider “Neglected” to tell me about this overage until months later, stating that they had a glitch in their billing.  But going 90Mbit over my 10 for almost 30 solid days makes for a SEVEN THOUSAND DOLLAR bandwidth bill.

Crap.  So now I’ve rapidly taught myself how to limit bandwidth in VMWare (something I should have been doing the whole time) but I have a mad fight on my hands to try and get this provider to see that they’ll bankrupt me if they pursue this, and that won’t be good for either of us.

I hope they see logic.  Because if not, I have to explain to 50-75 bloggers why their sites are going down.  And I *WILL* name names.

Um…yes?  Kind of an obvious question in the grand scheme…  In fact I’m really surprised it’s not done more often.

In the grand scheme of things.  Why wouldn’t a small business use something that’s FREE and gives them the ability to maximize their (usually minimal) hardware investment?

I have a customer, one of those little “side” gigs we all take on, a friend of a friend said “I have someone who needs a new server.” and it all went from there.

So I sell them an old Dell 2U (PowerEdge 2650) I’ve got floating around, migrate them off their antique HP ML160, their single server provided domain services, file-server.  SQL and a number of other minor services.

Now that it’s time to upgrade.  Should we spend hours and hours installing a new server, migrating files, SQL, changing all of the ODBC connectors on every workstation to point to the new server?

Or P2V the old server to the new server and be done with it.

VMWare offers a lot of options over and above the obvious.  Once the P2V is done, boot, and you’re done.  Then you can build a second domain controller, seperate off some of the minor functions, maybe even seperate off the fileserver and sql servers.

From an admin standpoint, dual network connections, thinly provisioned luns, and most importantly, the ability to power-cycle a server from 2700 miles away without having to wonder if it’s going to come back up, the ability to remotely mount ISO images as CDrom’s for upgrades.

How easy would it be to do remote software upgrades or install a new Windows server if you could insert a CD remotely?

VMWare ESXi is *FREE* people.

Downtime last night – This time (almost) planned.

In my constant effort to stay no more than a couple of steps behind current technology, I have started aquiring Dell 1850′s to replace my 2650′s.

I decided to go with the 1850′s because they are smaller form-factor, which means that they’ll fit in my cramped little rack and allow me room to add the third rack of disks to the Clariion.  I had originally purchased the 2650′s because not having a SAN in the basement I needed something in the way of real storage.

One happy side-effect is that apparently the PC3200 DDR2 memory that the 1850′s requires is in much higher supply than the PC2100 that the old 2650′s took.  This makes it cheaper to run more member and thereby fewer servers.

The whole thing was sold to the finance committee as a power saving/cost saving effort.  The best part (and real reason) is that this upgrade has enabled me to go from ESX 3.5 to vSphere4.  (The 2650 doesn’t support vSphere4 due to it’s lack of 64 bit support)

So far my impression is vSphere4 is that it’s a pretty solid upgrade to ESX 3.5, however I’m sure that once I get a chance to dive into it I’ll find a lot more coolness burined under the covers.

Oh – and the 2650′s are probably hitting ebay soon, should anyone want them – let me know ahead of time and I’ll cut you a sweet deal for being a reader.

Ok, I’m curious as to whether anyone has an answer for this.

Why don’t more people boot VMWare ESX from the SAN?

It occurred to me the other night that I have 2 36G drives in each of my servers that I use possibly 10G of, when I already have a High-Availability storage solution at my fingertips.  I’ve got plenty of storage space, not even including the vault drives.

So I tried it.  I took one of my off-line VMware boxes. (I use DPM so at any given time 2 of my 3 VMWare hosts are probably in StandBy mode) and popped the drives out of it.

I turned it on, went into the BIOS and disabled the onboard RAID controller and enabled the boot BIOS on one of the Emulex HBA.s

I created an 18G lun on the clariion and assigned it to the host as LUN0 and poof, I have a boot disk.

Worked like a charm.  The one surprise (pleasant) is that VMWare seems aware of the multi-pathed boot device even without any form of powerpath on the system.  (That was my biggest concern)

So now I have my VMWare infrastructure running on a host with ZERO fixed-disk drives spinning in it.

So has anyone else tried this and know of any gotchas involved that I may not have run across yet?  I’ve done windows and Linux native boot-from-san many many times, but this is my first attempt at VMWare.

I’ve not however tried pulling a path to see just HOW resilient it is…I should probably should try that before I convert the other two systems to diskless operation, right?

Ok, first off, full-disclosure. I am a VMWare (NYSE:VMW) stockholder. Not much, but enough that I sat up and took notice when the news hit this week.  I also work indirectly for EMC Corporation (NYSE:EMC), so I have my loyalties there as well.

VMWare CEO Diane Greene Departs

The news that CEO and co-founder Diane Greene of VMWare was sacked doesn’t surprise me in the least. I’m curious however as to whether this was at EMC’s behest or the stockholder’s, because I’d like to know who is panicing. If it’s the stockholders I am going to start scooping up shares as fast as I can fund my brokeredge account, if it’s EMC, I’m going to have to ask myself “What does EMC know that they aren’t telling?”

When a CEO of a company is so unceremoniously fired, it’s almost universally about PR. Kind of like a lay-off doesn’t actually do any good other than to convince the investors that they are trying to do something. If EMC is responsible for the departure of Diane Greene, it’s because they want the perception in the market to be that “things are changing for the better”

Change is not always for the better.

I have great hopes for VMWare, not as an investor but as someone who really does see the benefits of x86 virtualization. Seriously. High-end systems have been doing multi-partition operations for decades. Mainframe, IBM, Sun, HP. All of them have some form of virtualization, although they call it “partitioning” or something along those lines.

It’s about bloody time that the x86 systems could do it to. The good news is that VMWare makes it realistic, and affordable.

This site is run on an ESX cluster. Included are my internal infrastructure, Exchange, Blackberry, WebMail, MailScanners, Webserver, Active Directory, even a fileserver, all handled using VMs. I also run about a half-dozen “test” systems as well, ranging from Solaris, Linux, to Windows 2008.

I can honestly say that I absolutely could not run my environment the way it is currently without VMWare.  (As it is my power bills are in the neighborhood of $500/month including the additional 10,000 BTU’s of AC I have pumped into my office)

The long and the short of it, I cringe at the idea of having to do this using separate systems, and it amazes me that more companies haven’t embraced the technology sooner.

VMWare currently holds about an 80% market share, world-wide. Most analysts have stated that the closest competitor, Microsoft, doesn’t hold a candle in either reliability or functionality. (VMotion anyone?)

That being said, I know why VMWare is having the issues it is. I remember one company in particular, two CIO’s in a row were adamant against VMWare. Preferring instead to waste money on horribly underutilized, overpriced hardware for an environment that was primarily windows.

They spent MILLIONS of dollars on a “state of the art” datacenter for a windows-based application. (Don’t look at me, wasn’t my decision) It could have easily been done in less space, using less power, less cooling, and more redundancy using VMWare and no-one would have noticed the difference.

Why did they balk at the idea? Because they (incorrectly) perceived it as something new. Maybe the software was new, though VMWare has been around for what in the computer world is ages. ESX server and VI are on version 3.5 now, with some amazingly solid functionality built right in.

VMware needs to get off their collective tails and remind the CxO level community that Server Virtualization is hardly a new concept, it’s been in place for years. And if you can put a VMWare based datacenter together for 60% of the up-front cost and a significantly decreased monthly operating budget, maybe it’s a concept that should be taken seriously.

Given the savings in energy alone, in this age of skyrocketing energy prices, this should be taken very seriously.

A few have noticed the site was down for an extended period this week.  I learned a few things this week.

I set up my FC system and was so excited to get it moving that I neglected to adequately test my equipment.  I bought used equipment, with used drives, and put real data on them after a whopping 2 days of light testing.  I never stress-tested the drives, didn’t do any kind of exercizing of them to validate that they were worthy of production data.

I also neglected to functionally test the array.  While it did offer the ability to configure a hot-spare, I didn’t check to see if the hot-spare was functional before I moved data over to it.  (Seeing that it was configured was enough for me)

So what happened was this.  I was running on the system and all was well until a drive failed.  The hot-spare didn’t invoke on it’s own, and while one drive was in a failed state, the second drive failed.  Needless to say I lost half my luns and three of them were corrupted beyond repair. 

Luckily I’m one of the old hold-outs.  I have a tape backup system consisting of a Veritas 6.0 environment with an ATL tape library.  I was able to restore to within 48 hours of the failure using tape.

My *NEW* storage back-end of consists of a Dell 2650 with 5x 146G drives.  I installed CentOS5 with a 512GB NFS-mount partition and mounted them to my VMWare servers.  The most interesting part is I realized that by bonding the network interfaces I’m getting the same bandwidth I got out of the 2x 1Gig fibrechannel ports.

Not being a network guy though, does anyone have any suggestions for optimising NFS for storage applications?

Ok, I’ve been playing more and more with VMware lately.  All of it personal, because the work opportunities just haven’t really presented themselves.

In relocating one of my “production” servers to the Fibre array I purchased recently, I ran into a problem.  I realized that I was doing it wrong and tried to cancel out of a disk move.

Every vmware vmfs disk is made up of two parts.  The actual virtual disk is contained in a file ending in “-flat.vmdk” then there is a header file that is named the same way, minus the “-flat”.

In my particular mistake somehow the -flat file got moved but the header file didn’t.  So when I went to re-mount the disk under the VM, it was just gone.

To give you an idea of the level of panic that was going on, the name of the disk that was lost was “finance.vmdk”.  Yes, this is the root disk of the server that runs my accounting package for work.  Not a happy time for me”

I played with it, I scoured the vmfs volumes to ensure that it didn’t get redirected to the wrong lun, I searched VMWare’s knowledge base (a useless endeavor) and was getting ready to rebuild the server when I had an idea.

I renamed the remaining flat file to “finance-temp-flat.vmdk” and went into the console and created a new disk of exactly the same size.  I then deleted the -flat file that was created, and renamed ‘finance-temp-flat.vmdk” to “finance-flat.vmdk” .

I restarted the virtual machine, and lo and behold, it booted without effort.

I then immediately shut it down and backed it up.

I then exhaled.

-SG

I got the notification a while back that VMWare ESX v3.5 is out.  I read everything I could about it to determine if it was going to work on my almost out-of-date hardware (This site as well as most of my infrastructure runs on a pair of Dell 2650 servers running ESX)

So very carefully, I shut-down the VM’s on the smaller server and boot from the 3.5 CD.

I have to say, as always, I’m impressed.  The VMWare upgrade went absolutely without incident, took about 35 minutes, and added a few new options that I think are just about the coolest thing.

One of the new features is the ability to import a virtual appliance directly from the VMWare Virtual Appliance Marketplace.

So the long and the short of it, let’s say you need to put a nagios box online in short order.

Go to http://www.vmware.com/appliance and search for “nagios”

Select from one of the 319 search results – find the solution that best fits your environment.

Download the appliance – then use the Import menu to import it directly to your server.

Simple as that.

Or is everyone else busy too?

Haven’t had time to scratch my arse lately.  Though I did find the time to pair down my MP3 library to about 70Gig.  Amazing the stuff you accumulate when you have the room for it.  Stuff that you absolutely would never listen to in a thousand years.

Getting over the emotional blockage and hitting the delete key is a hard thing to do, harder still is controlling yourself once you start deleting and realize how easy it is.

I’ve been playing with VMWare mounted over NFS lately.  The best part I’ve found is that it makes the virutal machines very portable.  I have a few VMWare servers seeing a single NFS volume as server area, and once I get the system established on it, I can import the VM to any of the VMWare servers as I need to.

Pretty cool.  Pity that they still need to be backed up.  (You still can’t back-up the underlying NFS filesystem and expect to get a restartable copy of your server, unless you are using snap technology)