Post virtualization thoughts

I think that I have achieved my goal of reducing the number of machines on which my network runs. There is plenty of good about this:

less power consumption therefore a greener planet.
less hardware to maintain.
an ability to tune the RAM for each virtual to exactly what I want. Thus, if the web server is using 203MB then I can set up a virtual machine with say 233MB of RAM and use the remaining ( you’d normally have to put 256MB physically into a machine so I can scrimp 23MB for another virtual machine) somewhere else.
I get better utilization out of my hardware (ie things like the processor are now running at 50-60% instead of 5-6%).
I can ‘freeze’ each virtual PC rather than having to completely shut down the machine If I need to do maintenance.
I can copy/backup a virtual machine by simply copying files. Sure they are big files but now if I want to migrate to faster hardware I do a simply copy and then fire up the virtual PC on the bigger, better, faster machine – upgrade done.
I can quickly isolate a virtual PC from the network by removing the mapping of the virtual network card from the physical network card. This is great for maintenance tasks that may affect the network.

There are obviously some bad things as well:

Disk performance is slower since all machines share a single physical disk which gets thrashed more.
If I get a corrupt virtual PC image then I lose the whole virtual PC.
I have a single piece of hardware that can still fail and if it does ALL my virtual machines are out of action.
Virtual technology doesn’t give as good performance as physical machines.
Converting physical machines to virtual machines does take some fiddling.
Working with virtual PC files requires much greater transfer times since the files are GB’s in size. A simply cut and paste can take 10 minutes.

Some other comments I’d make on the process I took:

Sure I could have used Windows Virtual Server but Virtual PC is quicker and doesn’t require IIS. Also the virtual PC images are more easily moved. Not being an expert in Virtual Server I’m sure eventually that the virtual PC images will end up in Virtual server, my thinking is that if machines are already virtual PC’s then they are going to be easier to move to virtual server should I choose.
VMware conversion is something that needs more research, I have done it very successfully with workstation images using Shadowprotect but servers appear to be a different kettle of fish.
Virtual PC’s don’t like non-Windows environments. Shadowprotect boots in virtual PC but man is the network transfer slow.
If you are migrating a production environment spend the time and do it properly, don’t try and do it off the top of your head. You’ll make mistakes and the conversion process will take twice as long. Sit down and define the steps you can take and what the roll back is.

In conclusion, I have no doubt that virtualization is the way of the future, it has too many advantages to ignore. Virtualization can work in an SMB environment but there are still some considerations to take into account (eg speed). Now that everything is converted I’ll keep posting what I find as I’m sure I’m bound to uncover some more interesting lessons.

The transformation is almost complete – Part 3

This is the final part in the saga of my intention to migrate all my network equipment (servers and workstations) into virtual machines on a single piece of hardware.
In our last episode you may remember that I had ended up doing a swing migration of my SBS 2003 server onto a new machines. After a few hiccups I had it all working. Now the final task was to migrate the stand alone ISA 2004 server I use as a firewall device.
ISA 2004
Ok, so this machine does nothing except host ISA 2004 as a firewall and web proxy. It only has a small disk and very few apps installed. My concern was because it has two network cards that there might be issues (and I was right).
So, the first attempt was again to do a Storagecraft image of the original machine and then simply do a restore to a new clean Microsoft Virtual PC (no more attempts to migrate to Vmware, two strikes were enough to convince me that I needed to do more research to understand the process). After imaging the server I restore into a Virtual PC and Windows booted but I started having all kinds of issues with ISA 2004. My guess is that this stemmed from changing both network cards in the machine simultaneously. Now I could have sat down and tried to resolve things but since this was a firewall machine and I’d never be completely sure whether I had fixed everything, I decided that it would be better (and quicker) to rebuild a new machine from scratch. Besides, there wasn’t much software to install and once I had ISA 2004 running I “should” be able to simply import the rules from the old ISA box straight into the new box (in theory).
After installing Windows Server and then ISA 2004 I exported the firewall rules from the old server and attempted to import them into the new server. On attempting this I was greeted with the following :

Hmmm, not good, catastrophic failure eh? Thinking, thinking, thinking. Bing! Ah ha, the new ISA 2004 server doesn’t have ISA 2004 Service Pack 3 installed. Installed that and now the import works! Yeah.
Problem was that ISA still wasn’t working correctly. When I looked at the rules I saw that they still referred to the old listener, so I changed that, still no go. I cleaned up the rules, removing what I didn’t need. Still no go. I checked the configuration and network cards. Still wouldn’t work. When all else fails try a reboot. Guess what? It worked after that. So even if you make changes to ISA 2004 you may still need to reboot for them to take effect.
Ahhhhh, finally done. All the machines are now virtualized and I can dispose of all the old hardware. It had taken a long while and there were plenty more bumps in the road that I expected but I had managed to do what I had set out to achieve.
In my next post I’ll summarize what I found along the way with some more thinking about the whole virtualization concept as I think it has particular relevance in the SMB market. For the record I’ve gone from 6 different pieces of hardware into a single piece. If that doesn’t cut my electricity bill I don’t know what will!

The transformation is almost complete – Part 2

This is the second part of my story of attempting to migrate all my machines (servers and workstations) into virtual machines actually housed on one physical piece of hardware (you know to stop global warming and save the whales man).

If you can remember our last episode I had managed to finally migrate my web server into Microsoft Virtual PC using Shadowprotect, that wasn’t exactly the way that I’d planned to do it but at least it was done and I had removed one piece of hardware from my network. Next on the agenda was my SBS server.

SBS Server

So having failed with my initial attempt to convert to a virtual machines using Storagecraft and VMware I decided to try again, since this time I didn’t have dynamic disks on my SBS server. So I imaged the SBS server and attempted to convert it in Vmware. Unfortunately, once again the conversion failed with some obscure error. Damm, not again. Ok, abandon the VMware option, roll on Virtual PC. So I started to do a Storagecraft hardware independent restore of my SBS image to a clean Virtual PC. Problem was it was excruciatingly slow, too slow for me. So scratch that idea since I had a lot of data on my SBS box.

At this point I was beginning to question the whole migration process, it was worse than having teeth pulled. Time to take a deep breath and have a think about this for a while. After some peppermint tea and a nice lie down I deiced that perhaps the best method was to migrate my SBS 2003 installation to SBS 2003 R2. Sorry not migrate but S.W.I.N.G. using Jeff Middleton’s method. That would keep the active directory but I’d get a nice new cleanly upgraded server. Yeah baby, let’s do it.

So Jeff’s method is basically to introduce a temporary domain controller into your existing domain and replicate the existing active directory to that machine. You then detach it from the production network and build a new network around this copied active directory. There are a few critical steps with Jeff’s migration, firstly like turning off the Windows 2003 firewall (forgot about that the first time since it re-enables itself on a reboot – bugger) and secondly to ensure that during the migration you make the domain control a global catalogue server (forgot that the second time – again, bugger). Both of these oversights meant that I had to go back and do the swing migration again (why am I so stupid? I should have really concentrated on what I was doing rather than just doing it off the cuff, which you always pay the price for!).

Finally, I had a good copy of my active directory and I installed SBS 2003 R2 onto the virtual machine. Typically you know the swing migration has had a problem during replication if the Exchange Server component of SBS won’t install. At last, a clean SBS box. I copied over the data that I wanted and the Exchange mail stores (which took a little while) but the great thing is that with the swing migration the Exchange databases simply load. After a little more fiddling (adding customized ISA 2004 rules, installing anti-virus and tweaking Exchange to keep the spammers out) I was done – phew.

Once again, one of the biggest advantage of virtual machines is the ability to switch the network cards in and out of the real network. In this way I could work on my migrated SBS server with it clashing the existing production server. When I was ready I simply shut down the production SBS server and brought the virtual SBS server up in its place (with the virtual network cards actually connect to the real network). Another big advantage of virtual machines is the ability to adjust the amount of memory that each server uses. So after a while I actually adjusted the RAM used by both migrated servers down to give me the ability to host more virtual PC’s on this one piece of hardware.

Other benefits of ‘swinging’ on to a new SBS server? Bye, bye CRM 1.2. Yeah!! Why? Because it wouldn’t uninstall. The ability to create a bigger boot partition (to handle those upcoming Windows Server 2003 service packs – really had to scramble to get SP2 on my machine). The opportunity to remove all the other crap that I had accumulated on my server over the years from testing this and that. Now I have a simple but extremely functional SBS server.

Two servers down, maybe this will work after all! Tune in to the next episode to get the the low down on my migration of a stand alone ISA 2004 box.

The transformation is almost complete – Part 1

Over the Christmas / New Year period I planned to undertake the biggest change to my network structure so far. I decided that I wanted to reduce the total amount of hardware in my shop by using virtualization technology. This basically meant migrating 5 physical machines (4 servers and 1 workstation) onto a single piece of hardware. As they say we have the technology to build it but here is my story of the experience.

Prior

Ok, so the first thing I needed was decent machine to host all these virtual machines on and one with plenty of RAM. So I started with a name brand server, RAID 5 with 4GB of RAM. I install Windows Server 2003 Enterprise Server to allow access to RAM above 4GB (which I don’t have initially but I do want to be able to scale up to more virtual machines should I want to). After installing Windows, applying updates and installing the suppliers monitoring software I was ready to do my first my migration.

Now, the plan was to make this as simple as possible and from what I could tell the easiest way was to use Storagecraft Shadowprotect to take an image of the whole server and then simply convert this image into a VMware machine, which it does support. So, in theory, image, convert, run, nothing could be simpler eh? Here’s what actually happened next.

Stage 1 – Web Server

After imaging the server using Shadowprotect I attempted to convert the image into VMware. Half way through the process I received an error about a disk driver (scsiport.sys) but I chose to continue thinking that I could deal with this afterwards. Problem was a little further down the conversion process the whole thing crapped out. Bugger, what’s the issue? A little bit of investigation pointed to the fact that I had (stupidly) converted the basic disks to dynamic disks on the original server. Why the hell did I do that all those years ago? Now sure, I could “unconvert” them but I already had an image so I thought I’d try option two. You know onwards and upwards (to infinity and beyond is the catch cry isn’t it?).

Option two was to do a hardware independent restore using Storagecraft. So I booted the Storagecraft CD in a clean VMware machine and had issues. Damm. Not being a real Vmware expert I decided it was time for option three – Microsoft Virtual PC 2007, as my failures were beginning to REALLY PISS ME OFF. Storagecraft booted fine in Virtual PC and I did a TCP/IP mapping to my saved server image and commenced the restore. Lesson 1 – Storagecraft restores to Virtual PC are slow! But they do work.

So with the image restored to a new Virtual PC I rebooted the Virtual PC expecting everything to work just fine – WRONG. For starters, for some reason, all the drives were outta whack (ie C: was D: and D: was C: and so on). so the system booted but I couldn’t even run Computer Manager in Administrative tools to restore the correct drives letters (the server had a C: which held Windows and D: that held everything else). Damm. After some more fiddling around with the boot record I got C: drive in the right place, after which I could run Computer Manager and get D: correctly assigned.

Finally, the web server was back in operation with no major errors in the logs. (Ahhh, That’s better). So I now shut down the actual web server and bring the new virtual web server on line and it works! One of the really good things about virtual technology is that you can redirect the network cards to actual or virtual network cards. Thus, I could work on the web server with the same IP address as the original one but with the virtual network card not actually connected to the real network. When I was ready, all I did was shutdown the real server and change the virtual PC’s network card to connect to the actual physical network card so it can now be seen on the network.

As I basked in glow of the first “successful” migration I mulled over the challenge of the next migration, my SBS server. Surely, that won’t take as long as now I know what to look for and this server DOESN”T have dynamic disks!

As they say boys and girls, be sure to stay tuned to the next episode to see what actually happened.

Latest news on new version of SBS

Here’s some more information about what is coming down the pipeline with SBS Cougar. Seems like the Premium edition will allow the installation of 2 server to split applications like SQL and terminal services.

On that score David Mackie raises some interesting questions and issues with the Premium installation in his blog, especially given the new virtualization technology that will be available in Windows Server 2008.

I’m sure that we’ll hear more about all this as the product near launch (June 2008 time frame) and I’m sure things will change and I wouldn’t be surprised if some MAJOR things change! Time will tell.

When a blue screen of death can be helpful

Got a call from a client who was having regular BSOD (the dreaded Windows Blue Screen of Death), basically meaning they had no option but to reboot their system. Did a quick Windows Update and virus scan remotely but the problem persisted, so an onsite visit was the next option.
The next step was to do some analysis of the actual memory.dmp file that is created when Windows crashes. So I copy this file onto my laptop and run The Windows debugger that you can download from Microsoft to analyse these files. The results did produce something interesting :

Now the lines that I’ve highlighted are errors with files kallenylab4-4db6.sys, kirkjtkkd174f-3545.sys and ortyeras37cd.sys. The final line of the debugger says that the crash was probably cause by kallenylab4-4db6.sys.
Now I don’t know about you but when I see files likes these I sorta know that it is a virus/trojan/malware. So I went searching for the files but couldn’t find them using a normal file search (and yes I had display the hidden and system files options turned on). I know the files are there so I did a bit of googling and found some information that indeed confirmed the files were trojans and had to be removed in safe mode. Even better, this trojan had implemented some cloaking or root kit technology so the files weren’t displayed under normal Windows, but the good old crash dump told me they were there.
Seems like this trojan comes from an “greeting card” email that asks the user to download a file happynewyear2008.exe from a web site. Once the user has downloaded the file the trojan installs. Now I go back to the user and query them about downloading this file from a web site and they confirm they did that because it looked like something fun. Ah, ok, that little bit of fun has just cost you a few hours of my time.
When will users realise that they SHOULDN’T download something they don’t know about? You have the most sophisticated security software in the world installed but if the user overrides this then it is all to no avail. The people who write these trojans know that and that’s why this sorta stuff is always going to be a problem. It is a human problem, not a technology problem.
However, the moral of the story is that sometimes a Windows Blue Screen of Death can be of benefit, especially when it indicates you have a trojan on your system!

Russian Roulette

Seems that everywhere I go these days I hear users saying that they are going to convert to a Mac because Windows is such a pain. Even scarier is that they believe that with a Mac they won’t need anti-virus or patching! Ah, hello, who told you that? Ah those Apple ads is their reply.

Firstly, those Apples ads are exactly that, advertisements to get you to buy the product. They are paid for by the people supplying the stuff not some independent third party. Of course they are going to tell you want you want to hear. They want you to buy the product. So even before we start credibility from these ads = 0! (but they are funny – see the latest ones here. Especially check out the one called Podium in light of my previous post on Vista)

Next, all hardware and software is developed by human beings. Yes, they are generally smart than the average human being but they are still humans. They can’t foresee every ramification and variation that their product will be exposed to. So no matter what is that is developed by humans, it is subject to flaws and these flaws need to be addressed with updates and patches (Mac included).

Next, the bigger the market share the bigger the target. If you only have 10% of the market why, as a bad guy, would I bother writing something to attack you? I get much greater chances of return if I attack the other 90% of the market. However, as that market share increases then I begin to reevaluate my strategy. This is even truer if you propose that the more uninitiated users are moving towards something like the Mac. As a bad guy if more uninitiated users are there then my potential return is even greater so I am going to devote more time to attacking that segment.

I could go on and on. I also acknowledge that in many ways Macs are better for users BUT don’t believe for a second that they are not vulnerable and shouldn’t be protected in a way a PC is protected. If you don’t believe that then you are playing Russian roulette, because it is only a matter of time before you get hit.

For a good article on the overall issues of Mac security click here.

One little check box

So having recently installed Windows Server Service Pack 2 on a client’s SBS 2003 R2 box over the Christmas/New Year break (yes, I know but they were too busy to allow it any other time) it was only upon their return did I strike the following strange problem.

A shared HP printer on the server was showing offline. Strange, all the other HP printers off the server were fine. When I attempted to print a page to the offline printer the job just sat in the queue. I killed all the print jobs and restarted the Print Spooler service and then a test print worked. However, when I asked a user to try and print again the printer was offline again. Did a quick Google and couldn’t find anything so rather than muck about I thought that I’d call HP since surely they had seen this before.

After being accidentally hung up on by the first technician I told the next technician in great detail what I had done and that I suspected the issue to be something related to Windows Server Service Pack 2. He suspected a corrupt driver. Ok, that is possible I suppose. So I deleted and recreated the shared printer on the server (resetting all the page sizes to A4! Why oh why doesn’t this happen immediately??). Guess what? same problem. Next he got me to create a new printer on a workstation to see if I had the same problem. Ah yes, same problem. Next he wanted to delete the printer, restart the server and start hacking the registry.

At this point I had to put my foot down and say that there were users on the system and I believed the real issue was linked to Service Pack 2. He told me to wait on hold while he checked something. Fine, while you’re doing that I’ll Google some more. Guess what? I found the issue! Guess what? It is related to Windows Service Pack 2. Here’s the solution :

Apparently Windows Server 2003 SP2 has some changes in the way SNMP is handling printer queues. It now does multiple SNMP threads for the printer queues instead of 1 round robin.

To resolve this, check if your printer’s SNMP is working properly.

To workaround, in the Printers and Faxes folder,

File > Server Properties
Goto Ports Tab > Click the offline Port > Configure Port
Uncheck “SNMP Status Enable”
OK
This will turn off SNMP querying and set the printer to always Online.

Anyway, after making the changes and checking that the printers work from the clients workstation my HP friend comes back from hold and I tell him that I’ve solved it. Oh, he says, yes Windows Server Service Pack 2 can cause those issues. Arrrggh… If you knew that why didn’t you tell me up front?

Support is never easy is it?