I have finally been able to get some free time and build out a SharePoint Development Farm. For the past six months I have been using a virtualized SharePoint farm running VMware workstation on top of Windows 7. While some thought went behind the initial design, the farm had been showing signs of fatigue and it was apparent that it really could not support the number of users needed. In addition to the DEV farm, I was running standalone SharePoint farms in VMware on my laptop. While this worked better than I expected, it certainly had its share of issues.
Like most consultants, I had VMs spread all over—VMs for sales demos, projects, and personal sandboxes. It always seemed that either a feature that I wanted to demo or some code that I wanted to work with was on a VM that was offline and I would constantly be switching back and forth between VMs. In addition to running a development farm, I was also using  VMware Workstation on my laptop for various development and project activities.  While this worked better than I expected, it had its share of issues. Requirements
There certainly are many areas to consider when building a farm, let alone a SharePoint farm. While this is my personal development lab (that I will end up using on client projects), I wanted to follow the best practices that I promote to my clients and define the goals and requirements up-front. It is worth mentioning that I was building this for myself and did have to stay within a specific budget.  It should not be considered a recommendation or blueprint for a production environment.  With that in mind, the following is a short list of the major goals for the new environment. Isolation The SharePoint farm needs to support several types of users—mainly designers and developers.  Many of their responsibilities overlap, however each user approaches their environment uniquely and the farm should accommodate their needs—users should not have to radically change their process to match what the farm provides.  A mixture of SharePoint 2007 and SharePoint 2010 environments is needed and isolation among them is needed.  While Sandbox Solutions in SharePoint 2010 seems provide a fair amount of isolation, it is a feature that is only available in SharePoint 2010.  Each user should have their own OS, database, and SharePoint farm. The environment should support a mixture of MOSS 2007 and MSS 2010 farms.
Scalable and Performant SharePoint provides a very flexible architecture that can easily scale out to handle load. In this environment however, load in terms of user requests for a particular farm is not as important.  In terms of scalability, the environment needs the ability to support additional farms. The farm should consist of multiple, independent farms each connecting to their own SQL Server instance. A centralized Active Directory domain should provide overall management of users across each farm. Creating new instances should be accomplished with as much minimal effort as possible.  The overall day-to-day management should be automated as possible so that only minimal manual intervention is needed. Specific farms should be fairly portable and not tied to any host machine; individual instances should be able to be moved to a new physical hardware with minimal effort.
Development The farm should be fairly flexible and allow developers complete freedom over their environment.  The farm should provide core services such as source control and continuous integration. The source control solution should be flexible and allow developers to work remotely without requiring a direct connection (e.g. VPN) to the source control repository. Solution While the requirements defined are fairly high-level and straightforward, I wanted to make sure that I at least captured them so that the environment would support them.  Given the fact that the old host machine (on Windows 7) was quickly overloaded, I decided that having multiple host servers in the farm would be ideal.  As I opted to set up a completely new farm, and I love shiny things, I chose the latest and greatest and went with Windows Server 2008 R2 and SQL Server 2008.  Although I have read a little on SQL Server 2008 R2, I have not kept up on it so SQL 2008 Standard was my choice. Software Application Package Operating System Windows Server 2008 R2 Standard Edition x64 (MOSS 2010 Beta 2 requires KB976472) Database SQL Server 2008 SP1 x64 (MOSS 2010 Beta 2 requires KB970315) Virtual Machine Management System Center Virtual Machine Manager 2008 R2 (SCVMM) SharePoint 2007 SharePoint 2007 Enterprise SP2 (Plus OCT09 Cumulative Update) SharePoint 2010 SharePoint 2010 Beta 2 Enterprise Development Tools (2007) Visual Studio 2008 SP1 Development Tools (2010) Visual Studio Ultimate 2010 Beta2 Source Control (Server) VisualSVN Server (Subversion 1.6.6) Source Control (Client) VisualSVN Client (Subversion 1.6.6)
Architecture As mentioned earlier, I wanted to have an environment that would handle the bulk of the workload and would not be subject to serious degradation of performance when running multiple virtual machines (and the occasional LFD2 break).  With the old Windows 7 VMware Workstation environment, disk access seemed to be the primary cause of poor performance so I wanted to insure that this would not happen again.  I also wanted to make sure that everyone was not at the mercy of a single VM host server.  I have always built my own servers so if this interests you, here are the specifications for the environment (it is a mixture of spare parts and some new equipment).

Server 1 is the primary Hyper-V host and with the Intel i7 offering quad core with Hyper-Threading, Windows detects that eight CPUs are available!—Hyper-Threading is a bit of an illusion, so there are really only four true CPU cores available, but it is still very fast nevertheless.  Host Server 2 is the secondary host and provides a secondary Hyper-V host and SQL Server Database.  The main reasons for including this in the farm was (1) I already had the hardware just lying around, and (2) it is a cost-effective way to offload guests on another machine.  During peak times, there can be a lot of activity on the farm and having a secondary host is helpful.

Disk Performance

I decided against reusing several hard drives that I had lying around even if they were only a year old.  I wanted all drives in the arrays to be completely identical.  I chose the Seagate Barracuda ES.2 drives.  The ES moniker designates “Enterprise Storage” and is probably a lot more marketing hype than anything, but I have read good things about them.  I chose them for three main reasons:

• They are rated at a phenomenal 1.2M hours MTBF.  This essentially means that (on average) these drives will run for 1.2 million hours before they start to fail.
• They are rated for 100% Duty Cycle (24×7).  This is something that many manufacturers neglect to mention.  While typical desktop drives are always on and spinning, they may not always be actually doing anything.  Manufacturers understand this and forgo the added expense of using higher quality (and higher cost) components.  Server drives on the other hand, may always be active.  Having a high duty cycle rating insures that if the drives are always active, you can still assume the MTBF rating.
• They come with a 5 year warranty.  Even with the understanding of MTBF and duty cycle, I cannot assume the drives will not fail.  Many drives come with a 5 year warranty, but I wanted to make sure that these did as well.

The drives are a little pricier than most drives, but I felt the cost was justified.  They are all 7,200 RPM and I did consider faster disks that are available.  The cost of 10,000 RPM drives would wreak havoc on my budget and the 15,0000 RPM drives would decimate it.  I did opt to use the Seagate VelociRaptor drives for system disks on both host servers.  Normally I would have recommended a RAID 1 (mirror) configuration for this, but the budget did come into play.  I did however, mitigate some risk by setting up a regular image snapshot of the system.

Jan 2, 2010 – Update:  A friend of mine pointed me to the Samsung drives that carry a 7yr warranty.

RAID Configuration

I have been using RAID-5 arrays for years and was always interested in testing RAID-10.  After some research, it was obvious that RAID-10 was going to be the array for the new farm.   If you are not familiar with RAID-10 (sometimes referred to as RAID-0 + 1), it is essentially a mirrored (RAID-1) striped array (RAID-0).   So you get the performance of striping, plus the added benefit of redundancy of mirroring (there is also some performance increase for certain operations when mirroring).  In my tests, I was seeing almost a 400% increase in read speed and about a 200% increase in write speed when using RAID-10 over RAID-5.

Backup and Recovery

Since the farm will be used by several friends and colleagues with whom I would still like to remain friends with, I wanted to make sure that their project data was safe.  Since I am on a budget and I had a Windows Home Server sitting idle most of the time, I decided to include it in the farm.  I was still paranoid about losing data say if my house burns down, I wanted to also include an offsite backup process.  I have been using Mozy for quite some time and have been pretty happy with it, so it’s included in the solution as well.

Backing up SharePoint can be accomplished in many different ways and I recommend that you have a mixture of STSADM, database, and file system/OS imaging.  Here’s the breakdown of what happens:

The backups are run on each of the respective servers with a Windows Scheduled Task that moves them to the Home Server.  The Home Server provides a very reliable storage 4TB storage array, but if it were to die, we could be in trouble.  With that in mind, I have Mozy routinely backup the STSADM and database backups.  However, due to the size of the VM image backups, it is impractical to continually send these over to Mozy.  While I did an initial push to Mozy with the baseline images (which took about 72 hours), I am relying on the combination of database and STSADM backups for fail-safe.

Virtualization

With the exception of the SQL Server database, everything is virtualized.  As each developer has their own virtualized development environment with their own SharePoint farm, these separate instances allow me to mix and match MOSS 2007 and MSS 2010 environments.  Surprisingly, I only needed to allocate 1GB – 2GB of memory for each instance (excluding the central SQL server).  Several users focus on work with SP Designer and use their own laptops or desktops remotely.  Several developers (including myself) however, spend a lot of time in Visual Studio so those instances typically get 3GB – 4GB.

I built out several baseline VMs for MOSS 2007 and MSS 2010.  Once I installed all the development tools and applications, I used sysprep to clean out the unique identifiers.  Sysprepping them allows me to easily spin up a new VM based on predefined configuration.  With those images saved, creating a new environment for a project or developer is about a 30 minute process:

• Create a new SQL server instance
• Create a new Hyper-V guest using a copy of the base image for the VHD
• Start the VM and configure the System Properties such as machine name (which Sysprep removed)
• Join it to the domain
• Run the appropriate SP Configuration Wizard.

VM Management

I manage all of the VMs with System Center Virtual Machine Manager 2008 R2.  Initially I wanted to use its VM Clone functionality to create new instances, but it did not work exactly the way I expected.  So for now, I manually make a copy of the baseline VHD and attach it to the new Hyper-V guest.  As I have two Hyper-V hosts, SCVMM does make managing the instances easier, but I do not use it for much else.

Remote Access

As our development teams are not all physically located in the same place, providing remote access to their environments was needed.  Obviously each developer can access their SharePoint farm through a standard web browser by pointing it their uniquely assigned port and configuring forwarding ports.  Sometimes however, the developer needs to get on the box so Remote Desktop was needed.   By default, Remote Desktop always uses port 3389.  I am using a personal grade firewall that does not provide any advanced features that you might find on an enterprise level Cisco appliance.  It does support Port Forwarding which allows me to redirect outside requests to specific internal machines based on the port number being requested.  Since you there is no easy way to redirect traffic on the same port to different machines, changing the port number that Remote Desktop uses allows me to get around this.   Changing the port is very easy:

• Start the Registry Editor (RegEdit.exe)
• Locate the HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Control \ Terminal Server \ WinStations \ RDP-Tcp key
• Change the PortNumber DWORD value from 3389 to the unique port number that you want to use (make sure it is in decimal)
• Ensure that any local firewalls (e.g. Windows Firewall) is configured to allow traffic on the new port
• Restart the server

Once the machine restarts, the machine should now be listening for RDP requests on the new port.  When a developer needs to remote into their machine, they simply specify their port.

Example:
mstsc /v:quartz:3397

I’d like to eventually change this process as it’s not an ideal solution.  I am considering either setting up a Microsoft ISA server or getting an enterprise-level firewall.  If anyone has any suggestions, please leave me a comment.

Development Process

As a long time developer and someone who has focused on the development side of SharePoint, I wanted to insure that the environment would support an Agile process that included a continuous integration (CI) build server.  For years developers have been following this process but I have not really seen it become widely adopted in SharePoint.  If you aren’t familiar with CI, as developers check in their changes, an automated process compiles and deploys the code to a test instance of SharePoint.  This allows you to always know when a recent check-in has broken something.  While this works great for regular projects, due to the nature of SharePoint’s design, it can be a little challenging.

The CI process is managed by CruiseControl.NET and is primarily responsible for monitoring the source control repository.  When it detects new source control commits (check-ins), it executes scripts that will build and deploy them to SharePoint.  Developers can use a monitoring program such as CCTray to stay updated on build progress.  If the latest code doesn’t build or deploy correctly, a notification (a system tray popup in the case of CCTray) informs everyone.  While CI isn’t always needed on every project, it proves to be very helpful when you are working with several developers.  If you are not new to development, you certainly know that developers are not always aware of what others are working on and at times, make a change that breaks someone else’s code.  It is actually pretty slick to easily see the status and activity of the various projects being worked on.  I highly recommend that you look at these free, open source, projects.  Both CC.NET and CCTray have been available for many years and support a wide range of source control systems.  The environment currently uses Subversion for source control but I have been starting to play with VS Team System 2010 Beta 2.  It is not as lightweight as Subversion, but I will probably move off of Subversion in the coming months.

In terms of architecture, a separate VM instance has been dedicated as the CI server.  Right now it just handles the monitoring, building, and deploying of changes, but in the future, I would love to include automated unit testing.

Build Process

Most of the configuration process is pretty straightforward and you can easily find sample configuration scripts but as expected, SharePoint-specific development can be a little tricky.  As I mentioned earlier, several developers focus on design work with Master Pages/Page Layouts and branding with CSS.  They prefer to store their assets in the Style Library of the SharePoint Publishing site.  While having them manually export all of these files to a local directory to commit them to source control is an option, it is tedious and error-prone.  Instead, I created a custom MSBUILD task (in C#) that uses the SharePoint API to grab the latest versions of their assets in the Style Library and commit them to the source control repository.  As all code is deployed using SharePoint Features and Solutions, deploying the assets back to the Style Library is all handled through SharePoint’s Feature Deployment architecture.  I plan on blogging about this process in a future article.

Conclusion

So far the environment is working very well.  I was really surprised by the performance.  I was so disappointed by VMware Workstation (and in all fairness, comparing it with Hyper-V is apples to orange), I was not expecting the new farm to run as efficient as it is.  This is not a big farm by any sense, but at peak times I can have up to eight developers/designers working without any noticeable performance hits.

There were a few hiccups along the way, but things have settled down and the developers seem pretty happy.  While this sounds like a lot of information to digest, it really did not take too long to set up and I probably spent about two weekends getting everything up and running.  I tried to automate as much of this as possible so I do not need to be involved in day-to-day maintenance.  I will post more about the Continuous Integration process in a future article.  I really believe that SharePoint can be used as an application development platform (especially with MSS 2010).  If you have any questions or suggestions, please let me know. I am always interested in tweaking and improving the process and I want to inspire developers to treat SharePoint as a true development platform.