SmartFTP is a fantastic FTP application which handles syncing files very effectively. This means that when you upload your entire website, SmartFTP will automatically detect changes and only upload what is required (instead of overwriting all of the files like some FTP applications do). For each project at TheFARM we have build scripts which run and create a time stamped ZIP package for each deployment environment with all of the necessary files formatted appropriately for each. Our deployment process then involves unzipping the contents of this file, opening up SmartFTP, connecting to the deployment destination and transfering all of the deployment files up (which SmartFTP synchronizes for us).

I thought it would be much more efficient if we automated this process. So we did some investigation and it turns out the SmartFTP conveniently has an API! So we decided to see if we could write a PowerShell script to use the SmartFTP api to automagically transfer/sync all of our deployment files in our Zip package to the necessary FTP site and with a bit of trial and error we managed to do it! Now, I’m not PowerShell expert or anything, and in fact this was my very first PowerShell script ever written so I’m sure this could all be done a bit better, but it works! I’m not going to go into detail about the SmartFTP api or how to write PowerShell stuff because this script will work with some basic requirements:

• You need both PowerShell and SmartFTP installed
• Currently this only supports the standard FTP protocol, but if you need SFTP, etc… you can just change the $fav variable’s ‘Protocol’ property
• The parameters, in this order are:
• destination
• the IP address, or host of your FTP server
• user
• the username used to login to the FTP server
• password
• the password used to login to the FTP server
• path
• The FTP path of where you want your files to go on your FTP server
• port
• The FTP port to use, default is 21
• source
• The source folder to copy to the FTP site, if not specified, uses the current directory that the PowerShell script is run from

Example usage:

FTPSync.ps1 123.123.123.123 MyUserName MyPassword 21 “C:\MyWebsiteFolder” “/websites/MyWebsite”

or you can just double click on the ps1 file and it will prompt you for these details.

So without further adieu, here’s the script!

#requires -version 2.0

# Define inputs
param (
    [parameter(Mandatory=$true)]    
    [string]
    $dest,
    [parameter(Mandatory=$true)]
    [string]
    $user,
    [parameter(Mandatory=$true)]
    [string]
    $pass,
    [parameter(Mandatory=$true)]
    [ValidatePattern('\d+')]
    [int]
    $port = 21,
    [parameter(Mandatory=$false)]
    [ValidateScript({ Test-Path -Path $_ -PathType Container })]
    [string]
    $source,
    [parameter(Mandatory=$true)]
    [ValidatePattern('\/+')]
    [string]
    $path
)

# get current folder
$currFolder = (Get-Location -PSProvider FileSystem).ProviderPath;
# set current folder
[Environment]::CurrentDirectory=$currFolder;
# if the source isn't set, then use the current folder
if ($source = "") {
    $source = $currFolder;
}

Write-Host "------------------------------------------------------" -foregroundcolor yellow -backgroundcolor black
Write-Host("{0, -20}{1,20}" -f "Destination", $dest);
Write-Host("{0, -20}{1,20}" -f "User", $user);
Write-Host("{0, -20}{1,20}" -f "Pass", "********");
Write-Host("{0, -20}{1,20}" -f "Port", $port);
Write-Host "";
Write-Host "Source:";
Write-Host $source;
Write-Host "";
Write-Host "Path";
Write-Host $path;
Write-Host "------------------------------------------------------" -foregroundcolor yellow -backgroundcolor black

# Create application
$smartFTP = New-Object -comObject SmartFTP.Application;
$smartFTP.Visible = [bool]0;
$smartFTP.CloseAll();

# create temp favorite item
$fav = $smartFTP.CreateObject("sfFavorites.FavoriteItem");
$fav.Name = $user + " @ " + $dest + " (temp favorite by cmdInterface)";
# 1 = FTP standard protocol
$fav.Protocol = 1;
$fav.Host = $dest;
$fav.Port = $port;
$fav.Path = $path;
$fav.Username = $user;
$fav.Password = $pass;
# forces it not to be saved
$fav.Virtual = "true";

# Add temporary favorite to SmartFTPs FavoriteManager
$favMgr = $smartFTP.FavoritesManager;
$rootFolder = $favMgr.RootFolder;
$rootFolder.AddItem($fav);

# Get the transfer queue
$queue = $smartFTP.TransferQueue;
# stop the queue if it isn't already
if ($queue.State -ne 1) { $queue.Stop(); } # Stopped = 1
# clear the queue
foreach($item in $queue.Items) { $queue.RemoveItem($item); }
# set the thread count for the queue
$queue.MaxWorkers = 20;
#enable logging
$queue.Log = "true";
$queue.LogFolder = $currFolder + "\\LOG";

# create new transfer item
$newItem = $smartFTP.CreateObject("sfTransferQueue.TransferQueueItem");
# set the item as a folder and copy operation, 
$newItem.type = 2;  #FOLDER = 2 
$newItem.Operation = 1; #COPY = 1

# Set the source
$newItem.Source.type = 1;  #LOCAL = 1
$newItem.Source.Path = $source;

# Set the destination
$newItem.Destination.type = 2; #REMOTE = 2
$newItem.Destination.Path = $path;
$newItem.Destination.FavoriteIdAsString = $fav.IdAsString; #links up to our connection favorite

# and finally add it
$queue.AddItemTail($newItem);

Write-Host "STARTING" -foregroundcolor yellow -backgroundcolor black;

$queue.Start();

while ($queue.Items.Count -ne 0) { 
    Write-Host "Processing...bytes transfered: " $queue.TransferredBytes;
    Start-Sleep -s 2; #wait 2 seconds
}

# store the total bytes
$totalBytes = $queue.TransferredBytes;

# cleanup smartftp app
$queue.Quit();
$smartFTP.Exit();

# parse logs

# regex to find "[DATE/TIME] STOR FILENAME
# which indicates a file transfer
$regex = new-object System.Text.RegularExpressions.Regex("\[[\w\-\:]*?\]\sSTOR\s(.+?)\[",,[System.Text.RegularExpressions.RegexOptions]::SingleLine);
$totalFiles = 0;
Write-Host "Files Transfered" -foregroundcolor cyan -backgroundcolor black
Get-ChildItem $queue.LogFolder -include *.log -Recurse | foreach ($_) { 
    $currFile = Get-Content $_.fullname;
    $match = $regex.Matches($currFile);    
    if ($match.Count -gt 0) {        
        foreach($m in $match) { Write-Host $m.Groups[1]; } 
        $totalFiles++;
    }
    remove-item $_.fullname -Force -Recurse ;
}

Write-Host "COMPLETED (total bytes: " $totalBytes ", total files: )" $totalFiles -foregroundcolor cyan -backgroundcolor black;

"------------------------------------------------------"

# cleanup COM
Remove-Variable smartFTP

Last year there was an article posted on TechChrunch about the problem of deploying your .snv folder live, it’s a really great way to give everyone your websites source code!

Recently though while tweaking our 2 Click ASP.NET Web Application Deployment with MSBuild to include a static folder consisting of the umbraco/ umbraco_client folders (which we leave excluded from the project to ensure performance of Visual Studio) I noticed that we were including the .svn folders!

We’re generating an ItemGroup like this:

<ItemGroup>
  <Umbraco Include="$(LocationWorkingWeb)\umbraco\**\*.*"/>
  <UmbracoClient Include="$(LocationWorkingWeb)\umbraco_client\**\*.*"/>
</ItemGroup>

Which recursively adds the files from those folder, including .svn.

Balls!

Sure it’s not really a problem, we’ve got no source code stored in those folders (and anyone who is putting their own source in umbraco or umbraco_client is asking for trouble), but by including them you’re pretty much doubling the size of the folder structure too!

Luckily it’s quite easy to solve. MSBuild has a build-in Exclude attribute, so you just need to change it to look like this:

<ItemGroup>
  <Umbraco Include="$(LocationWorkingWeb)\umbraco\**\*.*" Exclude="$(LocationWorkingWeb)\umbraco\**\.svn\**\*" />
  <UmbracoClient Include="$(LocationWorkingWeb)\umbraco_client\**\*.*" Exclude="$(LocationWorkingWeb)\umbraco_client\**\.svn\**\*" />
</ItemGroup>

It looks a bit weird, you’ve got to recursively exclude the recursive contents of the .svn folder :P

It’s all about making sure you only deploy what you should have on a production server, and it goes in hand with remembering that PDB != Product Deployable Bits.

A couple of months ago Shannon wrote a post on how we do 2 Click Deployments at The Farm (if you haven’t read this post I suggest you read it first as I make the assumption it’s been read and skip over a few sections that it covers). I’ve been working with him to try and improve our process of deployment, and one of those tasks has been moving away from using NAnt as the build runner to using MSBuild.

Why the shift? Well MSBuild has actually come a long way recently and despite common belief it’s not just for building .NET applications. In fact it can be used the same way that NAnt can be used, to execute arbitrary operations.
As mentioned in the original post we’ve used an internal tool for modifying the web.config (and other configuration files) but it had a bit a limitation as it meant that the files were always full of settings for all environments. It also wasn’t designed to update a single property of an XML node, you’d need to replicate the whole XML structure, even if it was just 1 attribute changing.

So I suggested that we migrate to a XSLT based config management, which is what I’d used previously. A former colleague of mine has a good post on how to set that up, it can be found here.

Moving to MSBuild also brought in another advantage, it would simplify our deployment process. Currently we’re using CruiseControl.NET to execute a NAnt script which in turn executed MSBuild.

Another goal was to allow us to deploy from the branch in source control, rather than from the trunk, this way you can deploy different branches if you want to test different functionality based on a branch without overriding the trunk. So we’ve updated our source control structure to look like this:

• trunk
• branch
  • v1.1
  • …
• build

The new addition is the build folder, this folder will generally only hold a single file, and that’s the master build information, which we’ll call cc.build. This build file is the one that CruiseControl.NET will look for and check out. It then contains the instructions to which branch to check out, and then handle the rest of the execution of the build. A sample looks like this:

<Project ToolsVersion="3.5" DefaultTargets="CheckoutAndBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>

  <PropertyGroup>
    <BranchVersion>v1.1_net</BranchVersion>
    <BuildType>_net</BuildType>
    <BranchFolder>$(BaseFolder)\$(BuildType)</BranchFolder>
    <RepoPath>file:///\svn/Client/branches/$(BranchVersion)</RepoPath>
  </PropertyGroup>

  <Target Name="CheckoutAndBuild">
    <RemoveDir Directories="$(BranchFolder)" />

    <SvnCheckout RepositoryPath="$(RepoPath)"
					LocalPath="$(BranchFolder)"
					ToolPath="C:\Program Files\CollabNet Subversion">
      <Output TaskParameter="Revision" PropertyName="Revision" />
    </SvnCheckout>

    <Message Text="Revision: $(Revision)"/>

    <Time Format="yyyyMMddtt">
      <Output TaskParameter="FormattedTime" PropertyName="buildDate" />
    </Time>

    <MSBuild Projects="$(BranchFolder)\cc.msbuild"
             Properties="CCNetWorkingDirectory=$(BaseFolder);BuildDate=$(BuildDate);DeployType=Dev;DeployLocation=$(BaseFolder)..\Deployment\Dev;BuildConfiguration=Debug;ShowMessages=True;IncludeSymbols=True" />
  </Target>

</Project>

So lets have a look at the break down of this file. First off, we’re using the MSBuild Community Tasks for tasks such as the SVN checkout, and a few other aspects which we’ll cover later.

When we call this MSBuild file from CruiseControl.NET and pass in a parameter, BaseFolder, which is the Working folder (see the Folder Structure section of the original post).
There’s some parameters combined so we get a path to where we’ll be checking out the branch, etc.

First thing is to remove any existing checkout folder, and then have SVN checkout the latest copy of the branch.
As Shannon mentioned in the original post we timestamp each sip package, so for this we’re using the Community Task date formatter to generate our timestamp.

Then it gets a bit tricky, because this file is a dumb file and doesn’t know what’s going to be done by the branch deployment file so we need to have a way in which we can actually do a build. This is done by using the MSBuild task which allows you to execute any specified MSBuild file(s). This is done by passing the file to execute into the Projects attribute. Since we’re going to execute the one within the branch we tell it to look there.
Then it’s just a matter of passing any properties you require into the MSBuild file, here we’ve got a bunch of different parameters which the target file can consume.

The MSBuild task is then replicated as many times for each different environment/ location which you want to build for.

Each branch maintains its own msbuild file, which is also called cc.msbuild so that we can keep the naming consistent across all locations. This file is the one which is responsible for:

• Compiling the project
• Modifying the config files
• Zipping the release

But this can easily be expanded upon, depending how the branch needs to be handled, there’s nothing that says it couldn’t also do the copying to the appropriate server, etc. Lets take a look inside this file.

<Project ToolsVersion="3.5" DefaultTargets="ZipFiles" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>

  <PropertyGroup>
    <LocationWorkingWeb>$(CCNetWorkingDirectory)\_net\Client.Web</LocationWorkingWeb>
    <ProjectWeb>$(LocationWorkingWeb)\Client.Web.csproj</ProjectWeb>
    <BuildFolder>$(DeployLocation)\$(BuildDate)\</BuildFolder>
  </PropertyGroup>

  <Target Name="ZipFiles" DependsOnTargets="FormatFiles">
    <ItemGroup>
      <ZipFiles Include="$(BuildFolder)**\*.*" Condition="$(IncludeSymbols) == 'True'" />
      <ZipFiles Include="$(BuildFolder)**\*.*" Exclude="$(BuildFolder)**\*.pdb" Condition="$(IncludeSymbols) == 'False'" />
    </ItemGroup>

    <Zip Files="@(ZipFiles)"
         ZipFileName="$(DeployLocation)\$(BuildDate).zip"
         WorkingDirectory="$(BuildFolder)"/>

    <RemoveDir Directories="$(BuildFolder)" />
  </Target>

  <Target Name="FormatFiles" DependsOnTargets="BuildWebProject">
    <MSBuild Projects="transformers.msbuild"
             Properties="XslFile=$(CCNetWorkingDirectory)\_net\CruiseControl\Web.config.xslt;OutputFile=$(BuildFolder)Web.config;InputFile=$(LocationWorkingWeb)\Web.config;Environment=$(DeployType)" />
  </Target>

  <Target Name="BuildWebProject">
    <MSBuild Projects="$(ProjectWeb)"
             Properties="Configuration=$(BuildConfiguration);OutDir=$(BuildFolder)bin\;WebProjectOutputDir=$(BuildFolder)"
             Targets="Clean;Build;ResolveReferences;_CopyWebApplication"/>
  </Target>
</Project>

This file, again uses the MSBuild Community Tasks, but its also a whole lot smarter about what’s happening for this particular build. I’ve specified that the DefaultTargets of the file is the ZipFiles which is really the final task which I want to execute.
One really nice thing about MSBuild is that you can specify a DependsOnTargets attribute for a target, which states that the particular target(s) must have executed before the called one will execute. So by looking at how the dependency is configured the project will have been compiled and the files formatted will both have been done. Makes it so much simpler to execute something, rather than having a ‘runner’ target which is just responsible for calling a bunch of targets in order we can just have them done in order of dependency.

So we’ve got a BuildWebProject target which will again call an external MSBuild file, in this case our csproj file, passing in some of the information which was given to us from the ‘master’ file, in a manner in which Visual Studio itself would have done.

Next we format the files. This goes off to an external MSBuild file, so we can use the same operations for every single file we want to format. The contents of transformers.msbuild is as follows:

<!-- ROBOTS IN DISGUISE -->
<Project ToolsVersion="3.5" DefaultTargets="Xslt" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
  <ItemGroup>
    <XslFileWithParams Include="$(XslFile)">
      <environment>$(Environment)</environment>
    </XslFileWithParams>
  </ItemGroup>
  <Target Name="Xslt">
    <Xslt RootTag="" Inputs="$(InputFile)" Output="$(OutputFile)" Xsl="@(XslFileWithParams)" />
  </Target>
</Project>

This is a basic task file which adds a parameter (named envrionments) to our XSLT and then uses the MSBuild Community Tasks to do the transformation.
Note – this is a slight deviation from Alistair’s blog post, we don’t have a different XSLT per machine, we have a single XSLT which we use the parameter to determine what to do. This makes it easier to see all the options we want to change per environment.

Lastly the dependencies hierarchy throws us back to the ZipFiles target, where we gather all the files together that we want (and there’s an option to exclude the pdb files, after all PDB != Product Deployable Bits ;) ), call the Zip task (make sure you specify a working folder otherwise you’ll end up with a crazy hierarchy in the Zip!) and then we delete the folder which we built to.

Conclusion

MSBuild is a good tool which can be used for doing operations other than just compiling a .NET project, as you can see here we’re using it to pull files down out of SVN or transform a file via XSLT. This doesn’t have anything to do with compilation, or .NET for that matter.

It make take a bit of work to get set up, there was quit a bit of frustration vented during this process but now that it’s working there’s no looking back.