Also See...

You should also check out my non-technology blog at:

http://www.bobsbasement.net/

Some Useful and Obscure FTP Configuration Settings

I get a lot of question about various configuration settings for the IIS FTP service, and most of the settings that I discuss with people can be configured through the FTP features in the IIS Manager. That being said, there are some useful configuration settings for the FTP service which I periodical send to people that have no user interface for setting them. With that in mind, I thought I would write a quick blog to point out a few of these obscure settings that I personally use the most-often or I send to other people.

Using Domain Name Syntax as an FTP Virtual Hostname

I use this setting on all of my FTP servers because it seems a little more natural to me. Here's the scenario: the IIS FTP service supports two kinds of hostnames:

  • "Real" FTP hostnames - these are real FTP hostnames that are specified by using the FTP HOST command (as defined in RFC 7151)
  • "Virtual" ftp hostnames - these are FTP hostnames that are specified as part of the FTP USER command

Real FTP hostnames are pretty straight-forward: an FTP client specifies the hostname with a HOST command when a user is connecting to the server. Once the IIS FTP service receives that command, the FTP service routes the FTP session to the correct FTP site.

That being said, the FTP HOST command is still rather new, so only a handful of FTP clients currently support it. Because of that, you can use FTP "virtual" hostnames with the IIS FTP service. By default that syntax uses the "vertical line" or "pipe" character to differentiate between the hostname and user name. For example:

  • "ftp.contoso.com|username"
  • "ftp.fabrikam.com|username"

When you are specifying your FTP credentials in your FTP client, you would enter your username like the preceding examples. While this syntax is valid for both the IIS FTP service and the underlying FTP protocol, it seems a little odd to most users (including me). With that in mind, we added a configuration setting for the FTP service that will allow you to use the more-familiar domain\username syntax like the following examples:

  • "ftp.contoso.com\username"
  • "ftp.fabrikam.com\username"

To enable this feature, use the following steps:

  1. Open a command prompt as an administrator.
  2. Type the following commands:
    cd /d "%SystemRoot%\System32\Inetsrv"
    appcmd.exe set config -section:system.ftpServer/serverRuntime /hostNameSupport.useDomainNameAsHostName:"True" /commit:apphost
    net.exe stop FTPSVC
    net.exe start FTPSVC
  3. Close the command prompt.

More information about this feature is available in the IIS configuration reference at the following URL:

FTP Credential Caching

The FTP service caches user credentials for successfully-authenticated user sessions in order to help improve login performance, and I wrote the following detailed blog about this a couple of years ago:

Credential Caching in FTP 7.0 and FTP 7.5

I don't want to re-post an old blog, but I have sent several people to that blog over the years, so I thought that it was worth mentioning here since it seems to be referenced quite often. The problem that people seem to run into the most is that their old password is still valid for FTP after they have changed it, and this is caused by the FTP service caching their user credentials.

This is especially annoying for me personally when I am working on a development computer where I am creating an authentication provider. Unless I disable credential caching on my development computer, I can never seem to get any work done. To resolve this issue, I disable credential caching for the FTP service by using the following steps:

  1. Open a command prompt as an administrator.
  2. Type the following commands:
    cd /d "%SystemRoot%\System32\Inetsrv"
    appcmd.exe set config -section:system.ftpServer/caching /credentialsCache.enabled:"False" /commit:apphost
    net.exe stop FTPSVC
    net.exe start FTPSVC
  3. Close the command prompt.

The blog which I mentioned earlier goes into more detail about setting a custom timeout interval for credential caching instead of disabling the feature entirely, and all of the settings for FTP credential caching are in the IIS configuration reference at the following URLs:

FTP Client Certificate Authentication

FTP Client Certificate Authentication is an often-overlooked feature of the IIS FTP service, and I think that this is due to two reasons:

  1. There is no user interface to configure the required settings
  2. Configuring the required settings is very difficult

My second reason cannot be understated; I usually have to set up FTP Client Certificate Authentication once or twice a year in order to test various scenarios, and each time I do so I am reminded of just how difficult it can be to get everything right, and equally how easy it is to get something wrong.

Fortunately I took the time a couple of years ago to write a blog which documents everything that it takes to configure the FTP service, and I have used my notes in that blog several times. In complement to my blog on the subject, Vivek Kumbhar wrote an excellent blog series with additional steps to configure your Active Directory for certificate authentication. With that in mind, here are all of the requisite blog posts that you would need to set up this feature:

As I have mentioned before, configuring this feature is not for the faint-of-heart, but it can be very beneficial from a security standpoint.

For more information about the settings that are required for FTP Client Certificate Authentication, see the following articles in the IIS configuration reference:

That wraps it up for today's post. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

Posted: Jun 30 2014, 16:37 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: Configuration | FTP | IIS
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Simple Utility to Calculate File Hashes

I have to download various files from time-to-time, and it's nice when websites provide checksum hashes so I can validate that the file I just downloaded matches the version on the server. (ON a related note, I wrote a blog several years ago which showed how to create a provider for the IIS FTP service which automatically creates checksum files when files are uploaded to a server; see my Automatically Creating Checksum Files for FTP Uploads blog post for the details.)

In order to calculate hashes for files that I have downloaded, several years ago I wrote a simple command-line application for Windows which uses several of the built-in algorithms in .NET's System.Security.Cryptography. And while I realize that there are probably other tools that provide this same functionality, I have used this little utility for years, and I've had several people ask me for copies. With that in mind, I thought that it might make a nice blog topic if I shared the code with everyone. (Note: It's a really simple sample; the .NET framework does all the real work for this application.)

Without further fanfare, here's the source code. In order to use this code sample, you need to create a new C# project in Visual Studio and choose the Console Application template. When the new project opens, replace the template's code with the following:

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Security.Cryptography;

class Hash
{
    static void Main(string[] args)
    {
        // Verify the correct number of command-line arguments.
        if (args.Length != 2)
        {
            // Show the help message if an incorrect number of arguments was specified.
            ShowHelp();
            return;
        }
        else
        {
            byte[] hashValue = null;
            // Verify that the specified file exists.
            if (!File.Exists(args[1]))
            {
                // Show the help message if a non-existent filename was specified.
                ShowHelp();
                return;
            }
            else
            {
                try
                {
                    // Create a fileStream for the file.
                    FileStream fileStream = File.OpenRead(args[1]);
                    // Be sure it's positioned to the beginning of the stream.
                    fileStream.Position = 0;
                    // Use the specified hash algorithm.
                    switch (args[0].ToUpper())
                    {
                        case "MD5":
                            // Compute the MD5 hash of the fileStream.
                            hashValue = MD5.Create().ComputeHash(fileStream);
                            break;
                        case "SHA1":
                            // Compute the SHA1 hash of the fileStream.
                            hashValue = SHA1.Create().ComputeHash(fileStream);
                            break;
                        case "SHA256":
                            // Compute the SHA256 hash of the fileStream.
                            hashValue = SHA256.Create().ComputeHash(fileStream);
                            break;
                        case "SHA384":
                            // Compute the SHA384 hash of the fileStream.
                            hashValue = SHA384.Create().ComputeHash(fileStream);
                            break;
                        case "SHA512":
                            // Compute the SHA512 hash of the fileStream.
                            hashValue = SHA512.Create().ComputeHash(fileStream);
                            break;
                        case "BASE64":
                            // Compute the BASE64 hash of the fileStream.
                            byte[] binaryData = new Byte[fileStream.Length];
                            long bytesRead = fileStream.Read(binaryData, 0, (int)fileStream.Length);
                            if (bytesRead != fileStream.Length)
                            {
                                throw new Exception(String.Format("Number of bytes read ({0}) does not match file size ({1}).", bytesRead, fileStream.Length));
                            }
                            string base64String = System.Convert.ToBase64String(binaryData, 0, binaryData.Length);
                            Console.WriteLine("File: {0}\r\nBASE64 Hash: {1}", fileStream.Name, base64String);
                            hashValue = null;
                            break;
                        default:
                            // Display the help message if an unrecognized hash algorithm was specified.
                            ShowHelp();
                            return;
                    }
                    if (hashValue != null)
                    {
                        // Write the hash value to the Console.
                        PrintHashData(args[0].ToUpper(), fileStream.Name, hashValue);
                    }
                    // Close the file.
                    fileStream.Close();
                }
                catch (Exception ex)
                {
                    Console.WriteLine("Error: {0}", ex.Message);
                }
            }
        }
    }

    // Display the help message.
    private static void ShowHelp()
    {/>        Console.WriteLine("HASH.exe <hash algorithm> <file name>\n\n" +
            "\tWhere <hash algorithm> is one of the following:\n" +
            "\t\tBASE64\n\t\tMD5\n\t\tSHA1\n\t\tSHA256\n\t\tSHA384\n\t\tSHA512\n");
    }

    // Print the hash data in a readable format.
    private static void PrintHashData(string algorithm, string fileName, byte[] array)
    {
        Console.Write("File: {0}\r\n{1} Hash: ", fileName,algorithm);
        for (int i = 0; i < array.Length; i++)
        {
            Console.Write(String.Format("{0:X2}", array[i]));
        }
        Console.WriteLine();
    }/>}

When you compile and run the application, you will see following help message when you specify no command-line parameters:


HASH.exe <hash algorithm> <file name> Where <hash algorithm> is one of the following: BASE64 MD5 SHA1 SHA256 SHA384 SHA512

When you specify one of the supported hashing algorithms and a filename, the application will display something like the following example:


C:\>hash.exe SHA1 foobar.zip File: C:\foobar.zip SHA1 Hash: 57686F6120447564652C20426F6220526F636B73

That's all there is to it. As I mentioned earlier, it's a pretty simple sample. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: May 15 2014, 19:22 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

What I Do For A Living

It seems that I have always had a difficult time explaining to people what I do at Microsoft. It's not that I'm unsure about what I do - the details of my job have always been crystal-clear to me and I love what I am doing. It's just that I can't find a way to explain things in a way that doesn't result in blank stares from anyone who isn't a geek. (This problem isn't limited to me, though; my non-technical wife simply responds "I have no idea what he does" when someone asks her what I do for a living.)

Here's a perfect example: when I was a Program Manager on the Internet Information Services (IIS) team, people would often ask me what I did for Microsoft, and I would reply with something like, "I help design and implement the web publishing protocols for Microsoft's web server."

Other Person: [Blank Stare]

I would attempt to remedy the situation by adding, "You know, I design Microsoft's implementation of Internet technologies like the File Transfer Protocol, WebDAV, and the FrontPage Server Extensions."

Other Person: [Blank Stare]

In a sometimes-futile effort to salvage the conversation from complete disaster, I would interject, "You like to use the Internet, right? Well, your computer is on one side of the Internet, and my team helps build the other side of the Internet. That's kind of what I do."

That comment would usually be met with a slight spark of recognition, which was sometimes followed by a half-muttered, "That's nice."

At one time or other during my tenure as a Program Manager on the IIS team I was responsible for a smattering of disparate technologies; things like FTP, WebDAV, FPSE, FastCGI, PHP, URL Rewrite, IIS Express, Log Parser, etc. Most of those technologies garnered little to no interest for the average person, and many of my coworkers found them pretty boring as well. Just the same, I personally found every one of those technologies completely fascinating. (Why else would I spend eight years trying to get just one new command added to FTP?)

A couple of years ago I left the IIS program management team and I joined the writing team which is responsible for documenting Microsoft's ASP.NET framework; and if you have to ask what that means, then you are probably not interested in the answer.

Still, people would ask me what I do for Microsoft, and I would try to explain my job with statements like, "I document the Application Programming Interfaces (or APIs) for Microsoft's ASP.NET."

Other Person: [Blank Stare]

I would try to nudge the conversation along by saying things like, "I help people write web code."

Other Person: [Blank Stare]

Skipping ahead in the conversation, I would usually make a last-ditch attempt by stating, "Let's say you wanted to create a website; if so, you might read something that I wrote in order to help you get started."

Sometimes this remark would illicit a hint of acknowledgment, but usually I just got another blank stare.

This leads me to a few days ago. My wife and I were at dinner, and a waiter asked me what I did for a living. In the back of my mind I started to say something like, "Well, these days I'm documenting a set of APIs that Java programmers will use with Microsoft Azure technologies [blah blah blah]..."

But what actually came out of my mouth was, "I could explain it to you, but I'm pretty sure you wouldn't want me to. Trust me."

I like that answer. I think I'll stick with it in the future. :-)

Posted: Apr 28 2014, 23:47 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: Random Thoughts
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

FTP ETW Tracing and IIS 8 - Part 2

Shortly after I published my FTP ETW Tracing and IIS 8 blog post, I was using the batch file from that blog to troubleshoot an issue that I was having with a custom FTP provider. One of the columns which I display in my results is Clock-Time, which is obviously a sequential timestamp that is used to indicate the time and order in which the events occurred.

(Click the following image to view it full-size.)

At first glance the Clock-Time values might appear to be a range of useless numbers, but I use Clock-Time values quite often when I import the data from my ETW traces into something like Excel and I need to sort the data by the various columns.

That being said, apart from keeping the trace events in order, Clock-Time isn't a very user-friendly value. However, LogParser has some great built-in functions for crunching date/time values, so I decided to update the script to take advantage of some LogParser coolness and reformat the Clock-Time value into a human-readable Date/Time value.

My first order of business was to figure out how to decode the Clock-Time value; since Clock-Time increases for each event, it is obviously an offset from some constant, and after a bit of searching I found that the Clock-Time value is the offset in 100-nanosecond intervals since midnight on January 1, 1601. (Windows uses that value in a lot of places, not just ETW.) Once I had that information, it was pretty easy to come up with a LogParser formula to convert the Clock-Time value into the local time for my system, which is much easier to read.

With that in mind, here is the modified batch file:

@echo off

rem ======================================================================

rem Clean up old log files
for %%a in (ETL CSV) do if exist "%~n0.%%a" del "%~n0.%%a"

echo Starting the ETW session for full FTP tracing...
LogMan.exe start "%~n0" -p "IIS: Ftp Server" 255 5 -ets
echo.
echo Now reproduce your problem.
echo.
echo After you have reproduced your issue, hit any key to close the FTP
echo tracing session. Your trace events will be displayed automatically.
echo.
pause>nul

rem ======================================================================

echo.
echo Closing the ETW session for full FTP tracing...
LogMan.exe stop "%~n0" -ets

rem ======================================================================

echo.
echo Parsing the results - this may take a long time depending on the size of the trace...
if exist "%~n0.etl" (
   TraceRpt.exe "%~n0.etl" -o "%~n0.csv" -of CSV
   LogParser.exe "SELECT [Clock-Time], TO_LOCALTIME(ADD(TO_TIMESTAMP('1601-01-01 00:00:00', 'yyyy-MM-dd hh:mm:ss'), TO_TIMESTAMP(DIV([Clock-Time],10000000)))) AS [Date/Time], [Event Name], Type, [User Data] FROM '%~n0.csv'" -i:csv -e 2 -o:DATAGRID -rtp 20
)

When you run this new batch file, it will display an additional "Date/Time" column with a more-informative value in local time for the sever where you captured the trace.

(Click the following image to view it full-size.)

The new Date/Time column is considerably more practical, so I'll probably keep it in the batch file that I use when I am troubleshooting. You will also notice that I kept the original Clock-Time column; I chose to do so because I will undoubtedly continue to use that column for sorting when I import the data into something else, but you can safely remove that column if you would prefer to use only the new Date/Time value.

That wraps it up for today's post. :-)

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Apr 09 2014, 03:17 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

FTP ETW Tracing and IIS 8

In the past I have written a couple of blogs about using the FTP service's Event Tracing for Windows (ETW) features to troubleshoot issues; see FTP and ETW Tracing and Troubleshooting Custom FTP Providers with ETW for details. Those blog posts contain batch files which use the built-in Windows LogMan utility to capture an ETW trace, and they use downloadable LogParser utility to parse the results into human-readable form. I use the batch files from those blogs quite often, and I tend to use them a lot when I am developing custom FTP providers which add new functionality to my FTP servers.

Unfortunately, sometime around the release of Windows 8 and Windows Server 2012 I discovered that the ETW format had changed, and the current version of LogParser (version 2.2) cannot read the new ETW files. When you try to use the batch files from my blog with IIS 8, you see the following errors:

Verifying that LogParser.exe is in the path...
Done.

Starting the ETW session for full FTP tracing...
The command completed successfully.

Now reproduce your problem.

After you have reproduced your issue, hit any key to close the FTP tracing session. Your trace events will be displayed automatically.

Closing the ETW session for full FTP tracing...
The command completed successfully.

Parsing the results - this may take a long time depending on the size of the trace...
Task aborted.
Cannot open <from-entity>: Trace file "C:\temp\ftp.etl" has been created on a OS version (6.3) that is not compatible with the current OS version


Statistics:
-----------
Elements processed: 0
Elements output: 0
Execution time: 0.06 seconds

I meant to research a workaround at the time, but one thing led to another and I simply forgot about doing so. But I needed to use ETW the other day when I was developing something, so that seemed like a good time to quit slacking and come up with an answer. :-)

With that in mind, I came up with a very easy workaround, which I will present here. Once again, this batch file has a requirement on LogParser being installed on your system, but for the sake of brevity I have removed the lines from this version of the batch file which check for LogParser. (You can copy those lines from my previous blog posts if you want that functionality restored.)

Here's the way that this workaround is implemented: instead of creating an ETW log and then parsing it directly with LogParser, this new batch file invokes the built-in Windows TraceRpt command to parse the ETW file and save the results as a CSV file, which is then read by LogParser to view the results in a datagrid like the batch files in my previous blogs:

@echo off

rem ======================================================================

rem Clean up old log files
for %%a in (ETL CSV) do if exist "%~n0.%%a" del "%~n0.%%a"

echo Starting the ETW session for full FTP tracing...
LogMan.exe start "%~n0" -p "IIS: Ftp Server" 255 5 -ets
echo.
echo Now reproduce your problem.
echo.
echo After you have reproduced your issue, hit any key to close the FTP
echo tracing session. Your trace events will be displayed automatically.
echo.
pause>nul

rem ======================================================================

echo.
echo Closing the ETW session for full FTP tracing...
LogMan.exe stop "%~n0" -ets

rem ======================================================================

echo.
echo Parsing the results - this may take a long time depending on the size of the trace...
if exist "%~n0.etl" (
   TraceRpt.exe "%~n0.etl" -o "%~n0.csv" -of CSV
   LogParser.exe "SELECT [Clock-Time], [Event Name], Type, [User Data] FROM '%~n0.csv'" -i:csv -e 2 -o:DATAGRID -rtp 20
)

Here's another great thing about this new batch file - it will also work down-level on Windows 7 and Windows Server 2008; so if you have been using my previous batch files with IIS 7 - you can simply replace your old batch file with this new version. You will see a few differences between the results from my old batch files and this new version, namely that I included a couple of extra columns that I like to use for troubleshooting.

(Click the following image to view it full-size.)

There is one last thing which I would like to mention in closing: I realize that it would be much easier on everyone if Microsoft simply released a new version of LogParser which works with the new ETW format, but unfortunately there are no plans at the moment to release a new version of LogParser. And trust me - I'm just as depressed about that fact as anyone else. :-(

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Apr 08 2014, 08:35 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

FTP Clients - Part 13: WinSCP

For this next installment in my series about FTP clients, I want to take a look at WinSCP, which is an open source FTP/SFTP client that is available from the following URL:

http://www.winscp.net/

For this blog post I used WinSCP 5.5.1, and it was available for free when I wrote this blog post. That being said, WinSCP's author (Martin Prikryl) takes donations. (And I think that it's a worthy cause; I like to support independent development work.)

WinSCP 5.5 Overview

When you open WinSCP 5.5, you will first see the Login dialog box, which will be empty until you add some sites to the list. The Login dialog allows you to create folders so you can categorize your sites, and the user interface is comparable to what you would expect in a Site Manager for other FTP clients.

Fig. 1 - The opening Login dialog in WinSCP 5.5.

When you are adding FTP sites, you have three choices for the protocol: FTP, SCP, and SFTP; you also have four choices for encryption: No encryption, TLS/SSL Implicit encryption, TLS Explicit encryption, and SSL Explicit encryption. (I'll discuss those later.)

When you open a site for which you did not save the password, (which I highly recommend), you will be prompted for your password.

Fig. 2 - The WinSCP 5.5 Password dialog.

Once your FTP site is opened, the main application window is displayed, and it resembles a two-column file explorer interface with local and remote folders, which you might expect from a GUI-based FTP client. (Note: WinSCP refers to this as it's "Commander" interface.)

Fig. 3 - Local and Remote Folders.

That being said, if you change your application preferences, you can change the user interface so that it uses a single-column file explorer interface with a folder tree, which might be useful if you would rather use the FTP client as a drag-and-drop repository. (Note: WinSCP refers to this as it's "Explorer" interface.)

Fig. 4 - Remote Folder Tree and Files.

WinSCP 5.5 has support for automation through .NET and COM, and documentation about automating WinSCP 5.5 programmatically is available on the WinSCP website at the following URL:

WinSCP .NET Assembly and COM Library

There are several detailed automation examples on the WinSCP website that are written in C#, VB.NET, PowerShell, JavaScript, VBScript, etc., and the documentation is quite good. If you need to do a lot of FTP scripting and you are looking for a good way to automate your FTP sessions, you might want to consider this FTP client.

If you don't want to write a bunch of code, you can also automate WinSCP from a command line, and the documentation about that is available on the WinSCP website at the following URL:

WinSCP Command-line Options

Another great feature about WinSCP is that it can be downloaded as portable executables, which makes it easy to copy between systems. This is a great feature for me since I like to keep a collection of handy utilities in my SkyDrive/OneDrive folders.

Using WinSCP 5.5 with FTP over SSL (FTPS)

WinSCP 5.5 has built-in support for FTP over SSL (FTPS), and it supports both Explicit and Implicit FTPS. To specify which type of encryption to use for FTPS, you need to choose the appropriate option from the Encryption drop-down menu for an FTP site.

Fig. 5 - Specifying the FTPS encryption.

Once you have established an FTPS connection through WinSCP 5.5, the user experience is the same as it is for a standard FTP connection. That being said, I could not find a way to drop out of FTPS once a connection is established, so FTPS is an all or nothing option for your sessions.

Using Using WinSCP 5.5 with True FTP Hosts

True FTP hosts are not supported natively, and even though WinSCP 5.5 allows you to send post-login commands after an FTP site has been opened, I could not find a way to send a custom command before sending user credentials, so true FTP hosts cannot be used.

Using Using WinSCP 5.5 with Virtual FTP Hosts

WinSCP 5.5's login settings allow you to specify the virtual host name as part of the user credentials by using syntax like "ftp.example.com|username" or "ftp.example.com\username", so you can use virtual FTP hosts with WinSCP 5.5.

Fig. 6 - Specifying an FTP virtual host.

Scorecard for WinSCP 5.5

This concludes my quick look at a few of the FTP features that are available with WinSCP 5.5, and here are the scorecard results:

Client
Name
Directory
Browsing
Explicit
FTPS
Implicit
FTPS
Virtual
Hosts
True
HOSTs
Site
Manager
Extensibility
WinSCP 5.5.1 Rich Y Y Y N Y N/A
Note: I could not find anyway to extend the functionality of WinSCP 5.5; but as I said
earlier, it provides rich automation features for .NET, COM, and the command-line.

That wraps things up for today's blog. Your key take-aways should be: WinSCP 5.5 is good FTP client with a lot of options, and it has a very powerful automation story. As I mentioned earlier, if you have to do a lot of FTP automation, you should really take a look at this FTP client.

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Feb 28 2014, 15:57 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: FTP | SSL
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Error 0x80070005 When Calling the FTP FlushLog Method

I had an interesting question earlier today which I thought was worth sharing. One of my coworkers was trying to use the code sample from my Programmatically Flushing FTP Logs blog, and he was getting the following error:

Unhandled Exception: System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
   at Microsoft.Web.Administration.Interop.IAppHostMethodInstance.Execute()
   at Sample.Main() in c:\Projects\FtpTests\Program.cs:line 25

I knew that the code sample in my blog worked perfectly when I originally wrote it, so I figured that my coworker must be doing something wrong. (And every developer has said "It works on my computer..." at one time or other.) But I decided to give him the benefit of the doubt, so I copied the source code from my blog into a new Visual Studio project and I ran it.

Much to my surprise, I saw the same error that my coworker was seeing if I didn't step the code through with a debugger.

When I stepped through the code in a debugger, I saw the following error message:

At this point I was thinking, "What the heck? I know this code was working before..." I started to wonder if we had released a breaking change to the FTP service sometime during the past two years, but then it suddenly dawned on me: I hadn't started the FTP service on my computer.

[Duh.]

That was the source of the problem: I usually have the FTP service configured for manual startup on my development computers, but the FTP methods to start and stop FTP sites and flush the FTP logs do not work when the FTP service is not running. Once both of us started the FTP service on each of our systems the problem went away.

I hope this helps. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Jan 31 2014, 06:59 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: FTP | Extensibility
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Custom Post-Build Events for Compiling FTP Providers

I've written a lot of walkthroughs and blog posts about creating custom FTP providers over the past several years, and I usually include instructions like the following example for adding a custom post-build event that will automatically register your extensibility provider in the Global Assembly Cache (GAC) on your development computer:

  • Click Project, and then click the menu item your project's properties.
  • Click the Build Events tab.
  • Enter the following in the Post-build event command line dialog box:
    net stop ftpsvc
    call "%VS100COMNTOOLS%\vsvars32.bat">nul
    gacutil.exe /if "$(TargetPath)"
    net start ftpsvc

And I usually include instructions like the following example for determining the assembly information for your extensibility provider:

  • In Windows Explorer, open your "C:\Windows\assembly" path, where C: is your operating system drive.
  • Locate the FtpXmlAuthorization assembly.
  • Right-click the assembly, and then click Properties.
  • Copy the Culture value; for example: Neutral.
  • Copy the Version number; for example: 1.0.0.0.
  • Copy the Public Key Token value; for example: 426f62526f636b73.
  • Click Cancel.

Over time I have changed the custom post-build event that I use when I am creating custom FTP providers, and my changes make it easier to register custom FTP providers. With that in mind, I thought that my changes would make a good blog subject.

First of all, if you take a look at my How to Use Managed Code (C#) to Create a Simple FTP Authentication Provider walkthrough, you will see that I include instructions like my earlier examples to create a custom post-build event and retrieve the assembly information for your extensibility provider.

That being said, instead of using the custom post-build event in that walkthrough, I have started using the following custom post-build event:

net stop ftpsvc
call "$(DevEnvDir)..\Tools\vsvars32.bat"
gacutil.exe /uf "$(TargetName)"
gacutil.exe /if "$(TargetPath)"
gacutil.exe /l "$(TargetName)"
net start ftpsvc

This script should resemble the following example when entered into Visual Studio:

This updated script performs the following actions:

  1. Stops the FTP service (this will allow any copies of your DLL to unload)
  2. Loads the Visual Studio environment variables (this will add gacutil.exe to the path)
  3. Calls gacutil.exe to forcibly unregister any previous version of your FTP provider
  4. Calls gacutil.exe to forcibly register the newly-compiled version of your FTP provider
  5. Calls gacutil.exe to list the GAC information for your FTP provider (this will be used to register your DLL with IIS)
  6. Starts the FTP service

Let's say that you created a simple FTP authentication provider which contained code like the following example:

using System;
using System.Text;
using Microsoft.Web.FtpServer;

public class FtpTestProvider :
    BaseProvider,
    IFtpAuthenticationProvider
{
    private string _username = "test";
    private string _password = "password";
    
    public bool AuthenticateUser(
        string sessionId,
        string siteName,
        string userName,
        string userPassword,
        out string canonicalUserName)
    {
        canonicalUserName = userName;
        if (((userName.Equals(_username,
            StringComparison.OrdinalIgnoreCase)) == true) &&
            userPassword == _password)
        {
            return true;
        }
        else
        {
            return false;
        }
    }
}

When you compile your provider in Visual Studio, the output window should show the results of the custom post-build event:

When you examine the output information in detail, the highlighted area in the example below should be of particular interest, because it contains the assembly information for your extensibility provider:

------ Rebuild All started: Project: FtpTestProvider, Configuration: Debug Any CPU ------
FtpTestProvider -> c:\users\foobar\documents\visual studio 2012\Projects\FtpTestProvider\bin\Debug\FtpTestProvider.dll
The Microsoft FTP Service service is stopping..
The Microsoft FTP Service service was stopped successfully.

Microsoft (R) .NET Global Assembly Cache Utility. Version 4.0.30319.17929
Copyright (c) Microsoft Corporation. All rights reserved.

Assembly successfully added to the cache
Microsoft (R) .NET Global Assembly Cache Utility. Version 4.0.30319.17929
Copyright (c) Microsoft Corporation. All rights reserved.

The Global Assembly Cache contains the following assemblies:
FtpTestProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=eb763c2ec0efff75, processorArchitecture=MSIL

Number of items = 1
The Microsoft FTP Service service is starting.
The Microsoft FTP Service service was started successfully.

========== Rebuild All: 1 succeeded, 0 failed, 0 skipped ==========

Once you have that information, you simply need to reformat it as "FtpTestProvider, FtpTestProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=eb763c2ec0efff75" in order to enter it into the FTP Custom Authentication Providers dialog box in the IIS Manager, or by following the steps in my FTP Walkthroughs or my Adding Custom FTP Providers with the IIS Configuration Editor blogs.

That wraps it up for today's post. As always, let me know if you have any questions. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Oct 26 2013, 03:53 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

A Quick Discussion of Active and Passive FTP Connections

I get a lot of questions about Active versus Passive FTP Connections, specifically when people are configuring their FTP firewall settings as described in my articles like Configuring FTP Firewall Settings in IIS and FTP Firewall Support, and I get related questions when people are trying to figure out why they can't use the command-line FTP.EXE utility that ships with Windows over the Internet. With all of this in mind, I thought that I would put together a quick blog that explains Active and Passive FTP connections and why those matter when you are connecting to an FTP server over the Internet.

Here is the briefest description that I can think of to describe the different between Active and Passive FTP:

  • When you are using Active FTP, your FTP client determines how your data connections will work.
  • When you are using Passive FTP, your FTP server determines how your data connections will work.

That sums up the whole concept into two bullet points. Unfortunately, neither of those bullet points are the least bit significant to you if you don't understand what they actually mean, so I will describe them in detail below.

How Active FTP Works

The following output shows the communication between an FTP client and FTP server using Active FTP to retrieve a simple directory listing, and I manually color-coded the output so that the client and server's responses would be a little easier to see:

OPEN ftp.contoso.com
Resolving ftp.contoso.com...
Connect socket #1920 to 169.254.10.1, port 21...
220 Microsoft FTP Service
HOST ftp.contoso.com
220 Host accepted.
USER robert
331 Password required for robert.
PASS **********
230 User logged in.
PWD
257 "/" is current directory.
PORT 169,254,10,2,72,50
200 PORT command successful.
LIST
125 Data connection already open; Transfer starting.
drwxrwxrwx 1 owner group 0 Feb 15 19:26 aspnet_client
-rwxrwxrwx 1 owner group 689 Jan 31 22:27 default.htm
226 Transfer complete.
Transferred 106 bytes in 0.008 seconds
QUIT
221 Goodbye.

In the beginning of this exchange, the client connects to the server from one of its ephemeral data ports to the server's port for the FTP command channel. After negotiating the FTP host, username, and password, the client retrieves the name of the current directory. So far all of the client/server communication has taken place over the FTP command channel, and up to now the conversation is identical to Passive FTP, but that is about to change.

The client's next task is to request a directory listing, which is denoted by the LIST command. The server will return its response to the client over the data channel, so before FTP client can send the LIST command, the client has to specify whether to use Active or Passive. In this example, the client has specified Active FTP by sending a PORT command. The syntax for this command is PORT A1,A2,A3,A4,P1,P2, where A1 through A4 are octets of the client's IPv4 address, and P1/P2 are two bytes that make up a 16-bit (0-65535) port address on the client. (Note: if you are using IPv6, there is a similar EPRT command that works with IPv6 addresses.)

Here's what the information in the PORT command means: the FTP client is essentially telling the FTP server, "For the upcoming data transfer, you need to talk to me at this IP address on this port." This means that the FTP client is actively in control of how the subsequent data communication is going to take place.

If we analyze this information, you can easily see why Active FTP will often fail to work over the Internet. As a relevant example, if you were to use the FTP.EXE client that ships with Windows, it can only use Active FTP. So when a client computer requests something from the server that needs to use the data channel, the client computer sends its IP address via a PORT command. If the FTP client is behind a firewall or NAT server, then the client is going to send its internal, LAN-based address, to which the FTP server will more than likely fail to connect. For example, if you are on a LAN that uses a NAT server and you have a 192.168.0.nnn IPv4 address, that IP address is invalid over the Internet, so the server will never be able to establish a data connection to your client to send the data. (Note: This is the reason why many customers contact me with the following problem description: "I can use FTP.EXE to connect to my server, and everything works until I try to retrieve a directory listing, then it hangs until the connection times out." What is actually happening is the FTP server is trying to connect to the FTP client's IP address and port that were specified by the PORT command, but the connection does not succeed because the server cannot connect to the private IP address of the client.)

How Passive FTP Works

The following output shows the communication between an FTP client and FTP server using Passive FTP to retrieve the same directory listing as my previous example, and once again I manually color-coded the output so that the client and server's responses would be a little easier to see:

OPEN ftp.contoso.com
Resolving ftp.contoso.com...
Connect socket #2076 to 169.254.10.1, port 21...
220 Microsoft FTP Service
HOST ftp.contoso.com
220 Host accepted.
USER robert
331 Password required for robert.
PASS **********
230 User logged in.
PWD 
257 "/" is current directory.
PASV
227 Entering Passive Mode (169,254,10,1,197,19).
LIST
Connect socket #2104 to 169.254.10.1, port 50451...
150 Opening ASCII mode data connection.
drwxrwxrwx 1 owner group 0 Feb 15 19:26 aspnet_client
-rwxrwxrwx 1 owner group 689 Jan 31 22:27 default.htm
226 Transfer complete.
Transferred 106 bytes in 0.008 seconds
QUIT
221 Goodbye.

As I mentioned in my earlier example, the beginning of this conversation is identical to Active FTP: the client connects from one of its ephemeral data ports to the server's port for the FTP command channel. After negotiating the FTP host, username, and password, the client retrieves the name of the current directory as in my earlier example - and here's where the difference begins.

Once again the client's next task is to request the directory listing, which is still denoted by the LIST command. But in this second example, the client has specified Passive FTP by sending a PASV command. The server responds to this command with a reply that is in the format of 227 Entering Passive Mode (A1,A2,A3,A4,P1,P2), where A1 through A4 are octets of the server's IPv4 address, and P1/P2 are two bytes that make up a 16-bit (0-65535) port address on the server. (Note: if you are using IPv6, there is a similar EPSV command that works with IPv6 addresses.)

Here's what the information in the response to the PASV command means: the FTP client is essentially telling the FTP server, "For the upcoming data transfer, you need to tell me which IP address and port I should use to talk to you." This means that the FTP client is passively allowing the server to control how the subsequent data communication is going to take place.

If we analyze this information, you can easily see why Passive FTP often works over the Internet; when the FTP server is in control of the communication parameters, it doesn't matter whether the FTP client is behind a NAT server, because the server is telling the client how it should communicate with the server.

All of this leads to an obvious question: what happens when both the server and the client are behind NAT servers or firewalls? This is where a little bit of configuration comes into play. If you read my Configuring FTP Firewall Settings in IIS and FTP Firewall Support articles, you would notice that you can configure the IIS FTP service to tell the FTP client which IP address to use; when your FTP server is located behind a firewall, you would configure your FTP server to send the external IP address of your firewall, and then you would configure your firewall to route FTP requests on that IP address to your FTP server. Since the FTP server is sending the external IP address of your firewall, the client knows how to communicate to the FTP server even though it is behind a firewall, even if your server is using a LAN-based internal IP address.

In Closing...

Having explained everything in my preceding examples, you should now understand what I meant earlier when I described the difference between Active and Passive FTP with these two simple points:

  • When you are using Active FTP, your FTP client determines how your data connections will work.
  • When you are using Passive FTP, your FTP server determines how your data connections will work.

I hope this clears up some questions you might have about Active versus Passive FTP, why you sometimes need to configure your firewall settings for your FTP service, and why the built-in FTP client for Windows seldom works over the Internet.

That wraps it up for today's blog post. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: May 24 2013, 16:29 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: IIS | FTP
Tags: ,
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Adding Custom FTP Providers with the IIS Configuration Editor - Part 2

In Part 1 of this blog series about adding custom FTP providers with the IIS Configuration Editor, I showed you how to add a custom FTP provider with a custom setting for the provider that is stored in your IIS configuration settings. For my examples, I showed how to do this by using both the AppCmd.exe application from a command line and by using the IIS Configuration Editor. In part 2 of this blog series, I will show you how to use the IIS Configuration Editor to add custom FTP providers to your FTP sites.

As a brief review from Part 1, the following XML excerpt illustrates what the provider's settings should resemble when added to your IIS settings:

<system.ftpServer>
  <providerDefinitions>
    <add name="FtpXmlAuthorization"
    
 type="FtpXmlAuthorization, FtpXmlAuthorization, version=1.0.0.0, Culture=neutral, PublicKeyToken=426f62526f636b73" />
    <activation>
      <providerData name="FtpXmlAuthorization">
        <add key="xmlFileName"
        
 value="C:\inetpub\FtpUsers\Users.xml" />
      </providerData>
    </activation>
  </providerDefinitions>
</system.ftpServer>

The above example shows the settings that are added globally to register an FTP provider. Note that this example only contains the settings for my custom provider; you would normally see the settings for the IisManagerAuth and AspNetAuth providers that ship with the FTP service in the providerDefinitions collection.

To actually use a provider for an FTP site, you would need to add the provider to the settings for the FTP site in your IIS settings. So for part 2 of this blog series, we will focus on how to add a custom provider to an FTP site by using the IIS Configuration Editor.

Having said all of that, the rest of this blog is broken down into the following sections:

Before continuing, I should reiterate that custom FTP providers fall into two categories: providers that are used for authentication and providers that are used for everything else. This distinction is important, because the settings are stored in different sections of your IIS settings. With that in mind, let's take a look at the settings for an example FTP site.

Step 1 - Looking at the configuration settings for custom FTP providers

The following example shows an FTP site with several custom FTP providers added:

<site name="ftp.contoso.com" id="2">
  <application path="/">
    <virtualDirectory path="/"
    
 physicalPath="c:\inetpub\www.contoso.com\wwwroot" />
  </application>
  <bindings>
    <binding protocol="ftp"
    
 bindingInformation="*:21:ftp.contoso.com" />
  </bindings>
  <ftpServer>
    <security>
      <ssl controlChannelPolicy="SslAllow"
      
 dataChannelPolicy="SslAllow" />
      <authentication>
        <customAuthentication>
          <providers>
            <add name="MyCustomFtpAuthenticationProvider" />
          </providers>
        </customAuthentication>
      </authentication>
    </security>
    <customFeatures>
      <providers>
        <add name="MyCustomFtpHomeDirectoryProvider" />
        <add name="MyCustomFtpLoggingProvider" />
      </providers>
    </customFeatures>
    <userIsolation mode="Custom" />
  </ftpServer>
</site>

If you look at the above example, you will notice the following providers have been added:

  • A custom FTP authentication provider named MyCustomFtpAuthenticationProvider has been added to the ftpServer/security/authentication/customAuthentication/providers collection; this provider will obviously be used by the FTP service to validate usernames and passwords.
  • A custom FTP home directory provider named MyCustomFtpHomeDirectoryProvider has been added to the ftpServer/customFeatures/providers collection; this will be used by the FTP service for custom user isolation. Note the mode for the userIsolation element is set to custom.
  • A custom FTP logging provider named MyCustomFtpLoggingProvider has been added to the ftpServer/customFeatures/providers collection; this will be used by the FTP service for creating custom log files.

As I mentioned earlier, you will notice that the settings for FTP custom providers are stored in different sections of the ftpServer collection depending on whether they are used for authentication or some other purpose.

Step 2 - Navigate to an FTP Site in the Configuration Editor

Open the IIS Manager and click on the Configuration Editor at feature the server level:

Click the Section drop-down menu, expand the the system.applicationHost collection, and then highlight the sites node:

If you click on the Collection row, an ellipsis [...] will appear:

When you click the ellipsis [...], IIS will display the Collection Editor dialog box for your sites; both HTTP and FTP sites will be displayed:

Expand the ftpServer node, which is where all of the site-level settings for an FTP site are kept.

Step 3 - Add custom FTP providers to an FTP site

As I mentioned earlier, custom FTP providers fall into two categories: providers that are used for authentication and everything else. Because of this distinction, the following steps show you how to add a provider to the correct section of your settings depending on the provider's purpose.

Add a custom FTP provider to an FTP site that is not used for authentication

Expand the customFeatures node, which is located under the ftpServer node for an FTP site; this collection defines the custom providers for an FTP site that are not used for authentication, for example: home directory providers, logging providers, etc. When you highlight the providers row, an ellipsis [...] will appear:

When you click the ellipsis [...], IIS will display the Collection Editor dialog box for your custom features (providers). When you click Add in the Actions pane, you need to enter the name of an FTP provider that you entered by following the instructions in Part 1 of this blog series:

Once you enter the name of your FTP provider in the Collection Editor dialog box for your custom features, you can close that dialog box. The Collection Editor for your sites will reflect the updated provider count for your FTP site:

Important Note: If you are adding a custom FTP Home Directory Provider, you have to configure the mode for FTP's User Isolation features. To do so, you need to expand the userIsolation node, which is located under the ftpServer node for an FTP site. Once you have done so, click the mode drop-down menu and choose Custom from the list of choices:

When you close the Collection Editor dialog box for your sites, you need to click Apply in the Actions pane to commit the changes to your IIS settings:

Add a custom FTP authentication provider to an FTP site

First and foremost - there is built-in support for adding custom authentication providers in IIS Manager; to see the steps to do so, see the FTP Custom Authentication <customAuthentication> article on the IIS.NET website. However, if you want to add a custom FTP authentication provider to an FTP site by using the IIS Configuration Editor, you can do so by using the following steps.

Expand the security node under the ftpServer node for an FTP site, then expand the authentication node, and then expand the customAuthentication node; this collection defines the custom authentication providers for an FTP site. When you highlight the providers row, an ellipsis [...] will appear:

When you click the ellipsis [...], IIS will display the Collection Editor dialog box for your custom authentication providers. When you click Add in the Actions pane, you need to enter the name of an FTP authentication provider that you entered by following the instructions in Part 1 of this blog series:

Once you enter the name of your FTP authentication provider in the Collection Editor dialog box for your custom authentication providers, you can close that dialog box. The Collection Editor for your sites will reflect the updated authentication provider count for your FTP site:

When you close the Collection Editor dialog box for your sites, you need to click Apply in the Actions pane to commit the changes to your IIS settings:

Summary and Parting Thoughts

As I mentioned in part 1 of this series, I admit that this might seem like a lot of steps to go through, but it's not that difficult once you understand how the configuration settings are organized and you get the hang of using the IIS Configuration Editor to add or modify these settings.

Disabling Custom User Isolation

In the Add a custom FTP provider to an FTP site that is not used for authentication section of this blog, I added a step to specify Custom as the User Isolation mode. Since this is something of an advanced feature, there is no user interface for enabling custom user isolation; this was a design decision to keep people from breaking their FTP sites. Here's why: if you enable custom user isolation and you don't install a custom Home Directory provider for FTP, all users will be denied access to your FTP site.

That being said, once you have enabled custom user isolation, the option to disable custom user isolation will "magically" appear in the FTP User Isolation feature in the IIS Manager. To see this for yourself, you would first need to follow the steps to custom user isolation in the Add a custom FTP provider to an FTP site that is not used for authentication section of this blog.

Once you have enabled custom user isolation, highlight your FTP site in the list of Sites pane of IIS Manager, then open the FTP User Isolation feature:

When you open the FTP User Isolation feature, you will see that an option for Custom now appears in the list of user isolation modes:

This option will appear as long as custom user isolation is enabled. If you change the user isolation mode to something other than Custom, this option will continue appear in the list of user isolation modes until you navigate somewhere else in IIS Manager. Once you have changed the user isolation mode to one of the built-in modes and you navigate somewhere else, the Custom option will not show up in the FTP User Isolation feature until you follow the steps to re-enable custom user isolation.

Additional Information

If you want additional information about configuring the settings for FTP providers, you can find detailed reference documentation at the following URLs:

Each of these articles contain "How-To" steps, detailed information about each of the configuration settings, and code samples for AppCmd.exe, C#/VB.NET, and JavaScript/VBScript.

As always, let me know if you have any questions. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: May 02 2013, 11:31 by Bob | Comments (0) RSS comment feed |
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: IIS | FTP | Extensibility
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us