Web Services: Automated feature testing using Specflow and Babelfish

example project at github

I recently started implementing BDD using Specflow for one of our customers.

Basically Specflow is a BDD framework written in .NET inspired by Cucumber and the tests are written in the same language (Gherkin)

One of the things we would like to include in this are the testing of soap service endpoints.

If you are a .NET developer you probably know all about the generation of proxy classes by using the wizard in visual studio or invoking wsdl.exe on the wsdl source file. This could be a problem and result in needless work, so I looked into compiling the proxy classes in memory during runtime.

Authentication

There exist a number of free and open soap web services out there, some that are used in my examples are taken from www.webservicex.net.

But of course we do have services that require some form of authentication. The following snippet illustrates how to perform forms authentication using the System.Net.WebClient and Babelfish.

public string FormsAuth(Uri url, string username, string password)
{
    var response = DownloadData(url);

    INode document = new Babelfish.HTML.HTMLDocument(Encoding.ASCII.GetString(response));

    var postData = string.Empty;

    var formElements = document.Find(n => n.Name.ToLower() == "input" && new[] { "hidden", "button", "submit", "checkbox", "radio", "text", "password" }
    .Contains(n.Attributes["type"]));

    foreach (var formElement in formElements)
    {
        postData += postData.Length > 0 ? "&" : string.Empty;

        var name = formElement.Attributes["name"];
        var value = formElement.Attributes["value"];

        switch (formElement.Attributes["type"])
        {
            case "text":
                postData += string.Format("{0}={1}", name, HttpUtility.UrlEncode(username));
                break;
            case "password":
                postData += string.Format("{0}={1}", name, HttpUtility.UrlEncode(password));
                break;
            default:
                if (!string.IsNullOrEmpty(value))
                    postData += string.Format("{0}={1}", name, HttpUtility.UrlEncode(value));
                break;
        }
    }

    Headers.Add("Content-Type", "application/x-www-form-urlencoded");
    string result;
    try
    {
        result = Encoding.ASCII.GetString(UploadData(url, "POST", Encoding.ASCII.GetBytes(postData)));
    }
    catch (WebException ex)
    {
        result = ex.ToString();
    }
    return result;
}

Web Service instantiation

Now when we have access to the wsdl definition via our modified WebClient we can use that to compile a SoapHttpClientProtocol object.

The following snippet is from the class ArbitraryWebservice and shows how to compile a wsdl stream into such an object given a valid type.

private static SoapHttpClientProtocol GenerateService(Stream stream, string serviceName)
{
    var serviceDescription = ServiceDescription.Read(stream);
    var serviceDescriptionImporter = new ServiceDescriptionImporter
    {
        ProtocolName = "Soap12",
        Style = ServiceDescriptionImportStyle.Client,
        CodeGenerationOptions = CodeGenerationOptions.GenerateProperties
    };

    serviceDescriptionImporter.AddServiceDescription(serviceDescription, null, null);

    var codeNamespace = new CodeNamespace();
    var codeCompileUnit = new CodeCompileUnit();

    codeCompileUnit.Namespaces.Add(codeNamespace);

    var warnings = serviceDescriptionImporter.Import(codeNamespace, codeCompileUnit);

    if (warnings > 0)
        throw new Exception(string.Format("{0}", warnings));

    using (var codeDomProvider = CodeDomProvider.CreateProvider("C#"))
    {
        var compilerParameters = new CompilerParameters(new[] { "System.dll", "System.Web.Services.dll", "System.Web.dll", "System.Xml.dll", "System.Data.dll" }) { GenerateInMemory = true };

        var results = codeDomProvider.CompileAssemblyFromDom(compilerParameters, codeCompileUnit);

        if (results.Errors.Count > 0)
            throw new Exception(string.Join(", ", results.Errors.Cast<CompilerError>().Select(e => e.ToString()).ToArray()));

        var service = results.CompiledAssembly.CreateInstance(serviceName);

        return service as SoapHttpClientProtocol;
    }
}

Method invocation

The following snippet is also from the class ArbitraryWebservice.

Basically it attempts to map the given methodName to a method in the instantated service object. Then it iterates trough the parameters for that method and match each one to one of the elements in the args array.

The last part is to cast/convert complex parameter types to the correct type using a extension method (basically JSON serialization/deserialization).

public object Call(string methodName, MethodParamDescriptor[] args)
{
    var method = _service.GetType().GetMethod(methodName);

    if (method == null)
        throw new MissingMethodException(string.Format("this service contains no method named {0}", methodName));

    var methodParams = new List<object>();

    foreach (var p in method.GetParameters())
    {
        if(!args.Any(a => a.Name == p.Name))
            throw new ArgumentException("expected parameter not supplied", p.Name);

        methodParams.Add(args.Where(a => a.Name == p.Name)
            .Select(a => a.Value.Convert(p.ParameterType))
            .FirstOrDefault());
    }

    return method.Invoke(_service, methodParams.ToArray());
}

Specflow

Finally with the above in place we can write feature descriptions using Specflow and execute them as a normal unit test.

The complex object comparisons are done with the help of a Babelfish extension method

You can compile and run these tests yourself by downloading the solution from here.

Feature: Webservices
	In order to know what is going on
	As a developer
	I want to instantiate any arbitrary web service and invoke availible methods and assert their return values

@mytag
Scenario: Weather
	Given webservice named Weather with the wsdl url http://ws.cdyne.com/WeatherWS/Weather.asmx?wsdl
	When I call the method GetCityForecastByZIP with the parameter ZIP=36003
	Then I expect it to return an object matching content {City: 'Autaugaville', WeatherStationCity: 'Selma'}

@mytag
Scenario: Speed
	Given webservice named ConvertSpeeds with the wsdl url http://www.webservicex.net/ConvertSpeed.asmx?WSDL
	When I call the method ConvertSpeed with the parameters speed=100 and FromUnit=centimetersPersecond and ToUnit=feetPersecond
	Then I expect it to return the number 3.2808398950131235

@mytag
Scenario: Whois
	Given webservice named whois with the wsdl url http://www.webservicex.net/whois.asmx?WSDL
	When I call the method GetWhoIS with the parameters HostName=google.com
	Then I expect it to return a string containing TERMS OF USE: You are not authorized to access or query our Whois

@mytag
Scenario: country
	Given webservice named country with the wsdl url http://www.webservicex.net/country.asmx?WSDL
	When I call the method GetCountries with no parameters
	Then I expect it to return html containing <NewDataSet><Table><Name>Kiribati</Name></Table><Table><Name>Cocos (Keeling) Islands</Name></Table></NewDataSet>

@mytag
Scenario: GeoIPService
	Given webservice named GeoIPService with the wsdl url http://www.webservicex.net/geoipservice.asmx?WSDL
	When I call the method GetGeoIP with the parameters IPAddress='209.85.148.147'
	Then I expect it to return an object matching content {IP:'209.85.148.147', CountryName:'United States', CountryCode:'USA'}

@mytag
Scenario: BarCode
	Given webservice named BarCode with the wsdl url http://www.webservicex.net/genericbarcode.asmx?WSDL
	When I call the method GenerateBarCode with the parameters BarCodeText=123456 and BarCodeParam={Height:1,Width:1,Ratio:5,FontName:'Arial',FontSize:10,BarCodeImageFormat: 'PNG'}
	Then I expect it to return a base64 string matching iVBORw0KGgoAAAANSUhEU...AAAAAAAAAAAAAAAAAAAAAAAAAAAAA==

Ze-Chat using Websockets and GO!

I decided to do something, well different. So I ended up with this redistributable online chat based on Websockets.

The client is of course just HTML and Javascript but I decided to try and write the server software using GO

The result is a basic but totaly redistributable chat, you don’t even need to have a webserver installed. Just compile and go!

The server is initially set to listen to port 8181 as you can see above, just browse to http://localhost:8181 using any browser supporting Websockets (like Chrome and Firefox 4 Beta)

If you wanna try this you can download the zipped source below, you will need to have Go installed for either Linux or Windows depending on what you are.. running, duh

Just download the source and unzip, if you are using GNU/Linux you can build and start the server using either the build or buildx64 scripts.

Windows:

  1. Download and unzip http://code.google.com/p/gomingw/downloads/list to e.g C:/Tools/go
  2. Add C:/Tools/go/bin to your PATH
  3. Add new Environment Variable GOROOT and set it to C:/Tools/go/
  4. Add new Environment Variable GOBIN and set it to C:/Tools/go/bin
  5. Download the ze-chat source and unzip it somewhere
  6. open a new command prompt and navigate to that directory
  7. type and execute ’8g server.go’
  8. type and execute ’8l -o server.exe server.8′
  9. type and execute ‘server.exe 8181′ , replace 8181 with any other port you would like
  10. done, use either Chrome or Firefox 4 Beta, open a tab/window, type in the address “http://localhost:8181″

Explorer.exe says “no such interface supported”

When I started up my laptop this morning at the office I noticed my 2nd monitor did not display my extended desktop.

I run Windows 7 so the first thing I attempted was to right click my desktop and select “screen resolution”

Immediately I was prompted with the following dialog box.

So.. crap! my windows installation seemed to be fubar, how do I fix this without reinstalling everything “which is Microsoft support first answer to everything”

How to fix this?

I tried lots of stuff like running “svc /scannow” that did find that tcpmon.ini was corrupted but it was unable to fix it because the backup copy was also corrupted hmm.

Maybe rebooting with the windows 7 installation DVD, repair and run “svc /scannow” from there and restore the file fresh from the DVD? nope, it would not even run because the initial run that failed from windows placed some ‘will repair at next boot’ flag. of course the repair at boot fails so the flag remains, moment 22

The solution

After searching the web for a while I found this thread post where the solution was to re-register every .dll on the system partition.

All that was needed was to start a console as admin (start->type ‘cmd’->hold CTRL+SHIFT and press enter)

Paste the line below and press enter

FOR /R C:\ %G IN (*.dll) DO “%systemroot%\system32\regsvr32.exe” /s “%G”

I recommend setting the “Always combine, hide labels” option in the “Taskbar and Start Menu Properties” if you do not allready have this by default (I didn’t)

Because running the above command will cause hundreds of alert/error boxes to pop up which you can ignore/close but it is easier to close them all grouped.

This seemingly fixed my problem, rebooted just to verify and and yup,  fixed

Displacement Field – more fun with javascript and canvas

Just another fun/play/experiment with javascript, canvas and some number crunching.

http://www.creamdog.se/DisplacementField/

JSCloth – fun with javascript and canvas

Just wanna post a link to my latest experiment, simulating cloth/fabric using javascript and canvas.

Obviously you need a browser that supports the <canvas> element, should work fine in Chrome, Firefox, Safari or Opera. I do not recall at the moment if IE9 supports it, but I think so.

Anyhow,

It is pretty slow in Firefox 3.6.10 (15 fps textured, 30 fps wireframe)

It is faster in Firefox 4 Beta, but just barely (18-20 fps textured, 35-40 fps wireframe)

Chrome 6.0.472.63 is the fastest browser (50 fps textured, 80 fps wireframe)

http://www.creamdog.se/JSCloth/

Secret plans

Andrey stumbled upon my secret factory in Rotebro, plans for world domination foiled once again! I will get you yet Zhukov!

Visual Studio copy, paste & freeze

The problem

After I installed Visual Studio 2010 I started experiencing that the IDE would freeze for 20+ seconds after I pasted a code snipped in markup edit mode. I had the same issue on my home computer also running Visual Studio 2010.

As you may suspect this became a bit frustrating quite fast.

Fumbling in the dark

I found some MS post suggesting re-installing Visual Studio / Windows or whatever but really, who would do that, really?

Also some MS people suggested that the issue may be not enough RAM for the copy paste operation. Since I have 4GB on both computers and monitoring the memory usage during copy paste at ~2GB it did not really feel like the culprit in this case.

I did find some promising theories out there, like add an exception to your antivirus not to scan your project files – did not work.

Maybe uninstalling Microsoft Visual Studio Web Authoring Component? Seemed like a good idea in general since its just some Frontpage crap – but didn’t help much.

I did find a hotfix from Microsoft with the description (Patch Available for Cut or Copy displaying “insufficient memory” error in VS 2010) not quite my scenario but hey, they seemed related so I installed it, took a long time - did nothing.

The solution

By this time I was quite fed up with this shit, I could not get any real work done with Visual Studio behaving like this. So I started looking at warnings and stuff and did find that there was a typo in the web.config in a sectiongroup.

<configuration>
  <configSections>
    <sectionGroup name="system.web.extensions" type="System.Web.Configuration.SystemWebExtensionsSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35">
      <sectionGroup name="scripting" type="System.Web.Configuration.ScrpingSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35">
       ...
      </sectionGroup>
    </sectionGroup>
  </configSections>
</configuration>

The sectionGroup “scripting” had a typo in in its type attribute, “ScrpingSectionGroup” instead of “ScriptingSectionGroup”.

After I had corrected this everything worked fine.

I had no idea that a typo in web.config could screw up the IDE so bad. Keep in mind that even that web.config had this typo, the project compiled and ran just fine.

Sigh

PyBlocks

Some time ago I had the need to generate order confirmation emails using templates populated with order data such as ordered items, pricing and customer information.

One way is of course to serialize the order to XML and transform it via XSL but I had an urge to try something new.

So I decided to use IronPython and parse templates with {py py} blocks.

You can download the example PyBlocks solution here

Example PyBlocks template

Hello {py Console.Write(customer.Givenname+ ' ' + customer.Surname) py}!

The package will be sent to the following adress

  {py
  Console.WriteLine(customer.Street + ' ' + customer.StreetNumber)
  Console.WriteLine(customer.ZipCode + ' ' + customer.City)
  py}
Regards
//Some online store

Rendering the template

var engine = Python.CreateEngine(AppDomain.CurrentDomain);
var scope = new ScriptScope(engine, new Scope()).Engine.GetClrModule();

scope.SetVariable("customer", new Customer
                                  {
                                      City = "Stockholm",
                                      Givenname = "Steve",
                                      Street = "AppleRow",
                                      StreetNumber = "23",
                                      Surname = "Mobs",
                                      ZipCode = "14565"
                                  });

Console.WriteLine(PyBlocksRenderer.Render(engine, scope, template));

Output

Hello Steve Mobs!

The package will be sent to the following adress

  AppleRow 23
  14565 Stockholm

Regards
//Some online store

PyBlocksRenderer.cs

    public class PyBlocksRenderer
    {

        /// <summary>
        /// Renders the template using the specified engine and scope.
        /// </summary>
        /// <param name="pythonEngine">The python engine.</param>
        /// <param name="scriptScope">The script scope.</param>
        /// <param name="template">The template.</param>
        /// <returns></returns>
        public static string Render(ScriptEngine pythonEngine, ScriptScope scriptScope, string template)
        {
            //Replace tabs with 4 spaces using a string extension method
            template = template.ReplaceTabsWithSpaces();

            //Fetch a MatchCollection using a string extension method
            var pythonBlocksMatchCollection = template.GetPythonBlockMatches();

            //contains the matching {py..py} block as a key and the result as value
            var pyBlockMatchResults = new List<KeyValuePair<string, string>>();

            foreach (Match pythonBlockMatch in pythonBlocksMatchCollection)
            {
                //Fetch matching code block code and normalize indentation using a string extension method
                var pythonCode = pythonBlockMatch.Groups["code"].Value.NormalizeIndentations();

                //Add import to enable writing to the console from within the pythonCode
                pythonCode = string.Format("from System import Console{0}{1}", Environment.NewLine, pythonCode);

                //Execute script
                var pyResult = ExecuteScript(pythonEngine, scriptScope, pythonCode);

                //Fix result indentations
                var line = template.GetLineAt(pythonBlockMatch.Index);
                if (string.IsNullOrEmpty(line.Trim()))
                {
                    var indentation = line.GetIndentation();
                    var pyLines = pyResult.GetLines().ToList();
                    var lines = from l in pyLines select pyLines.IndexOf(l) > 0 ? l.Prepend(" ", indentation) : l;
                    pyResult = string.Join(Environment.NewLine, lines.ToArray());
                }

                //Add the result of this execution to the list
                pyBlockMatchResults.Add(new KeyValuePair<string, string>(pythonBlockMatch.Value, pyResult));
            }

            //find / replace template py blocks with results
            foreach (var replace in pyBlockMatchResults)
                template = template.Replace(replace.Key, replace.Value);

            return template;
        }

        /// <summary>
        /// Executes the script.
        /// </summary>
        /// <param name="engine">The engine.</param>
        /// <param name="scope">The scope.</param>
        /// <param name="code">The code.</param>
        /// <returns>code output</returns>
        public static string ExecuteScript(ScriptEngine engine, ScriptScope scope, string code)
        {
            //save reference to standard Console out
            var stdOut = Console.Out;

            //1: redirect the console to write to a stringwriter
            //2: execute the script
            //3: return the contents of the stringbuilder
            //4: reset Console out to the saved reference above
            //Exception: return the exception message
            try
            {
                var sb = new StringBuilder();
                Console.SetOut(new StringWriter(sb));
                engine.Execute<string>(code, scope);
                return sb.ToString();
            }
            catch (Exception ex)
            {
                return string.Format("#! {0}", ex.Message);
            }
            finally
            {
                Console.SetOut(stdOut);
            }
        }
    }

StringExtensions.cs

    public static class StringExtensions
    {
        /// <summary>
        /// Replaces the tabs with spaces.
        /// </summary>
        /// <param name="str">The STR.</param>
        /// <returns></returns>
        public static string ReplaceTabsWithSpaces(this string str)
        {
            return str.Replace("\t", "    ");
        }

        /// <summary>
        /// Gets the python blocks.
        /// </summary>
        /// <param name="template">The template.</param>
        /// <returns></returns>
        public static MatchCollection GetPythonBlockMatches(this string template)
        {
            return Regex.Matches(template, "{py(?<code>.*?)py}", RegexOptions.IgnoreCase | RegexOptions.Singleline);
        }

        /// <summary>
        /// Normalizes the indentations.
        /// </summary>
        /// <param name="str">The STR.</param>
        /// <returns></returns>
        public static string NormalizeIndentations(this string str)
        {
            str = str.ReplaceTabsWithSpaces();
            var lines = from line in str.GetLines() where !string.IsNullOrEmpty(line.Trim()) select line;
            var minimumIndent = lines.GetMinimumIndentation();
            return string.Join(Environment.NewLine, (from line in lines select line.Substring(minimumIndent)).ToArray());
        }

        /// <summary>
        /// Gets the lines.
        /// </summary>
        /// <param name="str">The STR.</param>
        /// <returns></returns>
        public static string[] GetLines(this string str)
        {
            return Regex.Split(str, "\\r?\\n");
        }

        /// <summary>
        /// Gets the minimum indentation.
        /// </summary>
        /// <param name="lines">The lines.</param>
        /// <returns></returns>
        private static int GetMinimumIndentation(this IEnumerable<string> lines)
        {
            var minimumIndent = int.MinValue;

            foreach (var line in lines)
            {
                var indent = line.GetIndentation();
                minimumIndent = minimumIndent > indent || minimumIndent == int.MinValue ? indent : minimumIndent;
            }

            return minimumIndent > int.MinValue ? minimumIndent : 0;
        }

        /// <summary>
        /// Gets the indentation.
        /// </summary>
        /// <param name="str">The STR.</param>
        /// <returns></returns>
        public static int GetIndentation(this string str)
        {
            return Regex.Match(str, "^\\s+").Value.Length;
        }

        /// <summary>
        /// Gets the line at.
        /// </summary>
        /// <param name="str">The STR.</param>
        /// <param name="index">The index.</param>
        /// <returns></returns>
        public static string GetLineAt(this string str, int index)
        {
            return str.Substring(0, index).GetLines().LastOrDefault() ?? string.Empty;
        }

        /// <summary>
        /// Prepends the specified STR with value n times.
        /// </summary>
        /// <param name="str">The STR.</param>
        /// <param name="value">The value.</param>
        /// <param name="times">The times.</param>
        /// <returns></returns>
        public static string Prepend(this string str, string value, int times)
        {
            for (var i = 0; i < times; i++)
                str = value + str;

            return str;
        }
    }

Setting up a EPiServer 6 project from scratch

Prerequisites

  1. Microsoft Visual Studio 2008
  2. EPiServer 6.0 CMS (Download and install from here)
  3. MS SQL Server Express (Download and install from here)

Prepping a EPiServer database

Start EPiServer Deployment Center and choose “Install SQL Server Database” > run

Setting up the EPiServer 6 project in Visual Studio

Create a new EPiServer 6 project in Visual Studio,  File > New > Project > C# > EPiServer

ConnectionStrings.config

Alter the ConnectionStrings section to match your database setup

Properties > Web

In Visual Studio, right click your website project and choose “properties”

In the “Web” tab, clear the Virtual Path.

I had problems accessing the site with the default port 6666 so I changed it to 5859

EPiServer.config

Change the “siteUrl” of your site to “http://localhost:5859″

Launch

Now you should be able to launch the project and login at http://localhost:5859/EPiServer/CMS/Edit/

Create a simple page template

Visual Studio – /Templates/Pages/SimplePage.aspx

In your EPiServer project, create a Templates folder and a Pages folder beneath that one (you can structure/name this however you want)

In the pages folder create a new Web Form, name it SimplePage and change the base class to EPiServer.TemplatePage

EPiServer

Launch your project and navigate to http://localhost:5859/EPiServer/CMS/Admin/

Choose the “Page Type” tab and click “Create New Page Type”

Type a name for your new Page Type and set the File Name to “/Templates/Pages/SimplePage.aspx”

Click the save button and you are done

Google Wave – Reversi game

This is the classic board game Reversi as a Google Wave Gadget. Players takes turns placing white/black stones on the grid. Stones are only allowed to be placed to an allready existing stone on the board, if a stone is surrounded on either sides by stones of oposing color that stone will change to that color. the player with the most his/hers colors on the board when the board is full Wins.

Insert this gadget into a wave by creating a new blip/response, click the green puzzle icon, type/paste the url below and click Add.

http://www.creamdog.se/googlewave/go/go.xml

If you are interested you can download the source code here