• Meet the users expectations and requirements
• Deployable and supportable in a manner acceptable to the operations group
• Maintainable in a manner acceptable to the development group

The purpose of testing can be reduced to verifying these statements about the program are in fact true. In many ways these can be considered the Prime Directives of a testing approach.

 “System Testing” is one type of testing that can be used to verify this. It is typically done in an environment that closely matches the production environment, and performed by people who will act as the eventual consumers (Users, Administrators, etc.). At this level, only the externally visible aspects of the program are considered, and the tests typically fall into one of three categories:

• Script based Manual Testing
• Exploratory Manual Testing
• Automated Testing

This approach does have some limitations including:

• Testing cannot begin until the program is in a sufficiently mature state
• Regression testing is time consuming and therefore costly
• Setting up specific scenarios (especially error scenarios) can be very difficult

As a result, other types of testing get introduced to minimize the impact of these limitations. But the question remains what types of testing are best for different scenarios.

“Functional Testing” breaks the system down to individual elements that map directly to the (wait for it….) Functional Specifications, or for Agile teams, User Stories.  The primary advantage of this is that is correlates very well with the concepts of “deliver working software” without having to necessarily have the whole system (or even large parts of it) complete. With the judicious use of Mock objects we can easily address most of the limitations with system level testing. A large number of teams focus on this type of testing during each iteration (sprint) and use this as the primary methodology of determining with when something is “complete”;  the completed items are regularly propagated into the Test/QA environment, where system type testing can be performed.

Before continuing, let us think about what “perfect” testing would be.  I propose the following as a working definition: “Perfect testing would insure that any change made to the software, which would cause a violation of any of the Prime Directives, is caught by one or more failing tests”.  However, we know that nothing is perfect, and that even if this is the goal, it can not be reached even with infinite resources dedicated to testing. What we are left with is a continuum. At one end, a minimal effort expended on testing with a high probability of issues which incur associated costs.  At the other end,  an extremely high investment in testing with the goal of minimizing the number and severity of any issues which slip past the tests.

The “sweet spot” is where we have minimized the sum of both the testing costs and “issues” costs. Of course, the type of system has a major impact on finding this spot. Some of the projects I work on are Industrial Automation, where serious injury or loss of life is a distinct possibility if certain types of malfunctions occur. Other systems such as banking and trading may incur millions of dollars in financial losses in the event of certain  defects. What I have repeatedly found working in different domains (vertical markets), is that many of the methodologies in these demanding environments can be applied, with modification, to “normal” business systems where they are not usually considered; and that this can be done with a minimal cost, often resulting in overall savings.

I have come to the conclusion that while Functional Testing is nearly always a key component, it is not sufficient to minimize the chances of violating a Prime Directive by making a change to the code. This is because Functional testing only captures one of the 5 “W”’s:

• Who – Which piece of code is responsible for a given item
• What – What does the code accomplish
• When – When is the operation actually performed (Aggressive Loading, Lazy Evaluation, etc.)
• Where – Where do the inputs come from?, where do the outputs go?, where are there side effects?
• Why – What drove a specific approach? Why were others eliminated?

It is important to remember the qualifier as “by making a change to the code”. Initially determining these items is definitely a human activity; and the best way to achieve this is through a solid design methodology with a strong emphasis on code-reviews (pair-programing can also be very helpful in many cases).  What I am specifically addressing are the changes that occur to a piece of code, for any one of a number of reasons that “seem like a good idea”, but fail to take into account that (presumably) the code was developed the way it was for a specific reason.

As a specific example, I was recently on a project where there was a small (never more than 20 items) collection that had a huge number of replacements (one item being removed, a different item being added)  occurring at a very high rate. There were also a fair number of places where the entire collection was iterated over, and a few places where an element was being accessed by its “key”. The original implementation was a .NET List<T> so that locating an item for replacement and accessing an element by key were both done by a linear scan.  One of the developers was working with code that interacted with the collection, and noticed  the  linear scans of the collection.  It looked like a good idea to replace the List<T> with a Dictionary<K,T>, because of this change it was also necessary to add some synchronization (lock) code. The code was reviewed, the change was made, the Functional tests all passed, and even the Application level Load/performance tests passed.

About two weeks after the code was deployed to production, there were various performance anomalies that were causing a multitude of problems. It took about two days to track the problem back to this change. The initial thought was that it was the locking that caused the problem, but that did not account for all of the issues. The original developer was contacted [he had moved to a different team], and after some time thinking about it remembered the reason he had chosen the List<T> with the linear operations – the overhead of managing the index far outweighed the cost of the sequential scans.

Many will argue that the code should have been commented, or that the load testing should have been enhanced, and neither of these are incorrect. But, we all know that comments are often not read, or become out of date; and, in this particular case, setting up a load test for this specific condition at the application level would have been extremely difficult. Had there been a test at a lower level designed to detect the conditions that caused the original author to choose the specific implementation, this entire episode would have been virtually eliminated.

This is only one simple sample, but the general concept applies in many situations. In order to address these, I generally recommend the following types of tests be implemented (or at least considered):

“Granular Unit Testing” (which I often will refer to as GUT testing) These tests focus tightly on specific code elements at the lowest level practical, typically individual methods and properties of each class. Heavy use of Mock object, and even Reflection to setup the desired states is quite common. In many environments, this type of testing has fallen out of favor; being viewed as “testing the trivially obvious”; Yet, there is significant value in this type of testing. The ability to quickly execute a focused test can reduce the “cycle time” from minutes down to seconds for quick verification of the change. The tests themselves often provide useful information about the amount of work to setup a given item (often revealing tighter than intended coupling), and can also be useful as “sample use-cases” for new developers coming up to speed. In order to reduce the costs in developing these tests, some automation is helpful. Here are Dynamic Concepts Development Corp. we have developed a custom toolkit (hopefully to be released as a product later this year) that makes it very easy to setup the initial conditions and have a clean separation between the manually created elements and the generated code.

“Design Rule Check Testing” (commonly referred to as DRC tests) These tests have the greatest degree of separation from the “functional” aspects. The goal here is to minimize the risk that the code has deviated from the intended/desired design.  Some “tests” can be accomplished by having a robust set of StyleCop/FxCop/etc. rules. Others may perform more detailed analysis on the code. Items that are often a good target focus (some specifically for .Net): Resource Utilization, Thread Safety Identification (even for applications that are currently single threaded), Complexity of Object Graphs (which may a bigger impact on Garbage collection than the number of temporary objects created!).

“Negative Testing”  (i.e. testing things that don’t happen!) These tests seem to be the the rarest of them all, and typically need explanation. As a quick example consider a class that exposes 4 events. Changing a specific property should raise one of the events. Validating that this event is raised at the appropriate time is likely to be in the Functional tests and/or the GUT tests. But most of these tests will not catch a bug where the property change triggers other events in addition to the one that is specified.  This can be very significant. At the low end of the impact spectrum it can create unnecessary processing overhead -especially if the handler[s] are heavy, or there are a large number of handlers registered. At the high end of the impact spectrum, it may reveal a significant defect in the logic of the code under test which might not be caught in other conditions. As with the GUT tests, it is very helpful to have a tool which can compare initial state with final state as well as register for events which should not fire.

When all of these are put together to form a multi-dimensional approach to testing, I have seen drops in defect rates of up to 60%. If you are currently engaged on a greenfield (initial development) project, I strongly recommend this approach. If you are currently on a brownfield (ongoing maintenance/upgrade) project, then an incremental adoption is typically the best bet. When one component is being updated, take the time to make sure that the testing for that component (not just the elements you are currently changing) have a solid set of tests before making the change, validate that the change will cause test failures to the existing tests, and also look for cases where the old code is capable of passing the relevant test suite. This last step goes a long way in ensuring that the changes you are making are thoroughly tested.

void Sample<T1>( Thing a, Thing  b, T1 data)

can be logically transformed to and from

void Sample<T1>( Thing b, ref  T2< Thing,T1> encapsulatedData)

by simply creating a simple wrapper contain such as the one below:

class T2<TARGET, ARG2, DATA>
{
    public T2(TARGET target, ARG2 arg2, DATA originalData)
    {
        Target = target;
        Arg2 = arg2;
        OriginalData = originalData;
    }

  internal TARGET Target { get; private set; }
   internal ARG2 Arg2 { get; private set; }
   internal DATA OriginalData { get; private set; }
}

class Helper<X> : IThingDispatchTarget<T2<X,MyData>> where X : Thing
{
   public void Dispatch( Thing b, ref  T2<X, MyData> encapsulatedData) {…}
}

We already know how to perform a a method invocation based on runtime type for a method with the signature of “EncapsulatedSample” so it would seem we are very close
to our goal. In fact, if we were dealing with C++ templates rather than C# generics,  we would indeed by on the last steps of our journey. Unfortunately, even though our T2 helper class is specialized for a specific derived class, within the generic body, the type is still treated as if it were of the type specified in the where clause. Thus any usage of encapsulatedData.FirstParameter within our Dispatch(…) method [or indeed within the Helper class implementation] will be treaded as a Thing rather than a specific derived class such as: Asteroid or Planet. So much for being able to simply dispatch using the methodology we have been covering.

Alas, there is no way to accomplish our goal using purely “static” code while avoiding
significant duplication, and/or runtime conditional code. This series would not be a useful “Power Programming” tip if it ended here, and it doesn’t. Compiled Expression Trees can come to our rescue! The following code will generate a delegate that will invoke the proper overload:

private staticAction<TARGET, ARG1, ARG2, DATA> GetRealAction<TARGET,ARG1, ARG2, DATA>()
{    
   MethodInfo mi = typeof(TARGET).GetMethod(
            “Execute”, 
             new[] {typeof(ARG1),typeof(ARG2),typeof(DATA)});     
   ParameterExpression target = Expression.Paramete(typeof(TARGET),target”); 

   ParameterExpression arg1 = Expression.Paramete(typeof(ARG1),“arg1″);

   ParameterExpression arg2 = Expression.Parameter( typeof(ARG2),“arg2″);

   ParameterExpression data =  Expression.Parameter( typeof(DATA),“data”);     

   MethodCallExpression mce = Expression.Call(

                                target, mi, arg1, arg2, data);    

   Expression<Action<TARGET, ARG1, ARG2, DATA>> lambda =    

              Expression.Lambda<Action<TARGET, ARG1, ARG2, DATA>>

             (mce, target, arg1, arg2, data);    
   return lambda .Compile();
}

With this helpful utility method, we can create all that we need as part of the
type initialization of the Helper class:

static Helper()

{
    sr_ThingAction = GetRealAction<RealClass, X, Thing,  MyData>();

  sr_SpaceshipAction = GetRealAction< RealClass, X,  Spaceship, MyData>();

    sr_PlanetAction = GetRealAction< RealClass, X, Planet, MyData>();    
    sr_SunAction = GetRealAction< RealClass, X, Sun, MyData>();
    sr_AsteroidAction = GetRealAction< RealClass, X, Asteroid,MyData>();
}  
private static readonly Action<RealClass, X, Thing,MyData> sr_ThingAction;

private static readonly Action<RealClass, X, Spaceship, MyData> sr_SpaceshipAction; private static readonly Action<RealClass, X, Planet, MyData> sr_PlanetAction; private static readonly Action<RealClass, X, Sun, MyData> sr_SunAction;

private static readonly Action<RealClass, X, Asteroid, MyData> sr_AsteroidAction;

And complete our Helperclass implementation with:

public void Dispatch( Thing b, ref T2<RealClass, ARG1, MyData> data)
{ b.Dispatch( this,  ref data); }  

public void Execute(Thing instance, ref  T2<RealClass,ARG1,MyData>data)

{ sr_ThingAction(data.Target,data.Arg1, instance, data.OriginalData); }

public void Execute(Spaceship instance, ref T2<RealClass, ARG1, MyData>data) { sr_SpaceshipAction(data.Target, data.Arg1,instance,data.OriginalData); }

 
public void Execute(Planet instance, ref T2< RealClass, ARG1, MyData>data)

 { sr_PlanetAction(data.Target, data.Arg1, instance, data.OriginalData); }

public void Execute( Sun instance, ref T2< RealClass,ARG1, MyData> data)
{ sr_SunAction(data.Target, data.Arg1, instance, data.OriginalData); }  

public void Execute( Asteroid instance, ref T2<RealClass, ARG1, MyData> data) { sr_AsteroidAction(data.Target, data.Arg1, instance, data.OriginalData); }

Conclusion

Well, it took a bit of work but we finally made it. The only remaining question is was the effort worth it. Running the above code where we have 1,000 elements (of equally distributed derived types) and running a loop that invokes a method for every combination (1,000,000 method invocations), gives the following numbers:

The chart clearly shows that dispatching to specific methods based on runtime type
of parameter is not “cheap”, but the timing difference is significant. Using dynamic created double the overhead of out multi-dispatch implementation, and we reduced the loop time from over 18.4  seconds to 9.5 seconds. If this were a smaller collection (say 1000 invocations per iteration) this would translate to a difference in frame rates (assuming this  loop was the only calculation occurring) of 54 FPS and 105 FPS.

Additionally, although the use cases where this implementation is worth the effort may be small; we did uncover a very powerful technique for being able to tread generic arguments as their actual type rather than being constrained to the functionality exposed by the common base [i.e. the type specified in the where clause]. In the long run is is the discovery of these “gems” that are potentially applicable to a wide variety of use-cases that provides the biggest gains in exploring “Power Programming” techniques.

I hope you have all enjoyed this journey, and am looking forward to hearing your
comments and feedback. The next series of posts will be focused back on ALM, but
I am hoping to produce about one “Power Programming” series per month.

First, dynamic is a great tool and is an excellent choice for many situations where the goal is to have different methods execute based on the runtime type of a parameter. Slightly reworking his same to use our classes we have:

void DynamicMethod(dynamic thing)

{

    SomeMethod(thing);

}

Indeed, it is very “clean” to look at, no modifications were required to the class hierarchy, and in fact this approach will handle additional parameters and even dispatch based on multiple parameters with ease. Alas, as science fiction writer Robert Heinlein said (he did not originate the acronym but definitely popularized it, as did Larry Niven) TANSTAAFL  – “There ain’t no such thing as a free lunch.”.  There are in fact two side-effects of choosing to use dynamic:

• Security – As Peter pointed out, there does not even have to be a relationship between the instance. Provided a target can be resolved, it will be used. In our case, this risk is minimal; but in the example Peter used with deserialization the risk can be much higher. Lets look at a sample very similar (but with significant difference) to Peter’s:

       IFormatter  formatter = new  BinaryFormatter ();
       dynamic shape = formatter.Deserialize(stream);
       shape.Execute();

If someone can hijack the stream (perhaps it is reading from a disk file), then the the object type can be completely replaced by altering the stream. Providing the object type has an execute method that takes no parameters, the code will run. Now, imagine if the de-serialized object is of type SelfDestructSequence instead of a sub-class of Shape.

• Performance – Assuming we are constrained by a performance specification, the use of dynamic may have serious ramifications.  Measurements I have performed (using Stopwatch over 10 million executions and then averaging 10 runs after throwing out the fastest and slowest) show that dynamic is typically twice as slow as the original sequence of casts and compares, and 15 times as slow as the double dispatch used in the last post.

It is critical to put this performance measurement into context. If the overall operation is of significant time, or the number of calls through the dispatching mechanism is low, then the difference will be meaningless in the overall scheme of things. However, if the target method is very fast, and there are going to be large numbers of calls [such as in our later example where 10,000 instances are operated on in all possible combinations for a total of nearly 10 million calls] and there is a tight performance specification, then this must be taken into account.

With that out of the way, we can return to the original topics. First, lets look at scenarios where the target method requires one or more parameters in addition to the one that is being specially treated.  This is relatively simple, just change the signatures of the “IDispatchTarget” interface, along with the dispatch methods within the class hierarchy. But what should the new signatures be???

At first glance, it may appear easiest to modify the code to match the parameter types for the specific use-case. I recommend against this approach because it has a habit of growing out of control very quickly. If the parameters change, then the interface and class hierarchy will be impacted.  it gets even worse if the same class hierarchy needs to be dispatched to different methods in different classes. Either there will have to be multiple different interfaces (with a “Dispatch” method for each, or there will be multiple methods in the interface for the same object type, which usually leads to forced implementation of “NotSupported” methods.

Using generics helps the situation, but only a little. You still need different implementations for different numbers of parameters, and face the same issues as in the previous scenario.

What I have found to be the best solution for the majority of cases, is to use a single generic parameter, and pass it as a ref parameter. This gives rise to the following:

interface IThingDispatchTarget<T>

{

   void Execute(Thing instance, ref T data);

   void Execute(Spaceship instance, ref T data);

   void Execute(Planet instance, ref T data);

   void Execute(Sun instance, ref T data);

   void Execute(Asteroid instance, ref T data);

}

 

The reason for using pass by reference, is that it gives use the ability to efficiently package an transfer multiple items within the data parameter by using a struct, without the overhead of creating additional instances on the heap, or by copying the contents of the struct as would be required if it was pass by value. To complete the code sample..

class RealClass : IThingDispatchTarget<MyData>

{

    public void SomeMethod(Thing thing, ref MyData data)

    {

       thing.Dispatch(this, ref data);

    }

   void IThingDispatchTarget<MyData>.Execute(Thing instance, ref MyData data) { }

   void IThingDispatchTarget<MyData>.Execute(Spaceship instance, ref MyData data) { }

   void IThingDispatchTarget<MyData>.Execute(Planet instance, ref MyData data) { }

   void IThingDispatchTarget<MyData>.Execute(Sun instance, ref MyData data) { }

   void IThingDispatchTarget<MyData>.Execute(Asteroid instance, ref MyData data) { }

}

That wraps up handling additional parameters, leaving use with the topic of being able to dispatch based on the runtime type of multiple parameters. Since this post is getting a little on the long side, we will continue with it next time…

The following code illustrates a trivial case of the issue:

The above code will always call the overload that takes a Thing because that is the declared type of sample1, even though it is obvious that the instance is a Spaceship. Of course this trivial example could easily be modified to declare sample1 as a Spaceship, but that would be cheating. Since the overload that takes a Thing will always be called, many developers resort to variations on the following pattern:</p.

This code smells bad in so many ways. There are more lines of code for plumbing than there are for actual work. There are many redundant casts. If a new derived class is created, this code must be manually tracked down and altered. There is a better was and it is known as Double Dispatch. This pattern has been known for at least 20 years, and there is documentation on it readily available on the web. The classic implementation is to create an interface to the code that has the method in question add one virtual method to the base class of the hierarchy which is overridden in each of the derived classes. The result will look something like the following:

Ah, the sweet smell of success. We only need to create the interface and add the methods to the hierarchy once, regardless of how many places in the application there is the need to perform different processing based on the runtime type of the parameter. There are no casts anywhere in the code. There is no conditional plumbing in the class(es) that do the work. The code is also faster. Timing tests run on the above show that double dispatch has only 12.8% the overhead of the nested if; furthermore, the overhead of double dispatch is fixed regardless of the number of classes in the hierarchy, while the nested if with cases grows linearly with the number of types.

Conclusion

I can almost hear readers saying “This is a good technique, but it has been known for a long time….is it really worthy of the title Power Programming?”. The answer to that is no! This post is just setting the stage for discussing some of the limitations with this simple implementation. When we continue, we will cover some topics that I have rarely found published:

• How do we extend this when there are additional parameters involved that must be passed?
• How do we extend this when we want to dispatch to different targets based upon multiple parameters?

This series will focus on specific techniques that address recurring issues that are found in certain classes of applications. Many of these will be performance related, so examining the adage “Premature Optimization is the Root of all evil”  is  very relevant.

The actual statement written by Donald Knuth was:

“Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”  – Computing Surveys, Vol. 6, No 4, December 1974

I had started writing software just over two years before just over two years before this was written (I did not actually encounter the article until nearly three years after it was written) on a DEC PDP-8 with 16 K (that is 0.0000152587890625 Gig for modernists) of real (ferromagnetic) core memory. Even with this limited amount of memory it was responsible for administrative tasks of a 6,000 student school district. Combined with a clock speed that was less than 1MHz, writing the smallest, fastest possible code was a requirement. These days, most of my actual coding work is on projects that are extremely demanding in some fashion, often in terms of raw performance and I consistently find myself dealing with “our opportunities in that critical 3%”.

The first step in looking at performance is to have a specification. Without a documented specification it is impossible to say that something is “too slow” and therefore a candidate for performance optimization. The second step is to perform detailed measurements of items that fail (or are dangerously close to) these specifications. In many cases, a focusing on a very small part of the operation under consideration will yield the desired benefits. In other cases, a redesign may be necessary in order to have a robust, maintainable codebase that meets the specification.

As an example of the former case; Last year my firm was contracted by a major financial company who were having problems with application performance. Historically they had programmed in C++, but had recently moved to C#.  The codebase was well designed with SOLID principles applied throughout, the algorithm implementation was straight forward, and although there were some general issues with the code, the “killer” problem was the performance of this one algorithm.

When I first arrived, the team was nearly evenly divided into three camps:

• “We told you .Net is too slow…we have to go back to native code”
• “The algorithm has to be redesigned or at least re-implemented”
• “We have to abandon OO principles and had write the code in an optimized fashion”

I asked them if they had profiled the code, and they answered yes. I then asked to see how they profiled the code and they showed my the following:

Stopwatch sw = new Stopwatch();
sw.Start();
CallProblematicAlgorithm();
sw.Stop();
int measurement = sw.ElapsedMilliseconds;

They had not done ANY actual profiling of where the time was being spend within the various parts of the code. I fired up my favorite profiler (Red-Gate ANTs 6.0), and within a few minutes identified that replacing elements in a generic Dictionary<K,V> collection was responsible for a large amount of the time. I then did some research (aka I asked questions) and found out that the collection was indeed highly volatile, elements were frequently referenced by their key (although there were actually more Add/Removes’ than Keyed retrieval), but most importantly, the design constrained the collection to never have more than than about 20 entries in the collection!!!!

Because of the size of the collection, and the overhead in maintaining an indexed collection, I made one simple change…to a plain old generic List<T>, with a simple iteration to find the proper element when “keyed access” was required. Not surprisingly the time required to add or replace an element went down dramatically, the times to remove an element went down slightly, and the indexed access was just about the same.

This one simple change, was sufficient to reduce the overall time so that it met the performance specification with a good safety margin. [I only wish some of the other issues on that project were so easily resolved].

The key takeaway is that it really was a small part of the code that was critical, and that their original views on the subject were significantly off-the mark [I am being kind], and had they adopted any of them, they would have been in rather poor shape.

Conclusion

If you have done your due diligence, and determined that there are real performance issues and that specific code is a leading contributor to the problem,  then hopefully the techniques I will be posting here will be of help. It is unlikely that any of them will be directly adoptable in their presented form, but my goal is to provide the background so that they can be evaluated and even used as the basis for power programming techniques of your own.

As always, I am looking forward to comments on these items, and suggestions for ways to address specific issues that you are encountering.

For teams using Microsoft Visual Studio 2010 (or previous versions), there is great news. Microsoft Team Foundation Server 2010 is incredibly easy to get up and running, the default configuration is usable by a great many teams, customization can be approached incrementally, and perhaps best of all, the is a free trial available for download. Once the ISO is downloaded, it typically takes well under an hour (and can be as short as 15-20 minutes) to get a fully functional environment up and running.

For those who experienced either the 2005 or 2008 installation process, you are in for a major pleasant surprise. Not only is the process much faster, but in many cases you can simply step through the wizard driven installation and accept the defaults as they are presented.

The second most common thing I hear is “I’ve installed it, it looks good, but how do I now learn how to use the tools effectively?“. Once again, there is good news.

For people who want a detailed walkthroughs, I highly recommend Professional Application Lifecycle management with Visual Studio [ISBN :978-0-470-48426-5]. Written my Mickey Gousset, Brian Keller, Ajoy Krishnamoorthy and martin Woodword and published by Wiley Publishing (under the Wrox name), this book contains a wealth of information. It is the only book on the topic that I have brought into clients and said “Here is our starting point, as we adopt the material in this book, we will make customizations as required, but in general they should be minor and driven by unique requirements“.

Other people will instead (or in conjunction) want material that is focused on specific items. The Visual Studio ALM Rangers are a group approximately 70  Microsoft Employees and selected other individuals who “deliver practical out of band solutions for missing features or guidance on Visual Studio 2010”. [I am proud to be a member of this group, and my personal profile can be found here]. The following list highlights some of the guides that have been published.

•  Visual Studio 2010 Quick Reference Guidance – Compact guidance for the field to cover Team System and TFS capabilities
• Team Foundation Server 2010 Branching Guide – Guide includes discussions around branching concepts, strategies, and also focuses on practical hands-on labs.
• Team Foundation Server 2010 Monitoring Management Pack – Delivers the capabilities for professional monitoring and management of Team Foundation Server 2010 infrastructure components
• Visual Studio 2010 Architecture Tooling Guidance- Guide is focused on practical ways of effectively using Visual Studio 2010 Ultimate and other tools to create a new or revised design.
• Visual Studio 2010 Database Guide – Guide includes practical guidance for Visual Studio 2010 database projects.

Many of these guides include Hands on Labs [HOL] so that the material can be walked through interactively, and videos that provide visual impact.

In addition to the work by the Rangers, there are also a number o offerings by the Patterns & Practices group at Microsoft that directly relate to TFS. While many of them were written based on earlier versions of TFS, the majority of the material is applicable to TFS 2010. Finally there are Microsoft forums dedicated to TFS, Groups on Linked In, and many other public sources of useful information.

As part of this blog, it is my intention to regularly publish [I am planning on once a month, but open to more frequent updates if enough readers want them] summaries and links that I have found to be useful and relevant.

Requirements Gathering (User Story Development)

At first glance, it may not seem that there is anything to test at this point in time. However ensuring that the documented requirements accurately match the user requirements and are also consistent among themselves is extremely critical. If there is an undetected problem at this phase, it can easily cause significant impact on all of the other areas.

I have found that there are indeed a few activities that can be used to “test” this area.

1. Use light weight mockups to provide a visual walk-through. A large number of issues are often identified when “Seeing pictures” that do not surface by reading words. The most effective methods are using tools like SketchFlow or even PowerPoint. At early stages, mockups written in code often present more problems in the long run than they solve.
2. Record each item as a distinct artifact that is under revision control and provide links between related artifacts. TFS work items are the preferred way when using the Microsoft ALM tools.
3. Develop Test Plans and Specific Test Steps as each Requirement/Story is being formalized. This provides additional information about the expectations. I have found many instances where thinking about how the requirement is going to be tested as provided immediate refinement to the requirement itself.

Architecture/Design/Implementation

I have grouped these three items together because test driven approaches in these areas are typically the best understood. In a future post, I will be covering these areas in greater detail.

Quality Assurance

If Test Plans and Test Steps have been being developed since the beginning (see above) then ter is already a good start on determining the Acceptance Criteria that the QA team will be targeting. Careful thought should also be given to the types of tests that will be performed, with the three major catefories being:

• Scripted Manual Testing
• Automated [Coded UI] Testing
• Exploratory Testing

While most QA teams have a good handle on Scripted Manual Testing, the latter two categories are often overlooked or misunderstood. With Visual Studio 2010, it is simple to record actions while “driving” the application, and identifying the specific elements that are being examined along with their required values. These tests can then be repeated in an automated fashon to rapidly re-test many aspects of the system without the time consuming (therefore expensive) manual interactions. Having the tests run in a 100% repeatable manner also provides consistancy.

Formal Exploratory Testing is almost an oxymoron. By its very nature, it involves (a person) “wandering” through the application in a semi-structured or even random manner, looking for potential defects. Since there is no script or other definition of the testing activity, it can be hard to imagine this being formalized. However tooling such as Visual Studio 2010 once again provides some very helpful capabilities. The iteractions with the system can be recorded as a video that can be played back to see the exact steps to arrive at a specific point. IntelliTrace (background recording of internal state as the application runs) allows for the capture of elements such as stack traces, exceptions, state in the event that a potential defect is discovered. These features make it much simpler for the developer team to analyze reports that originate from QA.

When these activities are recorded into a central repository, there exists the capability to analyze the QA activities, and find out what testing methods are effective, what could be improved (in some cases, the improvement is a reduction in certain types of testing in favor of other types). Effectively we have reached the point where we can “test the testers” and reach a more harminic relationship between the development and test teams.

Maintenance/Defect Management/Deployment/Operations

Application lifecycle management does not end with the release of a version to production. For most application, the journey is just beginning as the time from initial release to final decommissioning can be orders of magnitude longer than the time from concept to initial release.

If solid practices and processed have been established during the initial development phase, there is a good deal of “metadata” about the project, including significant informantion about HOW the application reached the current state, and WHY decisions were made. Unfortunately too many companies treat the release milestone as “throwing the project over a wall”.  Customer (User) Support starts to use their own “issue tracking” system and Operations keeps their own internal records. Things begin to drift back into “islands of information” rather than a unified/comprehensive view.

To the suprise of many, these issues can be mitigated simple by “testing” the relationships between the various parties. Before the first “real” deployment, there should be mock deployments that are treated as any other development activity, with requirements, tasks, issues and bugs being recorded. This will provide helpful informtion to the deployment team. As the “real” deployments occur, these activities should also be tracked back in the same manner. Similar trials and integration with whatever system is being used for tracking customer issues should be applied.

Conclusion

Hopefully these two posts have provided some insight regarding integrating all of the various areas into a unified environment. The outcome is a consistent approach to capturing information in a form that can be analyzed, review by all parties (testing) at or near the time it is recording, being able to see relationships between items and validating (testing) that they are consistent, having easy access to reference and update the information as work progresses ensuring (testing) the current tasks align the the requirements and test plan, and finally having retrospectives on completed items with the focus on evaluating (testing) if the process/workflow can be improved.

To start off let’s quickly review the various activities that are part of the Application LifeCycle. These are going to be presented in a Waterfall style for simplicity, but each item also occurs within Iterative methodologies such as Agile/Scrum. the key ones here are:

• Requirements Gathering
• Architecture
• Design
• Implementation
• Quality Assurance
• Maintenance
• Defect Management
• Deployment
• Operations

Can each of these items be subjected to a process which establishes quantified metrics  that reflect both the quality and correctness of each item? It should be clear that conventional Unit Tests do not apply to all of these items; at best they can verify that a local aspect (e.g. a Class/Method) of implementation matches the (test writers perspective of) the appropriate design document. So what can we do?

For each of area, the goal is to create tests that are quantifiable and durable. The ability to quantify the measurements (beyond a simple pass/fail) is critical to tracking progress(eventually measuring the level of success that has been achieved) and for providing clear information on what items need to be addressed (along with the appropriate time to address them – in varying levels of detail) . Durability is important so that the test can be reapplied (ideally in an automated fashion) over the entire cycle.

There are many places where ”testing” is used outside of software development. Consider the (conventional) education process that most of us grew up on. The focus was to get the best grades as measured by different tests. Many of these tests measured rote memorization and not understanding of the subject matter. The result of this that many people graduated with high scores but without “quality and correctness” in their ability to utilize the subject matter (of course, the flip side is true where certain people DID understand the material but were not very good at taking this type of test).

One must also be careful of how the tests are organized and how the measurements are taken. If a test is in a multiple choice format, there is a significant statistical probability that a correct answer might be the result of a random guess. Also, in many situations, having the student simply provide a final answer can obscure many important elements. For example, on a math test, having the student simply provide a numeric answer (rather than showing the methodology) may result in a complete mismatch between the process and the result. It is hard to determine which is worse: The student who makes a simple arithmetric error at one step of a long process (resulting in a wrong answer) or The student who (without providing the “workflow”) uses a completely invalid approach, yet still comes up with the right number.

The “Wrong Process”/”Right Answer” is probably the single biggest problem in creating quality tests. Even very simple items can suffer from this. As an example consider the following code for a “straight line” calculation….Is it correct? (for Integral Points)

int Solve(int m, int b, int x) { return m * x + b; }

Most people would respond “Yes”. But let’s take the question one step further… Is it correct for all possible values of m,b,x??? Without additional information regarding constrains on “the possible values of m,b,x” the answer must be NO, there is the risk of overflow/wraparound that will produce an incorrect result!

To properly answer this question (i.e. Test the Code), one MUST be able to backtrack from the implementation through the design, and architecture all the way back to the requirements. And the requirement itself must be tested against the stakeholder(s). It is only when the bounding conditions are defined that it is possible to determine if the code is “Correct” and has “Quality”.

Yet, how many of us (myself included) have written such code without even thinking about it. In many canses we (think we) “know” what the bounds are, and that the code will be correct. As we all know, requirements change, “code reuse” causes implementations to be applied to different scenarios, etc. This leads directly to the types of system failures that plague so many projects.

This approach to TDD is much more holistic than ones which start by focusing on the details. The fundamental concepts still apply:

• Each item should be tested.
• The test should be defined/implemented before (or concurrent with) the definition/implementation of the actual item.

We also add concepts that expand the scope and alter the style by recognizing:

• There are many things beside “lines of code” that benefit from testing (measuring/evaluating in a formal way)
• Correctness and Quality can not be solely measured by “correct results”

Next time we will dig deeper into the 9 ALM areas listed at the beginning of this post and evaluate how each can be tested.

Last year Microsoft released Team Foundation Server 2010 along with Visual Studio 2010 Ultimate Edition, and with these two in combination the situation has drastically changed. At last there is a single environment that is capable of handling all aspects of ALM, and is also capable of dealing with migration and integration with existing systems to make the transition to a single solution much easier.

The possibilities (and practicalities) are nothing short of amazing, Architecture thru Testing integration? YES. Being able to correlate specific requirement items (and their history) to actual code (and code history)? YES. Identification of which tests will be potentially impacted by a given code change? YES. Resiliant Automated Testing of User Interfaces? YES. Automatic Deployment Management? YES. Integraton Level testing as part of (designated) Builds? YES.

I could easily double or triple the above list, but these items should be enough to get you thinking about the “pain points” your team and organization currently face and the fact that there IS a way to relieve the pain.

When teams are first introduced to these capabilities, there are a couple of common reactions.  Many can be grouped under the heading of “That’s Great! What do I have to do in order to experience this goodness?”), but a fair number fall into one or more of the following groups:

• We don’t need any of that. We are doing just fine editing our code, compiling it locally, testing it locally, and doing manual deployment.
• We already accomplish that (or at least the important parts) using a suite of [often OpenSource/Free] tools.
• That will make our jobs more difficult / take longer, and decrease efficiency.

If you are a member of the first two groups you are likely to be comfortable with the status quo and do not see the need for change. I recommend either a simple pilot project or a co-development effort (using both the existing methodology and an ALM approach in parallel) to identify and quantify specific areas for improvement. Trial versions of all of the tools are readily available, so this can be done without any capital expenditure.

If you are a member of the last group, it is likely that you have had a negative experience with adopting a formal comphrehensive process in the past. Unfortunately this is common, and my best suggestion is to keep an open mind, learn as much as possible about the capabilities, and (if possible) get an opportunity to work with a person or team who has successfully adopted ALM

This means that a context for this blog must be established which places some bounds on the mega-topic of Software Development. While here may be exceptions in certain posts, the following is the environment that will typically be considered:

• Microsoft Languages, Tools, and Platforms. Unless there are specific reasons: Code will be in C# 4.0, developed using Visual Studio 2010 Ultimate Edition.
• Small (<=5) to Medium (>5, <=20) Teams. Much of what will be written involves the recording and sharing of knowledge and this most commonly occurs when multiple people are involved. This is not meant to exclude the individual developer, who also needs to record this information if only for easly at reliable recall at a latter date.  Finally, since most large development efforts are broken down into smaller team effort, even the biggest teams will be able to leverage the information.
• Moderate to High Project Complexity. This is not really a “requirement”, but as project complexity increases the return on investment in processes and practices become much more obvious. Simple systems (so called “jelly bean” or “cookie cutter”) can definately benefit also, but getting tangible metrics on the value may be more difficult.

Within this scope are some key topic areas which will be categorized for easy of reference.

• Application Lifecycle Management[ALM] will be the the primary focus as this covers all aspects of the effort from the gathering of initial requirements/ideas, through development and deployment, and only finishes when the system is finally decommissioned.
• .NET Architecture & Implementationwill cover specific design choices that have been adopted within Dynamic Concepts Development Corp. and have been proven to have applicability to other scenarios.
• Tales from the Trencheswill cover specific situations that I have encountered that turned into key learning issues.

Next Up: ALM – What is it and Why Do I Care?