I'm throwing my hands in the air in despair.  I've drunk the Kool-Aid, bought the hype and taken to my code with a big test driven club.  I'm not supposed to feel this frustrated am I?  Despite my best attempts to chain it up in the basement of my subconscious mind there's a little voice in my head calling out to me...  You're wasting your time...

I'm Talking About You Internet

Before I go an offend a whole spectrum of TDD believers I want to narrow down my focus a bit and concentrate on offending just a specific group of them.  The problem I'm having is my productivity when working on front end web pages has gone down by an order of magnitude.

I'm sold on the domain, the business logic, the services and the delicately encapsulated, exquisitely patterned classes separated with the utmost of concerns.  I don't mind spending time here and I'm seeing returns so I'm happy.

When it comes to the web I'm used to working fast and light and moving on quickly.  The tools support this well or maybe it's that they don't support anything else well.  The problem is that now I'm trying to smother my code in an air tight blanket of tests my fast and light web interfaces are gasping for air and fast falling behind.  The real WTF here is that despite the totalitarian rule of test coverage a lot of bugs are still slipping through.

Use A Bigger Hammer

The big problem is that despite using MVP to TDD my ASP my HTML is still AFK with my CSS and AFAIK it's got a freaking great BUG in it.  What?  I mean the tests don't cover my views and this is a big thing in a world where where anything but Web 2.0 is sin and AJAX roams around freely with little regard for the desperate cries of the programmers trying to tame it.

So tests aren't working out, what should I do about it?  How about some more tests?  Enter, the Watin!

Now don't get me wrong because I think Watin and the like are great tools but my poor code has a hard enough time cowering under the shadow of the mighty xUnit empire without being taken by a surprise attack from behind.  Watin is great for general functional tests but it's not the right tool for achieving code coverage and it keeps trying to bully my refactoring into giving up it's lunch money.

Time To Sober Up

As this all spirals out of control the voice in my head only grows louder but now it's screaming out to me...

Why?  Why?!  Wwwwwwhhhhhhhhyyyhhyyyyyyyyyyyyy...

Huh.  Why?  Well because you told me to.  Yes you.  Then who?  Well someone did.  Testing is good so more testing must be better right?  Because it adds value...

Aha!  Value.  So if I'm testing because it adds value then it would make sense to stop when it stops adding value wouldn't it?  If I'm so far over the edge that I feel like I'm taking value away then it's time for an intervention.  I like to know what I'm measuring so here's a quick list of the places TDD adds value for me:

• It helps me write code with less bugs.
• It proves that the code I have tested works as I expect it to.
• It enables me to refactor more effectively when changes are needed.
• It helps me design better interfaces for users of my code.
• It makes the world a better place (?!).

So when don't these things apply?

Where Did My Value Go?

More and more of what I do with ASP.Net is getting wrapped up in whiz bang designers with draggy droppy widgety things that try to make the endless configuration files we've introduced more palatable.  The nerd in me doesn't want this to happen because he loves code and I'm inclined to support him because it's pretty hard to wrap a meaningful test around a configuration file.

Lately I'm trying my best to be cool and hang out with the Web 2.0 crowd.  That means the combined total of the HTML, CSS, JavaScript, AJAX and configuration files for any particular page add up to a lot more potential mistakes than there ever could be in the short code behind I need to support them.  I can test aspects of JavaScript with something like QUnit and do my best to cover the functional aspects with Watin but there is barely even a glimmer of hope for the CSS and visual aspects of the site.

If TDD doubles the value of my work but means it takes me twice as long to do it have I actually gained anything?  When it comes to the code heavy business domain layers of my applications I'm quite confident saying I'm getting more than double the value.   In the UI I'm covering less than half of the meaningful decisions that go into any particular web page so its much harder to justify.

A Hard And Fast Rule For Sometimes

Every time I say it depends a thousand clients cry out in pain so block your ears, here it comes again.  There are many different flavours of web and it's all about the right tool for the right job.  Since my brain isn't likely to last long if I go through this little episode of insanity on every project here's my definitive guide to what you might maybe possibly should do.

Your web projects will benefit from TDD when:

• Your project is big.
• Interesting things happen in the code behind for your pages.
• You will have to support, maintain or extend the project in future.
• You are really building an application that just happens to be exposed via the web.
• Bugs are a big deal and your client is prepared to pay to prevent them.
• Your team knows how to do it and you already have the processes in place.
• You unit testing is combined with acceptance testing to make sure you build the right thing.

It will cost you more pain than it's worth when:

• Your project is small.
• The pages in the site present content and don't have any interesting functionality.
• Introducing MVC or MVP patterns causes more complexity and introduces more potential bugs than it saves.
• You're going to deploy the code and forget about it.
• You are building visual marketing driven sites.
• Bugs are not the end of the world and the client is prepared to deal with them as they come.
• You have other quality control processes that are working effectively.

Since those sets of conditions exclude a few pretty common scenarios here's a couple where you've got bigger issues than whether or not to TDD:

• You don't know what your project is.
• Your client doesn't believe in bugs.
• You don't have any quality control processes at all.

A Peace Offering

So I'm still allowed to hang out with all the cool agile kids I should clarify what I didn't mean in the post title.  I'm trying to say that for me test driven development isn't a linear trade off against productivity.  It's more like some bizarre polynomial function they tried to get you to derive in that algebra paper you failed at university.  Used in the right places it's a tool that can add a tremendous amount of value to the work you do but that value quickly degrades in certain scenarios so put some of that gray matter to use and figure out what it means for you.

Certainly some of my frustration deserves to be directed at current tool sets because they aren't designed for the kind of workflow I'm trying to achieve.  One thing I'll be looking at in future is the tools and the way in which I'm applying them.  My hope is that I can shift the balance to a point where TDD offers me a lot more values in the untamed wilderness of my UI.

Labels: ASP.Net, Style, TDD

Session state is one of the easiest and most common places for ASP.Net developers to store user related data but I have noticed that not many developers fully understand its implications or the other options available.  Session state has some interesting behaviours and potential performance pitfalls that are not often noticed until an application is in production and affecting users.

In the interests of avoiding fixing these issues for you in the future I'm going to quickly run through some of the other places you can store data and my thoughts on when you might want to consider each of them.

Session State

Session state is provided by ASP.Net for each unique user of the application and data persists until that users session ends.  Session state is particularly convenient to use as stored data is available across all pages and requests for the user.

Any data you place in session state will consume server resources and if you have many users this can quickly become a problem.  Depending on how your site is configured you will either be consuming memory on the web server itself or resources of the state server.

When you place data in session state you need to consider whether you care if that data is lost.  There are a huge number of triggers that can cause the server to end a users session and some of them such as application pool recycles are beyond your sites control.  If you session is stored out of proc this is less of an issue.

The most common problem I have seen is developers placing data into session state when the data really relates to some action that the user is performing on a page.  It is not uncommon for users to work with multiple browser windows which can cause very difficult to trace problems with pages competing for session state.  Consider careful use of view state as an alternative.

View State

View state is a mechanism provided by ASP.Net that allows you to store data in the rendered HTML that is sent to and from the users web browser.  View state is best used for data that relates to the current state of a page.  It is not suitable for any information that needs to be available between different pages.

The most important thing to consider with view state is that you are trading server memory for bandwidth which lessens the load on the server but increases response times for the user.  Also be careful with security sensitive data as although view state is encrypted you are much safer keeping such data within the trusted environment of your server.

Unfortunately view state is enabled by default for ASP.Net controls so it is very easy for your HTML to grow out of control and become a real problem.  This is typically caused by repeating controls such as data grids.  Because of these issues developers are hesitant to use view state but with intelligent application it is very powerful.

In some ways view state is more durable than session state as data can sometimes be persisted on the client side even if the users session is lost.

Hidden Form Elements

A hidden form element is simply an HTML input tag with its type set to hidden.  View state is actually implemented using a hidden form element so all of the implications of using view state apply here.  Learning more about this method might provide you with a deeper insight into how the web really works when you strip away ASP.Net so it is important.  A little more detail is provided in part 2 of my series on web 2.0 forms.

Form elements can be especially useful for communicating with JavaScript code that will run in the clients browser.  Data can be sent to the browser and then modified by a script and sent back as part of a post back or even posted to an entirely different page.  This enables some scenarios that are not possible with the options provided by ASP.Net.

Data stored this way will not be encrypted unless you do it yourself so be security conscious.

Static Variables

For those developers who have not worked with ASP.Net before static variables seems like an obvious place to store data but my advice would be to use them as a last resort and only after you have carefully considered the alternatives.  In most cases the application cache is a better choice.

There are potentially serious performance issues with storing data in static variables as it bypasses all of ASP.Nets built in mechanisms for handling server resources.  Unlike the various caches available there is no way for ASP.Net to release static variables when the server is running low on memory which may mean your application ends up restarting more frequently.

The lifetime of an web site hosted in IIS is much less predictable than that of a traditional application where static data is guaranteed to be around as long at the application is running.  IIS may close and restart the application at any time for a number of reasons causing static data to be reset.  This might happen between pages requests in what a user considers to be a session.

If your application will be deployed in a server farm then you need to be aware that your static variables will no longer have a single instance but will instead exist on every server in the farm.

Application Cache

Application cache is a good choice when you want to cache data that is not relevant to any particular user but is important for performance reasons.  A typical example is loading a lookup list of countries from a SQL server and caching it on the server as it will be used frequently without changing.

There are some multi threading issues to consider with the application cache as multiple web requests will be accessing it at the same time but if you are using it for simple data caching this should not be too bad.  As with static variables you will have one application cache per server in your web farm so don't rely on the data to be unique.

Application cache is managed by IIS so you are allowing the server to manage its resources more effectively which is important for scalability.  This is relevant if you have a single application that must scale to a great load or many applications that are sharing a single server.

SQL Server

A SQL server is the most robust way to persist data that needs to survive even if the users session ends, the application is restarted of the server is rebooted.  SQL servers are designed with performance in mind but remember that you will still be making a round trip on the network which can add precious milliseconds to your response time.  This might not matter much to your users but it is using additional server resources that could be used to process other requests.

Accessing a SQL server directly might not be your best bet as both session state and ASP.Net profiles can be configured to use a SQL server for storage and provide a higher level interface.

ASP.Net Profile

The ASP.Net profile features added in .Net 2.0 are an excellent way of storing user specific data which needs to be more durable than typical session state.  The profile system lets you create strongly typed properties to store your data in which can make the programming experience much more pleasant.

The profile system can be configured with different providers but works with a SQL server out of the box.  If you are already using ASP.Net membership then I would strongly recommend looking at the profile system as they work very well together.

There are some additional setup steps required to use the profile system and once again you are consuming server resources but there are features which allow you to purge old profile data to keep storage requirements under control.

It's not immediately obvious but profile data is available for both anonymous and authenticated users.  I consider it to be a very strong alternative to data traditionally stored in user cookies.

Cookies

Cookies are a feature provided by the users browser and allow you to store small pieces of data on the clients machine.  The advantage of free storage is offset by a number of severe restrictions which mean cookies are only useful for small pieces of data where security and robustness are not issues.

The biggest restriction is that each cookie has a maximum size of 4KB.  If you are not careful even a users shopping cart data can fill 4KB so try to avoid any data that does not have a fixed size.  Cookies will be sent to the server with each request the client makes to your site so they will consume some bandwidth.  It is less common these days but browsers may still have cookies disabled and you will either need to provide an alternate mechanism for those users or at least inform them why your site will not work correctly.

The cookies stored on the clients machine are beyond your control so be prepared for them to be deleted or tampered with.  In particular avoid storing any security sensitive information as the cookies will be easily viewable by the user and potential attackers.  On the plus side cookies are normally around for much longer than the users session so are frequently used to remember information about the user that is still relevant the next time they visit.

A less obvious issue to consider is that most automated web agents such as search engines will not support cookies and thus will not be able to use your site if you rely on them.

Session state uses cookies to uniquely identify users by default so the issues outlined here apply to session state as well.

Query String

Storing data in the query string involves appending to the URL of a request being made.  It is particularly useful for communicating between different pages so might be an option for smaller pieces of data you were considering placing in session state.

The query string is one of the least reliable and least secure ways to communicate and it is not uncommon for query strings to arrive mangled or completely missing.  The big advantage to the query string is that it can be used to communicate between completely different sites and doesn't carry many of the security restrictions that other cross site communications do.

Be careful not to place too much in the query string as it is displayed to the user and the URL is intended to be a resource locator, not a database.

Other

This list is far from exhaustive but covers the most common alternatives.  Among the options I haven't covered are web services and integrations with various third party components that can store data for you.

It is also possible to use the basic options in unconventional ways such as implementing a per user store in the application cache or removing view state from the rendered HTML and storing it on the server instead.  These options are useful in specific scenarios but are often complicating things more than is necessary.  A good grasp of the basic methods will meet all but the most obscure of your needs.

Whichever option you use be aware of the security implications as even automated scripts are becoming very good at exploiting any weaknesses in your site.  Do not trust any data that is sent to you by the client, even it it has been encrypted.  While most users are fairly harmless you never know if their machines have been compromised without their knowledge.

Conclusions

Know what the options are and when to use them.  If session state is the right tool for the job then use it, otherwise find an alternative.

Labels: ASP.Net

I think the new ASP.Net MVC framework from Microsoft looks really good.  I also think MonoRail looks really good.  If I was in a product shop doing custom development from the ground up I'd be all over them but for better or worse I'm in a services role at the moment and that means working for clients that don't have endless budgets and timelines.

Typically this means we are building on top of SharePoint, EPiServer or another platform designed to be eighty percent "out of the box" (features, not effort).  At least at the moment all of the platforms we are working with are based around the existing web forms model and I doubt that is going to change any time soon as web forms is probably better suited to them.

I will certainly be keeping an eye on the MVC frameworks incase they are one day applicable but for now I'll be focusing my efforts on improving the way I work with the web forms model.  TDD, MVP, DDD and a whole truck load of other TLAs are applicable here but the real question is how well can we apply these ideas within the constraints of the platform providing the other eighty percent?

Labels: ASP.Net, MVC