I just have to point out this little snippet because it's one of those rare it just works experiences when you can solve a complex problem with a single line of code.

I'm toying with some ray tracing code, one of the classic hackers playgrounds.  Ray traced images can take a long time to render and I have a pretty modern machine so I wanted to make best use of my dual CPUs.  I began with a loop that looked like this:

While the rendering is in progress you can see the image gradually filled in from left to right.  Pretty simple but writing the code to divide the loop up into sensible pieces and distribute it to a pool of threads was going to be more work than the actual ray tracing itself.  Fortunately I happened upon the Parallel Extensions library which let me implement all of the above by referencing a .dll and changing a single line of code:

Now that's magic.  Rather than rendering the image one line at a time the library has decided that 4 threads would be optimal for my dual CPU configuration and chunked the work accordingly.  The image below is taken part way through a render and shows that the loop has been broken into segments of about 20 pixels each:

Now I have more time to spend sorting out the horrible colours on that sphere!  There are a couple of things I should mention before closing:

• You still need to be careful not to block the UI thread for any length of time.
• At some point you have to combine the results back into a single thread so you can output the image.
• Ray tracing is an ideal candidate for parallelism as each pixel can be rendered independently of any other but chances are your application is going to be a little trickier.
• The library is still a CTP so the usual warnings about pre release software apply.
• Immutable state is your friend.

There is a good amount of advice in the documentation that comes with the library and a number of articles online if you want to know more.

Labels: Parallel Extensions, Ray Tracing

Bookmark this on Delicious

This is a consolidation of my recent high level thoughts on how to mix test driven development with ASP.Net.  In my quest for Sanity Driven Development I've gone from having no tests to having too many tests and eventually settled into a nice middle ground.  The following guidelines assume certain types of projects:

• Bespoke or heavily customised web or intranet sites.
• Mid sized projects that need some process but still have tight budgets.
• Typical business focused sites dealing with data and dynamic functionality.

There are lots of projects that fall well outside of this scope but it covers the most common type for me, the one that is just big enough to be a real headache when you get it wrong.

Goodbye MVP

I've had a strained relationship with the MVP pattern and we've finally decided to go our separate ways.  This might be a bit controversial and goes against a lot I've been hearing for the last few years so give me a moment to explain exactly what I mean and why.

Introducing the MVP pattern to a typical ASP.Net site has one primary goal, increasing testability.  Yes, there are other significant advantages to the MVP pattern in general but you already have access to most of these with the ASPX file representing your view and the presenter logic in the code behind.  My argument is that with the rest of the development guidelines that follow there will be so little left in the code behind that the benefit of extracting it into yet another layer will not be worth the additional complexity.

There will be times when the MVP or MVC pattern solves a particular problem for you and by all means dust it off and apply it when the time is right.  I am simply saying that it probably doesn't offer enough value in the majority of your web pages.

Inexperienced web developers have a tendency to load all the code they can think of in to the code behind, probably because they were never taught any better.  The rest of this article is really about extracting that code from the code behind and opening it for better design and easier testing.

Start With The Domain

Incase you haven't heard, it's no longer cool to put your data access code in your code behind.  Assuming you have a data access layer of some kind going I strongly recommend you go and read the first half of Domain Driven Design by Eric Evans.  Personally I found the later parts of the book were less relevant but the core ideas behind the book are deeply important to the kind of projects we are discussing.

The most critical part is that you move all of your domain knowledge out of the user interface and into the domain layer where it belongs.  Here the business concepts driving the project should be presented independently of the technology being used to implement them.  This is where you can really let loose with the unit testing and get good returns because you are covering the knowledge that makes this project different from any other.

If you apply the ideas from domain driven design fairly liberally you should end up with very little interesting code remaining in your code behind files.  You should end up with a strong model backed up by a number of services which provide the business processes for the application.  Most of the code that remains in the code behind files is going to be specific to the implementation of the views in ASP.Net.

Consolidate Your Views

Any business application is likely to end up with a lot of forms.  These forms all tend to be fairly similar and building them can be a long and tedious process.  Unfortunately there are a huge number of details that go into these forms and they are often details that have a big impact on the end users.  Code behind files get filled with lines of code reading values from text boxes, validating them and converting them to other types to be passed to the data access layer.

Look for parts of your user interface that represent business or domain concepts and extract them into a set of common controls.  An example of such a concept might be a shipping company that frequently has users entering weights into the site.  Weight isn't a concept that is represented anywhere in the ASP.Net controls yet it is critical to the project and comes with it's own set of rules about how the user should be able to interact with it.

The naive approach is to represent weight as a text box on each page and include all the details that go along with it in both the ASPX and the code behind that make up that page.  Even a simple example such as weight needs client side validation, server side validation, text parsing, rendering and the layout of the actual controls presented to the user.

Take all of these details and wrap them in a single user control that can be reused throughout the user interface.  Make sure that the control takes weights as input from the domain and returns weights as output to the domain.  Place the control by itself on a simple page that serves as a test harness so you can use a tool like WatiN to get comprehensive coverage of the client side behaviours.  Complex behaviours such as parsing units of weight from input strings should be broken into independent components that can be unit tested in isolation from the control itself.

Creating these mini windows into the domain will simplify your general views and help you to create a much more consistent and powerful user interface.  Using combinations of smaller controls you may be able compose larger reusable controls which represent more complete ideas from the domain.  Don't forget that by doing this you reduce the amount of code in your user interface and therefore reduce the surface that is difficult to test effectively.

Apply Functional Testing

By now the code behind files should consist mostly of mapping user interface controls and events to the corresponding models and services in the domain.  These details define the behaviour of the view and unfortunately are tricky to unit test no matter how many layers of MVP you throw at them.

There is also a huge amount of detail left behind in the implementation of the site that we need to be able to verify.  As well as the ASPX files this includes a lot of configuration information wrapped in XML files and the settings of the server itself.

Use an automated functional testing tool such as WatiN to create broader user story style tests which cover the majority of these details.  Expect to settle for much less than complete coverage with these tools as the technology is still shaky and performance of the tests is likely to be slower than desired.  The advantage is that these tests are much more realistic as they closely mimic real user interactions with the site.

Expand Your Toolbox

As the web matures more and more of the features we add to our sites are implemented outside of the core programming languages we know how to test.  Anything labelled Web 2.0 is likely to be poorly covered by traditional methods but fortunately others have already tried to address many of the issues and have left behind clues if not complete tools we can use.

jQuery is an excellent example where the team has provided both an excellent JavaScript client library and a simple framework for writing and running client side JavaScript unit tests.  All that remains for us is to use tools such as WatiN to automate the execution of these tests and the collection of the results.  W3C and the various standards it promotes are becoming critical to delivery quality projects and once again there are tools and validators that can be easily automated to provide ongoing quality checks for your projects.

The number of technologies that go into a successful site has sky rocketed so try and find ways to automatically verify them.  Examples I haven't discussed include the way that search engines crawl the site and how the site will be used or abused by other programmers.

Conclusion

There are a few key themes in these ideas and it shouldn't be any surprise that they are the same ideas used in general software construction.  In particular I want to point out clean separation of concerns and careful management of dependencies between layers in your application.  Combine these with thorough but pragmatic testing using each tool until it stops delivering value and no longer.

Keep reading, keep learning and keep challenging anything that does not work for you.

Labels: ASP.Net, Style, TDD

Bookmark this on Delicious

There are a bunch of basic quality checks that should happen on every web page that goes live and they are tedious to do manually.  I am working on a few helpers to integrate these checks into an xUnit test suite for projects:

I have used the RowTest extension for NUnit as an example of how to test a number of URLs with as little fuss as possible.  The test is intended to sit alongside functional Watin style tests and be a basic check to stop developers handing rubbish off to QA.  There are lots of ways to structure the test but keep the following in mind:

• TimeTakenToExecute takes an Action so that you can a larger block of code or another function to it.  This makes it a nice generic timing helper for tests in general.
• GetResponse currently returns the response from a standard .Net WebRequest.  Reading the HTML back from a Watin session may be interesting as well though less generic.
• The ViewState is read from the html using a Regex which makes it a little less fragile with invalid HTML.  I need to figure out the exact set of characters that are legal in view state and tweak the pattern.
• The W3 validation constraint is actually calling the live W3 validator API and returning the errors it gives.  It only validates HTML at the moment though there is a possibility of adding checks such as CSS, accessibility and mobility in future.
• As mentioned in their documentation I need to install my own instance of the W3 validator to avoid abusing it and being blocked (again).

I'm interested to hear from you if you have any feedback or suggestions for similar things you'd like to be able to check.  I intend to push this out as a simple .dll you can link against where needed.

Labels: ASP.Net, TDD

Bookmark this on Delicious

This is my favourite kind of post because it spawned from a question somebody asked me earlier in the day.  The question was about storing ASP.Net viewstate on the server side to avoid the time and bandwidth used round tripping it to the client and back.  Though I was asked about implementation details at the time I want to step back and have a more complete discussion around whether this is a good thing to be doing or not.

Make Sure It's The Right Problem

The challenge is really a performance optimisation and we are normally after a few key factors which can be easily measured:

• Rendered size of the HTML.
• Time taken for the client to load the page.
• Bandwidth used by both the client and the server.

Each of these factors has a huge number of potential causes so it is important to make sure that viewstate isn't a misdiagnosis causing you to spend a lot of effort in the wrong place.  I do want to go into other possible causes some time but for now I'm going to assume that viewstate is the problem.  The ideal process for any sort of performance optimisation goes something like this:

1. Define a performance goal.
2. Measure performance.
3. If the goal is met, stop.
4. Make a change.
5. Go to 2.

Structured programming aside the most important thing is the measuring.  There is a very simple way to measure your viewstate that anyone can do with the tools they already have.  First browse to the target page in your web browser and view the source code.  Delete everything except the encrypted viewstate tag.  Save the file to disk and have a look at it's properties to see the size.  Here's an example from Intergen's homepage:

235 bytes, pretty good so I wont be spending any time on that.  This isn't a precise measure if you do it consistently it is quite effective for measuring the effect of changes.  You may want to measure the overall page size rather than just the viewstate as you typically change them together when optimising.  This is a pretty primitive way to measure things so I strongly recommend you go and grab something like Firebug for Firefox which is my number one web debugging tool.

Sweep It Under The Rug

The problem I have with the original proposal to store the view state on the server side is that is hiding the symptoms more than it is addressing the cause of the problem.  Viewstate simply shouldn't be big enough to cause a problem and if it is you need to look carefully at the way your pages are constructed.

Storing viewstate on the server has a number of subtle side effects you should carefully consider.  If you haven't done so already I suggest you have a read of my earlier post on different places to store data in ASP.Net.  In general you will find that you end up using server resources and processing time to reduce the rendered page size.  This might be a reasonable tradeoff but it also changes the characteristics of viewstate.

Viewstate is per rendered page instance so if you store it on the server you will need a way to match it to each instance that is sent to the client.  The first instinct is usually to push the data into session but it is becoming quite common for users to browse with multiple windows open so you will need to cache viewstate for multiple page instances at any one time.  The same problem is caused by the insidious back button.  Since you are doing this because your viewstate is too large the persistence of the data on the server is likely to cause it's own performance problems.

You can push the data out to a SQL database for more robust storage but you will then have the problem of deciding when to expire the data and remove it from the database.  This also adds two additional database queries to each page request which can be quite significant when you take network latency into account.

In general my advice is to leave the viewstate where it is and concentrate on making sure you don't have too much of it instead.

How Much Is Too Much?

Just how much viewstate you should allow in your pages depends a lot on who your target audience is.  In a tightly controlled intranet environment where all users are on high bandwidth connections no one is going to notice 50k of viewstate.  On the other hand mobile devices are becoming more common so watch out for the guy trying to view the company intranet on his cell phone from a back country paddock.  On the public Internet conditions are much less predictable and I would suggest setting a much lower limit for yourself such as 5k.  These are very generous limits and a lot of people would push you for a lot less but that's a call you need to make for your projects requirements.

Where possible always develop with viewstate in mind and make checking page sizes part of your normal testing cycles.  I highly recommend setting up a test environment that closely resembles your target audience including low bandwidth connections where appropriate.  This can be achieved using a number of tools such as applications you can install on your computer and certain firewalls.  With a little effort to keep things light you should be able to keep viewstate down to one or two kilobytes per page.

If you are really determined you can work  quite happily with no viewstate at all though ASP.Net makes this difficult out of the box.  Viewstate is a huge convenience feature that can save you a lot of time if used carefully.

Strategies For Reducing Viewstate

The best advice I can give for reducing viewstate is to find all of the data grids in your site and delete them.  Repeating controls such as a data grids, data lists, repeaters, grid views and drop down list are the number one culprit for generating viewstate.  Sometimes the problem isn't with the control itself but with controls nested inside it that are being repeated a number of times.

A lot of controls generate viewstate unexpectedly.  A great example if the label control which Microsoft suggests you use in all of its tutorials.  As well as saving all of its properties a label control generates viewstate for the text you place in it so you are usually better off using a literal control and formatting the text with CSS.  One label by itself isn't likely to cause a problem but when it is inside a repeating control it can be significant.

Every ASP.Net control has an EnableViewState property which you can set to false to make sure that that control doesn't generate any viewstate at all.  It's the next best thing to deleting the control entirely.  Personally I feel that this should default to false so that you have to think about it and turn it on only when necessary.

Since you can't just delete every offending control you need to be a little more cunning.  Start by asking yourself why you actually need viewstate and you'll probably find that there are a few key properties of each control that you are interested in.  Consider disabling the default viewstate for the control and storing these properties yourself.  Typical examples include sorting and paging information for lists of data.

Disabling view state often has the effect of making your data disappear on postback.  Think about this carefully because it's a good hint as to why you are getting so much viewstate in the first place.  Recently I have started to favour disabling viewstate and rebinding all data on a postback.  This usually generates more load on the database but ASP.Net has excellent support for server side data caching to mitigate this.

Finally remember that other web frameworks get along just fine without viewstate.  Viewstate and postbacks are simply wrappers on basic HTTP technology that trade flexibility and control for increased developer productivity.  As with all generalisations they come with drawbacks and in this case it's performance that takes a hit.

If you are really serious about the quality of your pages you need to get intimate with the layers underneath ASP.Net.  Some of my earlier posts on lightweight web forms might be a good starting point and they also include some information on reducing page size in general.  If you think you have a good handle on HTML I suggest book marking the HTML specification at W3 because it's packed with all the details and insights you need to achieve HTML Zen.

I am considering writing a little more about page size in general but if there are any particular topics you'd like me to cover in more depth just let me know.

Labels: ASP.Net

Bookmark this on Delicious

Have you ever noticed how fragile your productivity is as a developer?

I started working with Navision a couple of weeks ago which has it's own programming language called C/AL.  No one with any experience in it was around at the time and I felt like a child as I stumbled about trying to figure out how to do things I've been doing for years in C#.  Picking up a new technology (or old in this case) is a great bring any progress you were making to a grinding halt.

This week we had a power outage which throws a real spanner in the works for any IT professional whose whole reason for working suddenly goes offline.  Apparently bringing everything back online isn't as straight forward as pushing a button.

Today I attempted to log in to finish off a task I said would get done but some of the files I need have been misplaced.  I'm not sure if they've been deleted or just moved but even though I have lots of power the task is effectively a write off.

Things seem to be bigger and more complex than I think they should be.  Simplifying things may make them more or less fragile but it definitely makes it easier for me to grok them and therefore fix them when they are broken.  So next time you want feedback on something I'd like to ask for simple, light and consistent.

Cheers.

Labels: Productivity

Bookmark this on Delicious

One of the reasons I started this blog was so that I could experiment with a number of things including Google AdSense.  The way AdSense selects ads is by crawling your site on a regular basis and trying to determine keywords to match up with the ads.  I've been pretty impressed with the relevance of the ads placed so far as they have generally related to programming jobs, custom control vendors and web hosting.

As you probably noticed my previous post on finding the value in test driven web development was written with a little more, erhem, passion than normal.  I was particularly amused the first time I went back to the post to check the comment that had been left by one of my colleagues:

I'm not sure if it was total coincidence or not but it certainly looks like Google AdSense is doing it's job because the first ad, in prime position at the top left of the page, was this:

Oh dear.

Labels: AdSense, Blogging

Bookmark this on Delicious