ObjectSharp

Bring on the 3rd major release of WPF (3.5 SP1 Beta is ready!)

robburke.NET — Mon, 12 May 2008 17:25:10 GMT

A “Week of WPF” begins on Channel 9 with the announcement that the third major release of WPF (3.5 SP1) has gone beta! Tim Sneath’s blog entry has fantastic detail about what’s coming in this release and all the download/update details (Silverlighters: see caveat below). Folks attending my “WPF for Line Of Business apps” presentation on Thursday [...]...(read more)

Live Search is actually getting pretty good these days!

Barry Gervin — Mon, 12 May 2008 16:52:47 GMT

Live Search got a refresh recently and it's actually pretty good, dare I say may be even better than Google.

Performance: The load times are very snappy now and it feels pretty much on par with google performance.
Image Search: Functionality on Image Search has been superior for some time with the ability to size the thumbnails and I quite like the "add to scratchpad" feature but wish it was available for video and web content as well. The ability to refine the search by size, aspect ratio, colour, Style (photo vs. illustration) and Face (i.e. head & shoulders, just face, etc.) is brilliant. It's not 100% accurate, but it is still quite useful. Google can do a subset of this, but frankly, never noticed it until I went looking for it. These features are more discoverable in Live. I have to say that I'm in love with the fact that I can preview the context of the pages split screen so I don't leave my list of results. As far as relevancy goes, I'm going to have to give it a try for a week. Google does a better job finding pictures of me, but Live does a better job of finding black & whites of Darryl Sittler. Live found 12 good b&w's of the Sitler while google found absolutely nothing of relevance. Unfortunately, Live fails my vanity image search and google prevails with more pictures of me.
Video Search: I honestly have never used video search on live or google until this blog entry. Live has a nice feature where it can play videos from the results page by just hovering over the thumbnails. It can't do it for every video type, some you have to click on but it is a lovely feature when it works which from what I can tell is about 60% of the time. Google only manages to get 10 thumbnails per page where live gets twice that. Live brought back a few extra total results. It seems like google favours primarily youtube.
Map Search: Mileage seems to vary greatly depending on where you are searching. Live has the great integrated 3D maps with buildings, contoured terrain and birds eye view vs. google with street view as it's killer feature. For Canadians, google hasn't made any of these features available yet. Live has thrown us Canadians a few bones with birds eye view available in Vancouver, Calgary, Toronto, and Montreal. Live also gives us some traffic in Vancouver, Toronto, Montreal, and Ottawa.

Live maps has some great directional tools as well. I like the feature where I can just click on a point and ask for 1 click directions - which will give me generic instructions on coming to a location from all major available directions/arteries. This is great if you want to include directions on an invitation but you don't know where everybody is coming from. The Live Map collection editor and the ability to easily add pushpins and draw (with distances) on the map is great.

One thing I do like about Google maps is the ability to take a set of directions and drag segments around to tell it how I really want to get somewhere. All in all though, I much prefer (as a Canadian) Live Maps as the most

Plain Old Web Relevancy: Plain old web results are perhaps the most important criteria. I think this will be impossible to provide any meaningful observations after spending < 1 hour using live search. I'm going to give live a shot as my default search engine for the next week and see how it does but a few immediate observations:

Live wins on relevancy for the "Barry Gervin" vanity search, easily. By the bottom of the first page (items 7th-9th), Google is showing results that are irrelevant - in fact, no mention of Barry or Gervin on any of those pages.
Searching for some things that I know are on MSDN Forums - posted yesterday, google seems to be crawling that site faster than Live. Google still has a dedicated newsgroups search engine. I used to use this a lot more than I currently do so I have mixed emotions about live search. In fact, in some of my tests for searching things that I knew were known to exist in newsgroups, live search actually referred me to google groups - so it does appear that live is indexing the google groups search engine. I think the thing I find most appealing about google groups is not the content that it finds, but the way it presents it, showing me what group a result was found in, and then making the detailed results essentially a web based NNTP client. Live could do much more than they are doing now that's for sure, I'm just not sure that it really matters.
I'm making more and more use of google alerts to have things I am constantly tracking or researching searched and emailed to me. Live has no equivalent to this.

I'm going to give Live Search a trial as my default search engine for the next week or so and see how it goes. I'm optimistic.

Visual Studio 2008 SP1 Beta & SQL Server 2008

Barry Gervin — Fri, 09 May 2008 20:30:10 GMT

A quick heads up to let you know that VS 2008 Service Pack 1 is now available (links below). It typically takes a couple of months from this point before we'll see a final release.

This Service Pack includes new cool feature:

ADO.NET Entity Framework
ADO.NET Data Services
ASP.NET 3.5 Extensions (aka Cicero) which includes MVC Framework, ASP.NET Silverlight Controls, Dynamic Data, New Ajax Nav/History Controls.

One interesting point is that MS is going to simultaneously ship SQL Server 2008 which actually has a hard dependency on SP1.

I thought I’d take a moment to highlight some new features that Dev’s would care about in SQL Server 2008.

Change Data Capture: Async “triggers” capture the before/after snapshot of row level changes and writes them to Change Tables that you can query in your app. They aren’t real triggers as this asynchronously reads the transaction log.
Granular control of encryption, right through to the database level without any application changes required.
Resource Governor – very helpful when you allow users to write adhoc queries / reports against your OLTP database. Allows a DBA to assert resource limits & priorities.
Plan Freezing – allows you to lock down query plans to promote stable query plans across disparate hardware, server upgrades, etc.
New Date, and Time data types, no longer just DateTime types that you have to manually parse out the time or date to just get the real data you want.
DataTimeOffset – is a time zone aware datetime.
Table Value Parameters to procs – ever want to pass a result set as an arg to a proc?
Hierarchy ID is a new system type for storing nodes in a hierarchy….implemented as a CLR User Defined Type.
FileStream Data type allows blobish data to be surfaced in the database, but physically stored on the NTFS file system. ….but with complete transactional consistency with the relational data and backup integration.
New Geographic data support, store spatial data such as polygons, points and lines, and long/lat data types.
Merge SQL statement allows you to insert, or update if a row already exists.
New reporting services features such as access to reports from within Word & Excel, better SharePoint integration

Personally, haven't spent any time with SQL Server 2008 but that's a great set of new features that I can hardly wait to start using in real-world applications.

Downloads

· VS 2008 SP1 : http://download.microsoft.com/download/7/3/8/7382EA08-4DD6-4134-9B92-8585A5B07973/VS90sp1-KB945140-ENU.exe

· .NET 3.5 SP1 : http://download.microsoft.com/download/8/f/c/8fc1fe13-55de-4bf5-b43e-375daf01452e/dotNetFx35setup.exe

· Express 2008 with SP1:

o http://download.microsoft.com/download/F/E/7/FE754BA4-140B-413C-933F-8D35FB150F12/vbsetup.exe

o http://download.microsoft.com/download/F/E/7/FE754BA4-140B-413C-933F-8D35FB150F12/vcsetup.exe

o http://download.microsoft.com/download/F/E/7/FE754BA4-140B-413C-933F-8D35FB150F12/vcssetup.exe

o http://download.microsoft.com/download/F/E/7/FE754BA4-140B-413C-933F-8D35FB150F12/vnssetup.exe

· TFS 2008 SP1: http://download.microsoft.com/download/a/e/2/ae2eb0ff-e687-4221-9c3e-9165a942bc1c/TFS90sp1-KB949786.exe

Feedback Forum: http://go.microsoft.com/fwlink/?LinkId=119125

Attaching old Content Database to new Web Application in MOSS 2007 or WSS 3.0

max — Thu, 08 May 2008 20:20:36 GMT

If you have recently re-attached your content database to the new web application in SharePoint 2007 (whether because you have moved your SharePoint install or simply because you had to re-create the web application on your current server), you might have noticed a new error (Event ID: 5555; Event Source: Office SharePoint Server) occurring hourly in the Event Log of your web-front SharePoint server.

The description of the error tells you to run stsadm – o preparetomove, which doesn't seem to fix the problem, unfortunately. This command only prevents the error 5555, if you run it before moving and re-attaching your content databases. To fix error 5555, you need to run stsadm –o sync instead.

In my case, running the following command made error 5555 disappear:

stsadm -o sync -DeleteOldDatabases 0

As usual: ALWAYS BACKUP YOUR SHAREPOINT BEFORE MAKING ANY CHANGES TO ITS CONFIGURATION

The Entity Framework vs. The Data Access Layer (Part 1: The EF as a DAL)

Barry Gervin — Wed, 07 May 2008 04:06:37 GMT

In Part 0: Introduction of this series after asking the question "Does the Entity Framework replace the need for a Data Access Layer?", I waxed lengthy about the qualities of a good data access layer. Since that time I've received a quite a few emails with people interested in this topic. So without further adieu, let's get down to the question at hand.

So let's say you go ahead and create an Entity Definition model (*.edmx) in Visual Studio and have the designer generate for you a derived ObjectContext class and an entity class for each of your tables, derived from EntityObject. This one to one table mapping to entity class is quite similar to LINQ to SQL but the mapping capabilities move well beyond this to support advanced data models. This is at the heart of why the EF exists: Complex Types, Inheritance (Table per Type, Table per Inheritance Hierarchy), Multiple Entity Sets per Type, Single Entity Mapped to Two Tables, Entity Sets mapped to Stored Procedures or mapping to a hand-crafted query, expressed as either SQL or Entity SQL. EF has a good story for a conceptual model over top of our physical databases using Xml Magic in the form of the edmx file - and that's why it exists.

So to use the Entity Framework as your data access layer, define your model and then let the EdmGen.exe tool do it's thing to the edmx file at compile time and we get the csdl, ssdl, and msl files - plus the all important code generated entity classes. So using this pattern of usage for the Entity Framework, our data access layer is complete. It may not be the best option for you, so let's explore the qualities of this solution.

To be clear, the assumption here is that our data access layer in this situation is the full EF Stack: ADO.NET Entity Client, ADO.NET Object Services, LINQ to Entities, including our model (edmx, csdl, ssdl, msl) and the code generated entities and object context. Somewhere under the covers there is also the ADO.NET Provider (SqlClient, OracleClient, etc.)

To use the EF as our DAL, we would simply execute code similar to this in our business layer.

var db = new AdventureWorksEntities();
var activeCategories = from category in db.ProductCategory
                 where category.Inactive != true
                 orderby category.Name
                 select category;

How Do "EF" Entities Fit In?

If you're following along, you're probably asking exactly where is this query code above being placed. For the purposes of our discussion, "business layer" could mean a business object or some sort of controller. The point to be made here is that we need to think of Entities as something entirely different from our Business Objects.

Entity != Business Object

In this model, it is up to the business object to ask the Data Access Layer to project entities, not business objects, but entities.

This is one design pattern for data access, but it is not the only one. A conventional business object that contains its own data, and does not separate that out into an entity can suffer from tight bi-directional coupling between the business and data access layer. Consider a Customer business object with a Load method. Customer.Load() would in turn instantiate a data access component, CustomerDac and call the CustomerDac's Load or Fill method. To encapsulate all the data access code to populate a customer business object, the CustomerDac.Load method would require knowledge of the structure the Customer business object and hence a circular dependency would ensue.

The workaround, if you can call it that, is to put the business layer and the data access layer in the same assembly - but there goes decoupling, unit testing and separation of concerns out the window.

Another approach is to invert the dependency. The business layer would contain data access interfaces only, and the data access layer would implement those interfaces, and hence have a reverse dependency on the business layer. Concrete data access objects are instantiated via a factory, often combined with configuration information used by an Inversion of Control container. Unfortunately, this is not all that easy to do with the EF generated ObjectContext & Entities.

Or, you do as the Entity Framework implies and separate entities from your business objects. If you've used typed DataSets in the past, this will seem familiar you to you. Substitute ObjectContext for SqlConnection and SqlDataAdapter, and the pattern is pretty much the same.

Your UI presentation layer is likely going to bind to your Entity classes as well. This is an important consideration. The generated Entity classes are partial classes and can be extended with your own code. The generated properties (columns) on an entity also have event handlers created for changing and changed events so you can also wire those up to perform some column level validation. Notwithstanding, you may want to limit your entity customizations to simple validation and keep the serious business logic in your business objects. One of these days, I'll do another blog series on handing data validation within the Entity Framework.

How does this solution stack up?

How are database connections managed?

Using the Entity Framework natively itself, the ObjectContext takes care of opening & closing connections for you - as needed when queries are executed, and during a call to SaveChanges. You can get access to the native ADO.NET connection if need be to share a connection with other non-EF data access logic. The nice thing however is that, for the most part, connection strings and connection management are abstracted away from the developer.

A word of caution however. Because the ObjectContext will create a native connection, you should not wait to let the garbage collector free that connection up, but rather ensure that you dispose of the ObjectContext either explicitly or with a using statement.

Are all SQL Queries centralized in the Data Access Layer?

By default the Entity Framework dynamically generates store specific SQL on the fly and therefore, the queries are not statically located in any one central location. Even to understand the possible queries, you'd have to walk through all of your business code that hits the entity framework to understand all of the potential queries.

But why would you care? If you have to ask that question, then you don't care. But if you're a DBA, charged with the job of optimizing queries, making sure that your tables have the appropriate indices, then you want to go to one central place to see all these queries and tune them if necessary. If you care strongly enough about this, and you have the potential of other applications (perhaps written in other platforms), then you likely have already locked down the database so the only access is via Stored Procedures and hence the problem is already solved.

Let's remind ourselves that sprocs are not innately faster than dynamic SQL, however they are easier to tune and you also have the freedom of using T-SQL and temp tables to do some pre-processing of data prior to projecting results - which sometimes can be the fastest way to generate some complex results. More importantly, you can revoke all permissions to the underlying tables and only grant access to the data via Stored Procedures. Locking down a database with stored procedures is almost a necessity if your database is oriented as a service, acting as an integration layer between multiple client applications. If you have multiple applications hitting the same database, and you don't use stored procedures - you likely have bigger problems.

In the end, this is not an insurmountable problem. If you are already using Stored Procedures, then by all means you can map those in your EDM. This seems like the best approach, but you could also embed SQL Server (or other provider) queries in your SSDL using a DefiningQuery.

Do changes in one part of the system affect others?

It's difficult to answer this question without talking about the possible changes.

Schema Changes: The conceptual model and the mapping flexibility, even under complex scenarios is a strength of the entity framework. Compared to other technologies on the market, with the EF, your chances are as good as they're going to get that a change in the database schema will have minimal impact on your entity model, and vice versa.

Database Provider Changes: The Entity Framework is database agnostic. It's provider model allows for easily changing from SQL Server, to Oracle, to My Sql, etc. via connection strings. This is very helpful for ISVs whose product must support running on multiple back-end databases.

Persistence Ignorance: What if the change you want in one part of the system is to change your ORM technology? Maybe you don't want to persist to a database, but instead call a CRUD web service. In this pure model, you won't be happy. Both your Entities and your DataContext object inherit from base classes in the Entity Framework's System.Data.Objects namespace. By making references to these, littered throughout your business layer, decoupling yourself from the Entity Framework will not be an easy task.

Unit Testing: This is only loosely related to the question, but you can't talk about PI without talking about Unit Testing. Because the generated entities do not support the use of Plain Old CLR Objects (POCO), this data access model is not easily mocked for unit testing.

Does the DAL simplify data access?

Dramatically. Compared to classic ADO.NET, LINQ queries can be used for typed results & parameters, complete with intelli-sense against your conceptual model, with no worries about SQL injection attacks.

As a bonus, what you do get is query composition across your domain model. Usually version 1.0 of a convention non-ORM data access layer provides components for each entity, each supporting crud behaviour. Consider a scenario where you need to show all of the Customers within a territory, and then you need to show the last 10 orders for each Customer. Now I'm not saying you'd do this, but what I've commonly seen is that while somebody might write a CustomerDac.GetCustomersByTerritory() method, and they might write an OrderDac.GetLastTenOrders(), they would almost never write a OrderDac.GetLastTenOrdersForCustomersInTerritory() method. Instead they would simply iterate over the collection of customers found by territory and call the GetLastTenOrders() over and over again. Obviously this is "good" resuse of the data access logic, however it does not perform very well.

Fortunately, through query composition and eager loading, we can cause the Entity Framework (or even LINQ to SQL) to use a nested subquery to bring back the last 10 orders for each customer in a given territory in a single round trip, single query. Wow! In a conventional data access layer you could, and should write a new method to do the same, but by writing yet another query on the order table, you'd be repeating the mapping between the table and your objects each time.

Layers, Schmayers: What about tiers?

EDM generated entity classes are not very tier-friendly. The state of an entity, whether it is modified, new, or to be delete, and what columns have changed is managed by the ObjectContext. Once you take an entity and serialize it out of process to another tier, it is no longer tracked for updates. While you can re-attach an entity that was serialized back into the data access tier, because the entity itself does not serialize it's changed state (aka diff gram), you can not easily achieve full round trip updating in a distributed system. There are techniques for dealing with this, but it is going to add some plumbing code between the business logic and the EF...and make you wish you had a real data access layer, or something like Danny Simmons' EntityBag (or a DataSet).

Does the Data Access Layer support optimistic concurrency?

Out of the box, yes, handily. Thanks to the ObjectContext tracking state, and the change tracking events injected into our code generated entity properties. However, keep in mind the caveat with distributed systems that you'll have more work to do if your UI is separated from your data access layer by one or more tiers.

How does the Data Access Layer support transactions?

Because the Entity Framework builds on top of ADO.NET providers, transaction management doesn't change very much. A single call to ObjectContext.SaveChanges() will open a connection, perform all inserts, updates, and deletes across all entities that have changed, across all relationships and all in the correct order....and as you can imagine in a single transaction. To make transactions more granular than that, call SaveChanges more frequently or have multiple ObjectContext instances for each unit of work in progress. To broaden the scope of a transaction, you can manually enlist using a native ADO.NET provider transaction or by using System.Transactions.

Live Search Maps - now with Bird's Eye View in Toronto

Barry Gervin — Tue, 06 May 2008 18:36:19 GMT

It's amazing what MS will give you, if you give them a Team Canada Hockey Jersey.

Entity Framework Links for April, 2008

Barry Gervin — Fri, 02 May 2008 17:10:02 GMT

During the past month, Danny Simmons let us all officially know that SP1 of VS 2008/.NET Framework 3.5 will be the delivery mechanism for the Entity Framework and the Designer, and that we should see a beta of the entire SP1 very soon as well. No release dates yet.
Speaking of the next beta, there have been some improvements in the designer to support iterative development. Noam Ben-Ami talks about that here.
There is also a new ASP.NET EntityDataSource control coming in the next beta. Danny demo'd that at DevConnections, and Julie blogged about it here.
In April, Microsoft released the .NET 3.5 Enhancements Training Kit. This includes some preliminary labs on ASP.NET MVC, ASP.NET Dynamic Data, ASP.NET AJAX History, ASP.NET Silverlight controls, ADO.NET Data Services and last but certainly not least, the ADO.NET Entity Framework. Stay tuned for updates
Julie Lerman has created a spiffy pseudo-debug visualizer for Entity State. It's implemented as an extension method and not a true debug visualizer, but useful just the same.
Check out Ruurd Boeke's excellent post on Disconnected N-Tier objects using the Entity Framework. His sample solution is checked in to the EFContrib Project and he demonstrates using POCO classes, in his words "as persistence ignorant as I can get", serializing entities with no EF references on the clients, yet not losing full change tracking on the client - and using the same domain classes on the client and the server (one could argue this last point as being not being a desirable goal - but it does have it's place).

One Year Anniversary

max — Thu, 01 May 2008 18:13:00 GMT

Today was my one year anniversary in my new job as an IT Manager at ObjectSharp. It's been an interesting year; a year full of new experiences and surprises, mostly good ones.

One year ago I have traded a job at a public sector for one in a private sector. I have traded a stable and secure environment with clear guidelines, managerial or supervision rules and specific working times for a more open managerial style, openness to new ideas and more flexibility in rules with fewer guidelines. And, honestly, I have no regrets so far. After a year, I'm having more fun than ever. It's fascinating working with a team of exceptional people from a range of disciplines whose commitment to excellence is steadfast. At ObjectSharp, I got a chance to work with the newest and coolest technologies from Microsoft (often even before those technologies were available to the general public.) Here, I am working with a talented team of professionals who not only know how to get those technologies to work, but they know how to make them work better and more efficient. I love it!

On sad note, today I was expecting balloons, hugs, high fives, or even just a disgusted "I can't believe your still here". Nothing. Well, there is still time... ;)

Note: Later that day I did get a few high fives, no balloons though J

Some References on Extensible XML Schema Design

pmadziak — Tue, 29 Apr 2008 18:58:00 GMT

This topic came up in a WCF training session (i.e. ObjectSharp's WCF course) I am currently delivering so I wanted to post a few references on the subject here:

The first is "Designing Extensible, Versionable XML Formats" by Dare Obasanjo
http://www.xml.com/pub/a/2004/07/21/design.html?page=1

The second, slightly more academic treatment is "Extensibility, XML Vocabularies, and XML Schema", by David Orchard, an extensibility guru who has much to say on the topic:
http://www.xml.com/pub/a/2004/10/27/extend.html

And then finally a series of writings by Tim Ewald which are chock full of extremely powerful Versioning advice:

First his "Services and the Versioning Problem" article in the MS Architecture Journal:
http://msdn2.microsoft.com/en-us/arcjournal/bb245674.aspx

And then a series of blog entries:
http://pluralsight.com/blogs/tewald/archive/2006/04/14/21733.aspx
http://pluralsight.com/blogs/tewald/archive/2006/04/25/22704.aspx
http://pluralsight.com/blogs/tewald/archive/2006/05/05/22961.aspx
http://pluralsight.com/blogs/tewald/archive/2006/05/05/22963.aspx

Getting around the limitation on a number of controls on SharePoint page

max — Tue, 29 Apr 2008 14:53:42 GMT

Apparently there is a limit on a number of controls you can have on a SharePoint page. By default, no more than 200 controls are allowed on the SharePoint page. Not sure, why this limitation is there, but it is. To get around this problem, you need to increase the MaxControls setting in your web.config file:

</PageParserPaths>

</SafeMode>

In my case, I have MaxControls parameter set to 300.

Developing Streaming Pipeline Components - Part 5

nbarden — Mon, 28 Apr 2008 20:12:33 GMT

In the Part 4 I covered the XmlTranslatorStream in detail, explaining how it provides powerful coupling of an XmlReader and XmlWriter as it is read. This revealed how BizTalk manages to execute a pipeline of components in an efficient way - because it is actually reading a series of wrapped streams added by each component in the pipeline. However, what the last post did not disclose was how we actually perform any useful processing now that we know how to process data in a streaming mode.

This post I will cover how we can achieve this streaming based processing of data - and give some examples of how and where BizTalk manages to do this with its out of the box components.

...

As I eluded to towards the end of Part 4, processing data through the XmlTranslatorStream is pointless if you are simply providing an XML inbound stream, and reading from it an exact XML outbound stream, without performing some other processing function. All this is achieving is slowing down the movement of data as it moves through the BizTalk streams by adding an extra layer of overhead. However, because both the XmlReader and XmlTranslatorStream classes are very extensible, we can provide our own custom implementations of these to perform our processing.

In general, you have two main extensibility options:

Subclass XmlReader and perform your custom processing in there
- Then pass your custom XmlReader to the XmlTranslatorStream as its source of data
- Then set the XmlTranslatorStream as the message part's data
Subclass XmlTranslatorStream and perform your custom processing in there
- Then set your subclass of XmlTranslatorStream as the message part's data

Note that you can also perform a combination of both, but in most cases I think you would only really need to do one.

BizTalk ships with out of the box components that perform functionality such as:

XML Data validation
Data conversion (XML/Flat File)
Property promotion/demotion
Data manipulation (e.g. property demotion modifies the outbound data)

All of these pieces of functionality are performed efficiently using a streaming approach - using a mixture of the extensibility options above to perform this processing logic. Let's look at implementing some custom functionality, and placing the same logic in both a custom XmlReader and custom XmlTranslatorStream, to compare how each implementation would work.

Before we look at the code, I will go over some more details of how the XmlTranslatorStream works. When XmlTranslatorStream's ProcessXmlNodes method is called (see Part 4 for a detailed look at when this is called), it then advances its internal XmlReader and calls the TranslateXmlNode method. This method essentially performs a check on the current NodeType under the reader, and delegates to the appropriate TranslateXxx virtual method accordingly. It provides a TranslateXxx method for every type of XML node, which is called when the corresponding node type is being copied from the XmlReader to the XmlWriter (see Part 3 for sample code on using the XmlReader.NodeType property to allow XmlReader to XmlWriter data duplication).

By providing all of the TranslateXxx methods as virtual, the XmlTranslatorStream allows us to easily intercept the nodes & values that are written from the XmlReader to the XmlWriter. By allowing this, we can easily perform modification of the data as it is being written. We can easily choose to write our own values to the XmlWriter instead of the ones that were provided by the XmlReader, or we can choose not to write any data at all - essentially removing the data from the stream. This extensible design of the XmlTranslatorStream gives us very granular control over the output data.

Now, back to an example! BizTalk ships with the NamespaceTranslatorStream class, which performs a streaming replacement of a specified XML namespace, given the old namespace to find and the new namespace to replace it with. However, this stream is limited to only updating a single namespace per instance of the stream. If you need to replace more than one namespace, you need to wrap each NamespaceTranslatorStream with another NamespaceTranslatorStream and give each one separate namespaces to replace. This causes more overhead each layer of stream you add (note that NamespaceTranslatorStream derives from XmlTranslatorStream), and also makes the code that sets it up more verbose. What would be nicer would be to have a single class that performs namespace replacements for any number of namespaces. This can be easily achieved by creating our own subclass of XmlTranslatorStream.

Take a look at the following custom implementation of XmlTranslatorStream:

/// <summary>
/// Allows the streaming replacement of multiple namespaces.
/// Note: This class could be easily enhanced to also update the prefix being written (if the ns0 prefixes that BizTalk generates are not wanted!).
/// </summary>
public class MultiNamespaceTranslatorStream : XmlTranslatorStream
{
    private Dictionary<string, string> _namespaceReplacements;

    public MultiNamespaceTranslatorStream(XmlReader reader) : base(reader) 
    {
        Initialise();
    }

    public MultiNamespaceTranslatorStream(XmlReader reader, Encoding encoding) : base(reader, encoding) 
    {
        Initialise();
    }

    public MultiNamespaceTranslatorStream(XmlReader reader, Encoding encoding, MemoryStream outputStream) : base(reader, encoding, outputStream) 
    {
        Initialise();
    }

    public Dictionary<string, string> NamespaceReplacements
    {
        get { return _namespaceReplacements; }
        set { _namespaceReplacements = value; }
    }

    private void Initialise()
    {
        _namespaceReplacements = new Dictionary<string, string>();
    }

    protected override int ProcessXmlNodes(int count)
    {
        // This method is called when the buffer in the base stream cannot fulfill the amount of data being requested
        // This in turn causes the reader to advance, and the data from the reader to be written to the writer & underlying buffer
        return base.ProcessXmlNodes(count);
    }

    /// <summary>
    /// Override to replace the namespaces of attributes being translated.
    /// </summary>
    protected override void TranslateAttributeValue(string prefix, string localName, string nsURI, string val)
    {
        // Handle the default namespace declaration (has no prefix). Eg. xmlns="http://mynamespace"
        if (((localName == "xmlns") && (nsURI == "http://www.w3.org/2000/xmlns/")) && (NamespaceReplacements.ContainsKey(val)))
        {
            // Replace the ns with the one configured
            base.TranslateAttributeValue(prefix, localName, nsURI, NamespaceReplacements[val]);
        }
        // Handle non-default namespace declarations. Eg. xmlns:ns0="http://mynamespace"
        else if (((prefix == "xmlns") && (nsURI == "http://www.w3.org/2000/xmlns/")) && (NamespaceReplacements.ContainsKey(val)))
        {
            // Replace the ns with the one configured
            base.TranslateAttributeValue(prefix, localName, nsURI, NamespaceReplacements[val]);
        }
        else
        {
            // Does not match a namespace configured to be replaced
            base.TranslateAttributeValue(prefix, localName, nsURI, val);
        }
    }

    /// <summary>
    /// Override to replace the namespaces of elements being translated.
    /// </summary>
    protected override void TranslateStartElement(string prefix, string localName, string nsURI)
    {
        if (NamespaceReplacements.ContainsKey(nsURI))
        {
            // Replace the ns with the one configured
            base.TranslateStartElement(prefix, localName, NamespaceReplacements[nsURI]);
        }
        else
        {
            // Does not match a namespace configured to be replaced
            base.TranslateStartElement(prefix, localName, nsURI);
        }
    }
}

Note that the key methods to override for performing this particular task are the TranslateAttributeValue, which is called by the XmlTranslatorStream when an Attribute is being copied from its internal XmlReader over to the XmlWriter; and TranslateStartElement - called when the start of an Element is being copied. Also note that this class could be easily enhanced to also replace the prefix being written - for example if the ns0 style prefixes that BizTalk generates are not the prefixes that are needed.

This can be easily tested with the following code. Ensure that your sample file contains multiple namespaces to replace, and they match the ones configured:

public void TranslateData_CustomTranslatorStream()
{
    using(Stream output = new FileStream("MyOutputXmlFile.xml", FileMode.Create, FileAccess.Write, FileShare.Read))
    {
        Dictionary<string, string> nsReplacements = new Dictionary<string, string>();
        // Set the replacement namespaces
        nsReplacements.Add("http://originalNS1", "http://newwackynamespace.com");
        nsReplacements.Add("http://originalNS2", "http://anotherwackynamespace.com");
        nsReplacements.Add("http://originalNS3", "http://runoutofwackynames.com");
    
        using (XmlReader reader = XmlReader.Create("MyInputXmlFile.xml"))
        {
            using (MultiNamespaceTranslatorStream stream = new MultiNamespaceTranslatorStream(reader))
            {
                stream.NamespaceReplacements = nsReplacements;
                int value;
                while ((value = stream.ReadByte()) != -1)
                    output.WriteByte((byte)value);
            }
        }
    }
}

Now compare this to what you would have had to do to use BizTalk's built in NamespaceTranslatorStream:

/// <summary>
/// Translate a document with multiple namespaces using the NamespaceTranslatorStream. Wrap a NamespaceTranslatorStream per namespace to translate.
/// </summary>
public void Translate_MultipleNamespaces()
{
    using (Stream output = new FileStream("MyOutputXmlFile.xml", FileMode.Create))
    {
        using (Stream fileStream = new FileStream("MyInputXmlFile.xml", FileMode.Open, FileAccess.Read))
        {
            // Provide the old and replacement namespace for the first namespace.
            using (Stream stream1 = new NamespaceTranslatorStream(new PipelineContext(), fileStream, "http://originalNS1", "http://newwackynamespace.com"))
            {
                // Wrap the first stream with another NamespaceTranslatorStream for the second namespace.
                using (Stream stream2 = new NamespaceTranslatorStream(new PipelineContext(), stream1, "http://originalNS2", "http://anotherwackynamespace.com"))
                {
                    using (Stream stream3 = new NamespaceTranslatorStream(new PipelineContext(), stream2, "http://originalNS3", "http://runoutofwackynames.com"))
                    {
                        int value;
                        // Make sure the outer stream is the one being read!
                        while ((value = stream3.ReadByte()) != -1)
                            output.WriteByte((byte)value);
                    }
                }
            }
        }
    }
}

So, this simple custom class that is less than 100 lines of code long should help you start to see how real world problems can be solved using a streaming approach. This small class allows you to avoid loading the entire XML message into the DOM and updating via the inefficient in memory approach.

Now let's take a look at how we could have achieved the same result by utilising a custom XmlReader. The concept is similar in that we are changing the view of data at specific points - in this example, when we see data that is using a namespace we need to change. However, because the XmlReader exposes its view of the data via properties, rather than specific virtual methods augmented by parameters, this time we want to create an XmlReader that changes the view of data exposed by its properties.

This is performed by the custom XmlReader in the following code. Note that if this were to be used within BizTalk, this XmlReader would then be passed to the constructor of the XmlTranslatorStream:

public class MultiNamespaceTranslatorReader : XmlReader
{
    private bool _inAttribute;
    private XmlReader _wrappedReader;
    private Dictionary<string, string> _namespaceReplacements;

    public MultiNamespaceTranslatorReader(XmlReader wrappedReader) : this(wrappedReader, null)
    { }

    public MultiNamespaceTranslatorReader(XmlReader wrappedReader, 
        Dictionary<string, string> namespaceReplacements)
    {
        Initialise(wrappedReader, namespaceReplacements);
    }

    public Dictionary<string, string> NamespaceReplacements
    {
        get { return _namespaceReplacements; }
        set { _namespaceReplacements = value; }
    }

    protected XmlReader WrappedReader
    {
        get { return _wrappedReader; }
        set { _wrappedReader = value; }
    }

    protected bool InAttribute
    {
        get { return _inAttribute; }
        set { _inAttribute = value; }
    }

    public override int AttributeCount
    {
        get { return WrappedReader.AttributeCount; }
    }

    public override string BaseURI
    {
        get { return WrappedReader.BaseURI; }
    }

    public override void Close()
    {
        WrappedReader.Close();
    }

    public override int Depth
    {
        get { return WrappedReader.Depth; }
    }

    public override bool EOF
    {
        get { return WrappedReader.EOF; }
    }

    public override string GetAttribute(int i)
    {
        return WrappedReader.GetAttribute(i);
    }

    public override string GetAttribute(string name, string namespaceURI)
    {
        return WrappedReader.GetAttribute(name, namespaceURI);
    }

    public override string GetAttribute(string name)
    {
        return GetAttribute(name);
    }

    public override bool HasValue
    {
        get { return WrappedReader.HasValue; }
    }

    public override bool IsEmptyElement
    {
        get { return WrappedReader.IsEmptyElement; }
    }

    public override string LocalName
    {
        get { return WrappedReader.LocalName; }
    }

    public override string LookupNamespace(string prefix)
    {
        // Check if the namespace for this prefix has been replaced
        string ns = WrappedReader.LookupNamespace(prefix);
        if (NamespaceReplacements.ContainsKey(ns))
            return NamespaceReplacements[ns];
        else
            return ns;
    }

    public override bool MoveToAttribute(string name, string ns)
    {
        InAttribute = WrappedReader.MoveToAttribute(name, ns);
        return InAttribute;
    }

    public override bool MoveToAttribute(string name)
    {
        InAttribute = WrappedReader.MoveToAttribute(name);
        return InAttribute;
    }

    public override bool MoveToElement()
    {
        InAttribute = !WrappedReader.MoveToElement();
        return !InAttribute;
    }

    public override bool MoveToFirstAttribute()
    {
        InAttribute = WrappedReader.MoveToFirstAttribute();
        return InAttribute;
    }

    public override bool MoveToNextAttribute()
    {
        InAttribute = WrappedReader.MoveToNextAttribute();
        return InAttribute;
    }

    public override XmlNameTable NameTable
    {
        get { return WrappedReader.NameTable; }
    }

    public override string NamespaceURI
    {
        get 
        { 
            if(NamespaceReplacements.ContainsKey(WrappedReader.NamespaceURI))
                return NamespaceReplacements[WrappedReader.NamespaceURI];
            else
                return WrappedReader.NamespaceURI;
        }
    }

    public override XmlNodeType NodeType
    {
        get { return WrappedReader.NodeType; }
    }

    public override string Prefix
    {
        get { return WrappedReader.Prefix; }
    }

    public override bool Read()
    {
        return WrappedReader.Read();
    }

    public override bool ReadAttributeValue()
    {
        return WrappedReader.ReadAttributeValue();
    }

    public override ReadState ReadState
    {
        get { return WrappedReader.ReadState; }
    }

    public override void ResolveEntity()
    {
        WrappedReader.ResolveEntity();
    }

    public override string Value
    {
        get
        {
            // When reading an attribute (i.e. InAttribute is true), the Value property will
            // contain the actual namespace of the attribute.
            // E.g. For attribute: xmlns:ns0="http://mynamespace", Value = "http://mynamespace"
            if (InAttribute && NamespaceReplacements.ContainsKey(WrappedReader.Value))
                return NamespaceReplacements[WrappedReader.Value];
            else
                return WrappedReader.Value;
        }
    }

    private void Initialise(XmlReader wrappedReader, Dictionary<string, string> namespaceReplacements)
    {
        if (wrappedReader == null)
            throw new ArgumentNullException("wrappedReader");

        _wrappedReader = wrappedReader;

        if (namespaceReplacements == null)
            _namespaceReplacements = new Dictionary<string, string>();
        else
            _namespaceReplacements = namespaceReplacements;
    }
}

Note that our custom XmlReader just becomes a special type of "filter" over an existing XmlReader, which we wrap. We can then delegate any calls directly down to the wrapped XmlReader if it is dealing with data that we are not concerned with. Otherwise, we can intercept the call to the wrapped implementation, and return our own data instead.

This can be tested with the following code:

public void TranslateData_CustomTranslatorReader()
{
    Dictionary<string, string> nsReplacements = new Dictionary<string, string>();

    nsReplacements.Add("http://CBR.CBRInputSchema", "http://newwackynamespace.com");
    nsReplacements.Add("http://secondnamespace", "http://anotherwackynamespace.com");
    nsReplacements.Add("http://thirdnamespace", "http://runoutofwackynames.com");

    XmlWriterSettings settings = new XmlWriterSettings();
    settings.OmitXmlDeclaration = true;
    settings.Indent = true;
    using(XmlWriter writer = XmlWriter.Create(_output, settings))
    {
        using (XmlReader reader = XmlReader.Create(Constants.SampleFilePath_MultiNamespace))
        {
            using (XmlReader nsTransReader = new MultiNamespaceTranslatorReader(reader, nsReplacements))
            {
                // WriteNode writes out EVERY node from the reader to the writer
                // However, there is no way to get it to write just a SINGLE node
                writer.WriteNode(nsTransReader, true);
            }
        }
    }
}

After seeing the two different approaches, hopefully you can now see the how each allows for the custom processing of data by BizTalk by intercepting the inbound data as it is written back out. I personally believe that subclassing the XmlReader is the better approach, because the XmlReader was designed as a "view" of XML data, and it allows clients to see your modified view in an XML sense, as it is read. By subclassing the XmlTranslatorStream, you are changing the output stream of data for the next person who reads that stream. They then have to create their own XmlReader to read over that stream anyway, which you could have done for them already just by implementing your logic as an XmlReader. The main issue I believe with the XmlReader approach is that you need to have an understanding of the operations it provides, as you may be impacting how they need to function. Take my custom XmlReader as an example - I needed to override the LookupNamespace method, otherwise it would have behaved incorrectly. However, this may not be immediately obvious.

Well, that pretty much completes this series of 5 posts detailing how to develop streaming pipeline components. I hope that from Part 1 through to this fifth one I have managed to fill in a number of gaps in the "art" of developing streaming pipeline components, and save you the many hours that I spent investigating this technique. From what I have found, there is very little documented both officially and in blogs about how to develop proper streaming components, rather than the why, and also very little detail on what already comes with BizTalk that you can easily take advantage of. There are only a handful of posts that I could find indicating that others are aware of what you get out of the box in this area, and that they understand how to use them - see here, here and here . There are many more classes that ship with BizTalk which aid in developing your streaming components, which I have not yet covered. These include the classes that implement the features I pointed out at the top such as validation and property promotion & demotion. I plan to cover many of these in detail in future posts. However, they follow the pattern which I have described in the last 5 posts (most are implemented as custom XmlReaders or custom XmlTranslatorStreams), and as a result I will not add them as posts under the banner of this series. They are more or less a follow-on to this series. Over and out!

Team System Web Access 2008 SP1 CTP and Work Item Web Access 2008 CTP are now available

max — Mon, 28 Apr 2008 14:51:29 GMT

Team System Web Access 2008 SP1 CTP (Community Technology Preview) and Work Item Web Access 2008 CTP are now available for download. Even though it's only CTP version at the moment, don't let it discourage from using it, because it seems to be working very well. By the way, you still have to have Team Explorer 2008 installed to use TWSA and/or WIWA, which is not a big deal since it's free and most of us already have it installed anyway. For those who don't have Team Explorer 2008 installed, it can be downloaded for free from http://www.microsoft.com/Downloads/details.aspx?familyid=0ED12659-3D41-4420-BBB0-A46E51BFCA86&displaylang=en.

Team System Web Access 2008 SP1 has some really great features:

Single instance with multiple languages (9 languages are available: ENU, CHS, CHT, DEU, ESN, FRA, ITA, JPN, KOR)
Support for specifying field values in the URL for creating new work items (works in both TSWA and WIWA)
Share ad-hoc work item queries
Shelveset viewer
Improved search support

Team System Work Item Web Access 2008 CTP allows users to use some work item tracking feature without having TFS client access license (CAL.) A user without TFS CAL can use WIWA to:

Create new work items
Edit the work item you have created
See the list of work items you have created

However, user cannot:

See work items created by others
List, view, edit or run work item queries
Add, edit or remove work item links (except attachments and hyperlinks)
Access documents stored on the project portal
Access project reports
Access source control
Access team build

Specs for WIWA can be found at http://msdn2.microsoft.com/en-us/teamsystem/bb936702.aspx

The Entity Framework vs. The Data Access Layer (Part 0: Introduction)

Barry Gervin — Mon, 28 Apr 2008 03:20:00 GMT

So the million dollar question is: Does the Entity Framework replace the need for a Data Access Layer? If not, what should my Data Access Layer look like if I want to take advantage of the Entity Framework? In this multi-part series, I hope to explore my thoughts on this question. I don't think there is a single correct answer. Architecture is about trade offs and the choices you make will be based on your needs and context.

In this first post, I first provide some background on the notion of a Data Access Layer as a frame of reference, and specifically, identify the key goals and objectives of a Data Access Layer.

While Martin Fowler didn't invent the pattern of layering in enterprise applications, his Patterns of Enterprise Application Architecture is a must read on the topic. Our goals for a layered design (which may often need to be traded off against each other) should include:

Changes to one part or layer of the system should have minimal impact on other layers of the system. This reduces the maintenance involved in unit testing, debugging, and fixing bugs and in general makes the architecture more flexible.
Separation of concerns between user interface, business logic, and persistence (typically in a database) also increases flexibility, maintainability and reusability.
Individual components should be cohesive and unrelated components should be loosely coupled. This should allow layers to be developed and maintained independently of each other using well-defined interfaces.

Now to be clear, I'm talking about a layer, not a tier. A tier is a node in a distributed system, of which may include one or more layers. But when I refer to a layer, I'm referring only to the logical separation of code that serves a single concern such as data access. It may or may not be deployed into a separate tier from the other layers of a system. We could then begin to fly off on tangential discussions of distributed systems and service oriented architecture, but I will do my best to keep this discussion focused on the notion of a layer. There are several layered application architectures, but almost all of them in some way include the notion of a Data Access Layer (DAL). The design of the DAL will be influenced should the application architecture include the distribution of the DAL into a separate tier.

In addition to the goals of any layer mentioned above, there are some design elements specific to a Data Access Layer common to the many layered architectures:

A DAL in our software provides simplified access to data that is stored in some persisted fashion, typically a relational database. The DAL is utilized by other components of our software so those other areas of our software do not have to be overly concerned with the complexities of that data store.
In object or component oriented systems, the DAL typically will populate objects, converting rows and their columns/fields into objects and their properties/attributes. this allows the rest of the software to work with data in an abstraction that is most suitable to it.
A common purpose of the DAL is to provide a translation between the structure or schema of the store and the desired abstraction in our software. As is often the case, the schema of a relational database is optimized for performance and data integrity (i.e. 3rd normal form) but this structure does not always lend itself well to the conceptual view of the real world or the way a developer may want to work with the data in an application. A DAL should serve as a central place for mapping between these domains such as to increase the maintainability of the software and provide an isolation between changes in the storage schema and/or the domain of the application software. This may include the marshalling or coercing of differing data types between the store and the application software.
Another frequent purpose of the DAL is to provide independence between the application logic and the storage system itself such that if required, the storage engine itself could be switched with an alternative with minimal impact to the application layer. This is a common scenario for commercial software products that must work with different vendors' database engines (i.e. MS SQL Server, IBM DB/2, Oracle, etc.). With this requirement, sometimes alternate DAL's are created for each store that can be swapped out easily. This is commonly referred to as Persistence Ignorance.

Getting a little more concrete, there are a host of other issues that also need to be considered in the implementation of a DAL:

How will database connections be handled? How will there lifetime be managed? A DAL will have to consider the security model. Will individual users connect to the database using their own credentials? This maybe fine in a client-server architecture where the number of users is small. It may even be desirable in those situations where there is business logic and security enforced in the database itself through the use of stored procedures, triggers, etc. It may however run incongruent to the scalability requirements of a public facing web application with thousands of users. In these cases, a connection pool may be the desired approach.
How will database transactions be handled? Will there be explicit database transactions managed by the data access layer or will automatic or implied transaction management systems such as COM+ Automatic Transactions, the Distributed Transaction Coordinator be used?
How will concurrent access to data be managed? Most modern application architecture's will rely on an optimistic concurrency to improve scalability. Will it be the DAL's job to manage the original state of a row in this case? Can we take advantage of SQL Server's row version timestamp column or do we need to track every single column?
Will we be using dynamic SQL or stored procedures to communicate with our database?

As you can see, there is much to consider just in generic terms, well before we start looking at specific business scenarios and the wacky database schemas that are in the wild. All of these things can and should influence the design of your data access layer and the technology you use to implement it. In terms of .NET, the Entity Framework is just one data access technology. MS has been so kind to bless us with many others such as Linq To SQL, DataReaders, DataAdapters & DataSets, and SQL XML. In addition, there are over 30 3rd party Object Relational Mapping tools available to choose from.

Ok, so if you're not familiar with the design goals of the Entity Framework (EF) you can read all about it here or watch a video interview on channel 9, with Pablo Castro, Britt Johnson, and Michael Pizzo. A year after that interview, they did a follow up interview here.

In the next post, I'll explore the idea of the Entity Framework replacing my data access layer and evaluate how this choice rates against the various objectives above. I'll then continue to explore alternative implementations for a DAL using the Entity Framework.

Countdown to DevTeach

Barry Gervin — Sat, 26 Apr 2008 01:25:00 GMT

May 12th is just around the corner and I'm getting excited about DevTeach coming to Toronto for the first time. Guy says it all. DevTeach is unusual in that while the sessions are great, hanging around with the Speakers is what it's all about. I'm really looking forward to it. It's not too late to register. The 3 days of the main conference cost only $1200. Pre & Post cons are only $375/day. Some great sessions on Agile Development, .NET 2.0 through 3.5, .NET Futures, SharePoint, SQL, Architecture, BizTalk, and RockBand!

Hope to see you there.

It’s cool to be a geek…

dave — Fri, 25 Apr 2008 17:40:00 GMT

Do you remember when it was not cool to be a geek?

Not that I ever let it bother me. I have attended every PDC since 2001. It’s tough to be the coolest guy out of 8000.

Now a days it's cool to be a geek , people are coming out of the wood work to proclaim their geekyness with pride. Below are three examples I came across in the past few weeks.

1. A developer who changed his girlfriend’s favorite game (beleweled) so when she achieved a certain score it displayed display a proposal of marriage.

2. I remember when that one kid in a class with a laptop was the geek. Who’s the outcast now?

3. Cat 5 Wedding rings, need I say more.