Visualizing the benefit of forwardonly streaming in custom pipeline components

When doing custom pipeline component development you need to be aware of the forward-only streaming best practice. In short this means developing your pipeline components so that they do their logic either as a custom stream implementation or by reacting to the events availble to you through the Microsoft.BizTalk.Streaming.dll stream classes. Without ever keeping anything except the small stream buffer in Memory and without ever seeking the original stream. This is best practice from the perspective of resource utilization, both memory and processor cycles.

Microsoft.BizTalk.Streaming isn't available to reference out of the box in BizTalk Server 2006, you have to get it out of the GAC to be able to use it.

Now there are a couple of good writings about streaming pipeline components and some of the peculiarities you have to think about when developing them. I won't repeat more of it here. Instead the focus of this post is to make you aware of the difference between acting upon the stream as it is being read as opposed to reading through the stream in the execute method of the pipeline component (which is generally a bad idea, but necessary in some cases) by presenting a visual image of it for you.

 The scenario is the following: We have three custom pipeline components in a pipeline. The images below will show the difference between reading the stream the best practice way of reacting to what you need as the stream is being read once as opposed to reading the entire stream through in each pipeline component. I'm sure everyone is familiar with DebugView, and I will use it to display the trace statements outputed by the pipeline component.

Now basically what is being done here is that in the first pipeline component a stream wrapper is created, just so that we will get events from when the original stream is being read, in all three components we then use the events of the forward only eventing stream to react to events as the stream is being read. In the image above we can see that the original stream read event is called 3 times.

 This instead is what would happen if we read through the stream in every pipeline component, instead of once. As we can see the stream is read 10 times. Now 3 times or 10 might not sound that much - the stream is however intentionally small. The point is that it is read 3 times, instead of the one. You should also keep in mind that many streams are not naturally seekable, requiring temporary storage or the likes to make them so which only adds to the resource waste.