Mikael Håkansson : Testing

Ewan Fairweather is comming to BizTalk User Group Sweden

wmmihaa — Mon, 26 Apr 2010 11:42:00 GMT

Performance has been one of the more popular topics at the Swedish BizTalk user group, where we’ve had both Darren Jefford and Paolo Salvatori as previous speakers. This event is of a more practical and detailed level, and covers performance considerations from both a setup/configuration perspective as well as from a developer perspective. Ewan and I have been planning this event since mid January, but for various reasons we haven’t been able to do it until now.

BizTalk Server Performance: Best practice

Session #1 – Instrument your BizTalk Server

Optimizing and verifying your BizTalk Server installation is not an easy thing to do. The documentation is good but very extensive. This presentation aims to guide you through the most important operations you need to do in order to boost the performance of BizTalk.

Session #2 - Performance Optimization Patterns

This session will present architectural, design, and development patterns to improve BizTalk processing performance. "Performance" can be expressed by latency and/or throughput, and this session will cover aspects of both. This will include pipeline and orchestration patterns to increase throughput, reduce latency, and reduce memory usage during BizTalk processing. We will also cover the results from BizTalk CAT recent Perf engagements.

Speaker:

Ewan Fairweather

Ewan Fairweather has worked for Microsoft for five years. He currently works as a program manager in the Business Platform Division on the Customer Advisory Team (CAT) working on large scale Integration and OLTP SQL applications. Prior to this, Ewan spent three years working for Microsoft U.K. in the Premier Field Engineering team where he worked with enterprise customers, helping them to maintain and optimize their BizTalk applications. This included working in a dedicated capacity on some of the world's largest BizTalk deployments, predominantly within financial services.

Ewan coauthored the successful Professional BizTalk Server 2006 (Wrox, 2007) and has written many white papers for Microsoft including the "Microsoft BizTalk Server Performance Optimization Guide," which is available on the Microsoft Developers Network (MSDN) Web site. Prior to joining Microsoft, Ewan worked as a Cisco Certified Academy Instructor (CCAI) for a regional training organization, delivering advanced routing and networking courses. Ewan holds a bachelor's degree in computing with management from the University of Leeds. Outside of work, Ewan's hobbies include reading, taking part in as many sports as possible, and regularly going to the gym.

Ewan maintains his blog at http://blogs.msdn.com/ewanf

BizTalk Benchmark Wizard – New release

wmmihaa — Fri, 23 Apr 2010 12:25:00 GMT

The Benchmark Wizard was released earlier this year, and even though we got lots of good feedback we’ve also got requests for some changes.

If you haven’t yet heard about this tool, the purpose of it is to verify your BizTalk environment performs as expected. There are two different scenarios you can run, a Messaging and an Orchestration scenario. Each of the scenarios has been tested on various environments and configuration. The result of these tests has provided the tool with a set of KPI’s, which your test result will be benchmarked against. For more information about the Benchmark Wizard:

Benchmark your BizTalk Server (Part 1) - Overview

Benchmark your BizTalk Server (Part 2) – How to install

Benchmark your BizTalk Server (Part 3) – Drill down and analyse

The most common feedback is related to the fact that it was very difficult to meet the KPI’s. A reason for this is that the original tests where executed while global tracking was disabled. This was unfortunate as the DTA tracking has ~30% overhead. We have done a re-run of all the tests, and updated the KPI’s for the new version (shown at the bottom of this page).

Further more, there seams to be a question of how to interpret the result. What does “Succeeded” mean? Does it reflect the best possible result or good enough? To make this more clear, we’ve implemented the “stop light approach”, where if you’ve Succeeded you should be proud of yourself and make a blog post, while an Acceptable result is nothing to be ashamed of.

There has also been some other fixes such as the resetting all the counters when you re-run the test, and fixing the CPU counters to show correct values.

Special thanks to Microsoft and Ewan Fairweather for letting me use their test lab!

Download BizTalk Benchmark Wizard from CodePlex

Scenario KPI’s: Messaging Single and Multi Message Box

Scenario KPI’s: Orchestration Single Message Box

Test environment:

* Storage: EMC Clarion CX-240 ( 5 solid state drives )

Configuration:

Global tracking enabled
Partitioned the tempDb, BizTalkDTADb and BizTalkMsgBoxDb to as many files as CPU’s
Separated the BizTalk MessageBox database into multiple filegroups / files
Enabled the -T1118 flag on the SQL Service
Disabled Throttling on send and processing hosts
Updated the thread settings (CLR Hosting) on all hosts

WebCast - BizTalk Benchmark Wizard

wmmihaa — Mon, 01 Feb 2010 20:00:00 GMT

The webcast is moved to Cloudcast.net.

Overview:
MSDN Devcenter
Download:
Download from CodePlex.com
For more information:
Benchmark your BizTalk Server (Part 1)
How to install:
Benchmark your BizTalk Server (Part 2)
BBW Drill Down:
Benchmark your BizTalk Server (Part 3)

Benchmark your BizTalk Server (Part 2)

wmmihaa — Sun, 10 Jan 2010 10:00:00 GMT

This post describes the installation process for the BizTalk Benchmark Wizard application. For more information read part 1.

The complete installation will include the following components:

BizTalk Benchmark wizard – The client tool from which you will run the tests
Three Hosts – BBW_RxHost, BBW_PxHost, BBW_TxHost
Three Host Instances
Two Adapter handlers for NetTcp
One BizTalk Application
Two Receive Ports
Two Send Hosts
One Orchestration

The installation process

Download the bits from Codeplex, and proceed with the normal installation. As the setup completes, your first step would be to install the BizTalk artifacts. These artifacts are the BizTalk components and bindings which makes up the testing scenarios. This is a two step process as you need to set up the hosts before you import the bindings.

BizTalk environment

You can run the scenarios on either a single-server installation or a two server installation. Regardless of the number of BizTalk servers in you group, you should not run it with more than two “active” servers, as it will otherwise not be covered by the benchmark values.

1. Setup Hosts, Instances and Adapter handlers

The hosts, instances and handlers are created through running a script. You need to run the script file using “cscript” as shown below, and I apologize in advance to the poor error messages you might run into.

1. Open an command prompt window and navigate to [Installation folder]\Artefacts\BizTalk. By default, the installation folder is “C:\Program Files\Blogical\BizTalk Benchmark Wizard”.

2. In the BizTalk folder you’ll find a InstallHosts.vbs file. Execute it using the following parameters:

NTGroupName - The name of the Windows NT group.
UserName – The name of the user account running the service instances.
Password – The password of the user account running the service instances.
Receive Host – The name of the server where you want to run the receive host instance.
Send Host - The name of the server where you want to run the sen host instance.
Processing Host - The name of the server where you want to run the process host instance.

If you have a single box installation, your script command might look like this:

cscript InstallHosts.vbs "BizTalk Application Users" “\MyUser” “MyPassword” “BtsServer1” “BtsServer1” “BtsServer1”

If you have a multi server installation, your script command might look like this:

cscript InstallHosts.vbs "MyDomain\BizTalk Application Users" “MyDomain\MyUser” “MyPassword” “BtsServer1” “BtsServer2” “BtsServer2”

Running this script will create:

Three hosts (BBW_RxHost, BBW_TxHost and BBW_PxHost)
Three host instances
One send and one receive adapter handler for the WCF NetTcp adapter.

2. Import BizTalk MSI

1. Open the BizTalk Administration Console, point to the “Applications” node and import the BizTalk Benchmark Wizard.msi found in the same folder as the scripts above.

This will create a “BizTalk Benchmark Wizard” application along with all ports and orchestrations.

2. Finally, run the BizTalk Benchmark Wizard.msi on all BizTalk servers to add the assemblies to the Global Assembly Cache (GAC).

Done, click [Start]-> [All Programs]->[BizTalk Application Wizard] - [BizTalk Application Wizard] to start the application.

Related articles:

Benchmark your BizTalk Server (Part 3) by Ewan Fairweather

Good luck!

Benchmark your BizTalk Server (Part 1)

wmmihaa — Sat, 09 Jan 2010 23:00:00 GMT

Ever wondered if your BizTalk is all it can be?

Verifying your BizTalk Server installation is not an easy thing to do. So far the BizTalk Server 2009 Performance Optimization Guide is probably your safest bet. The Optimization Guide provides in-depth information for optimizing the performance of a BizTalk Server solution. However, it won’t help you evaluate you BizTalk installation at runtime. To do this, you’ll have to continue analyzing it using Performance Analysis of Logs (PAL).

I’m not saying these aren’t good tools. In fact they are great. They are however quite extensive, and will ultimately not answer the question: “Do I get the expected workload through BizTalk?”.

Ewan Fairweather, together with some other smart people at Microsoft, have put together a comprehensive study about scaling out BizTalk. The principal is simple, test the same scenarios with different environments and quantify the scale out capabilities of one to four BizTalk servers and one to three message boxes.

The BizTalk Server 2009 Scale Out Testing Study provides sizing and scaling guidance for BizTalk Server. However, you’d find it challenging to compare your environment to these numbers as you haven’t got access to the same testing scenarios. And even if you did, you still couldn’t be sure you’ve configured it the same, and that you have been running the equivalent LoadGen scripts.

Four months ago, I contacted Ewan to ask him if he had some testing scenario I could run to evaluate the environment I was currently working on. He didn’t, but seemed very aware of the lack of such a “tool”. One thing led to another and we came to the conclusion we should make it ourselves.

- Today, four months later, we are happy to announce that BizTalk Benchmark Wizard is publicly available on Codeplex.

The goal has been to make an easy to install and simple to use, wizard-like application with which one could test a BizTalk environment– and compare the result to the study. One of the challenges where to scope the project, and prevent ourselves from solving problems already addressed in tools such as LoadGen and PAL. For instance, BizTalk Benchmark Wizard is NOT…

…a load tool

Although it does create load, it only does so against ONE receive host. The application could work against multiple receive hosts, in fact the earlier versions did, but it required a much more complex setup process from the user. We came to the conclusion that if your environment measures up using only one receive host, it most likely would do so using multiple hosts.

By setting these limitations, it also simplifies the comparison of environments and benchmarking them against the result from the Microsoft Study.

…an analyzing tool

The tool does not analyze any eventual problems or bottlenecks. Neither does it give any hints or advice of how to solve them. It does however collect Perfmon counter data from each of the servers, both BizTalk and SQL. If your environment fails the test, you can analyze the data using the PAL tool.

How it works:

After the user has started the application and specified the BizTalk Group, the tool analyzes its configuration, finding all the BizTalk servers, Messageboxes etc.
Secondly, the user gets to select one of two scenarios: Messaging or Orchestration. Each scenario has a set of tested environments such as
- “Single server (2*Quad CPU, 4GB RAM)”
- “1*BTS (1*Quad CPU. 4GB RAM) + 1*SQL(1*Quad CPU, 8GB RAM)”.
- “2*BTS (2*Quad CPU. 8GB RAM) + 2*SQL(2*Quad CPU, 16GB RAM)”.
The user selects the environment which most resembles his/her own.
The user then starts the Indigo Service, a console application hosting a service which will be called from the BizTalk Send port.
As the user clicks “Run test”, the tool continues to start ports and orchestrations. It will also start the Perfmon collector sets if the user has chosen to create those.
As the test proceeds the user can monitor the counter values through the gauges (CPU utilization, Received msgs/sec and Processed msgs/sec). The default test duration is 30 minutes, with a warm-up of 2 minutes.
Finally, the user is presented a result, which is either Succeeded or Failed.

Related articles:

Benchmark your BizTalk Server (Part 2)

Benchmark your BizTalk Server (Part 3) by Ewan Fairweather

If you pass the test, you can proudly submit your result to the the High Score list. “E.W.N” seams to be the one to beat…

BizTalk migration – Testing

wmmihaa — Tue, 18 Mar 2008 18:04:00 GMT

When migrating from BizTalk 2004 (or possibly 2002) to 2006 one might feel tempted to do some refactoring at the same time, fixing things you (or somebody else) could have done better the first time. If the migration is done from a large BizTalk solution, you probably want to make sure everything works as it did before, without spending to much time testing. After all, it’s "just" a migration project.

So if what you are looking for, is some kind of regression testing, leaving your developers to focus on the migration task, you might find this post useful.

The basic principle here is letting messages, passing through the old environment, also passing through the new 2006 solution, and ending up being validated to match each other. It is important to understand that this testing process will only test the content of the message and not any possible connectivity problems. It will require you to deploy your 2006 ports using the File adapter on both receive and send port side. Upon deploying your solution to the production environment these ports would need to be changed to whatever binding you used before. An other important thing to understand is that this process only works with one-way ports.

To make this happen we need to add a pipeline component at the beginning of every pipeline in the old environment. This component will archive the incoming message to the location where the Receive Location in the new environment will pick it up. Hopefully you can achieve this by copying every pipeline to one new Visual Studio 2003 project, and add the pipeline component to the new pipelines.

The archive component should be designed to require as little system resources as possible, why it uses a Forward only event streaming manner, which basically prevents the stream from being read more than once. To learn more about ForwardOnly streaming, have a look at Johan's blog post.

Having the updated pipeline assembly, all we need to do is to (1) add an additional Send Port with identical filter settings, and (2) change the pipeline to the updated one, and (3) set the archive path to where ever you set the receive location to in the 2006 environment.

When you’ve done this, the orchestration will pick up the files from the Send Port locations and check if they are identical. The result will be sent to the database, along with the actual messages, for later use.

The orchestration is generic and can be used for every scenario. It has an parallel shape which will prevent it from continue until it receives two files with identical names. To ensure the names comes out with the same name, the pipeline component mentioned above, will promote the ReceivedFileName property (file-property namespace) if it is not set. This will be the case when the original message, for instance, is received from an MQ adapter.

It will then proceed and send the incoming messages to an archive, where you can later pick them up if you need to analyze its content. Next is a decision shape where the actual matching is done. The result will be inserted into a database table, which you may later query to get all failed messages (if any).

If you find this post of interest, let me know, and I'll post the source code.