Total Requests count in Load Test Results is Inconsistent

Leon Langleyben

Hi,

We are running load test with single WebTest in test mix and constant load of 5 users. Web test issues 3 requests per server. After load test completion we received Total Tests = 160, Total Requests = 483. To investigate additional 3 requests we used Request Plugin and Web Test plugin. In PreWebTest event we logged 164 tests, in PostWebTest, we logged 160 tests. It can explain Total Tests number. I guess, 4 tests were prepared but were not executed because the test time run over. But the RequestPlugin surprised us. We logged 486 requests in PreRequest event and 486 responses in PostRequest event. The same number of requests (486) was logged in IIS log.

Can anybody explain this inconsistency What exactly Total Requests and Total Tests numbers in Load Test Summary refers to

Thanks in advance,

Leon

Dennis Stone - MSFT

The Totals for both Requests and Tests indicate the number that completed. As your numbers indicated only the tests that completed were counted, but there were a few more that started and didn't get a chance to finish. Some of the requests from those incomplete tests were counted in your stats however because they did complete within the time allotted for the test.

Leon Langleyben

Thanks for Reply Dennis!

In this case Total Requests number is completely useless for me. It is different from actual requests number that hit web server. It is unclear and misleading.

In our scenario we need to check application error handling under load. We wanted to compare actual work performed by application in database gainst reported succesful requests. To do it now we have to create RequestPlugin, just to count actual requests number. How can it be consider as correct behavior

Dennis Stone - MSFT

For your scenario it may be better for you to run an iteration based loadtest, instead of running for a specified duration you can run a specified number of tests, then your numbers will always be consistent. There is an explanation of setting this up here: http://blogs.msdn.com/billbar/articles/533216.aspx

Leon Langleyben

Dennis, you are right this will work. I think our solution is more simple (just tog actual request trough RequestPlugin). Still I think both solutions are a bit ugly and artificial. I would prefer to see additional "Total Sent Requests" counter in summary.

Dennis Stone - MSFT

Are you able to use the IIS/Active Server Pages/Requests Total counter on the web server to track this rather than the request plugin or going iteration based

Leon Langleyben

I guess, I can in some cases, but not always I can be sure that load test running along against IIS (no other requests issued during test duration).

Dennis Stone - MSFT

If you cannot guarantee you're the only one using the server you're testing against then I agree that relying on the IIS counters is not the best solution and it looks like you will either have to continue using the current method you have or switch to the iteration based approach. Personally I like the iteration based approach a little better because you can then rely on the Requests data in the loadtest counters rather than having seperate data to read from your request plugin but you're right in that it may be a little more difficult to implement, although the article I previously reference should take care of most of the work for you. Iteration based load tests is something that we've added directly to the product for Orcas, which you can check out in our March CTP release if you wish.