Alan N

Hello,

Dennis Stone has posted what I think is a very useful LoadTestPlugin in this thread:

http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=1311729&SiteID=1

This plugin works great for warmup with a constant load test, but what if I needed to do the same with a goal-based test

I am using the goal-based functionality to make the test rate-driven. User load is being adjusted based on the Tests/Sec value so that the rate at which the requests are sent is independent of application response time. I could get this same result if ThinkTime would take milliseconds, but it only takes seconds as an integer.

Please let me know if the above plug-in could be re-worked for a goal-based test.

Thanks,

~Alan



Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Bill Barnett - MSFT

Mike Taute has a plug-in example that implements a goal-based load pattern for more than one perf counter. This probably does more than you need, but you may be able to trim it back to do what you need: http://blogs.msdn.com/mtaute/archive/2006/12/15/managing-goal-based-load-using-multiple-perfcounter-thresholds.aspx

Bill





Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Alan N

Bill,

I've tried implementing Mike's plug-in and found that it does not work for warm-up.

I belive the reason for this is that the ThresholdExceeded event is never triggered during the warm-up period.

What ends up happening is load just continues increasing until it reaches the upper bound (specified via context parameter) or the warmup period ends, and the load then starts adjusting to meet the threshold criteria.

Is there any way to enable ThresholdExceeded notification during the warm up period

Thanks,

~Alan





Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Bill Barnett - MSFT

Unfortunately no, and that did not occur to me when I pointed you towards Mike's plug-in.

It's certainly possible for your plug-in code to make the calls to read the performance counter values directly using the PerformanceCounter and related classes in the .NET System.Diagnostics namespace. Coding this may be more trouble than you want to go to because it's a bit cumbersome, though there is a pretty good example in the online docs on this page: http://msdn2.microsoft.com/en-us/library/system.diagnostics.instancedata.aspx.





Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Alan N

Bill,

How does Orcas handle the warm-up period with the new User Pacing Load model

Do all users still get launched simultaneously

What if I were to use an initialize test that sleeps for a random amount of time (shorter than warm-up) Would that prevent a large spike in generated requests

I have not yet installed Orcas, so I¡¯m just trying to get an understanding for what else has changed.

Thanks,
~Alan





Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Bill Barnett - MSFT

Alan, below is a description on the details of the User Pacing Load model taken almost verbatim from the original design spec. It works the same way during warmup, except that the number of virtual users is less.

"In a Pacing Test Mix, the total number of tests per hour determines the ¡°Test Start Interval¡±.

For example, consider the following Pacing Test Mix:

Test

Tests per user per hour

Test1

3

Test2

2

Test3

0.5

Test4

0.5

Each user should run a total of 6 tests / hour, so the ¡°Test Start Interval¡± is 10 minutes (or 3600 / 6 = 600 seconds).

The start time for the first test for each user is determined by choosing a random number between 0 and the ¡°Test Start Interval¡±. This staggers the start times of the first test for each user. The start time for subsequent tests is determined by performing the normal distribution function on the number of seconds for the ¡°Test Start Interval¡±, and adding that to the previous test start time. If the previous test does not complete before the scheduled start time, the test starts as soon as the previous test does complete.

The choice of the next test to start is determined using a weighted random selection based on the rates per hour, so that each virtual user does not run the test in the same sequence."





Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Alan N

Bill,
I was going to start a new thread, but figured it'd be easier to dig this up for reference.
Per your suggestion, we are now using Orcas for our testing.
We still have a problem with user ramp-up during warm-up period.

I have been using the LoadTestPlugin referenced in the first post, and it has eliminated my problem for constant-load tests.
There still doesn't seem to be a solution for goal-based tests. Today I've found that even the Plugin is flawed when a loadtest has more than one concurrently running scenario.

With that said, I have 2 questions:
1) Is it possible to modify the LoadTestPlugin to release control of the LoadTest.Scenarios[x].CurrentLoad at the end of the WarmUp period (this would allow me to use it with a gola-based test)
2) Is it possible to modify the LoadTestPlugin to work with a loadtest containing multiple scenarios

Thanks,
~Alan






Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Alan N

Scratch question #2
I wrote a switch statement that checks the number of scenarios in the LoadTest and adjusts the load for all of them.
Now, I can either keep the target number of users for every scenario equal or add more context parameters to specify the number of maximum users and increment number for each scenario.

Question #1 still stands. If it can be done, it would solve our biggest issue with VSTS.

Thanks,
~Alan




Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

slumley MSFT

As for your first question, the answer is no. Once you take control of the load for a web test, you are in control for remainder of the test.




Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Alan N

That's too bad.
In that case do you guys plan to implement an option to allow user ramp-up during warm-up period
We are running benchmarks, and we do not want user ramp-up included in the results. If we launch all users at the same time, we briefly overload the webserver and DB, and get a few unwanted errors. We've also managed to crash the web server on more than one occasion with this approach.

Thanks,
~Alan






Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Mandeep123

Hi Sir

Actually I came through your post but my query is different. I just want to know that what is warm up time period.

Please tell me in detail so that I can go for the further testing in the VSS testing Tool.

Hope I will get the answer soon.

Best Regards

Mandeep Singh






Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Chuck vdL

Warm up time is just that, an allowance for the servers in the 'system under test' (SUT) to get 'warmed up'. Now yes computers don't need to warm up temp wise like cars on a cold day, but they do need to load instances of runtime environments for scripts, create caches, establish connections to databases, etc. There are a lot of things that happen under the covers the first time a given page is hit, or transaction performed. Think of the warm-up period a bit like 'priming the pump' so to speak.

Web servers, and multi-tier systems can be notoriously slow to respond immediately after a restart. For a variety of reasons (like DB restores to the same initial conditions) it is not uncommon for all the boxen in a SUT to be restarted, or the services restarted, immediately before the start of the load test. Because you are controlling access to your SUT in order to be sure that the only load it is experiencing is that created by your test, the first requests they see will typicall come from your test scripts. The times for those initial transactions will often be WAY out of scale with a typical transaction later in your test, even with the same number of users. If your test doesn't push things to the point of failure, those first few transaction times can even exceed the transaction times for all of the rest of the test, which if they were included would really throw off any statistics reported for the test. The warm up time allows a way to get things rolling, but not include those initial transactions in your testing stats..

Another potential analogy would be in racing the difference between a 'standing start' and a 'rolling start' (aka 'flying start'). think of the warm up time as the time to get all the cars on the track, tires warmed up, and up to a reasonable speed before you really start racing and seeing how fast everone can go






Re: Visual Studio Team System - Web and Load Testing User load ramp-up during warmup period

Alan N

I like your analogies, Chuck.

There is another small issue I've found with the warmup period. If you set your test to run based on a number of iterations instead of a set time, some of the iterations are used up during the warmup. What I wanted to do was capture the test start time, and end time and use that to fetch the number of DB rows created between those times. With the number of iterations being known, I could just compare those two numbers to verify that no data is lost.
I would include the warmup period time in the start time, but we expect a few failures during the warmup period and they are not captured in the final results of a loadtest, so that doesn't work either.

I believe this should be a feature request.

Cheers,
~Alan