crichmon

There are two test environments that I'm working with: one is Windows Server 2003, the other is Windows Server 2003 x64. The tests on win2k3 x86 work as expected: testing on a test rig with test agents allows a larger request rate than just one local machine. However, the opposite appears to be true for the win2k3 x64 environment: test agents are maxing out cpu time and the net request rate is lower than if the tests were run on one local machine. What would cause this behavior Incorrect configuration setting Win2k3 x64 Something else

thanks,

Chris



Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

slumley MSFT

Do you have your agents configured to use server gc Check out this post for how to configure htis. Let me know if this makes a difference.

http://blogs.msdn.com/slumley/pages/improve-load-test-performance-on-multi-processor-machines.aspx






Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

crichmon

I modified the load agents per the listed instructions, however there was no change. When a load test is run locally in VSTS, the request rate is around 20. For the same user load, when the same load test is run with the rig setup (about 7 load agents), the request rate is around 4. Are there compatibility issues with load agents and Windows Server 2003 x64 The setup is almost identical to another setup on a set of Windows Server 2003 x86 machines; the 64-bit is the only difference that is obvious.





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

rakesh_brings_smile

I am seeing the same issue.Below is my scenario :

1. Run the load test with 20 webtests using a rig with 5 agent machines .Agent machines are low end machines( 512 MB RAM , 1 GB).

2. One of the agent machine goes down with System.OutofMemory error and then the userload reduces to 0.

3. Basically, the load test run aborts with the failure.

Should not the test continue ( request being sent from the other active agents to Server)

Any help will be greatly appreciated.





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

crichmon

Perhaps if the load test is attempting to generate a constant user load and when one or more of the agent machines is no longer able to run tests, then the entire test aborts because the user load changed and the constant user load requirement of the test was negated.

However, that is different from the issue that I'm discussing.

In one scenario, the test rig consists of Windows Server 2003 x86 installations (3.2ghz 4gb) with multiple agent machines, specifically 6 agent machines. There are no issues what-so-ever running load tests of any kind, and the test rig with multiple agent machines is able to generate a request rate greater than running the load test locally on one machine (expected).

In a different scenario, the envrionment consists of Windows Server 2003 x64 installations (3.2ghz 16gb) with multiple agent machines, specifically 7 agent machines (not that the specific number of machines matters). The issues are that all of the agent machines have 100% CPU usage while running tests, even with a low constant user load, the load test is not aborted, and the test rig with multiple agent machines is not able to generate a request rate greater than running the load test locally on one machine.

My current guess is that IIS might be required on the agent controller machine.





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

Bill Barnett - MSFT

crichmon: First of all, I can't think of any reason why IIS should need to be installed on the agent or controller machines, so I doubt that has anything to do with the problem.

The CPU usage on the 64-bit machines definitely warrants more investigation. It's possible that you have found a product bug that is unique to 64-bit machines, though that is not clear, and it would be surprising to me if that is the case. I suggest the following steps to narrow down the problem to a simpler form:

1. Run the load test on just one of the 64-bit agents (I assume the problem will still repro, but it make the environment simpler).

2. Use a step load pattern in your load test and start out with just one user. How many users do you need to run on one agent to get the CPU usage up to 100% Does the CPU usage go up gradually, or shoot up to 100% all of a sudden as the load test runs Is all of the CPU time on the agent being used by the QTAgent process, or is there some other process consuming CPU (You can add the Process / % Processor Time performance counter (using the QTAgent instance and other instances) to the load test graph.

3. What type of tests does your load test contain Unit tests or Web tests If they are Web tests, are they Coded web tests or not If you are using Coded web tests, try using a simpler declarative (non-coded) Web test to see if the problem still occurs.

rakesh: I agree with crichmon that your problem sounds different. Can you please start a new thread for your issue Thanks.





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

crichmon

The cpu usage appears to be 100% of one processor even for a user count of 1. The load test mix is a collection of generic tests running a C# application. The build used is the debug build. The exe application only uses 1 or 2% processor time on machines that have a visual studio installed, however uses 100% processor time for machines that don't have visual studio installed. (Also, a test of a load test of web test had the same request rate locally and on one agent machine.)





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

Bill Barnett - MSFT

So it sounds like you've determined that the problem only occurs when running your generic tests in the load test. To narrow this down further, what happens if you run each of the generic tests by itself (outside of the load test) What happens if you run your test executable on this machine without invoking it from VSTS at all





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

crichmon

The results vary on the applications installed on the machine. For the machines with the visual studio load agent installed, the application run by itself uses 100% cpu time. For the machines that have visual studio team system installed, the application run by itself uses only 1 or 2% cpu time. Perhaps there is something with the visual studio development install that affects the application during execution.





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

Bill Barnett - MSFT

I suppose it's theoretically possible that the load test agent is having some affect, but honestly I can't see how. It seems more likely that there is something else different about that machine (hardware, especially number of processors, OS, environment, virus scanners, etc.) that is causing the different behavior with the CPU usage. What if you stop or disable the load test agent temporarily on that machine (or even uninstall it - the reinstall doesn't take that long) Does the problem still occur





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

crichmon

There's a load test running that is prevent some experimentation for now. Another issue that is occuring from the load test is, over time, the tests have a longer duration, and therefore less tests (and less test rate) is actually tested. Stopping the load test and starting again starts the load test at the last request rate that it was on. Rebooting the machine returns the test rate to normal; perhaps there is a resource issue with the application.

The hardware configuration for the machines in the environment is the same. One difference that I noticed is that the machine that seems to run the test exe at the 'normal' rate (most of the time) is also running the "loadtest.vshost.exe" process. Currently my estimation is that the 32-bit characteristic of the compiled test application is part of the cause of the issues seen on the 64-bit machines.





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

crichmon

The problem still occurs with the uninstall of the load agent. The machines in the set have the same # of processors, environment, etc. as far as a I know, there is one machine that seems to run the C# load test app at about 2% cpu usage instead of 25%.





Re: Visual Studio Team System - Web and Load Testing tests via test rig test agents have lower request rate than testing locally?

Bill Barnett - MSFT

Sorry for not responding to this for a while. It certainly seems that your problem has something to do with your test program's performance on that machine and nothing to do with the VSTS testing software. I'm not sure how to help you - if this is still a problem perhaps you could post a new question to a different forum - I'm not sure which forum would be best, but it is problems one of the forums listed under ".NET Development" at http://forums.microsoft.com/MSDN/default.aspx SiteID=1.