Alan N

Hey All,

When attempting to run a LoadTest overnight, I found that the driver machine quickly (3 hrs) ran out of memory (1 Gig).

The LoadTest consists of one coded WebTest, which in turn consists of 6 requests with a total think time of ~10 seconds. User load is constant 10 users.

I ran the same test for an hour and monitored VSTestHost's private bytes, and here is what I found:

Screenshot

Looks like the Gen 2 heap size keeps growing indefinitely, eventually eating up all available memory.

What does VSTestHost store in memory My understanding is that the test results were written to a temp file, and then loaded into the SQL server at the end of the test.

Please help me find the memory hog.

FYI...there is a similar problem reported here:

http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=2286451&SiteID=1


Thanks,
~Alan



Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Bill Barnett - MSFT

There are a couple of things to check:

1. If you are running on a multi-processor test machine, make sure to configure the "Server GC" as described in Sean's blog post here: http://blogs.msdn.com/slumley/pages/improve-load-test-performance-on-multi-processor-machines.aspx. I doubt this is the cause of your problem since you are only running with 10 users and you have think time enabled, but this is a good thing to know anyway.

2. Does your coded Web test declare any static objects (especially like Lists/Hashtables/Dictionaries) that could be growing as the load test runs

3. Do any of the responses from your server use GZip encoding If so, you may be running into a known memory leak bug in load testing that is in VSTS 2008 Beta 2, but will be fixed in the RTM release.

You are partially correct about test results. Many test results are written to the database directly while the load test is running, but larger items (such as the details for a failed Web request) are written to temp files and loaded into SQL server at the end of the test. Is your load test results database on the local machine or have you configured it to point to a database on a remote machine





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Alan N

Bill,

1) Server GC is enabled

2) No such static objects are created

3) I do not believe our site does any GZipping at this time

My results store is on the local machine.

Thanks,
~Alan





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Bill Barnett - MSFT

Ok, some much for those theories!

I believe from another post that you are running VS 2008 Beta 2, right

I don't suppose your Web test hits a public site so that I could try to run it if you sent me the test project does it

If not, do you have some time to try to narrow down the problem For example, if you remove the requests from the coded Web test one at a time, or remove any requests that do something different than the others, does the memory leak still occur

Is there anything particulary unusual about your coded Web test What does it do that requires it to be a coded Web test rather than a declarative Web test





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Alan N

Bill,

You are correct, I am now using 2008 Beta 2. I have 2005 and 2008 installed side by side. I will try running the test tonight with 2005 to see if it behaves differently.

I have another WebTest with a single request (reservation time search) that I have been running at a much faster rate, and it does not show this behavior nearly as badly as the 'make reservation' test.

The reason for the coded WebTest is that to place a reservation, an anti-hacker ID must be generated by processing a cookie from one of the responses.

I post observations of a 2005 run tomorrow.

~Alan





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Bill Barnett - MSFT

Alan, any new info on this





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Alan N

Sorry Bill, I have not had time to play with it more. We've decided, for the time being, to run the tests in intervals which will allow the system to recover memory during break periods.
My last test confirmed that even a non-coded webtest with a single request eats up memory during an overnight run, but at an acceptable rate.
I will dig this thread up when I do have the time to get back to it.

In the meantime, you guys can try running some extended tests (12+ hours) and monitor memory usage by VSTestHost.exe and DevEnv.exe

Thanks,
~Alan




Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Ed Glas - MSFT

We've done that, successfully running a wide array of tests for 3 and 5 day tests without seeing a memory leak.

I'll mark this answered for now. Unfortunately 2008 is shipping, we won't be able to fix this until SP1. Of course we can send you a QFE, we'll need to work through support for that.

Ed.






Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Håkan Bj

The GZip encoding memory leak is it present in the 2005 release as well or is it just 2008 Beta 2

Having trouble with a crashing endurance test when using 2005, VSTestHost.exe uses a lot of memory. I haven't though found any thing in my plugins that could be the cause of this.

/Hakan





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Håkan Bj

I have seen a couple of threads about problem with the memory usage of VSTestHost.exe. Are there any known issues with the memory handling in 2005 Has anyone succeeded in using VSTS for longer tests or is it just not able to do the job

I'm running into the problem that I get a out-of-memory exception after a couple of hours test run. I have removed most of my plugins, set none as the Timing details storage, lowered the number of error details collected, lowered the request urls reported. I can however not see any improvments.

Any suggestions on what else to try

/Hakan





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Bill Barnett - MSFT

Are you running the tests on a multi-processor or multi-core machine If so, make sure that you configure the "Server garbage collector"; see this blog post for details: http://blogs.msdn.com/slumley/pages/improve-load-test-performance-on-multi-processor-machines.aspx.

Also, are any of the responses sent back by your Web server zgip encoded We did recently discover that there is a memory leak in VSTestHost.exe (or QTAgent.exe when running a remote test) that occurs with zgip encoded responses. This is fixed in VSTS 2008 (the RTM version which just shipped last week; this was not fixed in VSTS 2008 Beta 2).





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Håkan Bj

Yes I'm using a multi-processor and have set the server GC.

The responses I get is gzip encoded. So this leak is present in VSTS 2005 as well

/Hakan





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Bill Barnett - MSFT

Yes, the leak is present in VSTS 2005, but as I said it is fixed in the RTM version of VSTS 2008 which is now available for download at: http://msdn2.microsoft.com/en-us/vstudio/products/aa700831.aspx.

I strongly suggest you give it a try. There are many bug fixes and the list of new Web, load, and unit testing features can be found here: http://blogs.msdn.com/edglas/archive/2007/03/28/what-s-new-in-orcas.aspx. (Orcas was the code name for Visual Studio 2008.)

VSTS 2008 can be installed side-by-side with VSTS 2005.





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Håkan Bj

Hi

I have now tried out VS 2008 as well. Same result or it crashes even earlier. I have disabled all plugins but one and that one is just setting

System.Net.ServicePointManager.Expect100Continue = false;

Any ideas on how to get further and find the cause of the process running so high on memory.

/Hakan





Re: Visual Studio Team System - Web and Load Testing VSTestHost.exe memory usage

Håkan Bj

Hi again

I seem to have been able to close on the cause of this issue. I read somewhere else in the forum about someone having problem with memory when using connectionperuser.

I have 400 vusers in my test and changing to connection pool (300 users) made the memory stabilize on 500 mb (private bytes) with the connectionperusers settings it went up to 1,7Gb before the out-of-memory exception.

Could it be a memory leak when using ConnectionPerUser or does this setting just need very much memory

/Hakan