Johnathan Seal

Howdy,

I have been using VS TFS 2.0 now for about a year. Hear is a list of major pain points that I am requesting for the ORCAS release:

A fix to the web service code coverage issue: http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=6819&SiteID=1&PageID=1

A easy way to add custom tests to TFS: The manual tests are just flat txt files that are not easily searched/indexed. Adding a custom test with fields much like priority, component, expected run frequency, etc. would really help orgs out. Currently, adding a custom test is very painful.

Integration with Selenium RC (C#): since there is no functional testing type with VS test edition, this would be a good start.

Integration with NUnitForms (C#): again, since there is no functional testing type with VS test edtion, this would be good. Having to spend up to 20K for an external tool licence (ya, some are actually that expensive!) after the VS test edition license just does not add up in most org's budgets!

Parsable code coverage output files: there is big test value in the ability to parse and push particular block and arc coverage to a file or db.

Test Report integration with Share Point 2007: this is do-able right now, but having it out-of-box would be a big plus.

If there are others out there that read this post and have something to add, please do.

Cheers, Johnathan



Re: Visual Studio Team System - Testing A cry for help with VS ORCAS test edition

Bruce Taimana MSFT

Hope this helps:

A fix to the web service code coverage issue: http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=6819&SiteID=1&PageID=1.

>>>>>> We can look into it for future releases. Depending on what is needed, will depend on what will happen.

A easy way to add custom tests to TFS: The manual tests are just flat txt files that are not easily searched/indexed. Adding a custom test with fields much like priority, component, expected run frequency, etc. would really help orgs out. Currently, adding a custom test is very painful.

>>>>>> We are aware of this, and will look into it for future releases. There is high probability it will NOT be in the next version (what you called Orcas) of Team Test because of the small time frame to get the next version out, but will probably do a good, workable solution for the following one.

Integration with Selenium RC (C#): since there is no functional testing type with VS test edition, this would be a good start.

>>>>>> There is no reason why Selenium RC cannot integrate with VSTS. API's are public and documented for 3rd parties to do so. You may want to contact the owners about doing so.

Integration with NUnitForms (C#): again, since there is no functional testing type with VS test edtion, this would be good. Having to spend up to 20K for an external tool licence (ya, some are actually that expensive!) after the VS test edition license just does not add up in most org's budgets!

>>>>>> Same as the other integration question above.

Parsable code coverage output files: there is big test value in the ability to parse and push particular block and arc coverage to a file or db.

>>>>>> Code coverage results can be exported to xml files. Those should be parsable.

Test Report integration with Share Point 2007: this is do-able right now, but having it out-of-box would be a big plus.

>>>>>> You may want to check out this forum post. http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=1147147&SiteID=1

DISCLAIMER: The items in this post may not reflect the intent or ideas of Microsoft Corp.






Re: Visual Studio Team System - Testing A cry for help with VS ORCAS test edition

Johnathan Seal

Thanks Bruce.

One more thing that seems to be coming up over and over. We (the testers out there) need a way to parse/build .trx files. There are times that I want to pull out data or even insert data that TFS does not handle cleanly. With the Word 2007 XML document API, there is value in parsing the .trx file to get docx files out. Further, there is value in taking an external test result and creating a .trx file from it. Such a tool would not even need to be released with the next VS but as a downloadable tool. The bottom line on this is that it is needed yesterday.

Cheers, Johnathan





Re: Visual Studio Team System - Testing A cry for help with VS ORCAS test edition

Anutthara - MSFT

Johnathan - can you please elaborate more on what it is that you are looking to inject into the trx file Is it code coverage data Or are you trying to override results Or do you need to add more logging data Specific requirements will help us figure out if we have something already that will fix these issues.

Thanks

Anu






Re: Visual Studio Team System - Testing A cry for help with VS ORCAS test edition

Johnathan Seal

Howdy,

Case - 3rd party tools: There are a few automation frameworks that we are using that use 3rd party tools (NUnit Forms) or Selenium. I want to use them 'with' TFS. So basically, I will create a placeholder test in TFS - then run the 3rd party tool and create a .trx and upload to TFS. If I know what to modify, I can create the .trx and work with it.

Case - test case modification: There may be cases when false positives or false negatives occur in the test pass. We will need to modify the .trx (or the data cube on the warehouse). What is the best way to deal with this

Case - code coverage: the code coverage can be exported to XML. However, I do not believe this will give a good arc or path coverage. An in depth article as to what to look for in the .trx file would be of great value.

So thanks Anutthara for any help. The top case is the priority but I think each case has value.

- Johnathan





Re: Visual Studio Team System - Testing A cry for help with VS ORCAS test edition

Will Ballard

Hi there, I'm interested in implementing these features you describe as a plug in to VS Orcas as a followon to TestRunner (http://www.mailframe.net) in the Orcas timeframe -- very much aimed at providing NUnit testing compatibility. This would at least allow for NUnit forms and Selenium, but perhaps others.

Can you contact me at support@mailframe.net so I can ask you a few questions about how you'd like it to work