rpreston

I posted this in the testing forum initially but I haven't gotten any response so I thought I'd post it here.

I'm using the TestToolsTask in a team build to run unit tests in a C# test project. I'm building configurations that the test project isn't built in and that might be causing my build to fail even though I'm just seeing warnings and not errors.

Is there some way to specify which build configurations to run tests for

These are the lines I added to the team build script after downloading the TestToolsTask I found on Buck Hodges' blog:

<TestContainerInOutput Include="TestProject1.dll" />
<TestContainer Include="TestProject1.dll" />

This is the warning/error I'm seeing:

MSBUILD : warning : File "TestProject1.dll" not found .
The previous error was converted to a warning because the task was called with ContinueOnError=true.
TestToolsTask SearchPathRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Win32\Release\" PathToResultsFilesRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\TestResults" PublishServer="http://cltvsts01:8080/" TeamProject="Integrity" PublishBuild="NightlyBuild_20070919.3" BuildFlavor="Release" Platform="Win32"
Build continuing because "ContinueOnError" on the task "TestToolsTask" is set to "true".
TestToolsTask SearchPathRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Win32\Release\" PathToResultsFilesRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\TestResults" PublishServer="http://cltvsts01:8080/" TeamProject="Integrity" PublishBuild="NightlyBuild_20070919.3" BuildFlavor="Release" Platform="Win32"
Loading X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Win32\Release\\TestProject1.dll...
Starting execution...

I'm wondering if I could override the TestToolsTask and specify conditions If that wouldn't work maybe I could call mstest.exe explicity I'll try these ideas when I have time to.

Any ideas would be greatly appreciated. I've just been ignoring the fact that my builds always fail now, since it's just these warmings from the unit tests that causes it, but it's annoying.

Thanks,
Richard


Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

Aaron Hallberg - MSFT

In Team Build v1 there is no simple way to do this. You could override one of the core test targets, but be warned that this approach will probably cause your build to be broken when/if you upgrade to Orcas. If you want to try it out, however, here's how it would work:

<Target Name="CoreTest" Condition=" '$(RunTest)'=='true' and '%(ConfigurationToBuild.PlatformToBuild)' == 'MyPlatform' and '%(ConfigurationToBuild.FlavorToBuild)' == 'MyFlavor' " DependsOnTargets="$(CoreTestDependsOn)" >

<MakeDir Directories="$(TestResultsRoot)" Condition="!Exists('$(TestResultsRoot)')" />

<MSBuild Projects="$(MSBuildProjectFile)" Targets="RunTestWithConfiguration" Properties="BuildNumber=$(BuildNumber);Platform=%(ConfigurationToBuild.PlatformToBuild);Flavor=%(ConfigurationToBuild.FlavorToBuild);IsDesktopBuild=$(IsDesktopBuild)" />

</Target>

Note the check for your platform/flavor combination in the Condition of the target - this will cause it to execute only for the specified platform/flavor combination. If you want to do more than one, your condition will get a bit more complicated.

Note also that hooks to control this sort of thing already exist if you are using Team Build Orcas - we provide the Before/AfterTest targets, the Before/AfterTestConfiguration targets, etc.

-Aaron






Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

rpreston

Thanks, I'll give that try! I like to break things.

If this works, I'll just put a comment in my script that it might break with Orcas.




Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

rpreston

I'm not sure whether I'm supposed to put this in my build script or in the Targets file.

I tried it in my build script and I got this error:

error MSB4116: The condition " '$(RunTest)'=='true' and '%(ConfigurationToBuild.PlatformToBuild)' == 'Any CPU' and '%(ConfigurationToBuild.FlavorToBuild)' == 'Release' " on the "CoreTest" target has a reference to item metadata. References to item metadata are not allowed in target conditions unless they are part of an item transform.




Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

Aaron Hallberg - MSFT

Whoops - try moving the condition to the tasks within the target, rather than the target itself.

-Aaron






Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

rpreston

I tried putting the RunTestWithConfiguration target in my script and I put the condition in TestToolsTask. The build is successful now, but the tests aren't running.

Here is my condition:

" '$(IsDesktopBuild)'!='true' and '%(TestContainer.Identity)' != '' and '$(RunTest)'=='true' and '%(ConfigurationToBuild.PlatformToBuild)' == 'Any CPU' and '%(ConfigurationToBuild.FlavorToBuild)' == 'Release' "

Any ideas

Thanks,
Richard






Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

Aaron Hallberg - MSFT

Can you post the whole overridden target

-Aaron






Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

rpreston

Sure:

<Target Name="RunTestWithConfiguration" >

<TeamBuildMessage

Tag="Configuration"

Condition=" '$(IsDesktopBuild)'!='true' "

Value="$(Flavor)" />

<TeamBuildMessage

Tag="Platform"

Condition=" '$(IsDesktopBuild)'!='true' "

Value="$(Platform)" />

<!-- SearchPathRoot for not Any CPU -->

<CreateProperty

Condition=" '$(Platform)'!='Any CPU' "

Value="$(BinariesRoot)\$(Platform)\$(Flavor)\" >

<Output TaskParameter="Value" PropertyName="SearchPathRoot" />

</CreateProperty>

<!-- SearchPathRoot for Any CPU -->

<CreateProperty

Condition=" '$(Platform)'=='Any CPU' "

Value="$(BinariesRoot)\$(Flavor)\" >

<Output TaskParameter="Value" PropertyName="SearchPathRoot" />

</CreateProperty>

<!-- Test task for the end-to-end build -->

<TestToolsTask

Condition=" '$(IsDesktopBuild)'!='true' and '%(MetaDataFile.Identity)'!=''"

BuildFlavor="$(Flavor)"

Platform="$(Platform)"

PublishServer="$(TeamFoundationServerUrl)"

PublishBuild="$(BuildNumber)"

SearchPathRoot="$(SearchPathRoot)"

PathToResultsFilesRoot="$(TestResultsRoot)"

MetaDataFile="%(MetaDataFile.Identity)"

RunConfigFile="$(RunConfigFile)"

TestLists="%(MetaDataFile.TestList)"

TeamProject="$(TeamProject)"

ContinueOnError="true" />

<!-- Test task for the desktop build -->

<TestToolsTask

Condition=" '$(IsDesktopBuild)'!='false' and '%(MetaDataFile.Identity)'!=''"

SearchPathRoot="$(SearchPathRoot)"

PathToResultsFilesRoot="$(TestResultsRoot)"

MetaDataFile="%(MetaDataFile.Identity)"

RunConfigFile="$(RunConfigFile)"

TestLists="%(MetaDataFile.TestList)"

ContinueOnError="true" />

<!-- Translate the relative output paths into absolute paths and add them to TestContainerFullPath -->

<CreateItem Include="@(TestContainerInOutput -> '$(SearchPathRoot)\%(Identity)')">

<Output TaskParameter="Include" ItemName="TestContainer"/>

</CreateItem>

<!-- TestContainer tests for non-desktop builds. -->

<TestToolsTask

Condition=" '$(IsDesktopBuild)'!='true' and '%(TestContainer.Identity)' != '' and '$(RunTest)'=='true' and '%(ConfigurationToBuild.PlatformToBuild)' == 'Any CPU' and '%(ConfigurationToBuild.FlavorToBuild)' == 'Release' "

BuildFlavor="$(Flavor)"

Platform="$(Platform)"

PublishServer="$(TeamFoundationServerUrl)"

PublishBuild="$(BuildNumber)"

SearchPathRoot="$(SearchPathRoot)"

PathToResultsFilesRoot="$(TestResultsRoot)"

TestContainers="%(TestContainer.Identity)"

RunConfigFile="$(RunConfigFile)"

TeamProject="$(TeamProject)"

Categories="$(TestCategories)"

TestNames="$(TestNames)"

ContinueOnError="true" />

<!-- TestContainer tests for desktop builds. -->

<TestToolsTask

Condition=" '$(IsDesktopBuild)'=='true' and '%(TestContainer.Identity)' != '' "

SearchPathRoot="$(SearchPathRoot)"

PathToResultsFilesRoot="$(TestResultsRoot)"

RunConfigFile="$(RunConfigFile)"

TestContainers="%(TestContainer.Identity)"

Categories="$(TestCategories)"

TestNames="$(TestNames)"

ContinueOnError="true" />

</Target>






Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

Aaron Hallberg - MSFT

Ah... I was suggesting overriding the CoreTest target, not the RunTestWithConfiguration target. Something like:

<Target Name="CoreTest" Condition=" '$(RunTest)'=='true' " DependsOnTargets="$(CoreTestDependsOn)" >

<MakeDir Directories="$(TestResultsRoot)" Condition="!Exists('$(TestResultsRoot)')" />

<MSBuild Projects="$(MSBuildProjectFile)" Condition=" '%(ConfigurationToBuild.PlatformToBuild)' == 'Any CPU' and '%(ConfigurationToBuild.FlavorToBuild)' == 'Release' " Targets="RunTestWithConfiguration" Properties="BuildNumber=$(BuildNumber);Platform=%(ConfigurationToBuild.PlatformToBuild);Flavor=%(ConfigurationToBuild.FlavorToBuild);IsDesktopBuild=$(IsDesktopBuild)" />

</Target>

By the time you get to the RunTestWithConfiguration target, you'd need to check the $(Platform) and $(Flavor) properties. Basically what's going on here is that the CoreTest target recursively calls MSBuild (using the MSBuild task) on TfsBuild.proj (using the $(MSBuildProjectFile) property, which specifies the currently building project) for the RunTestWithConfiguration target. It does this for each unique combination of PlatformToBuild and FlavorToBuild in the ConfigurationToBuild item group, passing them in as $(Platform) and $(Flavor), respectively.

The CoreTest override given above will only call RunTestWithConfiguration for Any CPU / Release. If you want RunTestWithConfiguration called for each platform/flavor combination, you could keep overriding RunTestWithConfiguration instead and use the following condition:

" '$(IsDesktopBuild)'!='true' and '%(TestContainer.Identity)' != '' and '$(RunTest)'=='true' and '$(Platform)' == 'Any CPU' and '$(Flavor)' == 'Release' "

Hope this helps.

-Aaron






Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

rpreston

Ohhh, OK -- thanks! I'll try that.




Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations


Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

rpreston

Yep, that worked -- now the tests are only being run for the build configurations specified in the condition.

Now my problem is that I get a warning that the test dll can't be found even though the tests are being run.

When my TestContainer and TestContainerInOutput are like this:

<TestContainerInOutput Include="$(BinariesRoot)\Release\TestProject1.dll" />

<TestContainer Include="$(BinariesRoot)\Release\TestProject1.dll" />


I get this warning and the build fails even thought the tests are successfull:

MSBUILD : warning : File "X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Release\\X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Release\TestProject1.dll" not found .
The previous error was converted to a warning because the task was called with ContinueOnError=true.
TestToolsTask SearchPathRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Release\" PathToResultsFilesRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\TestResults" PublishServer="http://cltvsts01:8080/" TeamProject="Integrity" PublishBuild="NightlyBuild_20071022.3" BuildFlavor="Release" Platform="Any CPU"

When my TestContainer and TestContainerInOut are like this:

<TestContainerInOutput Include="TestProject1.dll" />

<TestContainer Include="TestProject1.dll" />


I get this warning and the build fails even thought the tests are successfull:


MSBUILD : warning : File "TestProject1.dll" not found .
The previous error was converted to a warning because the task was called with ContinueOnError=true.
TestToolsTask SearchPathRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Release\" PathToResultsFilesRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\TestResults" PublishServer="http://cltvsts01:8080/" TeamProject="Integrity" PublishBuild="NightlyBuild_20071022.2" BuildFlavor="Release" Platform="Any CPU"
Build continuing because "ContinueOnError" on the task "TestToolsTask" is set to "true".
TestToolsTask SearchPathRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Release\" PathToResultsFilesRoot="X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\TestResults" PublishServer="http://cltvsts01:8080/" TeamProject="Integrity" PublishBuild="NightlyBuild_20071022.2" BuildFlavor="Release" Platform="Any CPU"
Loading X:\Build\Integrity\NightlyBuild\BuildType\..\Sources\..\Binaries\Release\\TestProject1.dll...



It looks to me like there's an extra backslash being put in before the test dll name.









Re: Team Foundation Server - Build Automation Running Unit Tests for specific build configurations

rpreston

Thanks Aaron!

I got it to work by taking out the TestContainerInOutput node and leaving the full path to the DLL in TestContainer. I suspect that it would work without the path.

I got the idea to take out the TestContainerInOutput node from Buck Hodges blog entry about the TestToolsTask: http://blogs.msdn.com/buckh/archive/2006/11/09/updated-version-of-new-testtoolstask.aspx . I'm not sure what TestContainerInOutput does, if anything, but maybe I can live without it.