Shader states, etc? (HLSL effects not working right)

I'd like to call upon all the shader gurus here for some help :)

I've been experimenting with writing some of my own effects, meaning I'm a HLSL newbie, and have been using NVIDIA's FX Composer to do my work. After producing a cool (albeit simple) effect, I decided to try using it in my little model viewer. The problem is that the way it shows up through XNA is different than the way it shows up in FX Composer, and I can't figure out why.

You can find the code to my effect here: http://www.hiranipra.com/data/help_attach/gb1_shader.txt - it's just fairly simple environment mapping-ish stuff.

A screenshot of what it looks like in FX Composer: http://www.hiranipra.com/data/help_attach/FxComposer1.jpg - it's hard to tell, but the model has a nice shine to it, and as it rotates the shine changes. Looks like metal.

And the problem, a screenshot of what it looks like through my viewer: http://www.hiranipra.com/data/help_attach/Xna1.jpg - note the sharp edges on the shine areas.

So obviously I'm applying the effect right (as the model shows and such), and the textures are all there (when they aren't there is nothing displayed), but it's just not working.

The shader takes in two parameters - the worldViewProj matrix and the viewInverse matrix. Both of these are set each frame in my code:

effect.Parameters[ "worldViewProj" ].SetValue( _transformMatrix * transforms[ mesh.ParentBone.Index ] * _view * _projection );

effect.Parameters[ "viewInverse" ].SetValue( Matrix.Invert( _view ) );

This seems right, I think.

In an attempt to get more debugging information, I set the pixel shader to return the color of the cubemap. This gave the predicted results in FX Composer: http://www.hiranipra.com/data/help_attach/FxComposer2.jpg - but look at what it did in XNA: http://www.hiranipra.com/data/help_attach/Xna2.jpg - completely wrong!

Finally, I recorded a run of FX Composer with PIX and looked through the results - I don't see FX Composer doing anything different when it comes time to render my model (sets the texture states as I have them in the effect, and draws the vertices).

So then my question is, what else do I need to set to get things looking right Is this an issue with FX Composer setting states I don't know about, or something in XNA setting defaults that are different than normal, or my shader code doing something stupid (maybe all of the above :)

Try normalizing worldEyePos or OUT.texCoordMetal.

Nice Eve online model by the way. Extracted from the game

Unfortunately that didn't work :(

I was thinking it might be some divide by zero error, as I noticed that a lot of weird things happen when the video card executes stuff like that and gets NaN's, but I couldn't figure out where it would be coming from (and my pixel shader assembly skills aren't good enough to debug through the PIX disassembly).

Yeah, I wrote a .tri loader and a .blue loader - my hope is to get it converting the .blue shaders in to HLSL. It's nice because I can just add a .blue to my project and the content pipeline spits out a Model, an Effect, and a bunch of embedded Textures :)

Here is a random fix for you based on that HLSL has different layout of the matrix data than XNA. Was actually my first though on the matter but I wrote the normalization thing first. Hopefully this will work. If not..combine them...if it still doesn't work then...hmm...dunno...

effect.Parameters[ "viewInverse" ].SetValueTranspose( Matrix.Invert( _view ) );

Oh, and are you sure your texture came into XNA correctly. You should do some testing with that aswell.

Thanks for the answers - that bad Transpose would have messed me up :)

Unfortunately, I'm still having the same problem. With that view matrix issue fixed, things should be nearly identical between the two applications, so I'm still at a loss as to why they are producing different results. Is XNA doing something under the hood that is setting weird device states If I load the same model, with the same textures and effects, shouldn't the results in an XNA application be the same as in a normal D3D9 app

Tried transposing the other matrix aswell

Personally I do like this and then I don't have to worry:

switch( effectParam.ParameterClass )
{
case EffectParameterClass.MatrixColumns:
effectParam.SetValueTranspose( (Matrix)value );
break;
case EffectParameterClass.MatrixRows:
effectParam.SetValue( (Matrix)value );
break;
...
}

Malmer wrote:

Personally I do like this and then I don't have to worry: switch( effectParam.ParameterClass ) { case EffectParameterClass.MatrixColumns: effectParam.SetValueTranspose( (Matrix)value ); break; case EffectParameterClass.MatrixRows: effectParam.SetValue( (Matrix)value ); break; ... }

Nifty trick! I haven't started playing with efficiently calling the effects yet (by saving parameter references, etc), so I haven't looked at the object model yet. I bet there's a bunch of cool stuff that could be done with the annotations/semantics.

To exclude matricies as the problem, I went ahead and hand coded the values for worldViewProj and viewInverse to be the values that FX Composer is using (and of course I set my viewport to be the same size so that the projection matrix was valid). The LHS/RHS difference shouldn't come in to play here (someone let me know if that assumption is wrong), so things should look identical.... and they almost do. Whereas FX Composer is looking at the model from the front, my window looks at it from the back - same elevation and angle, just flipped along the Z axis. Weird!

But that should remove matrix problems from the equation.

I ran PIX again, and see it set textures - I know the cube map has loaded properly, as I can inspect it in PIX. Now that I'm getting more familiar with the shader debugger, I stepped through. Here is the code for the pixel shader, that is set to do just a texCUBE on the cubemap sampler at OUT.texCoordMetal and return the value:

where s0 is the cube sampler. While stepping through I notice the values of t1 (OUT.texCoordMetal) change, but the texld always returns (1,1,1,1) in r0.

So I ran FX Composer through PIX and stepped through the same shader, and found that the values of t1 were the same as in my shader, but texld returned real color values!

So what this means is that both my app and FX Composer seem to have the same cube texture loaded, are both using the same exact shader code, have the same input parameters to the shader, and seem to even be getting to the same point in the shader with the same values. The issue is that texld is returning the wrong values for valid input.

The good news is that I've gotten better at diagnosing shader issues, the bad news is I'm at a loss!

My next step, even though this is running in a very small testbed app separate from my main one, is to create another testapp that does nothing but load the mesh and the shader and try to display it. I'll post it up here and let you smart people see if you can reproduce the issue. I did run it on both my desktop with an NVIDIA GeForce 7600 and my laptop with a FireGL v5200, and it looked the same, but who knows.

Thanks again to everyone who is trying to help me out, I really appreciate it :)

Ben Vanik wrote:

To exclude matricies as the problem, I went ahead and hand coded the values for worldViewProj and viewInverse to be the values that FX Composer is using (and of course I set my viewport to be the same size so that the projection matrix was valid). The LHS/RHS difference shouldn't come in to play here (someone let me know if that assumption is wrong), so things should look identical.... and they almost do. Whereas FX Composer is looking at the model from the front, my window looks at it from the back - same elevation and angle, just flipped along the Z axis. Weird!

Not very weird. XNA uses a right handed coordinate system. DirectX (and probably FX composer) uses a left handed (generally that is). Check the DirectX docs for nice little pictures of the differences.

An other thing you might look into: Is your texture loading allright Use a spritebatch or something like that an draw the texture to the screen. Not entirely sure how you do that with a cubemap, but therte must be some very basic test you can do to make sure it gets imported correctly. I mean it really could be that which is your problem.

Yeah but I was under the assumption that the difference only mattered when building the matrices, and after that things would be ok (it's often been said here that if you don't like the new RHS world order, just build your matrices in LHS and then feed them in and there won't be a difference). Doesn't seem to be the problem anyway :)

More research!

Here is a PIX run for anyone who just wants to see that, as the solution may lie in there: http://www.hiranipra.com/data/help_attach/Run53.zip

Now for a series of pictures describing my findings, and where I've hit the wall:

First, what I see: http://www.hiranipra.com/data/help_attach/xRender.jpg

Next, the states getting set: http://www.hiranipra.com/data/help_attach/xState.jpg

The cube map, showing that it is loaded properly: http://www.hiranipra.com/data/help_attach/xCube.jpg

The VS running and producing the correct results: http://www.hiranipra.com/data/help_attach/xMesh.jpg

The PS, doing something retarded: http://www.hiranipra.com/data/help_attach/xShader.jpg

Notice that the sampler state is set to mirror on all 3 texture coordinates, so I shouldn't have to normalize anything and this is the behavior I want (can't believe I didn't remember that earlier - I blame it on being tired! :) Also notice that the cube map has only one mip level - I did this intentionally because I was worried that D3D was doing something silly and using the mip level with 2x2 or 1x1 (which would explain the results). For the VS, note TexCoord1, which is what is used to sample the cube map. I know x=y=z=w, and I plan to optimize it so that only one float is used, but it's easier like that for now :) Regardless, the data is correct, and matches FX Composer's results.

Finally, look at what the PS does. t1, TexCoord1, is correct, however the value sampled from s0 is completely wrong! When FX Composer runs and does the same thing (with a similar value), it gets back a valid color sample. And of course, after texld returns all the rest of the results are garbage.

(Also, if you look in the render shot, you'll see that there is both really light and dark grey - when it's dark grey, texld returns (.244,.244,.244) or something like that)

So..... pardon my language, but wtf :)

Here is the source and content (and executable) for the project I whipped up: http://www.hiranipra.com/data/help_attach/TestGame1.zip

If you look at the top of Game1 you'll see a constant Mode, change it to ShinyDiffuse to see what the effect really should do and how badly it looks because of this bug.

Thanks to anyone who has read this thread!

Hey...what about using samplerCUBE instead of just sampler when you set up the sampler.

Example here from DirectX SDK.

http://msdn.microsoft.com/library/default.asp url=/library/en-us/directx9_c/Goal_3___Environment_Mapping.asp

Malmer wrote:

Hey...what about using samplerCUBE instead of just sampler when you set up the sampler.

Nice catch! Unfortunately still no luck ;(

Yeah when I first saw it doing that I was kind of surprised too, but I guess it makes sense. If you look at the assembly generated: http://www.hiranipra.com/data/help_attach/xShader.jpg, you'll notice it sources from s0, which means the hlsl compiler is good enough to notice that under this specific case where I'm only using the cube map, it only uses one sampler. Very different than what I'm used to with the fixed function pipeline and worrying about each texture stage :)

Thanks for looking through my stuff, I really do appreciate it. I'm starting to get so frustrated that I may give up on using effects for this game :(

Ben Vanik wrote:

Yeah, I wrote a .tri loader and a .blue loader - my hope is to get it converting the .blue shaders in to HLSL. It's nice because I can just add a .blue to my project and the content pipeline spits out a Model, an Effect, and a bunch of embedded Textures :)

Any chance of getting a description of the formats I worked out a bit of the blue header awhile back, but had to give up on anything else because staring at the rest of it trying to figure out where the offset to get to the string table at the end was was taking far too much time...