Richard Kain

I was pondering, and something occured to me. Let's say I have a 3D model. Now lets say I want to enable 3D picking for that 3D model. That is to say, I want the mouse to be able to change its properties / response when it is over the 3D object, or clicking while over the 3D object.

I assume that one way to solve this would be to create a ray that originates from an equivalent point in front of the camera, and passes along a Vector3 through the scene. I could then test the ray to see if it colides with any 3D objects (or their bounding boxes) and return values for that particular object. But it also occured to me that this method for picking would be a huge pain in the neck, and would probably end up being far more costly in terms of processing. It would also make it impossible to divide a single model into separate selectable regions. Each mouse-selectable area would have to be divided up into a different 3D model. I do not consider that to be an optimal solution.

Then I came up with a different idea, and I think it would work better. But I would like some opinions as to its feasability. What if I rendered the scene twice Multi-pass rendering is becoming quite common, as it is the basis for many different kinds of special effects. (normal mapping, bloom, motion blur, etc...) Why couldn't I use an additional rendering pass to test for mouse-picking When I test to see what the mouse is over, what I really want is a 2D value anyway. If I render a region before testing to see whether the mouse is over it, I don't have to muck around with rays or 3D collisions or anything of that sort. I would just have to deal with pixels and colors. And if I use multiple passes, I can first render the scene as a testable image with specifically defined colors. 3D objects have textures. Those textures can be any color I choose. And if I disable all lighting and shaders for the pass, I can make those texture colors show up as 100% bright. So a car in the scene will be painted solid green. If the mouse test returns green, it will know that the mouse is over the car.

The only drawback is that the scene will have to be rendered twice. The first time it will render the entire scene with the mouse-test colors, and store the information in a buffer that isn't drawn to the screen. The second pass will render the scene exactly as it normally would, with its regular textures and lighting, and this WILL be drawn to the screen. This way I could have a fully animated scene that can be viewed from any angle, that will still have pixel-perfect accurate mouse testing. It would also allow me to have separete hit areas on a single model, by painting those areas with a different color. On the car, for instance, I could paint the door blue, the car body red, the window green, and the car handle yellow. And I could have each different element react differently to the mouse picking.

Does this proposal sound feasible




Re: XNA Game Studio Express 3D picking - possible solution?

waruwaru

You might be interested in this thread, it's probably not as a "pain in the neck" as you think.

http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=80853&SiteID=1

The 2 pass drawing thing would work, but by the time you go through the trouble of making things different colors, worrying about shading...etc, the math calculation might be faster. You could also use a simplified model for picking/collision detection purpose.






Re: XNA Game Studio Express 3D picking - possible solution?

Shawn Hargreaves - MSFT

I've actually implemented both of these approaches (geometrical math calculations versus rendering the whole scene) for editing tools I wrote in my previous job at Climax, so I can confirm from personal experience that both techniques work fine.

As to which is best, it really depends on your app. The advantage of the rendering approach is that it integrates well with whatever crazy rendering techniques you might be using. For instance if you have skinned animation done in the vertex shader, or displacement mapping, or billboard sprites, or using alpha test to cut out areas of a polygon, all those things just magically work with the hittesting without you having to do anything special to enable them.

There are some downsides, though:
  • Reading back from a rendertarget to the CPU can be quite slow on some cards. Not a problem if you're just testing a single pixel, but this can cause a noticable lag if you are doing region selects and want to know all the objects in a large area of the screen.
  • This is an image space technique, so if an object ends up smaller than a pixel in the rendered scene, you won't ever notice it. This can be a problem if you want to region select all the triangles or vertices in an area of a mesh. If you zoom too far out, the region select will randomly miss some of the triangles because they ended up too small to get even a single pixel.
  • The framebuffer only holds one color per pixel, so you can only ever select the closest object at that pixel. There's fundamentally no way to select all the objects inside a region, if you have multiple layers of objects and want to select even ones that are hidden behind others. This can be nasty if you're doing something like region selecting all the meshes that make up a car: you'll tend to miss the wheel that's on the wrong side and obscured by the car body.
Doing the tests using math computations, on the other hand, has no problems with very small or obscured objects, and the performance is nice and consistent for large region selects as well as clicking a single point. But you'll have to do more work if you want to support things like skinned animation or displacement mapping.

For what it's worth, I used rendering and reading back colors in the first tool I wrote, and math computations in the second and third tools. So you can probably guess which technique I ended up preferring :-)





Re: XNA Game Studio Express 3D picking - possible solution?

Darkside

Another possible solution is exampled on the XNAResources site, using color keys on a 2D plane, might help if you can translate your 3D pojection into a 2d, plane possiblly




Re: XNA Game Studio Express 3D picking - possible solution?

Richard Kain

The intention of this picking application was clearly not spelled out. I said "picking" and everyone seems to think I meant "real-time-strategy." What I meant was..."point-and-click adventure." In a point-and-click, there is no dragging and dropping, or selecting of multiple units at the same time. Since the camera would probably not shift as much, there isn't as much concern over smaller, obscured units. The kind of picking used in traditional point-and-click games was entirely based on color-keyed 2D planes. As such, creating a 3D game that plays like a 2D point-and-click adventure is a bit of a challenge. But if there were a way to use multiple passes to allow a 3D scene to be rendered into color-keyed 2D planes, it would be much easier to simulate the functionality of a traditional adventure game.

Of course, eventually, I would have to use some manner of raycasting anyway. After all, the characters can't be moved around the scene in 2D, now can they No, they have to move in 3D. And that would require useing a ray to get the Vector3 coordinates of where the ray intersects whatever 3D model is being used as the floor. But I think the color-keyed 2D plane rendering still has merit. The rendered key plane could even be adjusted depending on what control is currently active.






Re: XNA Game Studio Express 3D picking - possible solution?

waruwaru

Richard Kain wrote:
The intention of this picking application was clearly not spelled out. I said "picking" and everyone seems to think I meant "real-time-strategy." What I meant was..."point-and-click adventure." In a point-and-click, there is no dragging and dropping, or selecting of multiple units at the same time. Since the camera would probably not shift as much, there isn't as much concern over smaller, obscured units. The kind of picking used in traditional point-and-click games was entirely based on color-keyed 2D planes.

If that is the case, a bounding box/circle would've worked too. You are correct, each solution have its own pros/cons depending on requirements. Color-key can be a good solution for some apps/games.






Re: XNA Game Studio Express 3D picking - possible solution?

Richard Kain

The only thing that worries me about a bounding box/circle is that it wouldn't work well for objects that have irregular shapes. One of the benefits of the color-key layers used in point-and-click games was that each object in the game could have pixel-perfect hit detection. (no matter what shape or outline it happened to possess) The problem was that each differently colored hit area had to be painted. And painted graphics aren't nearly as nice for scaling. (unless you paint them really large, and in that case you lose quality when scaling down) 3D graphics can work very well for scaling. And they have obvious benefits for animation as well.

But a 3D model that has lighting switched off, can easily be rendered as a solid block of just one color. And with the proper mipmapping settings, it would be easy to have multiple color blocks assigned to the same model, just by painting a "color key" UV map. With this method, it would be possible to simulate the 2D color-key functionality of these older games, while rendering them using full-3D graphics. And the rendered 2D color-key would adapt to any conditions you made for it. You can change the camera angle, or place animated characters or props in the scene, and it will still give you pixel-perfect picking for your mouse cursor.

It isn't easy to describe with just words. I'll dig up a 3D scene I've worked on in the past and see if I can't cook up a visual exampe that I can link to.






Re: XNA Game Studio Express 3D picking - possible solution?

waruwaru

I don't think anyone is questioning the color-key solution. With the requirement you specified, color-key solution would be perfect. :)




Re: XNA Game Studio Express 3D picking - possible solution?

Darkside

in which case then the example on the XNAResources site should be what your looking for then, whic uses a colour key map for the picking you suggested

Darkside