Application Compatibility for Windows Vista
Only drivers and system files should be installed to sytem32 folder according to Vista Software Certification. Here are a couple thoughts that could be going on.
1) If this is a system file that is protected and registered in a different location, Vista will "lie" that it's install when it never is.
2)Your install may not have the permissions as "Admin" to install to that location. However I would expect a failure from your setup program.
Are you using an msi to install If so you can generate a log file with more details on the install process and find what went wrong. Is a script or custom action called durring the install process