Myria

VC8's security cookie system (/GS) is designed to not fail in the event of __security_init_cookie being called. However, there is a 1 in 4294967295 chance that it will fail if called twice on an x86-32 program.

This code from gs_support.c tries to prevent a second __security_init_cookie call from changing the cookie. On x86-32, it needs to remake the cookie if the module loader in XP put zero in the high word, so that the cookie is more secure.

if (__security_cookie != DEFAULT_SECURITY_COOKIE
#if defined (_X86_)
&& (__security_cookie & 0xFFFF0000) != 0
#endif /* defined (_X86_) */
)
{
__security_cookie_complement = ~__security_cookie;
return;
}

If that high word is zero, it will reinitialize the cookie on its own.

Later in the code, it tries to prevent a randomly-generated cookie from being DEFAULT_SECURITY_COOKIE or having a high word of zero, because if __security_init_cookie were called again, the above code would cause the cookie to be reinitialized. The code looks like this:

if (cookie == DEFAULT_SECURITY_COOKIE)
{
cookie = DEFAULT_SECURITY_COOKIE + 1;
}
#if defined (_X86_)
else if ((cookie & 0xFFFF0000) == 0)
{
cookie |= cookie << 16;
}
#endif /* defined (_X86_) */

There is a flaw in the x86-32 logic here. It tries to prevent the cookie having a high word of zero by ORing the high word with the low word. However, if the generated cookie is entirely zero, the cookie will remain zero, and a subsequent call to __security_init_cookie will set it to something else. In a program that calls __security_init_cookie twice, which would occur in a program that needs to override the entry point, /GS will falsely detect a buffer overflow under certain conditions.