garrytman


We just replaced an older Windows 2000 sp4 server with a new Window 2003 Standard Edition sp2 server to run our batch processing. We noticed that when the SQL command times out we now get the following error in the ADO command object error collection.

[DBNETLIB][ConnectionWrite (send()).] General network error. Check you network documentation.

Native Error: 11

SQL State: HY018

msado15.dll version 2.82.3959 is on this server.

On other servers running Windows 2003 Enterprise Edition sp1 the error on the ADO command object is normal with.

Timeout expired

Native Error: 0

SQL State: HYT00

msado15.dll version 2.82.1830 is on this server.

The SQL server they are talking to is on the other side of a firewall. It is SQL 2005 sp2 running on Windows 2000 sp4.

I have also tested this on a Windows 2003 sp2 server that doesn't have to cross a firewall and get the correct Timeout error. It also has the same version of msado15.dll as problem server.

The application is a collection of VB6 Components that are running in COM+ applications. I have isolated the test application to our one SQL interface component and have build a test vbs that can reproduce this on demand. The VB6 Components were not modified in any way for this change and have been working fine for many years prior to this.

All other aspects of our batch processing works as expected. Transactions are working fine. As long as the SQL doesn't timeout everything is normal. DTCPing was used to ensure both servers have correct ports open to allow DTC traffic. Checked the SynAttackProtect setting on SQL and this is not an issue since other clients have no problems.

One point I would like to add is that this new server was initially setup with a temporary server name and then was changed to use the name of the original server on move day. Don't know if this has any impact or not.




Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

Matt Neerincx


From a high level this sounds like an issue with the firewall, since the same client bits (and same server bits) work if they don't cross the firewall.

When the client times out, it sends a small packet called the attention packet to the server. When the server recv's this packet it sends back a response called the attention ack. It could be possible that the firewall is blocking these small packets for some unknown reason, but I have never seen this before. I would contact your firewall vendor to see if they can look at this at the network level.







Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

garrytman

I'm in the process of getting our network engineer on board to trace the traffic between these 2 servers. The only thing is that I have another server (Windows 2003 sp1) that must go through the same firewall and it works just fine.




Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

Matt Neerincx

One thing I recommend you do is run a trace on both the client and server machines simultaneously when you run the test.

This will tell you if some hardware in the middle is causing the problem.






Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

mitre

I am experiencing a very similiar/urgent issue as well. We have Biztalk 2002 installed on a windows 2000 server. The biztalk 02 databases reside on a windows 2003/sql 2005 server. We recently applied the new service pack (SP2) to the windows 2003 server and now intermittently revceive the "general network error" as well as many "unspecified" errors from the biztalk COM+ objects, these generally happen when the BT server is trying to retrieve coinfiguration from it's config database under load.

The only change that has been made has been the service pack. Are there any other COM+, Networking or the like modifications made by the sp2 patch that could cause such issues.

I'm at a loss but this sounds similiar.





Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

Matt Neerincx

This sounds more like the SynAttackProtect issue ->

See our blog for the gory details:

http://blogs.msdn.com/sql_protocols/archive/2006/04/12/a-special-gne-general-network-error-messages-when-running-sql-server-after-installing-service-pack-1-for-windows-server-2003-and-tcp-registry-key-synattackprotect.aspx

To be honest I would take the "disable SynAttackProtect" solution since in most cases your SQL boxes are inside corpnet you are very safe from this attack.






Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

garrytman

Our SQL server is running windows 2000 sp4 and we have several servers running our code that don't have this issue. Just in case I applied the SynAttackProtect setting and rebooted our SQL server with no change.

Today networking changed the firewall settings to allow all traffice to flow between the 2 servers. Same issue continues.

I have 2 trace files from the SQL server view of the conversation. I have to wait for the other side to be done. Maybe these can shed some light on the Issue. Is there a way for me to get them posted for review





Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

Matt Neerincx

Actually I was replying to mitre's post recommending SynAttackProtect. For your issue garrytman this would not apply, sorry for the confusion.

What sort of trace did you capture Netmon






Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

mitre

Hi Matt,

Thanks for your insight. We are monitoring for this as well. One thing that is unusual though is the server already had windows 2003 sp1 installed on it, we only started seeing the issue once sp2 was added. The symptoms are similiar as to what is described on the blog.

Unfortunately, I cannot add retry logic to the components since they are BT 02, blackboxed components being used by the bt server runtime to call our implmented AIC components





Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

garrytman

They are not Netmon captures. The network folks told of some shareware product called WireShark that I could use to look at them. But analyzing the traces is beyond my current skillset. I downloaded the shareware and I can look at the trace files. They have the extension .enc if that helps.

The only thing that stands out to me is one packet that is showing up in Red. It is from the client to the server a TCP packet [RST, ACK] that happens about 10 seconds after the call starts to the StoredProc. Which is the timeout setting in the my test script. Why the analyzer software marks this packet as red I don't know yet.





Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

garrytman

I put WireShark on 4 different machines and have come up with some interesting findings.

On the problem machine I am not capturing any TDS traffic to the SQL server only TCP on all the other clients there is a bunch of TDS traffic between the servers. I've checked my capture filters several times and don't think I am only filtering on TCP packets only. I will continue to look.

On good clients I see the Cancel Packet with the Response Packet from SQL server containing 'The statement has been terminated.'

Good clients include:

  • Windows XP sp2 (No firewall
  • Windows 2003 sp2 (No firewall)
  • Windows 2003 sp1 (thru same firewall)

Bad client is Windows 2003 sp2 thru firewall.

At least I have something concrete to work with now. I'll keep digging.





Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

Matt Neerincx

Yes, during a normal timeout what you should see is the following:

1. Client sends request to server (1 or more tcp packets).

2. A gap in the trace activity while the server processes the request up to the command timeout value.

3. When the client detects that command timeout expires, the client will send a small (8 byte) packet called the attention packet to the server.

4. You should then see an attention response packet from the server very shortly thereafter.

Note during the above exchange there is no need for client and server to tear down the connection, so if you see RST or FIN ACK on the connection something is going wrong.

If you see RST from client to server, most likely this is due to firewall resetting the connection. This is why you need to run tracing on the client as well. What you will see if the firewall resets is a RST from client to server in the server trace and a RST from server to client in the client trace (indicating something in the middle RST the connection).

If you see RST from client on both client and server side traces, then you know the RST came from client. RST can come from client in some cases, for example if client application terminates abruptly without closing the connection.






Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

garrytman

Here is were I'm at. I've upgraded NIC card driver. Noticed IWAM and IUSR were incorrect (because of server temporary name during initial install) I uninstalled IIS and reinstalled so IWAM and IUSR id's were now correct.

Still no relief. Problem seems to be in RPC area as I see no RPC traffic between the two servers. All the sniffer picks up is TCP traffic.





Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

Matt Neerincx

So you see a RST coming from client machine in trace on client machine

If this client code is running inside IIS, where is it running isapi filter ASP.NET application ASP application CGI application PHP application






Re: [DBNETLIB][ConnectionWrite (send()).] General network error. Occurs when SQL times out on new Windows 2003 sp2 server

mitre

We ran network sniffers as well and found nothing wrong. In our SQL profiler, on the biztalk object calls that fail, we see as the last line:

if @@TRANCOUNT > 0 COMMIT TRANS

Then we get an "unspecified" error back from the bt object. They are closed objects so I have no idea how they make calls to the db, whther they dispose properly etc etc...This error does not happen consistently. Again nothing in our code has changed, just the application of sp2 on the windows 2003 server which already had sp1. I did notice in the release notes that this sp2 patch "fixed" problems with MSDTC, DATA Access and COM+. My fear is one of these fixes caused a subtle change resulting in issues with our BT server environment and how it accesses the database on the win 2003 server.

I'm quickly running out of ideas to test.