Hi all,
I have an interesting problem that I have not seen before and need some help from you clever folks...(sorry for the long post !)
Production environment - two node cluster Windows Enterprise Server 2008 R2 with SQL 2008 R2 SP2 Standard in an active / Passive mode, this SQL server holds approx 150 databases from small 500MB to larger 60GB.
DR Environment - New DR cluster server (curently just 1 node - second to be added shortly) and we are in the process of mirroring the databases.
Both production server have two processors and 144GB of memory, the DR server has only a single processor and 76GB of memory
We have approximatly 100 databases mirrored but recently we are having problems mirroring the rest of the databases, the problems we are seeing are as follows:
Restoring a database to DR somtimes completes ok, in otrher instances the restore hangs at approx 90% and we need to restart the SQL server to free to the process. When this happens any refresh of the databases causes a SQL exception to be thrown and
i get a mdmp file produced.
If the database restore completes ok and i work through the mirroring process this all goes ok until the final point when i click on the 'start mirroring' at this point the screen flashes briefly and then the system hangs, sometimes a restart of
the SQL server will free then hung process, other time it requires a full reboot of the server to clear the issue.
In the above instances there is nothing reported in the SQL logs, the following errors are from the Application log file:
|
|
|
Keywords |
0x80000000000000 |
|
|
|
|
|
|
[
SystemTime] |
2012-10-31T06:50:18.000000000Z |
|
|
|
|
[sqsrvres]
CheckQueryProcessorAlive: sqlexecdirect failed |
|
|
|
4B4A0040010000000D0000004400420043004C00550053005400450052002D0044005200000000000000
|
|
|
|
Keywords |
0x80000000000000 |
|
|
|
|
|
|
[
SystemTime] |
2012-10-31T06:50:18.000000000Z |
|
|
|
|
[sqsrvres]
printODBCError: sqlstate = HYT00; native error = 0; message = [Microsoft][SQL
Server Native Client 10.0]Query timeout expired |
|
|
|
4B4A0040010000000D0000004400420043004C00550053005400450052002D0044005200000000000000 |
|
|
|
Keywords |
0x80000000000000 |
|
|
|
|
|
|
[
SystemTime] |
2012-10-31T06:50:18.000000000Z |
|
|
|
|
[sqsrvres]
OnlineThread: QP is not online. |
|
|
|
4B4A0040010000000D0000004400420043004C00550053005400450052002D0044005200000000000000 |
|
|
|
Keywords |
0x80000000000000 |
|
|
|
|
|
|
[
SystemTime] |
2012-10-31T06:50:18.000000000Z |
|
|
|
|
[sqsrvres]
printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL
Server Native Client 10.0]The connection is no longer usable because the server
failed to respond to a command cancellation for a previously executed statement
in a timely manner. Possible causes include application deadlocks or the server
being overloaded. Open a new connection and re-try the operation. |
|
|
|
4B4A0040010000000D0000004400420043004C00550053005400450052002D0044005200000000000000 |
further issue I see in the SQL logs is that memory is being trimmed, section of SQL logs:
10/31/2012 06:59:50,spid1s,Unknown,A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 1840 seconds. Working set (KB): 952240<c/> committed (KB): 2662000<c/> memory utilization:
35%.
10/31/2012 06:54:21,spid1s,Unknown,A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 1512 seconds. Working set (KB): 951076<c/> committed (KB): 2660912<c/> memory utilization:
35%.
10/31/2012 06:49:58,spid1s,Unknown,A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 1249 seconds. Working set (KB): 949972<c/> committed (KB): 2659888<c/> memory utilization:
35%.
10/31/2012 06:44:30,spid1s,Unknown,A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 920 seconds. Working set (KB): 948892<c/> committed (KB): 2658864<c/> memory utilization:
35%.
10/31/2012 06:40:07,spid1s,Unknown,A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 657 seconds. Working set (KB): 948348<c/> committed (KB): 2658352<c/> memory utilization:
35%.
10/31/2012 06:34:38,spid1s,Unknown,A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 328 seconds. Working set (KB): 937164<c/> committed (KB): 2641600<c/> memory utilization:
35%.
The odd part is that the server has 60GB+ of free memory
Thanks,
John