Hey,
Following my previous inquiry, just for testing purposes I ended up simplifying my replication test environment to just a publisher on one side of a 50 Mbps WAN and a subscriber on the other side.
The merge agent is set to run on a schedule from the subscriber since it is a pull subscription. The tables are the only article in the publication and the identities of the tables are manually managed with a 10 increment and a seed of 1 for the publisher
and 3 for the subscriber. The publication does nothing if tables are already present at destination at initialization.
----
Pushing the initial snapshot goes fine.
If I use the application on the subscriber's side, the replication works but at a rate between 2 and 10 rows per seconds. With higher threading on source and destination of the merge agent I can get double that speed. So data is uploading fine but very slowly.
If I use the application on the publisher's side, the merge agent fails on a deadlock if it is running more than 1 thread. The more thread, the faster it deadlocks. And the deadlocks are from the merge agent and not due to other process running.
Here is the deadlock info:
Deadlock graph <deadlock-list><deadlock victim="process67f01d498"><process-list><process id="process67f01d498" taskpriority="0" logused="9772" waitresource="KEY: 11:72057594040483840 (a70899768258)" waittime="117" ownerId="5597733" transactionname="user_transaction" lasttranstarted="2015-08-07T19:32:51.370" XDES="0x67884ad28" lockMode="U" schedulerid="2" kpid="4420" status="suspended" spid="157" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2015-08-07T19:32:51.387" lastbatchcompleted="2015-08-07T19:32:51.383" lastattention="1900-01-01T00:00:00.383" clientapp="Replication Merge Agent" hostname="DT-SVT-SQL-01" hostpid="5256" loginname="INSTREAM\oragain" isolationlevel="read committed (2)" xactid="5597733" currentdb="11" lockTimeout="4294967295" clientoption1="673253472" clientoption2="128056"><executionStack><frame procname="iTRANS.dbo.MSmerge_upd_sp_219F6C7A81E34933198189FCFFED442F" line="637" stmtstart="36758" stmtend="40220" sqlhandle="0x03000b00fc7f6c7667023a01eda4000001000000000000000000000000000000000000000000000000000000">
update [dbo].[msg] set
[ref_to_msg_id] = case when @p2 is NULL then (case when sys.fn_IsBitSetInBitmask(@setbm, 2) <> 0 then @p2 else t.[ref_to_msg_id] end) else @p2 end
,
[msg_from] = case when @p6 is NULL then (case when sys.fn_IsBitSetInBitmask(@setbm, 6) <> 0 then @p6 else t.[msg_from] end) else @p6 end
,
[msg_to] = case when @p7 is NULL then (case when sys.fn_IsBitSetInBitmask(@setbm, 7) <> 0 then @p7 else t.[msg_to] end) else @p7 end
,
[tx_style] = case when @p11 is NULL then (case when sys.fn_IsBitSetInBitmask(@setbm, 11) <> 0 then @p11 else t.[tx_style] end) else @p11 end
,
[tx_type] = case when @p12 is NULL then (case when sys.fn_IsBitSetInBitmask(@setbm, 12) <> 0 then @p12 else t.[tx_type] end) else @p12 end
,
[tx_version] = case when @p13 is NULL then (case when sys.fn_IsBitSetInBitmask(@setbm, 13) <> 0 then @p13 else t.[tx_version] end) else @p13 end
,
[translation_tag] = case when </frame></executionStack><inputbuf>
Proc [Database Id = 11 Object Id = 1986822140] </inputbuf></process><process id="process67f035498" taskpriority="0" logused="15640" waitresource="KEY: 11:72057594040483840 (bbd163ce615e)" waittime="115" ownerId="5597747" transactionname="user_transaction" lasttranstarted="2015-08-07T19:32:51.370" XDES="0x6756b0d28" lockMode="U" schedulerid="5" kpid="3344" status="suspended" spid="114" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2015-08-07T19:32:51.447" lastbatchcompleted="2015-08-07T19:32:51.447" lastattention="1900-01-01T00:00:00.447" clientapp="Replication Merge Agent" hostname="DT-SVT-SQL-01" hostpid="5256" loginname="INSTREAM\oragain" isolationlevel="read committed (2)" xactid="5597747" currentdb="11" lockTimeout="4294967295" clientoption1="673253472" clientoption2="128056"><executionStack><frame procname="iTRANS.dbo.MSmerge_upd_sp_219F6C7A81E34933198189FCFFED442F" line="170" stmtstart="7396" stmtend="8228" sqlhandle="0x03000b00fc7f6c7667023a01eda4000001000000000000000000000000000000000000000000000000000000">
update [dbo].[msg] set [ebxml] = @p14
from [dbo].[msg] t
where t.[rowguid] = @rowguid and
exists (select 1 from dbo.MSmerge_contents c with (rowlock)
where c.rowguid = @rowguid and
c.tablenick = 9868018 and
c.lineage = @lineage_old) </frame></executionStack><inputbuf>
Proc [Database Id = 11 Object Id = 1986822140] </inputbuf></process></process-list><resource-list><keylock hobtid="72057594040483840" dbid="11" objectname="iTRANS.dbo.msg" indexname="PK_msg" id="lock637059f00" mode="X" associatedObjectId="72057594040483840"><owner-list><owner id="process67f035498" mode="X"/></owner-list><waiter-list><waiter id="process67f01d498" mode="U" requestType="wait"/></waiter-list></keylock><keylock hobtid="72057594040483840" dbid="11" objectname="iTRANS.dbo.msg" indexname="PK_msg" id="lock638d81680" mode="X" associatedObjectId="72057594040483840"><owner-list><owner id="process67f01d498" mode="X"/></owner-list><waiter-list><waiter id="process67f035498" mode="U" requestType="wait"/></waiter-list></keylock></resource-list></deadlock></deadlock-list>
sa 34 2015-08-07 19:32:51.567
The second deadlock happening also for that merge agent run:
Deadlock graph <deadlock-list><deadlock victim="process67f044cf8"><process-list><process id="process67f044cf8" taskpriority="0" logused="10724" waitresource="KEY: 11:72057594040483840 (42c4334389b5)" waittime="819" ownerId="5620038" transactionname="user_transaction" lasttranstarted="2015-08-07T19:33:33.030" XDES="0x67884a3a8" lockMode="U" schedulerid="7" kpid="2040" status="suspended" spid="158" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2015-08-07T19:33:33.190" lastbatchcompleted="2015-08-07T19:33:33.190" lastattention="1900-01-01T00:00:00.190" clientapp="Replication Merge Agent" hostname="DT-SVT-SQL-01" hostpid="5256" loginname="INSTREAM\oragain" isolationlevel="read committed (2)" xactid="5620038" currentdb="11" lockTimeout="4294967295" clientoption1="673253472" clientoption2="128056"><executionStack><frame procname="iTRANS.dbo.MSmerge_upd_sp_219F6C7A81E34933198189FCFFED442F" line="218" stmtstart="10564" stmtend="11400" sqlhandle="0x03000b00fc7f6c7667023a01eda4000001000000000000000000000000000000000000000000000000000000">
update [dbo].[msg] set [payload] = @p15
from [dbo].[msg] t
where t.[rowguid] = @rowguid and
exists (select 1 from dbo.MSmerge_contents c with (rowlock)
where c.rowguid = @rowguid and
c.tablenick = 9868018 and
c.lineage = @lineage_old) </frame></executionStack><inputbuf>
Proc [Database Id = 11 Object Id = 1986822140] </inputbuf></process><process id="process67f035498" taskpriority="0" logused="15928" waitresource="KEY: 11:72057594040483840 (bbd163ce615e)" waittime="930" ownerId="5620125" transactionname="user_transaction" lasttranstarted="2015-08-07T19:33:33.127" XDES="0x6756b03a8" lockMode="U" schedulerid="5" kpid="3344" status="suspended" spid="157" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2015-08-07T19:33:33.190" lastbatchcompleted="2015-08-07T19:33:33.187" lastattention="1900-01-01T00:00:00.187" clientapp="Replication Merge Agent" hostname="DT-SVT-SQL-01" hostpid="5256" loginname="INSTREAM\oragain" isolationlevel="read committed (2)" xactid="5620125" currentdb="11" lockTimeout="4294967295" clientoption1="673253472" clientoption2="128056"><executionStack><frame procname="iTRANS.dbo.MSmerge_upd_sp_219F6C7A81E34933198189FCFFED442F" line="170" stmtstart="7396" stmtend="8228" sqlhandle="0x03000b00fc7f6c7667023a01eda4000001000000000000000000000000000000000000000000000000000000">
update [dbo].[msg] set [ebxml] = @p14
from [dbo].[msg] t
where t.[rowguid] = @rowguid and
exists (select 1 from dbo.MSmerge_contents c with (rowlock)
where c.rowguid = @rowguid and
c.tablenick = 9868018 and
c.lineage = @lineage_old) </frame></executionStack><inputbuf>
Proc [Database Id = 11 Object Id = 1986822140] </inputbuf></process></process-list><resource-list><keylock hobtid="72057594040483840" dbid="11" objectname="iTRANS.dbo.msg" indexname="PK_msg" id="lock65cbfc880" mode="X" associatedObjectId="72057594040483840"><owner-list><owner id="process67f035498" mode="X"/></owner-list><waiter-list><waiter id="process67f044cf8" mode="U" requestType="wait"/></waiter-list></keylock><keylock hobtid="72057594040483840" dbid="11" objectname="iTRANS.dbo.msg" indexname="PK_msg" id="lock63a46f700" mode="X" associatedObjectId="72057594040483840"><owner-list><owner id="process67f044cf8" mode="X"/></owner-list><waiter-list><waiter id="process67f035498" mode="U" requestType="wait"/></waiter-list></keylock></resource-list></deadlock></deadlock-list>
sa 26 2015-08-07 19:33:34.120
Question is:
* What am I doing wrong ?
* How can I fix it ?
Regards,
Olivier
Note: Production environment is standard SQL server. So no Enterprise solution :) Thanks