UPDATED : 2008-10-22 BY CHRIS MORTON
-------------------
Unfortunately at the time I published this article the data was truncated when I submitted it. Unfortunately that stuff is long gone.
Basically it worked like this:
I had a table that mapped the remote sql server (linked server) to the district id that it was associated with.
Then I had a bunch of stored procedures, one for each table that needed an import or update, in all of these stored procedures I had a districtid input parameter, which was resolved to the linked server name it belonged to by a user defined function.
From there the dynamic sql was constructed for example:
If district id was 1 then
Insert into localtable (columnlist plus the districtid) select (exact same column list plus the districtid) from [(linkedserverremotetable(fully qualified name))] where [(remotecriteria)] not in (sub query localtable).
The update works similarly.
Now I had something like 200 stored procedures doing this.
I wrapped them all up in a stored procedure that used a cursor to go through all the districtid's in the linkedsever table and execute them in order.
When I tested it on a local environment it worked very well and it to approximately 2 minutes to import about 2 million records.
This was my first piece of 'advanced' sql.
If I had to do it again I would change some things slightly, and probably use other technology.
However this logic did work, and if the technology is not available (i.e. you don't have enterprise licenses or have older technology)
It is a real shame that my article was truncated. It was eleven word pages long, originally. I have lost it long ago.
Thanks
Chris
-----------------------------
The Problem:
i need to join 19 databases on remote linked servers that do not support replication. the net effect must be the same as replication but without GUID's. i have to maintain all relationships and data integrity. the 19 databases are in different regions of the country, each on a different server and therefore can be identified by districtid and alias (the server name). the connections are not necessarily always connected and failures need to be handled. i have read only access to the databases. Only data that has changed must be updated. Only new data must be inserted. Errors must be traceable and failures must be able to be audited.
The Solution:
Set up your linked servers.
Create a table in the 'subscription' database (in this case [DIMSCONSOLIDATEDData]) that maps the district id's against an alais and an installation.
Use this script for example:
USE [DIMSCONSOLIDATEDData]
GO
/****** Object: Table [dbo].[General_District] Script Date: 08/07/2007 08:38:22 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[General_District](
[DistrictID] [int] IDENTITY(1,1) NOT FOR REPLICATION NOT NULL,
[District] [varchar](255) NOT NULL,
[ProvinceID] [int] NOT NULL,
[Enabled] [bit] NOT NULL,
[Alias] [varchar](90) NULL,
CONSTRAINT [PK_District] PRIMARY KEY CLUSTERED
(
[DistrictID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[General_District] WITH CHECK ADD CONSTRAINT [FK_General_District_General_Province] FOREIGN KEY([ProvinceID])
REFERENCES [dbo].[General_Province] ([ProvinceID])
GO
ALTER TABLE [dbo].[General_District] CHECK CONSTRAINT [FK_General_District_General_Province]
Create a districtid column in the 'replication tables' and make it a composite key.
for example:
USE [DIMSCONSOLIDATEDData]
G
