January 5, 2009 at 10:41 pm
Hello all,
I feel there's something fundamental I'm not understanding about merge replication and the updating of ranges.
We've had an issue whereby the transition from one range to the next isn't fluid. We get "identity range managed by replication is full" errors. The problem arises when the ranges are not sequential and we are inserting a batch of records. The records are inserted with sequential IDs from the identity seed and if larger than the amount left in the first range it fails on the identity constraint unless the next range immediately follows.
I've read in places that Range Threshold Percentage isn't used anymore (it's just there for show perhaps) Yet in others there are plenty of articles that explain how it works with 2005. This is confusing. My testing indicates that it isn't used as whatever I change the value to seems to have no effect (even when assuming the range size to be actually the sum of the two ranges).
The updating of the ranges only seems to occur when the MSMERGE_ins_ trigger runs. This is only going to run after the batch has completed and does not do a check of the curent identity seed against the "range start + threshold" - only against the range end.
I can't see how this can ever work.
Fortunately the customer using this enters records at either the publisher or the subscriber so the error doesn't occur as often as it could, but one of our other customers will be moving from sql 2000 soon so this is likely to change soon.
Version
Microsoft SQL Server 2005 - 9.00.3042.00 (Intel X86) Feb 9 2007 22:47:07 Copyright (c) 1988-2005 Microsoft Corporation Developer Edition on Windows NT 5.2 (Build 3790: Service Pack 1)
Running on Vista now but it also occurred on XP.
I've done a lot of searching but please just point me to the appropriate article/post if there is one.
Thanks in advance,
Michael.
January 6, 2009 at 8:45 am
mtrimboli (1/5/2009)
Hello all,I feel there's something fundamental I'm not understanding about merge replication and the updating of ranges.
We've had an issue whereby the transition from one range to the next isn't fluid. We get "identity range managed by replication is full" errors. The problem arises when the ranges are not sequential and we are inserting a batch of records. The records are inserted with sequential IDs from the identity seed and if larger than the amount left in the first range it fails on the identity constraint unless the next range immediately follows.
I've read in places that Range Threshold Percentage isn't used anymore (it's just there for show perhaps) Yet in others there are plenty of articles that explain how it works with 2005. This is confusing. My testing indicates that it isn't used as whatever I change the value to seems to have no effect (even when assuming the range size to be actually the sum of the two ranges).
The updating of the ranges only seems to occur when the MSMERGE_ins_ trigger runs. This is only going to run after the batch has completed and does not do a check of the curent identity seed against the "range start + threshold" - only against the range end.
I can't see how this can ever work.
Fortunately the customer using this enters records at either the publisher or the subscriber so the error doesn't occur as often as it could, but one of our other customers will be moving from sql 2000 soon so this is likely to change soon.
Version
Microsoft SQL Server 2005 - 9.00.3042.00 (Intel X86) Feb 9 2007 22:47:07 Copyright (c) 1988-2005 Microsoft Corporation Developer Edition on Windows NT 5.2 (Build 3790: Service Pack 1)
Running on Vista now but it also occurred on XP.
I've done a lot of searching but please just point me to the appropriate article/post if there is one.
Thanks in advance,
Michael.
Although Merge Replication is not my most used method I have to tell you that "automatic identity management" works only as well as you could possibly plan to unexpected behavior (many rows changed in batch that surpasses the "threshold") . I never liked the feature and Partition of PK based on location works much better without headaches of identity columns.
I know that the "merge agent" is the one that checks for the threshold but the effectiveness of the feature is flaky at best!
* Noel
January 14, 2009 at 9:07 am
We've been struggling with the same issues we're an MCP so we raised a ticket as MS....
Its been raised on this forum before
http://www.sqlservercentral.com/Forums/Topic339239-291-1.aspx#bm339851
Thank you for contacting Microsoft SQL Server Technical Support. Your case number is ####### and the following is a summary of the current issue. Please take a moment to review it and contact me with any comments you may have. My contact information and available hours are at the bottom of this message.
As we discussed, the issue you're facing below issue Once we provide the answer we will consider this incident completed and closed. We'll be working to resolve this specific issue through the course of the case. If I have misunderstood any aspect of the issue, please let me know.
[SCOPE]
ISSUE:
You have merge Replication with auto identity range enabled. The identity range is getting used up on the publisher when you insert data though below query with no of records greater than the Identity range of publisher.
INSERT INTO Table1(ContactID, [User])
SELECT TOP 995 '-1' AS ContactID, 'HELLO' AS [User]
FROM Table2
Error Message
==============================
Msg 548, Level 16, State 2, Line 2
The insert failed. It conflicted with an identity range check constraint in database , replicated table 'dbo.Calllog ', column ' ContactID '. If the identity column is automatically managed by replication, update the range as follows: for the Publisher, execute sp_adjustpublisheridentityrange; for the Subscriber, run the Distribution Agent or the Merge Agent.
The statement has been terminated.
Environment Information
==============================
Publisher\Distributor
SQL Server 2008 Standard Edition
Windows 2003Server
IMPACT:
Production server
CRITERIA OF RESOLUTION:
You want to identity the above error message is caused while inserting the data tough query with records which is greater than the Identity range of publisher.Is it by design or Bug.
STATUS: Pending Research
I have Reproduced this issue in SQL2008 and SQL 2005 at my end. We need to Check with Escalation team whether this is by design or a known issue.
ACTION PLAN:
NA
NEXT CONTACT:
13-01-2009
The upshot was that MS judged this was "by design" and they would not be persueing the issue but would come back to us with the reccommendec workaround.
In the meantime we are implementing two workarounds.
1.catch the error and run sp_adjustPublisherIdentityRange to reset the ranges
2.increase the identity ranges to a really massive value
January 15, 2009 at 8:51 pm
Although Merge Replication is not my most used method I have to tell you that "automatic identity management" works only as well as you could possibly plan to unexpected behavior (many rows changed in batch that surpasses the "threshold") . I never liked the feature and Partition of PK based on location works much better without headaches of identity columns.
I know that the "merge agent" is the one that checks for the threshold but the effectiveness of the feature is flaky at best!
Thanks Noeld.
Seems I missed your reply notification.
Personally I expect automatic identity management to work. It worked reasonably in sql2000 with the range threshold so it seems odd that they've broken it subsequantly. As we now see that it doesn't we may have to partition the PK based on location. This however is much more work and requires much more scripting.
As it turns out, we're about to take this approach with one of our tables whose IDs are needed for forms and must be 7 characters or less. In our original rolling out of replication years ago and really learning about it we used up well into 8,000,000 so now we have to seed the ID's for each site back. So I have a script ready to go to do this. For interest (and potential critism) - here it is. If we go down the path of set ranges it'll need to be dropped into a stored proc...
--###############################################################################################
--USE THIS TO SETUP IDENTITY FIELD RANGES AUTOMATICALLY
--SETUP THE TABLE SPECIFIC VARIABLES FIRST
--AND THEN BETWEEN EACH SITE SETUP THE RANGE AND SITE NAME
Declare @IDRangeMin int
Declare @IDRangeMax int
Declare @ConstraintName varchar(250)
Declare @TableName varchar(250)
Declare @IDField varchar(250)
Declare @SiteName varchar(250)
Declare @CurrentMaxID as int
--site specific info
Set @IDRangeMin = 15000
Set @IDRangeMax = 20000
Set @SiteName = 'Laverton'
--table specific info
Set @IDField = 'ReplID'
Set @TableName = 'tblRepl'
--The Rest created automatically from previous variables
Set @ConstraintName = 'Identity_Range_' + @TableName + '_' + @SiteName
Create Table #MaxID(ID int)
Insert into #MaxID
EXECUTE ('Select Coalesce(max(' + @IDField + '),' + @IDRangeMin + ') From ' + @TableName + ' Where ' + @IDField + ' > ' + @IDRangeMin + ' AND ' + @IDField + ' <= ' + @IDRangeMax)
Set @CurrentMaxID = (Select coalesce(ID,0) From #MaxID) + 1
Drop Table #MaxID
if Exists(Select * from sysobjects where name = @ConstraintName and xtype = 'C')
begin
Execute ('Alter Table ' + @TableName + ' Drop Constraint ' + @ConstraintName)
end
EXECUTE ('ALTER TABLE [dbo].[' + @TableName + '] WITH NOCHECK ADD CONSTRAINT [' + @ConstraintName + ']
CHECK NOT FOR REPLICATION (' + @IDField + ' > ' + @IDRangeMin + ' AND ' + @IDField + ' <= ' + @IDRangeMax + ')')
EXECUTE ('ALTER TABLE [dbo].[' + @TableName + '] CHECK CONSTRAINT [' + @ConstraintName + ']')
dbcc checkident(@TableName, 'noreseed')
dbcc checkident(@TableName, 'reseed', @CurrentMaxID)
dbcc checkident(@TableName, 'noreseed')
--###############################################################################################
cheers,
Michael
January 15, 2009 at 10:00 pm
alex (1/14/2009)
We've been struggling with the same issues we're an MCP so we raised a ticket as MS....Its been raised on this forum before
http://www.sqlservercentral.com/Forums/Topic339239-291-1.aspx#bm339851
Thank you for contacting Microsoft SQL Server Technical Support. Your case number is ####### and the following is a summary of the current issue. Please take a moment to review it and contact me with any comments you may have. My contact information and available hours are at the bottom of this message.
As we discussed, the issue you're facing below issue Once we provide the answer we will consider this incident completed and closed. We'll be working to resolve this specific issue through the course of the case. If I have misunderstood any aspect of the issue, please let me know.
[SCOPE]
ISSUE:
You have merge Replication with auto identity range enabled. The identity range is getting used up on the publisher when you insert data though below query with no of records greater than the Identity range of publisher.
INSERT INTO Table1(ContactID, [User])
SELECT TOP 995 '-1' AS ContactID, 'HELLO' AS [User]
FROM Table2
Error Message
==============================
Msg 548, Level 16, State 2, Line 2
The insert failed. It conflicted with an identity range check constraint in database , replicated table 'dbo.Calllog ', column ' ContactID '. If the identity column is automatically managed by replication, update the range as follows: for the Publisher, execute sp_adjustpublisheridentityrange; for the Subscriber, run the Distribution Agent or the Merge Agent.
The statement has been terminated.
Environment Information
==============================
Publisher\Distributor
SQL Server 2008 Standard Edition
Windows 2003Server
IMPACT:
Production server
CRITERIA OF RESOLUTION:
You want to identity the above error message is caused while inserting the data tough query with records which is greater than the Identity range of publisher.Is it by design or Bug.
STATUS: Pending Research
I have Reproduced this issue in SQL2008 and SQL 2005 at my end. We need to Check with Escalation team whether this is by design or a known issue.
ACTION PLAN:
NA
NEXT CONTACT:
13-01-2009
The upshot was that MS judged this was "by design" and they would not be persueing the issue but would come back to us with the reccommendec workaround.
In the meantime we are implementing two workarounds.
1.catch the error and run sp_adjustPublisherIdentityRange to reset the ranges
2.increase the identity ranges to a really massive value
Well that's discouraging. I'm somewhat astonished to find that it doesn't handle this. Every application we've created does bulk inserting so I assume it's the norm, and yet it hasn't been catered for.
re approach 1. I assume that you then rerun the sql. Are you catching this error in your app then?
re approach 2. I assume these are being set to those so massive as to never be used up between reinitialisations/rereplicating. Workable where the id's are never seen by the user.
An alternative solution is altering the MSMERGE_ins_GUID trigger so that I we the range threshold percentage to trigger the new range before it hits the end rather than the max range ID. Although this is sort of neat with it tying directly into the current structure, many triggers would have to be updated (lots of work), and if not done right potentially puts the whole of the subscription a dodgy state.
Yet another alternative is to run a job every night that checks each table against the threshold and run sp_adjustPublisherIdentityRange accordingly. This requires ranges to be set a little larger than they usually would be. The advantage here is that it is self contained and the worse that can happen is it fails and the existing behaviour re occurs. The threshold seems to be in sysmergearticles which can link through to MSmerge_identity_range. I'm leaning towards this pending further investigation/discussions.
January 19, 2009 at 8:35 am
According to http://msdn.microsoft.com/en-us/library/ms152543.aspx, the range threshold percentage is only used for subscribers running earlier versions of SQL Server, or SQL Server Compact 3.5SP1. So it's more of a backwards compatibility thing.
Further to my colleague Alex's reply, we are catching the errors in code, and running sp_adjustpublisheridentityrange, then attempt to re-run the same insert query once more only.
Also, in our case, those tables for which the user can see (and uses) the identity value have rows added one at a time, and not so often, so are not badly affected by this.
From what I have figured out, the constraint / trigger method for auto-managing identies is fundamentally flawed in that it is always possible to break it with multi-record inserts. It seems to be a bit of a nailed on botch job, build onto the server rather than integrated into it. In the ideal world, the "query engine" would be range aware, but it is not, and instead it blindly increments the identity column, and leaves it to a flawed post-insert trigger to check the last used identity value, and a constraint to lock you out when it fails.
The trigger is flawed in that it will only refresh the ranges if the last used identity happens to be the last value of the 2nd range, which I think is not the intended behaviour. This is fine for single record inserts, but for multi-record inserts it is more likely to fail than not.
I think it is possible to fix this by changing the line:
else if IDENT_CURRENT('[dbo].[Table1]') >= @next_range_end
to
else if IDENT_CURRENT('[dbo].[Table1]') >= @range_end
Then at least the ranges would be refreshed once the identity is in the 2nd range - much like the merge agent does when it runs.
However, if there is a gap between the 2 ranges - and I've seen this happen - then even this will fail, as the constraint will prevent the trigger running when values in the gap are inevitably used.
The old range threshold percentage method was better for this due to there being only one range, and therefore no no chance of gaps.
I too was astonished that Microsoft would allow such hack methods be used in a flagship product like SQL Server, through 2 versions even. I spent many days investigating the issue, believing that I must be doing some wrong and that there was something fundamental that I didn't understand.
But, like Alex mentioned, they have confirmed that this is a flaw in both 2005 and 2008, and even agree that it is a bit rubbish. But they can't justify fixing it because not enough people have been affected by it - nobody else apparently having reported it to them. And without irony, they added that any fix would require a lot of testing and they need to justify the amount of effort this would take.
Slightly bemused by this response, I confirmed that I would like them to provide a work-around - which I am still waiting for, if only out of curiosity.
Who knows, maybe the person who is tasked with the work-around, will realise how badly done identity range mangement is, and will escalate the issue further than it has been so far.
Ok, rant over.
February 9, 2009 at 4:38 am
I've had ongoing discussions with Microsoft support, and progress has been made re the identity range management flaw.
The workaround they got back to me with was "There isn't one."
Then they suggested "Make to range very large."
Initially, they gave the flaw a status of "Design change request" - meaning they would think about fixing it for the next version of SQL Server.
Finally, it's now been accepted as a bug in 2005 and 2008, which gives it a chance of a it being fixed for these versions. I'm not holding my breath though...
-----------------------------------------------------------------------------------
Hi Martin,
It was my pleasure to serve you during your Microsoft SQL Server issue. . Based on our last conversation, I will close this case. If this is pre-mature, please let me know as soon as possible.
Problem Description:
===========================================================================================================================
You have merge Replication with auto identity range enabled. The identity range is getting used up on the publisher when you insert data though below query with no of records greater than the Identity range of publisher.
INSERT INTO CallLog(ContactID, [User])
SELECT TOP 995 '-1' AS ContactID, 'HELLO' AS [User]
FROM Contacts
Error Message
==============================
Msg 548, Level 16, State 2, Line 2
The insert failed. It conflicted with an identity range check constraint in database , replicated table 'dbo.Calllog ', column ' ContactID '. If the identity column is automatically managed by replication, update the range as follows: for the Publisher, execute sp_adjustpublisheridentityrange; for the Subscriber, run the Distribution Agent or the Merge Agent.
The statement has been terminated.
Environment Information
==============================
Publisher\Distributor
SQL Server 2008 Standard Edition
Windows 2003Server
Resolution
1] We reproduce this issue in SQL server 2005 \2008
2] This issue is identified as a bug with SQL server 2005\2008
Inclusion
As this case in bug in SQL server I have marked this case for non decrement. Thanks for your cooperation
February 9, 2009 at 9:49 am
I just tripped on this forum while searching for the same issue and am very upset to find it has not been resolved with microsoft fully, especially as a flagship product.
We've been having this issue for over a year and I have never had a chance to get at it, as it happens about once a month and I just manually adjust the range.
For the people who have just created a global job to go around and check the tables how are you doing that?
Are you just selecing all the tables out of the master database and looping through them and running the command on them at say 3am? Or is there another table you're searching for all tables involved in a publication and looping those?
February 9, 2009 at 4:18 pm
It seems a reply I posted a couple of weeks ago didn't make it. I'll give an abridged version.
After thinking more on this I'm now thinking Noelds approch of avoiding identity range management entirely is the go. For the moment we'll just live with the issues (fortunately the only customer currently using 2005 and replication is related to us and it's light use).
When our major customer changes over shortly we're going to create ranges manually, seeding the identity range and creating constraints as a fail safe using the script I posted above (or similar) wrapped in a stored proc.
We'll create a identity range replicated table with site, table name, range start and range end in it. Having it replicated means only having to administer one table on the main publisher database. The ranges will be created large enough to really never need to be updated. We'll create another stored proc that cycles through the table for the given sites ranges and creates the range and seeds the identity based on the current maximum within that range.
I can't see any reason why that wouldn't work but welcome any that anyone can think of. I'm not keen on changing code to capture errors and rerunning inserts. Sounds like too much work, and an extra layer of complexity I'd rather leave out of the app.
cdooley,
I'm going to create my own table because I need to store more information. Otherwise to get the list of tables I'd loop through sysobjects (which is still available in 2005 though it's really called something else) and look for tables (xtype 'U' off the top of my head) that start with 'tbl' but then all our tables start with that - it's harder if you don't have a standard prefix. A little more specific, and actually better, would be to link from msmerge_identity_range to sysmergearticles on artID and to sysmergesubscriptions on subID (the last one is only really needed for the publisher as the subscribers will only have their entries) to give you your list.
February 9, 2009 at 5:08 pm
Thanks for the reply. If someone here has existing code to update all of the tables it would be greatly appreciated. My desk is so full of stuff I started taking a look at this today and it's just another project I don't have a lot of time to spend on at the moment.
Our system is highly dependent on this type of identity management since we have many subscribers, potentially ranging in the thousands in the near future. There are ~20 tables that depend on the identity for inserts occurring at the subscriber to propogate back to the distributor, with transactions happening several times per minute per subscriber. It isn't that large of a headache right now with errors happening about once a month, but soon I have a feeling it will be happening a lot more.
It would be nice if software that costs 10's of thousands of dollars worked properly in the first place.
February 10, 2009 at 5:31 am
http://technet.microsoft.com/en-us/library/ms181527.aspx:
EXEC sp_adjustpublisheridentityrange @publication=publication_name
The above will adjust all tables in the publication publication_name where the current identity is outside the first range.
I assume the error was occuring when running an insert query against the publisher rather than a subscriber.
February 10, 2009 at 7:32 am
It looks like I am getting errors on both sides. Thanks for the note on the adjustrange, I was un-aware I could do it for a whole publication.
Is there any way to trigger it on the subscribers to pull a new range? This seems to be where most of the issues are happening. On one of the databases I have the replication happening every 10 minutes, and it seems like if you run out of ID's before the 10 minutes (I think it's 10,000 ID blocks given out) then it will error.
So am I reading it right that the subscriber will only get a new range when it's filled up it's current range down to the last ID, and then on the next merge agent running it will download a new range?
What is the fix for something like this?
Can you elaborate on the two ranges you're talking about? I need to re-read back up on some of this stuff, but from what I remember I thought the subscriber got one range, and then when it reached your percentage it would request another one. But if it's waiting until the last range, when it requests another one, the first one would be gone anyway, so you wouldn't have a backup.
Or do I have it all wrong?
February 10, 2009 at 11:00 am
The way I understand it (and I may be wrong in places) is:
Subscribers running version 2005 and above no longer use a single range, and a percentage.
Instead they are allocated 2 ranges, which may or may not be consecutive.
I believe the theory is that when one range is used up, the 2nd range becomes the 1st range, and a new 2nd range is allocated.
This is not quite implemented using a constraint/trigger mechanism. When the constraint allows a batch of rows to be successfully inserted, a post insert trigger runs, and amongst other things checks the following:
1) If the last used identity is exactly on the last value in the 1st range, the identity is reset to the 1st value in the 2nd range. This is to account for a possible gap in between the ranges.
2) If the last used identity is on or beyond the last value in the 2nd range, then the ranges are refreshed.
This all works fine when one row at a time is inserted, as each new identity will get checked in the trigger.
When more than one row at a time is inserted, there is a good chance that some of the identities used will fall outside of the ranges. This is because the pre-insert identities are generated without regard to the ranges - simply each row to be inserted has an identity +1 the previous row in the batch. This is the critical design flaw here.
When any identity used is outside both ranges, the constraint will prevent the entire batch of rows from being inserted, and the insert trigger will not execute. No more inserts are possible on this table until its ranges are refreshed.
So, on a subscriber, if there is a gap between the two ranges, and you are inserting more than one record at a time, it is highly likely that eventually identities falling between the two ranges will be used resulting in an error, a failed insert, and a table which is locked for further inserts.
If this is what is happening for your subscribers, then I don't know of an easy solution which will prevent this happening, and the only thing you can do post-error is to run the merge agent to refresh the ranges.
If the identity makes it safely into the 2nd range (more likely if there is no gap), then the next time the merge agents runs, the ranges should get refreshed.
If the merge agent doesn't run in time, then you are relying on the insert trigger to do this, which will only happen if you are inserting one row at a time, or if a multi-row insert happens to luckily use the last identity on for its last row.
We were having trouble running insert queries at the publisher, and The way we have dealt with it is to catch the errors in program code wherever a multi-row insert occurs, check for the particular error code, run sp_adjustpublisheridentityrange, and re-run the insert query. It's not ideal, but it seems to work.
The alternative for you is to increase the frequency of the merge agent running, and/or increase the range for the tables with the problem. Even then, I think you'll still run into problems for multi-row inserts where there is a gap between the ranges, as here the identity is less likely to make it safely into the 2nd range.
February 10, 2009 at 3:59 pm
I think we're fine on the multiple inserts.
Usually we have a batch of inserts, but they are not in the same statement (one stored procedure with one insert ran 10x) for example when a options screen is saved. It saves option1, then option2, then option 3 etc but one row at a time.
It seems it is erroring out then probably at the end of the second identity range.
I can definitely see both of the ranges assigned to the subscriber.
Are you saying that the ranges are not refreshed until the second range is fully exhausted? I would imagine the best case scenario would be is when range 1 is exhausted and the second range takes over, the next replication agent would trigger the pull down of only 1 range. Then you always have a full range to spare.
If this worked properly the only issue I could see happening is if you inserted 2,001 records without the merge agent firing before the last record is inserted. And in that case, simply enlarging the assigned ranges would help.
But if the thing is that BOTH ranges are not refreshed until the 2nd is exhausted, then you would have a lot of issues. I wouldn't think this is the case though because how would you obtain two separate ranges then? If you're always requesting 2 at a time (when your 2 used ones were filled up) then it would give you ranges (example: 5000-6000 and 6000-7000)...but my ranges are more around 5500-6500 and then 15500-16500 (my distributor gets 10,000).
February 10, 2009 at 4:53 pm
Martin,
I was going to disagree on the statement 'If the last used identity is exactly on the last value in the 1st range' as I thought it did a '>=' check. But after rechecking the stored proc I am once more astonished to see you're correct. This is unforgivable. I understand the complexity here is that the identity may have moved on past the start of the second range but it isn't hard to check the max identity within the second range and set the current identity to that. A slightly more complex set of 'if' statements including the range threshold would of course be even better and solve all these issues. When I'd initially heard that 2005 were going to use two ranges I thought, great, it'll solve the 2000 identity range issues where the connection is down for a short time. But it seems not a lot of thought went in to the logic.
I had another thought. I earlier said it'd be too hard to change all the triggers. However it is possible to change the script from which all the triggers are created. The script is found here: 'sp_MSaddmergetriggers_internal' in the model database. Unfortunately this means needing to manage this stored proc, and modify it, perhaps differently if it has been changed, every time a new SP is rolled out. For myself I'm not sure it's worth it but could be with a more detailed structure/greater number of subscribers.
cdooley,
If you are entering one record at a time a new range should be created next time the merge agent runs. That is, the second range becomes the first and a new second range is created. This is from testing. I haven't found where this is happening (and haven't looked very closely) but, confusingly, the merge insert trigger on the table doesn't seem to umm trigger this. Perhaps i'm reading it wrong.
Also, if anyone doesn't know, I've just modified an articles range in the publication properties and it is used on the next range update. It doesn't wait for a reinitialisation or new subscription to use it. This is handy as a temporary fix to reduce the number of errors.
Viewing 15 posts - 1 through 15 (of 25 total)
You must be logged in to reply to this topic. Login to reply