Random cube failures / not logged in AS processing log
-
son of a b-tch! my last thread post attempt didn't go thru - now to rewrite my post again......sh-t!
We have been experiencing seemlingly random cube processing failure that always rerun successfully the next day. Nothing appears in the AS processing log for these failures. The next day, I am able to successfuly run the failed step by either a) running it manually in Analysis Manager b) re-running the SQL job, or c) running the specific DTS package or task step. They always work fine the next day. Also, we are running parallel prod/test systems and the failure have never been on both simultaneously. It's always one or the other. And the failure rarely repeats the following night.
The failed processing tasks have been (so far) 1) failed virtual cube processing 2) failed cube partition processing 3) failed time dimension (shared) processing. Each is a full process (no incrementals).
We have 2 sets of prod/test parallel server setups (so 6 servers total):
Prod - 1 dedicated SQL Server with 4GB RAM, 1 dedicated Analysis Server with 2GB RAM
Test - 1 shared SQL / AS server with 4GB RAM
We are running the /3GB switch on the 4GB servers and have AWE enabled for SQL. There is plently of free memory on all servers (>1GB). We have SP3a installed on all servers except for one test server which I applied SP4 yesterday.
any clues on what may be causing the failures?
any clues on why they are not logged in the processing log?
thanks in advance!
-
^bump^
-
^bumped^
Hey Jay,
What does the processing log tell you (ie if there isn't an error raised, does it actually indicate that it ran through to completion? maybe no last entry lines? ??)
Are you able to tell roughly what time the errors/issues are occuring? And if so, have you checked the system and apps logs (even security log) on the offending server(s) for around that time?
Cheers,
Steve.
-
hi steve,
the processing log is missing info, which is part of the problem, since i have no clues on the actual cause of the failures. in the log, it will show the dimension processing steps (which always succeed) then the section where i would expect to see the cube processing info is missing. it seems that the cubes that are partitioned are always missing log info (whether successful or not). virtual cubes are missing failure log messages. non-partitioned non-virtual cubes have all their info available in the logs. (for all cases, this is when processing via a DTS package using an OLAP task - processing via Analysis Manager produces log messages). i'm almost willing to bet that there is a bug in AS or DTS that causes this missing log behaviour. there's nothing special in the servers' event viewer logs either.
anyhoo - the cubes and date/time of occurrance is random. it happens only occasionally, but in june it got bad. since my original message, i've gone through the DTS packages that process these particular cubes and streamlined them a bit (these were written by various consultants and the logic/technique used varied from cube to cube). i actually haven't had a failure since then so perhaps i sorted something out. or maybe it's just waiting awhile again for the next error.
here is my log of the cube failures, as you can see there does not appear to be a pattern, different servers, different cubes, on different dates. we only had one instance where 2 things failed on the same day (14 June) and that was on two different servers.
(sorry bout the format...)Server Date Cube NotesTest SWDE01SRV014 3-Apr Total inventory cube virtual cube
Prod CH1SRV24 22-May Total inventory cube virtual cube
Test CH1SRV36 25-May Total inventory cube virtual cube
Prod SWDE01SRV013 7-Jun Commercial cube partitioned by year
Prod SWDE01SRV013 8-Jun Commercial cube partitioned by year
Prod CH1SRV24 12-Jun Security cube partitioned by year
Test CH1SRV36 13-Jun Total inventory cube virtual cube
Prod CH1SRV24 14-Jun Time dimension (security)
Test CH1SRV36 14-Jun Total inventory cube virtual cube
Prod SWDE01SRV013 15-Jun Commercial cube partitioned by year
i'll post again if i have more failures or more information about the problem.
thanks!
jay
-
Hey Jay,
Haven't fully digested what you've written yet but it did remind me of something I read once where I think the AS processsing log goes into the DTS log when called from DTS (i think this was a Dave Wickert thing, so possibly the Ops guide?) Could be worth a look in to your DTS package logs (they are logging yeah?!?) to see if there's any clues in there.
Cheers,
Steve.
Viewing 5 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply