thxs Lowell the only concern I have in my files it may or may not have headers as well some file has Row Delimiter "CR LF"" and some has "LF"....

Jeff I have considered excel as simple table format columns and rows...no formatting is consider in my scri[t. probably after some tweaking it may work:-)

Describe what you mean by 'flat structure'.

Flat Table with all headers from files, if no header it should come with Column0, Column1, Column2...& so on.

Are the files all in different...

you are correct! I have periodic data modification which leads us for full "HUGE" refresh or full fact load. We have certain scenarios where we apply incremental load (when data...

Eirikur Eiriksson (4/1/2015)

---

Quick questions, what is the max memory settings for the sql server, what other services are running on the server (turn off anything not needed!)?, are you using...

I won't be able to publish all details...but hope this helps

I ran the performance counter and checked the reports and it shows "the system is experiencing excessive memory pressure". I am not sure which other counter should I look into:doze:

Using lookup transformation with FULL cache will add an overhead on memory whereas we have other application accessing the same server. another concern using lookup will be using 26 lookup...

we do not have flat files as I mention source and target both are SQL

Thanks Eirikur,

1) Data sizes? - 30 GB

2) Available high performance storage space? 1TB disk, 97 GB sql memory, 16 processors

3) Type of storage available? - RAID 1

> Already using staging...

you solution worked Hansi perfectly, thank you for quick response! and thanks everyone for sharing your thoughts

The only point I was concerned about the behavior of SQL Query. I was...

As per my understanding SQL should compare the value one by one if that it the case it should have consider the matching value. Can you please explain the behaviour

thanks guys you saved me!

addition to this I am also searching how many times the word has been repeated within the same column

below is the script

;WITH

CTE

AS(SELECTdistinct pattern

FROM @phrase w

CROSS...

We tried both SSIS and T-SQL, both options are taking time. With T-SQL yes dropping index before load and creating after load.

As you correctly mentioned the time take is while...

Yes

I know the best practices say about delta load but we have business requirement to load each time we receive bulk data.