File import to table in Unix

  • I am struck in one point as we gets files in unix box and we don't have any windows server .

    Issue - We get pipe'|' separated text file, usually this can be bcp in to table if the structure exactly matching with table .

    Here structure is same but some of the rows having issue with row delimiters which cause overlapping to next line.

    This file opens in excel perfectly no issue.

    I need to find a solution how to fix this. This we cannot say forrmat issue as this issue with only on unix env. Is there any way we can use ssis ? please advice if any other solution.

  • Is there any way we can use ssis ?

    To do what, precisely?

    As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin (1/12/2014)


    Is there any way we can use ssis ?

    To do what, precisely?

    As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?

    Sample data looks like below, for 3rd row rowdelimiter is not correct.

    |USD|1542.52|14536|xxxx

    ||1542.52|14536|

    EUR|1542.52|14536|xxxx

    I m not sure how does SSIS Fit into Unix, please advice me if SSIS fits into unix.

    I can create an ksh scripts ,is ssis can be called from ksh ?

  • sql crazy kid (1/12/2014)


    Phil Parkin (1/12/2014)


    Is there any way we can use ssis ?

    To do what, precisely?

    As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?

    Sample data looks like below, for 3rd row rowdelimiter is not correct.

    |USD|1542.52|14536|xxxx

    ||1542.52|14536|

    EUR|1542.52|14536|xxxx

    I m not sure how does SSIS Fit into Unix, please advice me if SSIS fits into unix.

    I can create an ksh scripts ,is ssis can be called from ksh ?

    Sorry but I know very little about Unix. But I'm pretty sure that SSIS can be used to process and push data into a Unix database, assuming the connectivity can be resolved.

    But SSIS needs to be installed on a Windows server - is that a show-stopper for you? It's also quite expensive to justify if you are not already using SQL Server.

    Does your sample data represent two or three rows of actual data?

    If three rows, it looks like the column delimiters (not row delimiters) are incorrectly placed. That's just bad data and it should be fixed at source, not by you - in my opinion.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin (1/12/2014)


    sql crazy kid (1/12/2014)


    Phil Parkin (1/12/2014)


    Is there any way we can use ssis ?

    To do what, precisely?

    As you have given no examples of the data which is giving you problems, providing any sort of useful answer is close to impossible. How does SSIS fit into the architecture of your Unix environment?

    Sample data looks like below, for 3rd row rowdelimiter is not correct.

    |USD|1542.52|14536|xxxx

    ||1542.52|14536|

    EUR|1542.52|14536|xxxx

    I m not sure how does SSIS Fit into Unix, please advice me if SSIS fits into unix.

    I can create an ksh scripts ,is ssis can be called from ksh ?

    Sorry but I know very little about Unix. But I'm pretty sure that SSIS can be used to process and push data into a Unix database, assuming the connectivity can be resolved.

    But SSIS needs to be installed on a Windows server - is that a show-stopper for you? It's also quite expensive to justify if you are not already using SQL Server.

    Does your sample data represent two or three rows of actual data?

    If three rows, it looks like the column delimiters (not row delimiters) are incorrectly placed. That's just bad data and it should be fixed at source, not by you - in my opinion.

    Here is few more details

    Backend - SQL server 2008 R2,SSIS installed

    OS - Unix Env

    We have many process running on unix ,connecting to sql server , BCP into sql tables , all are working fine .

    Above sample data is a representation of actual data . But really there is a correction required on data file , but in case we are using this in windows env this not an issue . Either with SSIS or Informatica this works perfectly with no issue as we have column to column mapping . Since that we cannot ask to fix that issue.

  • Here is few more details

    Backend - SQL server 2008 R2,SSIS installed

    OS - Unix Env

    We have many process running on unix ,connecting to sql server , BCP into sql tables , all are working fine .

    Above sample data is a representation of actual data . But really there is a correction required on data file , but in case we are using this in windows env this not an issue . Either with SSIS or Informatica this works perfectly with no issue as we have column to column mapping . Since that we cannot ask to fix that issue.

    OK. Then please answer my questions about your sample data. At the moment, it looks like:

    c1 (blank),c2,c3,c4,c5

    c1 (blank),c2 (blank),c3,c4,c5 (blank)

    c1 (missing),c2,c3,c4,c5

    My point? If columns can be missed out randomly from your source data, there is NO WAY ON EARTH to get it fixed by SSIS, which is lacking the necessary Paranormal Transformation.

    If you are saying that only certain columns are missing, under certain circumstances, you need to explain the logic of what needs to be done please.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin (1/12/2014)


    Here is few more details

    Backend - SQL server 2008 R2,SSIS installed

    OS - Unix Env

    We have many process running on unix ,connecting to sql server , BCP into sql tables , all are working fine .

    Above sample data is a representation of actual data . But really there is a correction required on data file , but in case we are using this in windows env this not an issue . Either with SSIS or Informatica this works perfectly with no issue as we have column to column mapping . Since that we cannot ask to fix that issue.

    OK. Then please answer my questions about your sample data. At the moment, it looks like:

    c1 (blank),c2,c3,c4,c5

    c1 (blank),c2 (blank),c3,c4,c5 (blank)

    c1 (missing),c2,c3,c4,c5

    My point? If columns can be missed out randomly from your source data, there is NO WAY ON EARTH to get it fixed by SSIS, which is lacking the necessary Paranormal Transformation.

    If you are saying that only certain columns are missing, under certain circumstances, you need to explain the logic of what needs to be done please.

    Columns are not missing here . I have 5 column here ,looks like this.

    [Just a sample data ,but this is the format we load]

    col1 col2 col3 col4 col5

    USD1542.5214536xxxx

    1542.5214536

    EUR 1542.5214536xxxx

  • Columns are not missing here . I have 5 column here ,looks like this.

    [Just a sample data ,but this is the format we load]

    col1 col2 col3 col4 col5

    USD1542.5214536xxxx

    1542.5214536

    EUR 1542.5214536xxxx

    What would this sample data look like if there were no problems with delimiters?

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin (1/12/2014)


    Columns are not missing here . I have 5 column here ,looks like this.

    [Just a sample data ,but this is the format we load]

    col1 col2 col3 col4 col5

    USD1542.5214536xxxx

    1542.5214536

    EUR 1542.5214536xxxx

    What would this sample data look like if there were no problems with delimiters?

    I am getting output as like below when i do bcp command on unix.

    C1 C2 C3 C4 C5

    <NULL>USD 1542.5214536xxxx

    <NULL> <NULL> 1542.5214536EUR|1542.52|14536|xxxx

  • sql crazy kid (1/12/2014)


    Phil Parkin (1/12/2014)


    Columns are not missing here . I have 5 column here ,looks like this.

    [Just a sample data ,but this is the format we load]

    col1 col2 col3 col4 col5

    USD1542.5214536xxxx

    1542.5214536

    EUR 1542.5214536xxxx

    What would this sample data look like if there were no problems with delimiters?

    I am getting output as like below when i do bcp command on unix.

    C1 C2 C3 C4 C5

    USD1542.5214536xxxx

    1542.5214536EUR|1542.52|14536|xxxx

    I understand, but that is not the question I am asking.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin (1/12/2014)


    sql crazy kid (1/12/2014)


    Phil Parkin (1/12/2014)


    Columns are not missing here . I have 5 column here ,looks like this.

    [Just a sample data ,but this is the format we load]

    col1 col2 col3 col4 col5

    USD1542.5214536xxxx

    1542.5214536

    EUR 1542.5214536xxxx

    What would this sample data look like if there were no problems with delimiters?

    I am getting output as like below when i do bcp command on unix.

    C1 C2 C3 C4 C5

    USD1542.5214536xxxx

    1542.5214536EUR|1542.52|14536|xxxx

    I understand, but that is not the question I am asking.

    If there is no problem,output would be

    C1 C2 C3 C4 C5

    NULLUSD1542.5214536xxxx

    NULL NULL1542.5214536NULL

    EUR NULL 1542.52 14536 xxxx

  • If there is no problem,output would be

    C1 C2 C3 C4 C5

    NULLUSD1542.5214536xxxx

    NULL NULL1542.5214536NULL

    EUR NULL 1542.52 14536 xxxx

    Great, that helps.

    Now my question is: why in the above is there a NULL after EUR in row 3?

    That, surely, is a missing column, if the source is

    EUR|1542.52|14536|xxxx

    ?

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin (1/12/2014)


    If there is no problem,output would be

    C1 C2 C3 C4 C5

    NULLUSD1542.5214536xxxx

    NULL NULL1542.5214536NULL

    EUR NULL 1542.52 14536 xxxx

    Great, that helps.

    Now my question is: why in the above is there a NULL after EUR in row 3?

    That, surely, is a missing column, if the source is

    EUR|1542.52|14536|xxxx

    ?

    Thanks for getting into deeper

    Few of the rows , first column coming as empty . But do you think in SSIS or Excel this will be a issue as there is a column delimiter to treat the 1st column as empty.

  • Thanks for getting into deeper

    Few of the rows , first column coming as empty . But do you think in SSIS or Excel this will be a issue as there is a column delimiter to treat the 1st column as empty.

    You've avoided my question again!

    If your question can be restated as:

    Sometimes there will be four columns in a row, sometimes five columns.

    Will SSIS know which of the columns is missing?

    The answer is no.

    But if you know that there is a rule such as: if there are four columns in a row, add a fifth as Col1 and set its value to NULL, that can be done.

    I'm going to sign off now, by saying that if you can come up with the necessary logical steps to process your file (and I don't think that you're there yet), it will be possible to implement those steps in SSIS. It might require some scripting.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin (1/12/2014)


    Thanks for getting into deeper

    Few of the rows , first column coming as empty . But do you think in SSIS or Excel this will be a issue as there is a column delimiter to treat the 1st column as empty.

    You've avoided my question again!

    If your question can be restated as:

    Sometimes there will be four columns in a row, sometimes five columns.

    Will SSIS know which of the columns is missing?

    The answer is no.

    But if you know that there is a rule such as: if there are four columns in a row, add a fifth as Col1 and set its value to NULL, that can be done.

    I'm going to sign off now, by saying that if you can come up with the necessary logical steps to process your file (and I don't think that you're there yet), it will be possible to implement those steps in SSIS. It might require some scripting.

    Thank you for your time and effort put in. As you said this is a data issue ,we are done is our end corrected file and bcp into the table. Another think this is hard to connect SSIS from Unix .

Viewing 15 posts - 1 through 14 (of 14 total)

You must be logged in to reply to this topic. Login to reply