SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Fixed Length Flat File


Fixed Length Flat File

Author
Message
josetur12
josetur12
SSC Veteran
SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)

Group: General Forum Members
Points: 250 Visits: 392
Hi, I have a flat file with over 200 fixed length columns. I want to use SSIS to load the data into SQL Server.
Is there a way to easily load them into SQL Server without parsing each column?
I will appreciate any help. Thanks,
Revenant
Revenant
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: General Forum Members
Points: 12691 Visits: 5010
I am afraid there is no "silver bullet" solution.

You can use a Data Flow task, with several flows if data goes into several tables, or a Script Task. In a Script Task you could define a dictionary of offsets and lengths by table and column name and do the parsing in one simple loop, which IMO would be easier to maintain than 200+ individual substrings.

Sorry if my answer is too general. I could perhaps give a more specific opinion if I had more detail.
josetur12
josetur12
SSC Veteran
SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)SSC Veteran (250 reputation)

Group: General Forum Members
Points: 250 Visits: 392
Than you for the suggestion.
stevefromOZ
stevefromOZ
SSCrazy Eights
SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)

Group: Moderators
Points: 9856 Visits: 3757
Not sure I understand the problem?? Why not use the flat file connection, set it to fixed length for the format and load up the data?

Steve.
Revenant
Revenant
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: General Forum Members
Points: 12691 Visits: 5010
Yes, that would be a data flow. However, you need one dataflow per target table, and for 200+ that would be difficult (although not impossible) to maintain.
stevefromOZ
stevefromOZ
SSCrazy Eights
SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)SSCrazy Eights (9.9K reputation)

Group: Moderators
Points: 9856 Visits: 3757
However, you need one dataflow per target table, and for 200+ that would be difficult (although not impossible) to maintain.


Maybe I missed it, but I didn't see the requirement to load each column to an individual table. If you have (or want to create) a target table with the ~ 200 columns, that's easy enough to do. You can then set up the flat file connection as fixed width, and work with SSIS on it's guesses at the widths (ie you'll end up having to make changes where the guesses are wrong). Once you have this, you have a single data flow that will load all required fields into the target table.

Or, with more detail, posted by 'wcm' on 12-13-2010

1. Add a Data Flow Task to the Control Flow.
a)dbl-Click to enter the Data Flow Tab
2. Add a Flat File Source to the Data Flow Tab
a)Identify the Flat File Source in the Flat File Source Editor Dialog
3. Now That you have the Connection manager for the for the flat file created:
a)edit the Flat File Conn Manager
b)View the Advance Properties and you will see "Data Type" property and it's default is
String [DT_STR] and "OutputColumnWidth" is set to 50.
c) To change one column you can highlight the column in the middle list box, and make the change
d) you can also do multiplt selects to do several columns at once.


I'm still not seeing the issue?? Or the concern about 'parsing every field'.

Steve.
Revenant
Revenant
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: General Forum Members
Points: 12691 Visits: 5010
You do nolt have to load each column into a different table; however, if you are loading columns into more than one table (which is likely the case), you would create an SSIS Data Flow task that would be, in my experience, difficult to maintain.

Been there, done that. Thanks, I do not need another t-shirt.
WayneS
WayneS
SSC-Insane
SSC-Insane (22K reputation)SSC-Insane (22K reputation)SSC-Insane (22K reputation)SSC-Insane (22K reputation)SSC-Insane (22K reputation)SSC-Insane (22K reputation)SSC-Insane (22K reputation)SSC-Insane (22K reputation)

Group: General Forum Members
Points: 22216 Visits: 10657
josetur12 (3/11/2011)
Hi, I have a flat file with over 200 fixed length columns. I want to use SSIS to load the data into SQL Server.
Is there a way to easily load them into SQL Server without parsing each column?
I will appreciate any help. Thanks,


You could use T-SQL. Start with a table that contains each column (order, name, starting position, size). Then, utilizing BULK INSERT, load the data into a #temp table. Run a select to get the data.

This is a start... just expand upon it.
DECLARE @Columns TABLE (ColumnOrder INT, ColumnName varchar(20), StartingPosition smallint, ColumnSize tinyint);
INSERT INTO @Columns
SELECT 1, 'Column1', 1, 10 UNION ALL
SELECT 2, 'Column2', 11, 5 UNION ALL
SELECT 3, 'Column3', 16, 12;

DECLARE @TestData TABLE (RowID INT IDENTITY, RowData varchar(100));
INSERT INTO @TestData
SELECT 'y4gfbnenbeofnoihfgjq3fgwqgnq3ovqvqno' UNION ALL
SELECT 'qgbughqofniufgonqgoinvoqfqnbqnqn3qfo';

WITH cte AS
(
SELECT t.RowID, dt.*
FROM @TestData t
CROSS APPLY (SELECT ColumnOrder,
ColumnName,
ColumnValue = SubString(t.RowData, StartingPosition, ColumnSize)
FROM @Columns) dt
)
SELECT RowID,
Column1 = MAX(CASE WHEN ColumnName = 'Column1' THEN ColumnValue ELSE NULL END),
Column2 = MAX(CASE WHEN ColumnName = 'Column2' THEN ColumnValue ELSE NULL END),
Column3 = MAX(CASE WHEN ColumnName = 'Column3' THEN ColumnValue ELSE NULL END)
FROM CTE
GROUP BY RowID
ORDER BY RowID;



Wayne
Microsoft Certified Master: SQL Server 2008
Author - SQL Server T-SQL Recipes
If you can't explain to another person how the code that you're copying from the internet works, then DON'T USE IT on a production system! After all, you will be the one supporting it!
Links: For better assistance in answering your questions, How to ask a question, Performance Problems, Common date/time routines,
CROSS-TABS and PIVOT tables Part 1 & Part 2, Using APPLY Part 1 & Part 2, Splitting Delimited Strings

Revenant
Revenant
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: General Forum Members
Points: 12691 Visits: 5010
#Columns is what I meant by "dictionary of columns". I csan only guess but I think that there will be some parsing involved.
Jeff Moden
Jeff Moden
SSC Guru
SSC Guru (220K reputation)SSC Guru (220K reputation)SSC Guru (220K reputation)SSC Guru (220K reputation)SSC Guru (220K reputation)SSC Guru (220K reputation)SSC Guru (220K reputation)SSC Guru (220K reputation)

Group: General Forum Members
Points: 220980 Visits: 42002
josetur12 (3/11/2011)
I have a flat file with over 200 fixed length columns


We're just guess here because we can't see your file.

Do you have a record layout for this file and can you attach a text file with the first 10 rows? Of course, if you do attach a file, make sure it's not violating any privacy or company private concerns.

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
If you think its expensive to hire a professional to do the job, wait until you hire an amateur. -- Red Adair

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search