Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 12»»

SED and the Big Bad UNIX File Expand / Collapse
Author
Message
Posted Monday, December 19, 2005 5:06 PM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Sunday, March 2, 2014 9:45 PM
Points: 1, Visits: 46
Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/pward/sedandthebigbadunixfile.asp
Post #245399
Posted Tuesday, January 10, 2006 1:12 AM
SSC Journeyman

SSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC Journeyman

Group: General Forum Members
Last Login: Monday, September 29, 2008 5:10 AM
Points: 78, Visits: 43

Good stuff about dealing with big files, but for the delimiter it's easy enough to set LF rather than CRLF in the file's connection object. I do this for data from a well-known provider of financial information which seems unable to standardize on CRLF or LF for its various files.

Bill.

Post #249510
Posted Tuesday, January 10, 2006 2:20 AM
SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Monday, August 5, 2013 1:19 AM
Points: 245, Visits: 13
You could also achieve this result by a simple Shell script using RegEx-Replace and an ActiveX task before the actual transformation (dealt successfully with such things before)


Post #249522
Posted Tuesday, January 10, 2006 2:45 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, September 7, 2006 7:45 AM
Points: 7, Visits: 1
There are also two *NIX and DOS utilities to convert between the *NIX and DOS LF/CRLF formats. They are:

UNIX2DOS and DOS2UNIX

Simple, open source and widely available.
Post #249525
Posted Tuesday, January 10, 2006 5:41 AM


Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Tuesday, July 15, 2014 3:22 PM
Points: 352, Visits: 173
Yeah, I use UNIX2DOS & DOS2UNIX, very handy little utilities.


Post #249549
Posted Tuesday, January 10, 2006 7:02 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Tuesday, January 10, 2006 6:58 AM
Points: 1, Visits: 1
or open the file in an editor like Testpad and save it in Windows format.
Post #249581
Posted Tuesday, January 10, 2006 7:25 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Wednesday, August 21, 2013 12:35 PM
Points: 9, Visits: 69

Nice article. I would have never thought to use SED on Windows for that purpose.

I'll keep it in my favorites for just in case situations.




Thanks,

Edmond Shamon Larson

Post #249591
Posted Tuesday, January 10, 2006 7:43 AM
SSCrazy Eights

SSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy EightsSSCrazy Eights

Group: Moderators
Last Login: Friday, September 26, 2014 11:48 AM
Points: 8,370, Visits: 742

Great article. Fortunately I have not come across this yet but always keep this type of info around just in case.

For me the biggest issue has been mainframe files where the columns are fixed width but the file truncates to the line when a particular coulmn is not there. For me it is simply to import the whole thing into a single char column then export back to a new file and run thru the import as originally design. Would be nice to have a tool to auto check this and fix the file or data (source code would be great so I could just make a DTS object) while importing. Anyone got such an animal per chance?




Post #249595
Posted Tuesday, January 10, 2006 8:06 AM


SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Wednesday, February 19, 2014 2:01 PM
Points: 211, Visits: 556
Good article. I'll have to check out SED.

Regarding the bad record: if the record was embarassed, you could call it a rouge record. But I think you meant rogue.
Post #249611
Posted Tuesday, January 10, 2006 9:44 AM
Hall of Fame

Hall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of FameHall of Fame

Group: General Forum Members
Last Login: Yesterday @ 2:44 PM
Points: 3,475, Visits: 583

Hi,

Good article.

The problem is even more general then just importing from UNIX-generated files. I had recently to show one of the business ladies why the data import jobs for her application done by the third-party SW fails based on what files another company sends her. I created 5 files generated by DTS, VBScript and export from Excel using different row and field delimeters to show her the file will contan delimeters that a particular programmer specifies. Also, I showed to her that if the Comments field contains one of her row or field delimier characters her data import job will fail too with the message about incorrect number of fields.

We all have to determine what are row and field delimeters before setting up any file processing and after that to work with the file supplier to make sure they don't change their processes and technologies to produce the files.  Here is a part of the script that helps me to see what characters are used. It posts a message for each character (for the demo), re-write it to output into the file.

Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.OpenTextFile("C:\Temp\Files\MyTextFile.txt", 1)
Do Until objFile.AtEndOfStream
    strCharacters = objFile.Read(1)
    Wscript.Echo strCharacters & " " & Cstr(asc(strCharacters))
Loop

 




Regards,
Yelena Varshal

Post #249658
« Prev Topic | Next Topic »

Add to briefcase 12»»

Permissions Expand / Collapse