Multi-file U-SQL query doesn't return correct results

  • Michelle-138172

    Hall of Fame

    Points: 3232

    Looks pretty quiet here, but still hope to get some answers.

    I'm a new user to U-SQL. I have a query that queries files in 2 folders for aggregation, but the result comes back with only partial counts for the 2nd folder. If I filter the result with only the 2nd folder, the result looks fine. Wondering why this is happening. How can I get the full result back for the multi-files?

  • Lynn Pettis

    SSC Guru

    Points: 442116

    Michelle-138172 - Monday, March 19, 2018 9:56 AM

    Looks pretty quiet here, but still hope to get some answers.

    I'm a new user to U-SQL. I have a query that queries 2 files for aggregation, but the result comes back with only partial counts for the 2nd file. If I filter the result with only the 2nd file, the result looks fine. Wondering why this is happening. How can I get the full result back for the multi-files?

    Unfortunately, no one can see what you see.  Without more details anything we say will be just wild shots in the dark.

  • Michelle-138172

    Hall of Fame

    Points: 3232

    Thanks for the reply, Lynn. I actually meant 2 folders, not 2 files. It's almost like it's not reading all the files in the 2nd folder. 

    It's a simple count aggregation by date and another column. It works fine if I do a single date, but once I comment out the date filter, the result is skewed. Very puzzling.

  • Lynn Pettis

    SSC Guru

    Points: 442116

    Michelle-138172 - Monday, March 19, 2018 10:35 AM

    Thanks for the reply, Lynn. I actually meant 2 folders, not 2 files. It's almost like it's not reading all the files in the 2nd folder. 

    It's a simple count aggregation by date and another column. It works fine if I do a single date, but once I comment out the date filter, the result is skewed. Very puzzling.

    Hopefully someone else will be able to help, but with nothing to work with here I wouldn't even know where to start.

  • Michelle-138172

    Hall of Fame

    Points: 3232

    I added one more day's worth of data for testing. It looks like the output file is really just cutting off at a certain row number. In this case, it always ends at row 102. Is there any default row limit setting?

  • Lynn Pettis

    SSC Guru

    Points: 442116

    If you really want help you will need to post details about what you are trying to accomplish.  We aren't mind readers and we can't see what you are doing unless you show us.

  • Michelle-138172

    Hall of Fame

    Points: 3232

    OK, I figured this out. It was a rookie mistake. I was in the preview mode. When I fully downloaded the file, I got everything.

  • Lynn Pettis

    SSC Guru

    Points: 442116

    Glad you figured it out.  Not something I would have even thought to ask about.

  • Sue_H

    SSC Guru

    Points: 89891

    Michelle-138172 - Monday, March 19, 2018 1:50 PM

    I added one more day's worth of data for testing. It looks like the output file is really just cutting off at a certain row number. In this case, it always ends at row 102. Is there any default row limit setting?

    But no one can see your query so all anyone can do is guess. You'll get one post "are you using sample", another one "are you using offset", another one "are you using fetch". And it goes on and on like that and eventually people stop trying to guess. The more information people provide when posting, the more involved others others seem to be. It's the best way to get help when posting.

    Sue

Viewing 9 posts - 1 through 9 (of 9 total)

You must be logged in to reply to this topic. Login to reply