Using ADF Data Flow Derived Column transform against nested Delta structures

  • I'm trying (without success) to use a derived column transform within an ADF Data Flow where I've ingested a Delta table with nested structures. I'm struggling with the syntax needed to flatten out these structures and no column info is displayed despite me being able to preview the data.

    I thought I'd be able to use something like Name.FirstName to reference the structured column but it's giving me an error "Expression type could not be evaluated, correct the expression" in the expression editor

    Such a structure would be:

    {
    "ContactId":"1002657",
      "Name":{
    "FirstName":"Donna",
    "FullName":"Donna Brittain",
    "LastName":"Brittain"
    }
    }

    Data Preview working OK:

    ADF_Preview

    The structure of my Delta table:

    DeltaTableFormat

    The error I'm getting trying to reference a nested column:

    ADF_DerivedCol

    How can I reference a nested column such as Name.FirstName to flatten it out to FirstName and why is it not showing up in any of the mappings?

    It's also not an option to copy this data to another structure such as csv to flatten out as I have over 1 billion rows in this table

    • This topic was modified 4 weeks, 1 day ago by  cubemaster.
  • Thanks for posting your issue and hopefully someone will answer soon.

    This is an automated bump to increase visibility of your question.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic. Login to reply