I recently had a new pipeline fail. It was actually a copy of an old pipeline where I had made some adjustments into as part of a database migration. When triggered during an execution run, it failed saying some expression could not be parsed. When I went into the pipeline and triggered a debug, it immediately failed with the following helpful error message:

Right. Not the same error from the monitoring pane. I went into a copy data activity, and found the following expressions:

The parsing error was from the expression @item.Query. I changed this to @item().Query, but the debug still failed with the Bad Request error. Strangely enough, validating the pipeline doesn’t result in an error. After searching the web, I found that the Bad Request error pops up when something isn’t configured correctly in the pipeline. A space in a name where it doesn’t belong, an unused variable etc. After some digging around, I found that there were some user properties in the Copy Data activity that referenced some activities that I had removed earlier. I fixed their expressions to reference existing items:

Personally I’ve never used the user properties before. They seem to be quite useful though, as explained in this blog post by Cathrine Wilhelmsen.

Anyway, once I fixed the properties the pipeline ran without an issue. I would have been nice though if the error was a bit more clear (for example, expression X contains a reference that cannot be resolved), or that at least the validation of the pipeline would fail.

The post Azure Data Factory Pipeline Debugging Fails with BadRequest first appeared on Under the kover of business intelligence.