How To Inspect Items When Using The Execute Pipeline Activity In ADF/Synapse Pipelines

How To Inspect Items When Using The Execute Pipeline Activity In ADF/Synapse Pipelines

Ed Yau

Principal Architect - Data Solutions |Kerv Digital

Published 06/07/22 under:

Have a question about something here?

Get in touch

A workaround for ADF/Synapse pipelines when using the Execute Pipeline activity to inspect anything about the run from the calling pipeline

 

A longstanding issue I’ve found with ADF/Synapse pipelines is that if you try to use the Execute Pipeline activity, then it won’t really let you inspect anything about the run from the calling pipeline.

 

This can be generalised to state that within any individual pipeline, you’ll never really be able to understand much about other pipeline runs.

There’s actually a big Azure Feedback post about this, but unfortunately no one seems to have come up with any kind of workaround… So, here’s my workaround…

 

Just call the Synapse/Data Factory API from within the pipeline!

Setup the Rest API as a LinkedService:

 

 

  • Auth type = Managed Identity. It’s worth noting that AAD Resource is not the usual endpoint used for managed identity authentication (see this note)
  • For the base URL, just have workspace_name as a parameter for the linked service. This will be used later and will make this linked service agnostic to the environment you’re in.

 

Then create a dataset from this:

 

 

As you can tell, I’m  being really generic here and just passing the relative url through for now.

Again, it’s worth noting that workspace_name is a parameter here… we only define it in the actual pipeline.

 

Et voila!

Now we have a dataset we can use to get data from the Data Factory API. e.g.:

 

 

As you can now see, we can pass in the workspace name, so this linked service can be moved to new environments with no issue!

 

I actual had to do a double take when I first hit preview data…

 

 

We can now easily get the top level output of a run.

 

“But what about details of the inner activities” you ask…?

Glad you did! Just use this method and hey presto, we now have all the details of our pipeline run…

 

 

Since this is a REST dataset, we unfortunately can’t use it immediately in a Lookup or Get Metadata activity… but, we can easily persist this to a storage account, then look it up and understand anything about how our pipeline ran, from within a pipeline!

 

This was fun little workaround to create and saved me from a jam.

As I couldn’t find anyone else talking about having done this, I thought I’d leave it here in case you found it useful.

Related Articles

You might also be interested in

From our world to yours

Compliance monitoring under the spotlight

From our world to yours

The rise of business messaging apps and the need to stay on...

From our world to yours

Compliance: important ownership changes while adapting for flexible remote workforces

From our world to yours

Data silos and legacy compliance monitoring systems – barriers to effective surveillance...

From our world to yours

What it takes to top the Gartner CCaaS Magic Quadrant

From our world to yours

Why gamification is topping the bill for contact centre agent workforce management...

From our world to yours

8 contact centre trends to future proof your business

From our world to yours

Spotlight on Local Government – Where is CX heading for citizens?

From our world to yours

Think bigger with Microsoft Teams.

From our world to yours

Microsoft Teams is here to stay, what’s the next step?

From our world to yours

At Kerv we want to make customer-first really mean something…

From our world to yours

Different by design

From our world to yours

How Engagement Technology is Transforming the Membership Sector

From our world to yours

A modern CRM system needs a modern data platform

From our world to yours

What is… Microsoft Cloud for NonProfit?

From our world to yours

6 Back-office technologies modern supply chains should already be using

From our world to yours

Re-imagining CX in a golden age of integrations – combining Genesys and...

From our world to yours

Maximising CX value through AI-driven digital engagement

From our world to yours

Experience as a Service: What it is, why it’s important and where...

From our world to yours

Achieving empathy across digital channels

From our world to yours

Redesigning CX from the ground up

From our world to yours

Genesys EMEA Cloud Partner of the Year 2020

From our world to yours

Life at Kerv Digital As A Functional Consultant

From our world to yours

Life @ Kerv Digital as an Apprentice Power Platform Consultant

From our world to yours

How To Increase Efficiency With Dynamics 365 Supply Chain Management

From our world to yours

Supply Chain Control Towers – The Tech That’s Changing Logistics Forever

From our world to yours

Supply Chain Digital Twins – The What, The Who & The Why...

From our world to yours

On-Demand Logistics & The Tech That Makes It Possible

From our world to yours

What Is… Insurtech?

From our world to yours

What Is… Microsoft Cloud For NonProfit?

From our world to yours

16 Times You’ve Been Using Artificial Intelligence Without Realising

From our world to yours

Jack’s Worried – His Website Is Losing Him Dues Paying Members

From our world to yours

From Paper Based To Cutting Edge… With Zero Downtime

From our world to yours

Life at Kerv Digital as a UX Designer

From our world to yours

Sarah is Sad – Her Staff Aren’t Finding New Members Or Engaging...

From our world to yours

How To Squeeze Hidden Value From The Hidden Data You Didn’t Know...

From our world to yours

Reducing Member Churn & Delivering Member Insights With Data Science

From our world to yours

Life @ Kerv Digital As A Dynamics Functional Consultant

From our world to yours

The Effect Fintech Is Having On Our Everyday Lives

From our world to yours

Dealing With Ethical Walls In Tech… Ethically

From our world to yours

Life @ Kerv Digital As A DevSecOps Engineer

From our world to yours

IOT: Dragging The Future Of Healthcare Into Today

From our world to yours

How Will Tech Revolutionise Health Care Over The Next Half Century?

From our world to yours

Empowering Public Transport With Big Data

From our world to yours

Putting Patients First Vs. Cost Concerns

From our world to yours

Life @ Kerv Digital As A Power Platform Solution Architect

From our world to yours

What’s The Best CRM For The Membership Sector?

From our world to yours

How Remote Patient Monitoring Data Can Drive Health Efficiencies

From our world to yours

Is A Career In NonProfit Digital Transformation Rewarding?

From our world to yours

Six Back-Office Functions NonProfits Should Be Using Robotic Process Automation For

From our world to yours

How Data Storage Management Will Change In 2022

From our world to yours

Business Central Vs Sage – Which Does Your Organisation Need?

From our world to yours

How To Debug Something With A Rubber Duck

From our world to yours

Using Design Thinking To Empower Digital Transformations

From our world to yours

How Business Central Can Keep You GDPR Compliant

From our world to yours

What Can Business Central Actually Do?

From our world to yours

Why Data Classification Is Vital To Your Organisation (And How To Easily...

From our world to yours

How To Create UI Flow’s In Power Automate

From our world to yours

D365 BC Vs D365 FO: Let’s Settle This Once And For All!

From our world to yours

How To Drive Donor Engagement With Kerv Digital’s Free Powerups

From our world to yours

9 Awesome Benefits To The Microsoft Dataverse

From our world to yours

Canvas Apps Vs Model-Driven Apps

From our world to yours

How To Set Up Field Monitoring In Business Central

From our world to yours

Check Out The Benefits Of The Microsoft Catalyst IDEA Framework…

From our world to yours

Component Led Development, Or… How To Make Your Organisation Instantly More Resilient

From our world to yours

How To Achieve A Single Customer View In 5 Easy Steps

From our world to yours

Best Practice For Creating Cloud Flows With Microsoft Power Automate

From our world to yours

Legacy Estate Reduction… Or When To Get Rid Of Old Tech

From our world to yours

What Is Fintech?

From our world to yours

How To Get Better At: Online Continuous Personal Development (CPD)

From our world to yours

Virtual Exam Proctoring (Or How To Stop People Googling The Answers At...

From our world to yours

How Hyperautomation’s Benefiting PAO’s (Professional Accountancy Organisations)

From our world to yours

Cyber Security For Remote Working… How Everyone Can (And Has To) Pitch...

From our world to yours

What Is An Advanced Persistent Threat (APT’S) Attack?

From our world to yours

Visualising Your Data Differently With Power BI

From our world to yours

Database Marketing – What Is It & How Can You Benefit From...

From our world to yours

Microsoft Dynamics Cloud Licensing Options – What’s Available?

From our world to yours

Privacy By Design – What You Need To Know

From our world to yours

How To Connect To A Named Sandbox Environment

From our world to yours

What Are The Different Types Of Cloud Licensing Agreements?

From our world to yours

How To Export To Text Files From Microsoft’s Business Central SAAS

From our world to yours

Technical Debt – The What, Why, When & How Do I Get...

From our world to yours

Ensuring Business Continuity With The Microsoft Stack

From our world to yours

What’s The Best CRM For The NonProfit Sector?

From our world to yours

Rage Donations – How To Engage Past The ‘Now’

From our world to yours

What Is It & How To Avoid: Vendor Lock-In

From our world to yours

The Bad Guys Don’t Care You’re The Good Guys

From our world to yours

Build The Future Of The NonProfit Sector… Today

From our world to yours

Data Protection & Artificial Intelligence: Best Practice

From our world to yours

Shadow IT – 9 Things To Look Out For & 1 Unexpected...

From our world to yours

Organisational Debt & Why It Makes Digital Transformation Hard

From our world to yours

Discussing All Things RPA… Robotic Process Automation

From our world to yours

Creating A Low Code App Using PowerApps & The Power Platform

From our world to yours

Now’s The Time To Get Excited About Cognitive Search

From our world to yours

If You Don’t Have An Automated Deployment Process… You’re Already Obsolete

From our world to yours

Tips & Tricks To Creating Successful Volunteer Management Systems

From our world to yours

The Five Types Of Cyber Criminals

From our world to yours

Security-By-Design: Or… Better Safe Than Sorry!

From our world to yours

Leading With Technology In The Membership Sector

From our world to yours

What Does IAAS, PAAS & SAAS Stand For?

From our world to yours

What Is Business Architecture?

From our world to yours

What Is Data Gravity? (And How Your Organisation Can Benefit From It)

From our world to yours

How To Fix DateTime Stamps In Microsoft Dynamics 365

From our world to yours

6 Easy Steps For Promoting A Culture Of Cyber Security

From our world to yours

15 Cyber Security ‘Things’ To Safeguard Your Business

From our world to yours

Microsoft Dynamics 365: Settings In solutions

From our world to yours

13 Ways To Save Money When Using Microsoft Azure

From our world to yours

The Problems With Addresses In Microsoft Dynamics 365

From our world to yours

Understanding The Benefits Of Predictive Science In The NonProfit Sector

From our world to yours

Top 6 Digital Impacts On Membership Organisations

From our world to yours

The 7 Stages Of A Successful AI Project

From our world to yours

What’s The Difference Between UI And UX?

From our world to yours

8 Ways Your Business Can Increase Turnover With Big Data

From our world to yours

Dynamics 365 In NonProfits

From our world to yours

Kerv acquires TDS Global Communications Compliance Practice

From our world to yours

What is Microsoft’s Power Automate?

From our world to yours

What is InvestTech?

From our world to yours

Derek Is Stressed – His Purchasing Team Aren’t Coping With Long Winded...

From our world to yours

Life @ Kerv Digital As A Jnr DevSecOps Engineer…

From our world to yours

Life @ Kerv Digital As A Principal Architect

From our world to yours

Life @ Kerv Digital As An Azure DevSecOps Engineer…

From our world to yours

Life @ Kerv Digital As Head Of Client Services, Public Sector

From our world to yours

Life @ Kerv Digital As A Junior DevSecOps Engineer

From our world to yours

Life @ Kerv Digital As A Graphic Designer

From our world to yours

Microsoft Inspire 2022 – Here’s All The Big Announcements!

From our world to yours

The Need for WhatsApp Compliance Recording

From our world to yours

Life @ Kerv Digital As A UX Researcher

From our world to yours

Life @Kerv Digital As A Senior Delivery Manager

From our world to yours

Life @ Kerv Digital As A Senior Delivery Manager

From our world to yours

Data Silos and Legacy Compliance Monitoring Systems – Barriers to Effective Surveillance...

Have a question?

Leave your details and a member of the team will be in touch to help.
Type your first name here
Type your last name here
Type your phone number here
Type the company you represent here
By pressing send, you agree to our Terms and Conditions and Privacy Policy.
This field is for validation purposes and should be left unchanged.