I recently had the incredible opportunity to attend the European Power Platform Conference 2024 (EPPC24), where I witnessed an epic presentation by Damien Bird, a.k.a. DamoBird365. His session on “Elevate Your Power Automate Skills: Mastering Graph API, Array Manipulation & Xpath” was a full house, packed with eager attendees ready to soak up his expert insights.
Damien’s presentation was nothing short of inspiring. He showcased a game-changing trick to make flows that process large amounts of data run 10x faster—a revelation for anyone working with Power Automate. As a fan of his YouTube channel and blog, I felt incredibly lucky to be part of this session and witness his mastery in action.
In this article, I will attempt to do justice to Damien’s brilliant techniques and walk you through the process he demonstrated. Whether you’re a seasoned Power Automate user or just getting started, this guide will completely change the way you work with flows and change your understanding of how data is processed there.
Here are the key points we’ll cover, based on Damien’s presentation:
- Understanding Data Structures in Power Automate: We will start by solidifying our understanding of arrays and objects, as well as the difference between keys and values in JSON.
- Advanced JSON handling in Power Automate: Learn the differences between parsing JSON and writing expressions, and understand how to handle Power Platform API limits and throttling.
- Advanced Array Manipulation with Select and Xpath: Learn how to repurpose arrays and search nested arrays using Xpath to streamline your workflows.
In the following sections, I will delve into each of these points, providing detailed explanations and practical examples. Without further ado, let’s get started!
Understanding Data Structures in Power Automate
Power Automate extensively uses JSON data structures to handle various data manipulation tasks. In this section, we will solidify our understanding these data structures, specifically arrays and objects. We will then give you a brief overview of the tools you have in Power Automate to work with array that we will use in the following sections.
Introduction to Data Structures
Arrays and Objects in JSON:
- Arrays: Arrays are ordered collections of values enclosed in square brackets []. These values can be strings, numbers, objects, arrays, true/false, or null. Arrays are useful for handling lists of items where the order matters.
- Objects: Objects are collections of key-value pairs enclosed in curly braces {}. Each key is a string, and each value can be any valid JSON data type. Objects are useful for structuring data logically, similar to dictionaries in programming languages.
Example JSON Object:
{
“students”: [
{
“name”: “John Doe”,
“age”: 20,
“subjects”: [“Math”, “Science”]
},
{
“name”: “Jane Smith”,
“age”: 22,
“subjects”: [“English”, “History”]
}
]
}
In this example, students is an array containing objects. Each object has keys like name, age, and subjects .
Keys and Values in JSON
JSON (JavaScript Object Notation) is a lightweight data-interchange format that is easy for humans to read and write, and easy for machines to parse and generate. In JSON, data is organized into key-value pairs, where:
- Keys are strings that identify a specific piece of data within an object.
- Values can be any JSON-supported data type, including strings, numbers, objects, arrays, booleans (true/false), or null.
Specific Example of Key-Value Pairs
Let’s illustrate this with a practical example:
JSON Object Example:
{ “name”: “John Doe”, “age”: 30, “isStudent”: false, “contact”: { “email”: “[email protected]”, “phone”: “123-456-7890” }, “courses”: [“Math”, “Science”, “History”]}
In this example:
- “name” is the key, and “John Doe” is the value.
- “age” is the key, and 30 is the value.
- “isStudent” is the key, and false is the value.
- “contact” is the key, and the value is another JSON object:
{ “email”: “[email protected]”, “phone”: “123-456-7890”}
- “courses” is the key, and the value is an array of strings: [“Math”, “Science”, “History”].
Advanced JSON Handling in Power Automate
Now that we understand the basics of JSON data structure, and how to work with them in Power Platform, let’s delve a little deeper into how to work with it in Power Automate
Using the Parse JSON Action
The Parse JSON action in Power Automate is essential for converting JSON strings into JSON objects that can be easily manipulated within your flow. Here’s a step-by-step guide to using this action:
- Obtain JSON Data: The JSON data can come from various sources such as HTTP requests, files, or other actions within your flow (e.g., from SharePoint or emails)
- Add the Parse JSON Action:
- Insert the Parse JSON action in your flow.
- In the Content field, provide the JSON string that you want to parse. This can often be the body of a previous HTTP action or another JSON output
- Generate Schema: Click the “Generate from sample” button within the Parse JSON action to automatically generate the schema.
- Paste a sample JSON payload into the schema generation window. This helps Power Automate understand the structure and types of data it should expect.
- Refine Schema: Adjust the generated schema as necessary to ensure all data types and fields are correctly defined. This may involve adding or removing properties and setting data types correctly (e.g., string, integer, Boolean)
- Use Parsed Data: Once the JSON is parsed, the resulting data can be accessed and used as dynamic content in subsequent actions within your flow. This allows for more straightforward manipulation and utilization of specific JSON values.
Writing Expressions to Work with JSON
Directly working with JSON using expressions can sometimes be more efficient, especially for simple tasks where you don’t need the full parsing capabilities. Here are some common expressions and techniques:
- Accessing Values:
- Using Keys and Indices: To access specific items in a JSON object or array, use expressions that navigate through the structure. For example:
outputs(‘Parse_JSON’)?[‘students’][0]?[‘name’]
This expression retrieves the name of the first student from a parsed JSON object
- Handling Nested Structures: When dealing with nested arrays or objects, chain the key and index references appropriately to drill down to the desired value.
body(‘Get_items’)?[‘value’][rand(0,length(body(‘Get_items’)?[‘value’]))]?[‘Title’]
This example selects a random item from an array and retrieves its title
Power Platform API Limits
Power Platform, including Power Automate, imposes several types of limits to ensure service stability and fair usage across all users. These limits are categorized primarily into request limits and service protection limits.
Request Limits: Request limits define the maximum number of API requests a user can make within a specific period, typically a 24-hour cycle. These limits vary based on the licensing plan:
- Standard Users: Up to 40,000 requests per day.
- Non-licensed Users: Aggregated request limits based on product subscriptions.
- Capacity Add-ons: Additional capacity can be purchased in increments of 50,000 requests per 24 hours
Service Protection Limits: These limits ensure that no single user can monopolize resources and degrade service quality for others. The primary facets of service protection limits include:
- Number of Requests: Up to 6,000 requests per user in a 5-minute sliding window.
- Execution Time: Combined execution time of 20 minutes (1200 seconds) within a 5-minute window.
- Concurrent Requests: A maximum of 52 concurrent requests per user.
Techniques to Handle API Limits and Throttling
To efficiently manage these limits and avoid throttling, implement the following strategies in your Power Automate workflows:
- Optimize Data Retrieval and Processing:
- Filter and Sort Data at Source: Use query parameters to fetch only the necessary data, minimizing the payload and reducing the number of API calls.
- Batch Requests: Combine multiple operations into a single batch request to reduce the number of individual API calls. Be mindful of the execution time limits for batch operations.
- Implement Retry Policies:
- Use retry policies to handle transient errors and throttling. Configure retry intervals and maximum retry attempts to ensure smoother operation under high load conditions. Power Automate provides built-in options to set these parameters.
- Distribute Workloads:
- Multiple Flows: Split complex or long-running workflows into smaller, asynchronous flows. This helps distribute the workload and prevents any single flow from hitting the limits.
- Delay Actions: Introduce delays between successive API calls within a flow to avoid hitting the rate limits. This is particularly useful in loops or bulk processing scenarios.
- Monitor and Adjust Usage:
- Admin Center Reports: Utilize the Power Platform Admin Center to monitor API usage and identify flows that are approaching or exceeding limits. This helps in proactive capacity planning and adjustments.
- Handle Specific Query Throttling:
- Optimize queries to reduce database load, such as avoiding leading wildcards in filters and using indexed columns. For high-frequency automated operations, consider reducing batch sizes or adding delays between requests.
Expressions in Power Automate
Expressions in Power Automate allow users to perform complex data manipulations and transformations within their flows. These expressions are built using the Workflow Definition Language (WDL) and can handle a variety of tasks, including string operations, mathematical calculations, and data conversions.
Key Expressions:
- first()
- Purpose: Returns the first item from an array or string.
- Example: first(outputs(‘Compose_YourArray’))
- length()
- Purpose: Returns the number of items in an array or the number of characters in a string.
- Example: length(outputs(‘Compose_XPath_Result’))
- concat()
- Purpose: Combines multiple strings into one string.
- Example: concat(‘{ “root”: ‘, outputs(‘HTTP_Body’), ‘ }’)
- xpath()
- Purpose: Applies an XPath expression to XML content to extract data.
- Example: xpath(xml(outputs(‘Compose_with_XML’)), ‘//members/mail’)
- xml()
- Purpose: Converts a JSON string formatted correctly into XML.
- Example: xml(outputs(‘Compose_JSON_to_XML’))
- Context: Crucial for converting JSON to XML format after adding a root element.
- coalesce()
- Purpose: Returns the first non-null value from a list of arguments.
- Example: coalesce(outputs(‘Compose_YourArray’)?[99], ‘No value’)
- addProperty()
- Purpose: Adds or updates a property in an existing JSON object.
- Example: addProperty(outputs(‘Compose_YourObject’), ‘newProperty’, ‘newValue’)
- variables()
- Purpose: Retrieves the value of a specified variable.
- Example: variables(‘index’)
- replace()
- Purpose: Replaces occurrences of a specified string within a given string.
- Example: replace(outputs(‘Compose_String’), ‘[‘, ”)
- Context: Useful for cleaning up strings, especially when formatting output from XML.
- sort()
- Purpose: Sorts an array in ascending or descending order based on a specified property.
- Example: sort(outputs(‘Compose_YourArray’), ‘propertyName’, ‘ascending’)
- chunk()
- Purpose: Splits an array into chunks of a specified size.
- Example: chunk(outputs(‘Compose_YourArray’), 2
Advanced Array Manipulation with Xpath
Now that we have defined the tools that we need, here is a quick example on how to use xpath to make working with JSON much faster in Power Automate
Step 1: Set Up HTTP Request to Retrieve Data
- Initialize HTTP Request:
- Action: HTTP
- Method: GET
- URI: https://graph.microsoft.com/v1.0/groups?$select=id,mail,displayName&$expand=members($select=mail)
- Authentication: Properly configured to use OAuth with Azure AD or another suitable authentication method.
Step 2: Prepare Data for Conversion
- Compose to Modify JSON for XML Conversion:
- Action: Compose
- Inputs: Use the body from the HTTP action to wrap the JSON response with a root element for XML conversion: { “root”: @{outputs(‘HTTP_Body’)}}
Step 3: Convert JSON to XML
- Convert JSON to XML:
- Action: Use a custom script or a third-party connector that converts JSON to XML.
- Input: Pass the output of the previous compose action.
Step 4: Extract Data Using XPath
- Compose to Apply XPath Query:
- Action: Compose
- Input: Use XPath on the XML to extract needed information, such as email addresses:
- XPath Expression: //members/mail
Step 5: Initialize Variables for Loop Control
- Initialize Variables:
- Action: Initialize variable
- Name: index
- Type: Integer
- Value: 0
- Action: Initialize variable
- Name: count
- Type: Integer
- Value: Use the length() function to determine the size of the XPath results: length(outputs(‘Compose_XPath_Result’))
- Action: Initialize variable
Step 6: Implement Do Until Loop to Process Data
- Do Until Loop:
- Action: Do Until
- Condition: index < count
- Inside the loop:
- Process Each Item: Depending on the operation (e.g., create/update records in Dataverse or another system).
- Increment Index: Add a step to increment the index by 1 after each iteration.
- Inside the loop:
Step 7: Optimize Data Insertion Based on Results
- Conditional Insertion Logic:
- Before entering the loop, decide whether to batch process or individually process items based on the size of count.
- If count is less than a certain threshold, consider using batch operations instead of looping through each item.
Step 8: Handle Outputs and Logging
- Final Steps:
- Check outputs and log any relevant information or errors.
- Use condition checks to handle different outcomes, such as error handling or notifications on job completion.
To see this flow in action and find other amazing tips on how to further develop flows like this one, be sure to check out Damien’s original video:
Conclusion
In this blog, we’ve explored key aspects of his session, from understanding data structures in Power Automate, advanced JSON handling, to optimizing data operations with Select and Xpath. These insights are crucial for both novice and experienced users aiming to streamline their workflows and enhance efficiency. By applying these techniques, you can significantly improve the performance of your Power Automate solutions.
For more tips and tricks to boost your power automate skills and make your flows even more powerful, I highly recommend checking out Damien’s channel.