Data Visualization, Microsoft Technologies, Power BI

CDOT Bar Chart Makeover

As I was browsing Twitter today, I noticed a tweet from the Colorado Department of Transportation about their anti-DUI campaign. Shown below, it contains a bar chart that appears to have been presented in PowerPoint.

Tweet from CDOT containing a poorly designed bar chart

There are some easy opportunities to improve the readability of this chart, so I thought I would use it as an example of how small improvements can have a big impact on a fairly simple chart. I recreated the chart (as best I could) in Power BI and then made two revised versions.

Bar chart with 5 bars representing suspected impaired driving fatalities per year. The top left contains 3 logos. The title is on the top right. Each bar is a different bright color. The axis labels are very small.
My Power BI version of the original chart
Redesigned bar chart that removes the thick orange line, moves the title to the top left, and uses only one color across the bars.
My 10-minute makeover that increased readability
Bar chart with red bars and an annotation noting the upward trend from 2019 to 2020. A gray placeholder with a question mark has been added for 2021 with the note "Don't become part of this statistic."
My additional iteration that applied a clearer message and focus

Especially when making data visualizations for the general public —and especially when you want to get people’s engagement on social media— you need to reduce perceived cognitive load. Otherwise, people won’t even bother to read your chart. If your chart feels too busy or too complicated, many people in your intended audience will feel it is not worth the effort to even try to read it and will move on down their Twitter feed to the next Anakin and Padme meme.

Watch the video below for all of the details.

21-minute video showing how to improve upon a bar chart found on Twitter using Power BI
DAX, Microsoft Technologies, Power BI, Power Query

Calculating Age in Power BI

In week 26 of Workout Wednesday for Power BI, I asked people to calculate the age of Nobel laureates at the time they received the award. I provided some logic, but I didn’t prescribe how to create the age calculation. This inspired a couple of questions and a round of data validation as calculating age may be trickier than you think. In this post, I’ll explore some of the ways people have calculated age in Power BI and the edge cases where those calculations may not work.

In my solution video for Workout Wednesday, I used Power Query to calculate age. This was inspired by several blog posts and videos I had seen previously. There is an Age menu option in the Power Query editor under Date.

Calculating Age with the Power Query Editor user interface

When you select a date column and use that Age option, it calculates the duration between the selected date and the current date in days. You must then replace the current date with the second date column. Next you can choose Total Years under Duration, which divides the days by 365. Finally, you must round that number down to the next integer to get years.

If you follow Ruth’s video, you can do all of that in one step that creates a custom column with the final age value.

 Number.RoundDown(Duration.TotalDays([Date2] - [Date1])/365) 

That is the most common option in Power Query as there is no DateDiff function.

There are a few options for calculating age in DAX. Some people use the DATEDIFF function.

Age DateDiff = DATEDIFF([Date1],[Date2],YEAR) 

Another way I have seen is to use YEARFRAC function.

Age YearFrac = INT ( YEARFRAC ( [Date1], [Date2], 1 ) )

The way Marco Russo suggests is to use QUOTIENT.

Age Quotient (DAX): 

Age Quotient = 

VAR Birthdate = [Date1]

VAR ThisDay = [Date2]

VAR IntBirthdate = YEAR ( Birthdate ) * 10000 + MONTH ( Birthdate ) * 100 + DAY ( Birthdate )

VAR IntThisDay = YEAR ( ThisDay ) * 10000 + MONTH ( ThisDay ) * 100 + DAY ( ThisDay )

VAR Age = QUOTIENT ( IntThisDay - IntBirthdate, 10000 )

VAR CheckedAge = DIVIDE ( Age, NOT ISBLANK ( Birthdate ) )

RETURN

    CheckedAge

As Marco points out, many people were using YEARFRAC, but there is a bug in the DAX implementation that causes it to occasionally return an incorrect answer for this purpose.

Checking the Numbers

I created a Power BI file to demonstrate the differences in these four calculations. You can download the file here. The image below displays the results in several tests. For each row, I’m using Date1 as the birthdate and Date2 as the “as of” date. You’ll notice that I focused on leap years for a few cases.

Table in Power BI with 10 date ranges showing the results from the four calculations. 6 of the 10 rows have different results across the calculations.
Example date ranges and result of the four age calculations

There are six of ten date ranges that have different results across the different calculation methods.

In the second row, the Power Query age calculation says that Feb 29 to Feb 28 in the following year is a full year. This may or may not be what you want depending on your requirements. I’m noting the difference so you can be aware. A similar thing occurs in the fifth row going from Feb 29, 2016 to Feb 28, 2020, and again on the 9th row going from March 1, 2019 to Feb 29, 2020.

On the third row, notice that the DAX DATEDIFF function calculates Feb 29 to Feb 27 of the following year to be a full year, despite it being a day or two short. Depending on what you do with leap years, you might consider Feb 29 to Feb 28 in the following year to be a full year, but that third row result means DATEDIFF is probably not the calculation I want. We see a similar result going from March 1 to Feb 28 of the following year.

YEARFRAC calculates that Feb 29 to Feb 28 in the following year is not a full year, which may be desirable. But it counts Feb 29, 2016 to Feb 29, 2020 as only three years. And we see that March 1, 2000 to March 1, 2021 is only counted as 20 years. So even without starting on a leap year, we get some incorrect results. Small numbers seem to be correct until it gets to about 13 years.

Using the QUOTIENT function provides what I consider to be the most correct results. It calculates Feb 29 to Feb 28 of the following year to be less than a year. It calculates Feb 29, 2016 to Feb 28, 2016 to be three years and not four. And it calculates March 1 to Feb 29 of the following year to be less than a year.

Which to use?

The QUOTIENT formula produces the most accurate results if you don’t want Feb 29 to Feb 28 the next year to be counted as a year. DATEDIFF and YEARFRAC produce too many incorrect results for me to ever suggest using them. Since there is a DAX option that produces more correct answers, I would just go for QUOTIENT instead of either of these two.

UPDATE: There is a better alternative! Imke Feldmann reminded me that there is an Number.IntegerDivide function in Power Query. So let’s take the logic from Marco’s DAX calculation and move it to Power Query:

(BirthDate as date, EndDate as date) =>
let
BirthDateInt = Date.Year(BirthDate)10000 + Date.Month(BirthDate)100 + Date.Day(BirthDate),
EndDateInt = Date.Year(EndDate)10000 + Date.Month(EndDate)100 + Date.Day(EndDate),
Age = Number.IntegerDivide((EndDateInt - BirthDateInt),10000)
in Age

The Power Query custom column created by invoking this function should produce better compression than a DAX calculated column. This might not be significant for a small dataset, but we should be efficient when we can.

Accessibility, Data Visualization, Microsoft Technologies, Power BI

Zooming In on a Power BI Report

Have you ever tried to use your browser to zoom in on a visual in a Power BI report? If you simply published your report and then zoomed in, you might have experienced something like the video below.

Trying to zoom in on a report that is set to Fit to page can be confusing for users.

With the default settings of the report, when you zoom in, only the menus around the report change. This is because of report responsiveness and the View setting. By default, reports are set to Fit to page. Power BI is refitting the report to the page every time you zoom.

Why would we need to zoom in?

There might be accessibility or compliance reasons to allow people to zoom in. For instance, WCAG 2.1 Success Criterion 1.4.4 states “Except for captions and images of texttext can be resized without assistive technology up to 200 percent without loss of content or functionality.” People with low vision or other vision impairments might benefit from the ability to zoom within a report page.

Another reason might be that a user simply wants to focus on one chart at a time. Power BI does have a Focus mode. Unfortunately, it currently does a poor job of increasing the font sizes on the visual that is in focus, often rendering it unhelpful.

Column chart shown in Focus Mode in Power BI with large bars and tiny text
Power BI visual shown in Focus Mode

Edit: A helpful commenter pointed out that you can zoom in and out while in Focus mode. This works pretty well on many (but not all) visuals.

What Are Our Other Options?

There are a couple of workarounds for users who need to zoom in on visuals.

  1. We can set the report view or teach users to set the report view to Actual size. This then allows the browser zoom to work as anticipated. We probably don’t want to set all our reports to actual size because we would lose valuable screen real estate and diminish the experience for some users who don’t need to zoom. Having the report automatically fit to the user’s screen is usually helpful. But if users can change that setting as they need too, that might be ok. Here’s an example of how that works.
Setting the view on the Power BI report to Actual size allows users to zoom with the browser

2. We can use assistive technology to zoom. Both Windows and MacOS have built-in magnifier functionality. The downside to this is that using it would not satisfy WCAG 2.1 Success Criterion 1.4.4. I think there is still some gray area/lack of expertise as far as how people are making data visualizations WCAG compliant because it’s part text and part image/shape (although it’s not rendered on the page as an image in Power BI). I’m usually more concerned that users get the information they need an have a good experience. But I want to note this in case you are trying to be WCAG compliant and might run into this issue. Here’s an example of using the magnifier in Windows. You can still use the interactivity in the report. And you can change the size of the magnification window and the level of magnification.

The Windows Magnifier allows users to zoom in to part of the report page while retaining interactivity

3. Zooming in on the report page with a touch screen works fine. If users have tablets or laptops with a touch screen, they can use their fingers to zoom and it will behave as expected. Here’s a video that shows that experience.

Those are all the workarounds I’m aware of, but I’m interested to hear how you have worked around this issue. If you have other suggestions please leave them in the comments.

I found an existing idea about increasing the text size within visuals in focus mode on Ideas.PowerBI.com. I’ve added my vote to it, and I hope you’ll do the same.

Azure, Azure Data Lake, Microsoft Technologies, Power BI

Granting ADLS Gen2 Access for Power BI Users via ACLs

It’s common that users only have access to certain folders in an Azure Data Lake Storage container. These permissions are provided not through Azure RBAC (role-based access control) roles but through POSIX-like ACLs (access control lists).

The current Power BI documentation mentions only Azure RBAC roles, but it is possible to connect to a folder with permissions granted through ACLs.

You can manage ACLs through the Azure Storage Explorer application or in the Storage Explorer preview in the Azure Portal. As an example, I have a storage account with the hierarchical namespace enabled. In the container named filesystem1 is a folder called Test. Test contains 3 files, and I want a user to import Categories.csv into Power BI.

Azure Storage Explorer showing the mmldl storage account with filesystem1 selected. The Test folder in filesystem1 is selected and 3 files are shown.
Data lake storage account with files located in a folder called Test

If I select the Test folder and then select Manage Access, I can see that an AAD user named Data Lake User has been granted access and default ACLs. Note that the user needs at least Read and Execute. Write isn’t necessary if they don’t need to change the file.

The Manage Access window in Azure Storage Explorer. The user named Data Lake User is selected. Access and Default permissions are set to give the user Read, Write, and Execute.
Managing access on the Test folder for the Data Lake Access user

But with those permissions on the Test folder, I’m not able to connect to it from Power BI Desktop. If I try, I’ll get an error that says “Access to the resource is forbidden.”

Power BI error that says "Unable to connect. We encountered an error while trying to connect. Details: Access to the resource is forbidden."
Power BI error encountered when a user doesn’t have sufficient permissions to access a file in the data lake

This is because the user is missing some permissions. We need to grant Execute permissions on all parent folders up to the root (the container).

In this case, there is only one level above my Test folder. So I select the filesystem1 container, go to Manage Access, and grant it Execute permissions.

Manage Access window in Azure Storage Explorer showing permissions for Data Lake user on filesystem1. Execute is selected for both Access and Default permissions.
Adding Execute permissions to the parent container

Note that changing the Default ACL on a parent does not affect the access ACL or default ACL of child items that already exist. So if you have existing subfolders and files to which users need access, you will need to grant access at each parent level because the default ACLs won’t apply.

Thanks to Gerhard Brueckl for noting that I needed Execute permissions on parent folders when I got stuck in testing.

If you find yourself hitting that access forbidden message in Power BI when accessing a file in ADLS Gen2, double check the user’s Execute permissions on the parent folders.

Microsoft Technologies, Power BI, Workout Wednesday

Workout Wednesdays for Power BI in 2021

I’m excited to announce that something new is coming to the Power BI community in 2021: Workout Wednesday!

Workout Wednesday logo

Workout Wednesday started in the Tableau community and is expanding to Power BI in the coming year. Workout Wednesdays present challenges to recreate a data-driven visualization as closely as possible. They are designed to help you improve your skills in Power BI and Tableau.

How You Can Participate

Watch for the Power BI challenge to be published on Wednesdays in 2021. The challenge will contain requirements and a dataset. Use the dataset to create the desired end result.

Then share your workout! You can post your workout to the Data Stories Gallery or your blog, or just share a public link. If you aren’t able to share a public link – perhaps because that option is disabled in your Power BI tenant or you don’t have a Power BI tenant– a gif, a video, or even some screenshots are just fine.

To formally participate: Post to Twitter using both the #WOW2021 and #PowerBI hashtags along with a link/image/video of your workout. Include a link to the challenge on the Workout Wednesday site. And please note the week number in your description, if possible.

Community Growth

I’m looking forward to Workout Wednesdays for a couple of reasons. First, I think Power BI needs more love in the data visualization department. We need to be talking about effective visualization techniques and mature past ugly pie charts and tacky backgrounds. And I think Workout Wednesdays will help us individually grow those skills, but it will also foster more communication and sharing of ideas around data visualization in Power BI. That in turn will lead to more product enhancement ideas and conversations with the Power BI team, resulting in a better product and a stronger community.

Second, I’m also excited to see the crosspollination and cross-platform learning we will achieve by coming together as a data visualization community that isn’t focused on one single tool. There is a lot Tableau practitioners and Power BI practitioners can learn from each other.

Join Me In January

Keep an eye out on Twitter and the Workout Wednesday website for the first challenge coming January 6. While it would be great if you did the workout for every single week, don’t be concerned if you can’t participate every week. A solution will be posted about a week later, but nothing says you can’t go back and do workouts from previous weeks as your schedule allows.

I look forward to seeing all of your lovely Workout Wednesday solutions next year!

DAX, Microsoft Technologies, Power BI

DAX Logic and Blanks

A while back I was chatting with Shannon Lindsay on Twitter. She shares lots of useful Power BI tips there. She shared her syntax tip of the & operator being used for concatenation and the && operator being used for boolean AND, which reminded me about implicit conversions and blanks in DAX.

Before you read the below tweet, see how many of these you can guess correctly:

Blank + 5 = ? 
Blank * 5 = ?
5 / Blank = ?
0 / Blank = ?

In DAX, Blank is converted to 0 in addition and subtraction.

What about boolean logic? Do you know the result of the following expressions?

AND(True(), Blank()) = ? 
OR(True(), Blank()) = ? 
AND(False(), Blank()) = ? 
AND(Blank(), Blank()) = ? 

You can see the results as well as a few more permutations in the screenshot below.

Two tables in a Power BI report. The left table shows arithmetic operations involving blanks. For example, Blank + Blank = Blank, 0 * Blank = NaN, 5 * Blank = Blank, 5 / Blank = Infinity. The right table shows boolean operations involving blanks. True and blank = false, true or blank = true, false and blank = false, blank or blank = false
Read the left table as Number1 [operator] Number2, so 5 + Blank = 5. 5 * Blank = Blank. And 5 / Blank = Infinity. Read the right table as Bool1 [operator] Bool2, so True AND Blank = False and True OR Blank = True.

Why does this matter?

You need to understand the impact of blanks in your data. Do you really want to divide by zero when you are missing data? If you are performing a boolean AND, and your data is blank, are you ok with showing a result of False? Remember that your expression may produce undesired results rather than an error.

First, you need to be aware of where it is possible in your data to get a blank input. When you are writing your DAX measures, you may need to handle blanks. DAX offers the IFERROR() function to check if the result of an expression throws an error. There is also an ISBLANK() function that you can use to check for a blank value and a COALESCE() function to provide an alternate value when a blank value is detected.

But adding extra logic in your measures may have a performance impact. For example, the DIVIDE() function can handle divide by zero errors for you. But DIVIDE() may be slower than the / operator. The performance difference is dependent on your data and the expression you are writing. Alternatively, you can use an IF statement to check if an input value is greater than zero using the > operand. This can be quicker than checking for blanks or errors using other functions.

At the end of the day, producing the correct result is more important than fast performance, but we strive to achieve both. If you have any tips for handling blanks in DAX, please share them in the comments.

Accessibility, Data Visualization, Microsoft Technologies, Power BI

Stop Letting Accessibility Be Optional In Your Power BI Reports

We don’t talk about inclusive design nearly enough in the Power BI community. I was trying to recall the last time I saw a demo report (from Microsoft or the community) that looked like consideration was made for basic accessibility, and… it’s a pretty rare occurrence.

A woman, man, and another man in a wheelchair next to the Power BI logo.

Part of the reason for this might be that accessibility was added into Power BI after the fact, with keyboard accessible visual interactions being added in 2019 as one of the last big accessibility improvements. But I think the more likely reasons are that inclusive design requires empathy and understanding of how to build reports for people who work differently than ourselves, and Power BI accessibility features take time and effort to implement. While we can never make our reports 100% accessible for everyone, that doesn’t mean we should just not try for anyone.

Population statistics tell us that many of our colleagues have or will have a disability at some point, and many of them will be invisible. So even if you don’t see a report consumer with an obvious disability today, that doesn’t mean an existing user won’t acquire a disability or a new user with a disability won’t come along as people change roles in an organization. In addition to the permanent disabilities we normally think of, there are also temporary and situational disabilities that we should try to accommodate.

In order to start designing more inclusively, we need to increase conversation around accessibility requirements and standards for our reports. I fully understand that it can feel tedious or confusing as you get started. I hope that as Power BI matures, the accessibility features will mature as well to make it even easier to create a more accessible report by default. For now, the only way to make accessible Power BI report design easier for report creators is for us to start forming accessible design habits and to offer feedback to the Power BI team along the way.

My Accessible Report Design Proposal

This is what I would like to see from report creators in the community as well as within Microsoft. I’ll define what I mean by accessible report design in the next section.

  • Before publishing a report, implement accessible design techniques as thoroughly as possible.
  • For demonstrations of report design/UI techniques where you are providing a finished product at the end, implement accessible design techniques as thoroughly as possible.
  • For demonstrations of things that are not inherently visual, implement bare minimum accessibility or add a disclaimer to the report.
    Example: “Here’s a cool DAX technique that I just threw into a quick table or bar chart to show you the results. It hasn’t been cleaned up and made accessible (alt text, color contrast, etc.), but I would do that before publishing.”
  • For demonstrations of report design/UI techniques where you show only part of the process, implement bare minimum accessibility or add a disclaimer to the report. 
    Example: “This is the part of the report creation process about creating bookmarks, and before I publish to an audience, I want to make sure I’m following good design practices including accessibility.”

Power BI Report Accessibility

I have a full list of things to check here. That is the checklist that I use to ensure my report designs are generally accessible, when I have no specific compliance requirements or knowledge or any specific disabilities that need to be accommodated. In my opinion, this is what we should be doing in all of our reports because we want everyone in our intended audience to be able use our reports. You’ll find a very similar checklist on Microsoft Docs.

If you need to start smaller, you can go with my bare minimum accessibility and work your way up to the full list.

Bare Minimum Accessibility

This is the short list of the most impactful (according to me) accessibility changes you can make in your report. Use this because you have to start somewhere, but realize there is more we should be doing.

  1. Ensure text and visual components have sufficient color contrast
  2. Use descriptive, purposeful chart titles
  3. Avoid using color as the only means of conveying information
  4. Set tab order on all visuals in each page
  5. Remove unnecessary jargon and acronyms from all charts

Give It a Try

I just learned that the Power BI Community Featured Data Stories Gallery theme for September is Accessibility. So here’s your chance to win a free t-shirt and internet bragging rights by showing off your accessible design skills. You need to submit your report to the Data Stories Gallery by September 30th in order for your submission to be considered. But a well designed, accessible Power BI report added to the gallery is appreciated any time of year!

Accessibility, Data Visualization, Microsoft Technologies, Power BI

Fun with Power BI and Color Math

I recently published my color contrast report in the Power BI Data Stories Gallery. It allows you to enter two hex color values and then see the color contrast ratio and get advice on how the two colors should be used together in an accessible manner.

Screenshot of the Color Contrast calculator Power BI report. The report headline reads "How shoudl I use these colors together in my Power BI report?". There are 2 slicers that allow you to select colors by hex value. A contrast ratio is shown along with advice generated on how to use the colors.
Color contrast calculations in a Power BI report

I could go on for paragraphs about making sure your report designs are accessible and useful for your intended audience. But this post focuses on how I made this report.

The Calculations

Color contrast (as calculated in the WCAG 2.1 success criteria) is dependent on luminance. Luminance is the relative brightness of any point in a color space, normalized to 0 for darkest black and 1 for lightest white. In order to calculate color contrast you must first get the luminance of each color.

As an example, I have colors #F3F2F1 and #007E97. In this hex notation, often explained as #RRGGBB, the first two digits represent red, the second two digits are green, and the last two digits are blue. Each two digits is a value that represents the decimal numbers 0 to 255 in hexadecimal notation. The same red, green, and blue values can be represented in decimal notation as integers, and this is what is used to calculate luminance. #F3F2F1 is RGB(243, 242, 241), and #007E97 is RGB(0,126,151).

On a side note, there are places in Power BI where we can change the transparency of the color which is referred to as RGBA (where A represents opacity/transparency). But whenever you copy a hex color value out of the color palette in Power BI, you will just see the 6 digits without the A because the A is stored separately in the UI. When you set colors using DAX formulas, you can specify the A value.

The sRGB color space is non-linear. It compensates for humans’ non-linear perception of light and color. If images are not gamma-encoded, they assign too many bits or too much bandwidth to highlights that humans can’t distinguish, and too few bits to shadows to which humans are sensitive and would require more bits to maintain the same visual quality. To calculate luminance we have to linearize the color values.

For each color component (R,G,and B), we first divide our integer value by 255 to get a decimal value between 0 and 1. Then we apply the linearization formula:

  • if R sRGB <= 0.04045 then R = R sRGB /12.92 else R = ((R sRGB +0.055)/1.055) ^ 2.4
  • if G sRGB <= 0.04045 then G = G sRGB /12.92 else G = ((G sRGB +0.055)/1.055) ^ 2.4
  • if B sRGB <= 0.04045 then B = B sRGB /12.92 else B = ((B sRGB +0.055)/1.055) ^ 2.4

Note: You will find sources online that that incorrectly use the number 0.03928 in the linearization formula instead of .04045. My understanding is that this is incorrect for sRGB.

Then we plug those values in to calculate luminance:

L = 0.2126 * R + 0.7152 * G + 0.0722 * B

The luminance of #F3F2F1 is .8891. The luminance of #007E97 is .1716.

The final calculation is color contrast:

(L1 + 0.05) / (L2 + 0.05), where

  • L1 is the relative luminance of the lighter of the foreground or background colors, and
  • L2 is the relative luminance of the darker of the foreground or background colors.

The color contrast between #F3F2F1 and #007E97 is 4.24, and we usually write this as 4.24:1. You can check my math here.

The Dataset

The source data for the report is generated entirely in Power Query. It starts with a simple list of the integers 0 through 255. I placed this in a query called Values.

let
    Source = List.Numbers(0,256),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Column1", Int64.Type}})
in
    #"Changed Type"

My linearization function is called ColorConvert.

(colornum as number) =>
let 
    Source = if colornum < .04045 then colornum/12.92 else  Number.Power(((colornum+0.055)/1.055),2.4)
in
    Source

My main query is called color 1. This is where all the calculations through luminance are done.

let
    //Get values 0 - 255
    Source = Values,
    //Call that column R for Red
    #"R Dec" = Table.RenameColumns(Source,{{"Column1", "R Dec"}}),
    //Crossjoin to Values to get Green values 0 - 255
    #"G Dec" = Table.AddColumn(#"R Dec", "Custom", each Values),
    #"Expanded G Dec" = Table.ExpandTableColumn(#"G Dec", "Custom", {"Column1"}, {"G Dec"}),
    //Crossjoin to Values to get Blue values 0 - 255
    #"B Dec" = Table.AddColumn(#"Expanded G Dec", "B", each Values),
    #"Expanded B Dec" = Table.ExpandTableColumn(#"B Dec", "B", {"Column1"}, {"B Dec"}),
    //Get hexidecimal values for R,G,B
    #"R Hex" = Table.AddColumn(#"Expanded B Dec", "R Hex", each Text.End("00" & Number.ToText([R Dec], "x"),2)),
    #"G Hex" = Table.AddColumn(#"R Hex", "G Hex", each Text.End("00" & Number.ToText([G Dec], "x"),2)),
    #"B Hex" = Table.AddColumn(#"G Hex", "B Hex", each Text.End("00" & Number.ToText([B Dec], "x"),2)),
    //Concatenate to get full 6-digit Hex color value
    #"Changed Hex Type" = Table.TransformColumnTypes(#"B Hex",{{"R Hex", type text}, {"G Hex", type text}, {"B Hex", type text}}),
    #"Full Hex" = Table.AddColumn(#"Changed Hex Type", "Hex", each [R Hex] & [G Hex] & [B Hex]),
    //Convert integers to decimals and linearize
    #"R Lin" = Table.AddColumn(#"Full Hex", "R Lin", each ColorConvert(([R Dec]/255))),
    #"G Lin" = Table.AddColumn(#"R Lin", "G Lin", each ColorConvert(([G Dec]/255))),
    #"B Lin" = Table.AddColumn(#"G Lin", "B Lin", each ColorConvert(([B Dec]/255))),
    //Calculate luminance with the linearized values
    #"Luminance" = Table.AddColumn(#"B Lin", "Luminance", each 0.2126 * [R Lin] + 0.7152 * [G Lin] + 0.0722 * [B Lin]),
    #"Changed Luminance Type" = Table.TransformColumnTypes(#"Luminance",{{"Luminance", type number}}),
    //Create a column for hexidecimal value with the hash/pound at the beginning
    #"Hex Dup" = Table.DuplicateColumn(#"Changed Luminance Type", "Hex", "Hex With Hash"),
    #"Hex with Hash" = Table.TransformColumns(#"Hex Dup", {{"Hex With Hash", each "#" & _, type text}}),
    //Remove Hex and linearized RGB columns to keep model under 1 GB limit for Pro license
    #"Removed Columns" = Table.RemoveColumns(#"Hex with Hash",{"R Hex", "G Hex", "B Hex", "R Lin", "G Lin", "B Lin", "Hex"}),
    //Rename Hex with Hash to Hex
    #"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Hex With Hash", "Hex"}})
in
    #"Renamed Columns"

In order to allow users to choose two colors, I made a reference query to Color 1 called Color 2.

let
    Source = #"Color 1"
in
    Source

If you are interested in these Power Query scripts, you can get them from this Gist.

DAX Calculations

The color contrast calculation is a DAX measure because it is dynamically calculated based upon the colors selected in the report.

Color Contrast = 
If( Max('Color 1'[Luminance]) > MAX('Color 2'[Luminance]),
    Divide((Max('Color 1'[Luminance]) + 0.05) , (Max('Color 2'[Luminance]) + 0.05)),
    Divide((Max('Color 2'[Luminance]) + 0.05) , (Max('Color 1'[Luminance]) + 0.05))
)

The advice given based upon the color contrast ratio is also a DAX measure.

Advice =
IF (
    [Color Contrast] < 3,
    "Not enough contrast for text or non-text content, use only for decorative items",
    IF (
        [Color Contrast] < 4.5,
        "Appropriate for large text at least 18pt, bold text at least 14 pt, or non-text content",
        IF (
            'Color 1'[Color Contrast] >= 4.5,
            "Appropriate for any size text and any non-text content"
        )
    )
)

The example charts showing the two colors as foreground and background are SVG measures.

Chart 1 =
VAR Bkgrnd =
    MAX ( 'Color 1'[Hex] )
VAR Frgrnd =
    MAX ( 'Color 2'[Hex] )
VAR chart = "data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' width='100' height='100' viewBox='0 0 24 24' style='background-color:" & Bkgrnd & "'><path fill= '" & Frgrnd & "' d='M7 19h-6v-11h6v11zm8-18h-6v18h6v-18zm8 11h-6v7h6v-7zm1 9h-24v2h24v-2z'/></svg>"
RETURN
    chart
Chart 2 =
VAR Bkgrnd =
    MAX ( 'Color 2'[Hex] )
VAR Frgrnd =
    MAX ( 'Color 1'[Hex] )
VAR chart = "data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' width='100' height='100' viewBox='0 0 24 24' style='background-color:" & Bkgrnd & "'><path fill= '" & Frgrnd & "' d='M7 19h-6v-11h6v11zm8-18h-6v18h6v-18zm8 11h-6v7h6v-7zm1 9h-24v2h24v-2z'/></svg>"
RETURN
    chart

The check or x mark to indicate whether the colors can be used together in a graph or in text is created using Unicode characters.

UseInGraph =
IF ( [Color Contrast] < 3, "✗", "✔" )
UseInText =
IF ( [Color Contrast] < 4.5, "✗", "✔" )

The RGB value shown for each color in the report is a DAX measure because storing it in the model made the model size larger than 1 GB, which would have prohibited me from deploying the report and publishing it to the web.

RGB1 =
VAR R =
    SELECTEDVALUE ( 'Color 1'[R Dec] )
VAR G =
    SELECTEDVALUE ( 'Color 1'[G Dec] )
VAR B =
    SELECTEDVALUE ( 'Color 1'[B Dec] )
RETURN
    R & "," & G & "," & B

Check Out the Report

This post was an enjoyable combination of color, Power BI, and a bit of math. It was fun to make the report since it brought together my interests in accessibility and Power BI model optimization. At the least I’m hoping this gives you some exposure to how accessibility guidelines are applied to reports. If you are like me, you’ll find the color math fascinating and go down that rabbit hole.

Take a few seconds, pick some colors, and give the report a try.

Accessibility, Conferences, Microsoft Technologies, PASS Summit, Power BI

I’m Speaking at Virtual PASS Summit 2020

PASS Summit has gone virtual this year, but that isn’t keeping PASS from delivering a good lineup of speakers and activities. I’m excited to be presenting a pre-con and two regular sessions this year. I know virtual delivery changes the interaction between audience and speaker, and I’m going to do everything I can to make my sessions more than just standard lecture and demo to keep things interesting.

Building Power BI Reports that Communicate Insights and Engage People (Pre-Con)

If you are into Power BI or data visualization, check out my pre-con session. It’s called Building Power BI Reports that Communicate Insights and Engage People. Unless we’ve had data visualization training, the way we learn to make reports is by copying reports that others have made. But that assumes other people were designing intentionally for human consumption. Another issue is that we often mimic example reports from tool vendors. That can be very helpful with the technical aspects of getting content on the page, but we often overlook the design aspects of reports that can make or break their usability and effectiveness in communicating information. My pre-con will begin with discussion on how humans interpret data visualizations and how you can use that to your advantage to make better, more consumable visualizations. We’ll take those lessons and apply them specifically to Power BI and then add on some tips and tricks. Throughout the day, there will be hands-on exercises and opportunities for group conversation. And you’ll receive some resources to take with you to help you continue to improve your report designs.

Agenda slide from my pre-con session: 1) Defining Success, 2) Message & Story, 3) Designing a Visual, 4) Refine Your Report 5) Applied Power BI 6) Power BI Tricks 7) Wrap-Up
Agenda for my PASS Summit pre-con titled Building Power BI Reports that Communicate Insights and Engage People

This session is geared toward people that have at least basic familiarity with Power BI Desktop (if you can populate a bar chart on a report page, that’s good enough). If you have never opened Power BI Desktop, we might move a little fast, but you are welcome to join us and give it a try. If you are pretty good with Power BI Desktop, but you want to improve your data visualization skills, this session could also be a good fit for you. I hope you’ll register and join my pre-con.

Implementing Data-Driven Storytelling Techniques in Power BI

Data storytelling is a popular concept, but the techniques to implement storytelling in Power BI can be a bit elusive, especially when you have data values that change as the data is refreshed. In this session, we’ll talk about what is meant by story. Then I’ll introduce you to tool-agnostic techniques for data storytelling and show you how you can use them in Power BI. We’ll also discuss the visual hierarchy within a page and how that affects your story. You can view my session description here.

Inclusive Presentation Design

I’m also delivering a professional development session for those of us that give presentations. Most speakers have good intentions and are excited to share their knowledge and perspective, but we often exclude audience members with our presentation design. Join me in this session to discuss how to design your presentation materials with appropriate content formatted to maximize learning for your whole audience. You’ll gain a better understanding of how to enhance your delivery to make an impact on those with varying abilities to see, hear, and understand your presentation. You can view my presentation description here.

Other Pre-Cons from My Brilliant Co-Workers

If you aren’t into report design, my DCAC coworkers are delivering pre-cons that may interest you.

Denny Cherry is doing a pre-con session on Microsoft Azure Platform Infrastructure.

John Morehouse is talking about Avoiding the Storms When Migrating to Azure.

I hope you’ll join one of us for a pre-con as well as our regular sessions. With PASS Summit being virtual, the lower price and removal of travel requirements may make this conference more accessible to some who haven’t been able to attend in past years. Be sure to get yourself registered and spread the word to colleagues.

Azure, Azure Data Factory, Microsoft Technologies, Power BI

Refreshing a Power BI Dataset in Azure Data Factory

I recently needed to ensure that a Power BI imported dataset would be refreshed after populating data in my data mart. I was already using Azure Data Factory to populate the data mart, so the most efficient thing to do was to call a pipeline at the end of my data load process to refresh the Power BI dataset.

Power BI offers REST APIs to programmatically refresh your data. For Data Factory to use them, you need to register an app (service principal) in AAD and give it the appropriate permissions in Power BI and to an Azure key vault.

I’m not the first to tackle this subject. Dave Ruijter has a great blog post with code and a step-by-step explanation of how to use Data Factory to refresh a Power BI dataset. I started with his code and added onto it. Before I jump into explaining my additions, let’s walk through the initial activities in the pipeline.

ADF pipeline that uses web activities to gets secrets from AKV, get an AAD auth token, and call the Power BI API to refresh a dataset. Then and Until activity and an If activity are executed.
Refresh Power BI Dataset Pipeline in Data Factory

Before you can use this pipeline, you must have:

  • an app registration in Azure AD with a secret
  • a key vault that contains the Tenant ID, Client ID of your app registration, and the secret from your app registration as separate secrets.
  • granted the data factory managed identity access to the keys in the key vault
  • allowed service principals to use the Power BI REST APIs in in the Power BI tenant settings
  • granted the service principal admin access to the workspace containing your dataset

For more information on these setup steps, read Dave’s post.

The pipeline contains several parameters that need to be populated for execution.

ADF pipeline parameters

The first seven parameters are related to the key vault. The last two are related to Power BI. You need to provide the name and version of each of the three secrets in the key vault. The KeyVaultDNSName should be https://mykeyvaultname.vault.azure.net/ (replace mykeyvaultname with the actual name of your key vault). You can get your Power BI workspace ID and dataset ID from the url when you navigate to your dataset settings.

The “Get TenantId from AKV” activity retrieves the tenant ID from the key vault. The “Get ClientId from AKV” retrieves the Client ID from the key vault. The “Get Secret from AKV” activity retrieves the app registration secret from the key vault. Once all three of these activities have completed, Data Factory executes the “Get AAD Token” activity, which retrieves an auth token so we can make a call to the Power BI API.

One thing to note is that this pipeline relies on a specified version of each key vault secret. If you always want to use the current version, you can delete the SecretVersion_TenantID, SecretVersion_SPClientID, and SecretVersion_SPSecret parameters. Then change the expression used in the URL property in each of the three web activities .

For example, the URL to get the tenant ID is currently:

@concat(pipeline().parameters.KeyVaultDNSName,'secrets/',pipeline().parameters.SecretName_TenantId,'/',pipeline().parameters.SecretVersion_TenantId,'?api-version=7.0')

To always refer to the current version, remove the slash and the reference to the SecretVersion_TenantID parameter so it looks like this:

@concat(pipeline().parameters.KeyVaultDNSName,'secrets/',pipeline().parameters.SecretName_TenantId,'?api-version=7.0')

The “Call Dataset Refresh” activity is where we make the call to the Power BI API. It is doing a POST to https://api.powerbi.com/v1.0/myorg/groups/{groupId}/datasets/{datasetId}/refreshes and passes the previously obtained auth token in the header.

This is where the original pipeline ends and my additions begin.

Getting the Refresh Status

When you call the Power BI API to execute the data refresh, it is an asynchronous call. This means that the ADF activity will show success if the call is made successfully rather than waiting for the refresh to complete successfully.

We have to add a polling pattern to periodically check on the status of the refresh until it is complete.

We start with an until activity. In the settings of the until loop, we set the expression so that the loop executes until the RefreshStatus variable is not equal to “Unknown”. (I added the RefreshStatus variable in my version of the pipeline with a default value of “Unknown”.) When a dataset is refreshing, “Unknown” is the status returned until it completes or fails.

ADF Until activity settings

Inside of the “Until Refresh Complete” activity are three inner activities.

ADF Until activity contents

The “Wait1” activity gives the dataset refresh a chance to execute before we check the status. I have it configured to 30 seconds, but you can change that to suit your needs. Next we get the status of the refresh.

This web activity does a GET to the same url we used to start the dataset refresh, but it adds a parameter on the end.

https://docs.microsoft.com/en-us/resGET https://api.powerbi.com/v1.0/myorg/groups/{groupId}/datasets/{datasetId}/refreshes?$top={$top}

The API doesn’t accept a request ID for the newly initiated refresh, so we get the last initiated refresh by setting top equal to 1 and assume that is the refresh for which we want the status.

The API provides a JSON response containing an array called value with a property called status.

In the “Set RefreshStatus” activity, we retrieve the status value from the previous activity and set the value of the RefreshStatus variable to that value.

Setting the value of the RefreshStatus variable in the ADF pipeline

We want the status value in the first object in the value array.

The until activity then checks the value of the RefreshStatus variable. If your dataset refresh is complete, it will have a status of “Completed”. If it failed, the status returned will be “Failed”.

The If activity checks the refresh status.

If activity expression in the ADF pipeline

If the refresh status is “Completed”, the pipeline execution is finished. If the pipeline activity isn’t “Completed”, then we can assume the refresh has failed. If the dataset refresh fails, we want the pipeline to fail.

There isn’t a built-in way to cause the pipeline to fail so we use a web activity to throw a bad request.

We do a POST to an invalid URL. This causes the activity to fail, which then causes the pipeline to fail.

Since this pipeline has no dependencies on datasets or linked services, you can just grab my code from GitHub and use it in your data factory.