Consulting, DCAC, Microsoft Technologies

Types of data consulting engagements and what you can expect from them

There are multiple ways organizations can engage with a data (DBA/analytics/data architect/ML/etc.) consultant. The type of engagement you choose affects the pace and deliverables of the project, and the response times and availability of the consultant.

Two women are seated at a desk looking at a laptop screen. One woman is pointing at an object on the screen while the other looks on.
Photo from Unsplash

What types of consulting engagements are common?

Consulting engagements can range from a few hours a week to several months of full-time work. The way they are structured and billed affects how consultants work with clients. Here are some types of engagements that you will commonly see in data and IT consulting. .

Office hours/as-needed advising: This engagement type usually involves advising rather than hands-on work. This type of engagement may be used to “try out” a consultant to determine if they have the required knowledge and how the client likes working with them. Or it may be “after-care” once a project has been completed, so a client has access to advice on how to keep a new solution up and running. In this type of engagement, a consultant’s time is either booked in a recurring meeting or as needed.

Time & materials/bucket of hours: In a time and materials (T&M) engagement, a client has purchased a certain number of hours with a consultant that may be used for advising or hands-on work. (The “materials” part is for other purchases necessary such as travel or equipment.) While the client may identify some deliverables or outcomes to work towards, they are paying for an amount of time rather than a specific deliverable or outcome. This bucket of hours is often useful for project after-care where the consultant will be hands-on. It is also useful for a collection of small tasks that a client needs a consultant to complete. Clients commonly use these hours over several partial days or weeks rather than engaging the consultant full-time.

Retainer: When clients engage consultants on a retainer basis, they are paying for them to be available for a certain number of hours per week/month/quarter. This is commonly used for recurring maintenance work. It can also be used when a client knows they will have enough development tasks to engage the consultant each month, but they don’t know exactly what those tasks will be. This type of engagement is usually a 3- to 12-month commitment.

Project-based time & materials: In this engagement type, there is a specific project a client needs a consultant to complete or assist in completing. There is a scope defined at the beginning of the project with some rough requirements and an estimated timeline. If the work takes less time than anticipated, the client only pays for the hours used. If the work takes more time (usually due to unforeseen issues or changing requirements), the client and consultant will have to agree to an extension.

Project-based fixed-fee: In a fixed-fee project, the client and the consultant are agreeing to an amount of money in return for specified deliverables. This involves much more up-front effort in requirements gathering and project estimation than a time & materials engagement. This is because the fee stays the same whether the consultant finishes in the estimated time or not. If the fixed-fee project costs $100,000 and it ends up using $105,000 worth of consulting time & materials, the client does not owe the consultant $5000 (unless they have violated a part of the agreement and agreed to the additional hours before they were worked). In this case, the consultant simply makes less profit.

Staff augmentation: In a staff aug agreement, the consultant and client agree that the consultant will work a set number of hours, usually close to full-time, per week in a specified position. There are no stated deliverables, just expectations of hours worked and skills to be used.

Risks and rewards

Office hours

Office hours is the lowest level of commitment for both clients and consultants as it usually involves a small number of hours. As a client, you aren’t stuck for long if you find the consultant isn’t a good fit.

But if you don’t agree to recurring meetings, you are taking a chance that the consultant will not be available on the day and time you need them. You must understand that the consultant has other clients, many of whom are paying more money for more time with the consultant, who is “squeezing you in” between tasks for other clients. I personally enjoy these types of engagements because they are easy to fit in my schedule and don’t usually require a lot of preparation before meetings.

An office-hours engagement is not appropriate for complex hands-on work. It can be good for design and architecture discussions, or for help solving a specific problem. I’ve had clients successfully use office hours for help with DAX measures in Power BI. I’ve had helpful white-boarding sessions during office hours. But when something looks like it will become a full project, or there are urgent troubleshooting needs with high complexity, I usually suggest that a different type of engagement would be more helpful.

Bucket of hours

The biggest risk with the bucket of hours is scheduling and availability. It can be helpful to agree that the hours must be used by a specific date. This helps the consultant plan for those hours in their schedule and ensure that revenue will be earned in the period expected, rather than leaving the agreement open indefinitely. But clients must manage the hours to ensure they are used before the expiration date.

This type of engagement is best when there is effective communication between the client and consultant and deadlines are somewhat flexible. Since the consultant isn’t engaged full-time, they will have other deadlines for other clients that they must work around. This sometimes requires a bit of patience from clients. Unless it is specified in the agreement, clients usually can’t expect consultants to be immediately available.

I’ve seen two ways this type of engagement is used successfully.

  1. Clients clearly communicate tasks and deadlines each week and consultants deliver them at the end of the week, until the engagement is over.
  2. Clients use the hours for non-urgent support, where work consists mostly of paired programming or troubleshooting a system with which the consultant is familiar. Clients give the consultant at least a day’s notice when scheduling a meeting. The consultant only has tasks that arise as action items from the programming/troubleshooting work sessions.

While there may be a theme to the work, no one has agreed to deliver specific outcomes in this type of engagement. If that is needed, a project-based engagement may be more suitable. Buckets of hours are good for short-term tasks and support.

Retainer

Retainers involve a bit more commitment, as they usually last for multiple months. Often, retainer hours are discounted compared to time & material hours because the stability and long-term relationship are valued by the consultant.

At DCAC, we often use retainers for “DBA-as-a-service” engagements, where clients need someone to perform patching and maintenance, monitor databases, tune queries, and perform backups and restores. They don’t know exactly how many hours each task will require each month, but they are sure they will need a consultant for the agreed upon number of hours.

Retainer hours are often “use them or lose them”. If clients don’t give the consultant work to do, the hours won’t roll over to the next month.

Because retainers usually involve part-time work, it’s important to set expectations about the consultant’s availability. If a client needs the consultant to be available immediately for urgent support matters, that should be written into the agreement (e.g., “The Consultant will respond to all support requests within one hour of receipt.”).

It’s more difficult to do retainer hours for development projects. If the consultant uses all the hours for the month before a project is completed, the client either needs to find more budget for the extra hours or wait until the next month to resume project tasks. If there are real project deadlines, waiting a week or more to reach the start of the next month is probably not feasible. If you need a consultant to focus work on a single project with tight deadlines, it’s better to have a project-based engagement.

If the consultant is assisting other project members, and it is certain that the retainer hours will be used each month, it is possible to have a retainer for development efforts. I have a client who has a retainer agreement right now that has me perform a variety of small tasks each month. Sometimes it’s Azure Data Factory development and support. Some months involve writing PowerShell for automation runbooks. Other months, I help them troubleshoot Power BI models and reports. We meet weekly to discuss the tasks and assistance needed and plan tasks to ensure that we use the allotted hours each month without going over. But this only works because my client trusts me and keeps the communication flowing.

Project-based time & materials

This is the most common type of engagement I see in business intelligence/analytics consulting. In this project type, the agreement specifies expected deliverables and estimated effort for high-level tasks. While detailed requirements might not be determined up front, it is important to specify assumptions along with the scope and deliverables. If something violates an assumption, it will likely affect the time and cost it will take to complete the project.

With any type of project-based work, it might be helpful to include a discovery phase at the beginning of the engagement to better understand project requirements, constraints, and risks. After the discovery phase, the project estimate and timelines can be updated to reflect any new information that was uncovered. While this won’t keep scope and requirements from changing mid-project, it helps people plan a bit more up front instead of “going in blind”.

As with the other time & materials engagements, clients only pay for the hours used. So, if the project was estimated to take 200 hours, but it is completed in 180 hours, the client pays for 180 hours.

Project-based time and materials engagements often have consultants working full-time on the project, but that is not always the case. It’s important to establish expectations. Clients and consultants should discuss and agree to deadlines and availability for meetings and working sessions.

Project-based fixed fee

Fixed fee projects are all about deliverables and outcomes. Because of this, they carry the most risk for consultants. They require the most detailed agreements as far as scope, constraints, and assumptions. It is particularly important to include this information in the agreement and have both parties acknowledge it. Then, if something changes, you can refer to the agreement when discussing scope/cost/timeline changes.

It’s important for clients to read and understand the scope and assumptions. While it may be a less technical executive that actually signs the agreement, a technical person who can competently review the scope and assumptions on behalf of the client should do so before the agreement is signed.

Because it is easy to underestimate the work needed to complete the deliverables, consultants often “pad” their estimate with more hours than what they think is necessary to cover any unexpected complications. Most people underestimate effort, so if the actual hours were to be different, this would usually end up in the client’s favor. But it’s not uncommon to see large amounts of hours in an estimate in order to cover the risk.

It’s important for the person estimating the project to consider time required for software installation and validating system access, project management, learning and implementing unfamiliar technologies, knowledge transfer, design reviews, and other tasks that don’t immediately come to mind when estimating. If there are less experienced people working on the project, that could increase the hours needed.

I have learned that it takes less time to complete a project if it’s mostly me and my team completing the work. If I have client team members working with me, I usually have to increase the hours required, simply because I don’t know their personalities and skills and I’m not used to working with them.

Due to the risk of underestimation, many consultants do not like to undertake large fixed fee projects. Sometimes it’s better to break up a larger fixed-fee project into smaller phases/projects to reduce the risk. This is especially true when a client and consultant have not previously worked together.

Staff Augmentation

I personally do not consider staff aug to be true consulting. Staff aug is basically becoming a non-FTE (full-time employee) team member. It is a valid way to be a DBA/BI/ML practitioner, and many consultants do some staff aug work at some point in their careers. But it doesn’t necessarily require the “consultant” in the relationship to be consultative, and they may or may not have more skills than those present on the client team. Some companies treat staff augmentation as just a “butt in a seat”. But it’s also possible to be a knowledgeable and consultative consultant who happens to be working with a client through a staff augmentation agreement.

For independent consultants, the risk for this type of engagement is that it is likely full-time or close to it, which makes it difficult to maintain business with other clients. Having only one main client can be risky if something goes wrong. For consulting firms, there is an opportunity cost in allocating a consultant full-time to a client, especially if the consultant has skills that other consultants do not have. Depending on the length of the agreement, there is a risk that the consultant will feel like their skills are stagnating or be unhappy until they can work with a different client. For clients, having temporary team members can decrease consistency and institutional knowledge as people are only around for a few months to a year.

Choose an engagement type that matches the work you need done

You are more likely to have a successful consulting engagement if you go into it knowing the common risks and rewards. Many problems I have seen in consulting have been due to poor communication or trying to do work that doesn’t fit the engagement type. Whether you are a consultant or a client, it’s important to speak up if you feel like an engagement is not going well. There is no way to fix it if you can’t have a conversation about it.

Whether you are the client or consultant, you can propose changes to agreements before they are signed. If you find something is missing or concerning, speak up about it so everyone feels good about the agreement that is being signed. Consulting engagements are more successful when clients and consultants can support each other rather than having an adversarial relationship.

Azure, Azure Data Factory, DCAC, Microsoft Technologies

Slides and Video from Building a Regret-free Foundation for your Data Factory Now Available

Last week, Kerry and I delivered a webinar with tips on how to set up your Data Factory. We discussed version control, deployment, naming conventions, parameterization, documentation, and more.

Here’s our agenda from the presentation.

Slide showing top regrets of data factory users: Poor resource organization in Azure
Lack of naming conventions
Inappropriate use of version control
Tedious, manual deployments
No/inconsistent key vault usage
Misunderstanding integration runtimes
Underutilizing parameterization
Lack of comments and documentation
No established pipeline design patterns
List of top regrets from Data Factory users that they wish they had understood from the beginning

If you missed the webinar, you can watch it online now. Just go to the DCAC website, fill in the required fields with your info, and the video will be shown.

If you’d like a copy of the slides, you can download the PDF here. There is a list of helpful links at the end that you may want to check out.

I hope you enjoyed our webinar. Leave me a comment if you have other experiences with ADF where a design or configuration choice you made in the beginning was difficult or tedious to fix later. Help other ADF developers avoid those mistakes.

Data Visualization, DCAC, Microsoft Technologies, Power BI

Power BI for Communication and Marketing

We often focus on deep analysis and insights generated by machine learning when we talk about Power BI these days because it’s super cool and very fancy. But I think it’s important to remember that you can also use Power BI for simple communication of data. As humans, we are hard wired to process visual information faster than text alone. And it’s often more efficient and concise to convey information in a visual rather than text. We can show in one image what it takes paragraphs to describe. Outside of historical analysis and operational reporting, we can use data visualizations as an engaging way to convey simple facts in a communication or marketing effort. Many people do this with infographics.

Visualization can make simple numbers and facts more memorable and engaging. I enjoy using Power BI for this marketing communication purpose, so I made a report about the people I work with at Denny Cherry & Associates Consulting. I knew most of them before I started working at DCAC, so I was already aware that they were exceptional. But now you can see it, too, with our new Power BI report located at https://www.dcac.com/about-us/learn-more-about-us.

Screenshot of the home page of our About DCAC Power BI report

Data is some people’s preferred language. We can use it to discuss specific points as well as broader trends and comparisons. I have a sticker on my laptop that says “Please Talk Data To Me” for a reason. I can make sweeping statements about how my team is full of accomplished consultants, speakers, and authors. But it is much more impactful when you see the numbers: 27 Microsoft MVP awards and 8 VMWare VExpert awards over the years, a combined 113 years of experience with SQL Server, 86% of us having been a user group leader.

My Surface Pro tablet with a "Please Talk Data To Me" sticker on it
My Surface Pro tablet with a “Please Talk Data To Me” sticker on it

And because Power BI is interactive, you get to interact with my report, choosing which categories you want to learn more about. There is also a little guessing game built into the report. (Edit: Although there is currently a Power BI bug that I had to work around, so it’s helpful to know that the answers are limited to values 1-2500). But that is a blog post for another time, and for now you get a hint as to possible answers.) If I had just told you these stats, you might have listened, but you probably would have tuned out.

Our team even had fun browsing the report, seeing our collective achievements, and guessing who provided which answers.

Building the report

It was quick and easy to use Microsoft Forms to gather the data and then dump the responses to Excel. From there, I made a quick Power BI data model and put together my visualizations. The report is embedded on our website using a Publish to Web link. A few people have said they would like to do something similar for their organizations, so I’ll offer a couple of tips:

  1. Make sure the questions you ask aren’t violating any HR rules, and that employees know that some or all questions are optional. For example, I asked personal questions in the Fun Facts section about number of children employees have. This question was optional and our team felt comfortable answering it.
  2. Try to group your questions into categories so you can easily the separate sections and pages like I did. Then you can associate a color with each category.

If you do make something similar, I’d love to see it. Tweet me a link or screenshot at @mmarie.