Hard Conversations being a Technical Leader (Part 1): Stakeholders

Wed 19 October 2022

Conversations are Hard

I am an introvert. Many people that know me professionally will be surprised at this. I learned in late high school that extroverts in our society (and in most western cultures) get a lot more offered to them in life than introverts. This lead me down a path of learning how extroverts act, and mimicking them until I was able to run an extroverted facade constantly at work. (Thanks Tyler Weldon of Pahrump, NV fame).

This also meant that conversations were (and still are) a particularly difficult area for me. For a large part of my career, I maintained that conversations about hard subjects like engineering, infrastructure, operations, and maintenance should be governed by logic. I often took a role model of Scotty and Geordi LaForge from Star Trek around this. This was a big mistake for me.

In this series, we'll talk about hard conversations I've had, and how I've handled them, good and bad.

The truth about hard subjects

The truth around hard subjects is that most hard subjects, which we like to think of as being objective, really aren't. Everyone at the table has a different point of view on the subject, and this is crucial. Everyone can disagree and be correct! That's very difficult for many people, and was something I really didn't learn until after my first technical leadership role.

I'll give you an example. Say that you've been asked to create an application and host it for a customer. You have to work with several different teams. These teams often range from infrastructure (compute and networking), security, application development, operations, and management.

So, the application development team wants to make the most efficient application, and they build something, but its terribly inefficient when it comes to network utilization. Maybe it isn't as secure as it should be because that would hurt the user experience, so some security details have been relaxed. It isn't efficient in terms of database and storage either. Plus, let's face it, most of the time operations is forgotten about until the rubber meets the road, and this time is no different, and the administration of the app is a nightmare.

You have the best application from a compute efficiency perspective, and you go into the deployment meeting. The cloud engineering team has concerns, the network alone is going to be extremely expensive given your layout. Security has also put their foot down, and user experience or not, security is the utmost importance to them. Operations can see the late nights ahead attempting to reconfigure the application out of corner cases, and voices their opinion. Finally, management looking at all of this, suggests that the development team go back and handle the concerns.

All of this should be logical, and each piece is, but each piece is optimized differently. In my early career, I would dig in, and argue that my application was correct, and the other concerns were secondary to being efficient from a speed perspective.

Who is right here? Well... unfortunately everyone.

On top of all this, the security representative has just come back from a meeting where the company is going to slash his budget because they can't use metrics to determine value (common in security circles). That means that the disagreement with you is two fold, and may cause doubling down and overstating cases to get a sorely needed "win".

How do navigate this?

Difficult Conversations

A few tools have been extremely helpful to me in these circumstances. Without them, I probably would have continued to run into walls and have no way around them. They opened my eyes to conversations of hard subjects, but even more so, they helped me through much more difficult conversations around soft subjects. Those soft subjects are how business is done, and until you come to that conclusion, your life as an engineer will be, and stay, difficult.

The training I had in the 7 Habits of Highly Effective People really helped me process this. One habit was crucial here, "Seek first to understand, then to be understood". That is life changing once you think about it. In the example above, once you understand everyone's points, you can make your own in the context of the greater conversation. You realize everyone is really trying to do the best they can, and help, and it becomes much easier to talk through the issues.

Another book, Crucial Conversations, taught me how to maintain safety in these kinds of scenarios. Not only do I need to add my own thoughts and opinions to the "shared pool of meaning", but I also need to navigate the people aspects. The security rep isn't happy about something else, and may feel attacked by suggesting that some of his concerns are secondary to others. The book really goes through great lengths to show how you can restore safety by stepping out of the content of the conversation, and re-establish cognitive safety and trust.

Now that I was equipped, did the meetings go well?

Sometimes they did, when I was on the ball and understood my own emotional state, and that others really weren't trying to cause me to fail. Sometimes they didn't.

Below are some examples from my leadership career. Some were when I was the technical leader of a team, and had the responsibility to my teammates to represent their work well. Some were from when I was a Line Manager, and the conversations were more difficult because they didn't solely deal with objective subjects, but with people problems.

Difficult Stakeholders

Not too long ago I had the chance to work with a team at a large enterprise. The team was in charge of building an application that would process tens of millions of compliance records on a daily basis, and compare them to millions of rules to make sure systems, networks, and applications were compliant to contractual standards. This is a big task, and when I joined, the team had already been hard at work for over 3 years. The primary stakeholders were teams of people who had to address the compliance issues, quickly, usually in less than three days. My team, and the stakeholders, and all been pressed hard for results.

On top of all of that, my teams original leadership (a director), may not have been well intentioned. The director had a rule, that if the customer accepted the work of the team, then it was right and written in stone. Any issue the stakeholder's had with this work was considered a "new feature" request, and wouldn't be handled until later. Sometimes much later.

This lead to huge amounts of rework on the part of our customers. The worst part, it built a "we have to find every fault in the system every time" mentality, which meant getting code to production took months.

I joined the team to spearhead a new version of the application, using a completely new technology stack. This terrified the customers, because a new stack meant everything had to be write the first time, every time. What should have been a quick move to production, took a lot longer, and there were many heated conversations.

The tact I took here was to repair the broken trust we had with our stakeholders. After all, we were building this tool largely for them, and they had to be able to do their job. I'm big on reducing suffering, and if the new system I created only added more suffering, that was definitely a no-go. But how do you repair a bridge which had been burned, broken down, and barely existed without having 3 levels of management in the room?

The first thing we had to do was to change the paradigm. A great coworker, and inspiring teammate, named Rick Jones helped me with this greatly. We went to work making sure that the customer understood that issues with the work could be raised and addressed immediately, and not have to wait for the next delivery cycle. Sounds pretty normal, but for a dev team and stakeholders that didn't work like this, it took many weeks of conversations, one on one calls, and private communications to make them feel comfortable.

The new stack was ready to go within a couple months of development (the part we were tackling first, that is). The conversations around deployment and timelines started, and it was a slog to get through. At every turn the customer withheld information so they could have "Gotcha" moments when we promoted our code. The first shoe fell, and the customer, in a meeting, pointed out a mistake and said the whole project was suspect.

Now, was it a bug? Sure, it was. Could it have been avoided if the customer told us everything, absolutely. What did we do? We took it on the chin.

I asked for details, and dove right in. The customer wasn't used to having a spotlight shined so brightly on their concerns, but the attitude we took was a partner trying to help.

They said it was broken. That meant it was broken. No arguing about SLAs, no fighting back with "you didn't tell us that". It was simple, although it felt really crappy.

I worked with their senior analyst, got what we needed, and turned around a fix that made it to qual in less than 4 hours (usually a fix would take a month). We reran the days data, and handed back the data for review the same day. We even went a step further.

For the duration of the deployment proceedings, we build an analysis that would scan for the bug in all of the data, and report if the flaw reared its head again. We added it to our unit tests to make sure it never came back. After a week of running the data every day in qual, and getting those reports hand delivered, the customer started to realize that we meant business when it came to giving them something they could use.

It didn't fix everything. More conversations, harder than the first kicked up. At every turn, when they threw daggers at us, we caught them and made sure they landed. Eventually, after months of this, the customers started to realize we were a different team running in a different paradigm. Our goal was their goal, and the conversations became easier.

I left that team due to a once in a lifetime opportunity that came up. I miss my old teammates, and play D&D with many of them regularly still today! I have had the chance to see their relationship with their customer get better and better. Is it all good now? No, the customers still have issues they are working through, but their senior staff now believe the dev team is aligned with them, and the teams are generally working toward a common goal together.

Takeaways (TL; DR)

The take away here is that while your work is being judged by the people you are serving, it is difficult to keep in mind a couple things.

1. They may not know as much as you do on the technical side, and its your job to help. This does not mean lording the fact you know more over them, but mentoring them the same way you were mentored early in your career. Using their terms and definitions, not yours. Making it central to their understanding to bridge the gap.

2. They may not be nice, but that doesn't mean they are wrong. Our customers said some downright hurtful things, bordering on unprofessional, but they were hurt in the past. Understanding that helped, but it didn't make it easy. You have to step away from the content, create safety, and then get back into the content to understand the problem. That is the only way through.

3. Sometimes, you're wrong. Yes, I said it. You may have implemented exactly what was asked for, but that doesn't mean it is correct. Customer's have a hard time expressing themselves, they don't always get it right. It's ok if that happens. Just make sure you have a better understanding and try again.

4. Abuse is not ok. More than one personal conversation was had with difficult people on both sides to make sure that the conversations didn't turn abusive. It took a while, but was well worth it.

Final Thoughts

These types of conversations come up far more frequently than I would like to admit. When you become a leader in technology, this is is just one kind of issue you'll likely run into. Sometimes you'll have great customers that know what they want, other times you'll get nice customers that can't communicate. Every so often you'll have customers who are just mean. In all of these cases, its important you remember you control how you react.

The first of the 7 habits applies here. Be proactive. You control your happiness, and how you react to situations. A quote from a holocaust concentration camp survivor is permanently etched on my main whiteboard, and I see it every day:

The last of the human freedoms — to choose one’s attitude in any given set of circumstances, to choose one’s own way. - Viktor Frankl

In hard conversations, whether your team is being slashed, your application is being moth balled, or your getting yelled at by an unruly customer, you decide how you act. In the end, this is what usually determines your success. You can be the best engineer or manager in the world, but if you can't respond in a coherent, non-aggressive way when under fire, you wont go very far.

In the next article, we'll talk about under performance of teammates, how that can impact your team, your workload, and even your reputation. More importantly, I'll give an example and some tips about how to have that conversation.