Sub-Optimal Optimization

In an era when memes were not a thing, “wax on, wax off” was for me my first meme. All the kids had seen The Karate Kid and for weeks all you heard in school playgrounds were mini Karate Kid reenactments.

The idea of “wax on, wax off” was to develop mastery. By doing boring, repetitive chores, muscle memory developed. When Daniel-san finally reaches his breaking point, Miyagi tests Daniel-san, demonstrating that all the chores were core building blocks towards gaining essential karate skills.

This was my first lesson in optimization. Through a continuous cycle of repetition and learning from mistakes, you could become more efficient and effective. It reminds me of the egg sushi scene from Jiro Dreams of Sushi. It seems like borderline insanity, but in order to truly excel, it takes an immense amount of dedication.

It is common to hear enterprises talk about becoming fast like startups. There is a lot to admire about startup speed and execution ability when compared with how slow it can be to get things done in a large company. Lumbering and bureaucratic are words that are most often used when I speak to IT leaders in these organizations.

Tanya Reilly recently shared a story about organizational friction. We can all probably relate to what Tanya describes. I was helping out a team with similar communications challenges. Engineers needed information from other teams on API access. Over 50% of these requests never got a response, requiring escalation through layers of management until someone was nudged into responding. Requests often took weeks to resolve, causing work delays and missed project delivery dates.

It is therefore understandable that enterprises would want to adopt startup practices. Putting startup practices into a large organization however rarely results in change due to the scaling effect of maturing organizations. Successful companies undergo four distinct phases of growth:

  1. Seed phase — company launched with a few people to go from idea to product
  2. Startup phase — grows to several dozen people to identify product-market fit
  3. Scale phase — accelerates growth to achieve competitive positioning
  4. Steady phase — achieves market acceptance and top line growth slows

Startups usually do not have more than a hundred or so people. Communication and collaboration is rarely an issue. Everyone knows each other and the company is well aligned towards a singular purpose and set of values. As Peter Varhol shared, everyone understands the Big Picture and how their work aligns to the mission.

In the scale phase, the core value is growth. Because of the rush to expand, things break often, knowledge gets lost, recruiting is chaotic. It is both exciting yet frightening at the same time, but the company is still nimble enough to recover and move ahead. Values and purpose take a back seat to the growth engine.

As a company matures and takes a leading position in the market, growth slows. The breakneck pace of scaling is not necessary. In order to continue to maintain margins, companies begin to fix the manual, kludgy things done to support growth. The focus is to optimize to drive greater efficiency while lowering costs. Non-core work is outsourced, teams are restructured, and processes are created to automate and streamline work.

I call this progression “hyper focus, hyper scaling, hyper optimization”. The startup focuses on aligning everything towards purpose and values. The scaleup pours all resources into driving growth. The established enterprise needs to drive predictability and stability.

Optimization should account for issues of cross-team communication and collaboration. In practice, optimizing across teams and functions is difficult as there is often little incentive to collaborate. Instead optimization occurs within teams and functions, which leads to teams competing against each other for resources instead of cooperating on shared goals.

This has massive implications for software delivery performance. DevOps and DevSecOps have helped to raise awareness of the dysfunction and to address some of the breaks in collaboration across functional teams. Yet it is still common for an engineer at a startup to get an answer to an API question in minutes while another engineer in an enterprise has to wait a day, a week, or even longer to get a response.

This is the invisible work that undermines productivity. Hunting for answers, asking questions, searching documentation, interrupting colleagues, and chasing approvals. When an engineer gets stuck, waiting on others leads to switching tasks and adds to the ever growing queue of neglected work. The engineer should in theory be able to do some of this resolving, as Tanya points out:

“A key engineering skill is noticing what’s blocked, and figuring out why, following a thread of hints from person to person until you can see what’s not moving.”

This is what is often considered “glue work”, the engineering tasks that do not involve code or seem technical but are absolutely essential to the success of any software project. This includes documentation, team coordination, onboarding new team members, and tracking down answers when you are stuck.

Normally this would be the responsibility of an engineering manager. Many managers however are themselves overloaded with work or may not have the patience or skills to wade through organizational inertia.

How do you fight against the silos then? At the macro level, it comes down to challenging core beliefs. Most beliefs in an organization tend to be self-limiting, such as how fast it is to build new product features or how quickly code can be deployed into production. At the tactical level, implementing cultural hacks can provide quick wins without huge resistance.

What are cultural hacks? They are small changes to something in an organization that can have an out-sized positive impact. In the prior example where requests were not being responded to, the team created a chatroom channel just for engineering managers. One manager would triage and follow-up on requests in the channel for one month, then the role would rotate to another manager, and so on. This accomplished three things:

  1. Requests were resolved faster, shortening project delivery timelines
  2. Backlog of neglected work was reduced freeing up time for other work
  3. Team collaboration, motivation, and most importantly trust, increased

This may seem like a super obvious solution. Dysfunction and the hyper optimized nature of enterprises however squashes good ideas before they even have a chance. Teams become resigned to self-limiting beliefs about what is possible. However, if you can begin to identify and implement quick cultural hacks, you have a chance at nudging the culture in a positive direction.

What are some examples of processes that stifled your work or your team’s work? How did you go about addressing the process change and improving the situation?

Episode #8 — Jason Milkins on Building a High-Performing Culture of Engineering Excellence

We welcomed Jason Milkins to the show last week to share his experiences on building engineering teams & culture in enterprises from the ground up.

Check out past podcast episodes on Apple PodcastsGoogle Podcast, or wherever you listen to your favorite podcasts. Please like and subscribe 😁