For a software company, the Product Engineering organisation constitutes the core engine of the business. If a Product Engineering organisation cannot design, develop, test and deploy software features at scale, users quickly stop using the product, turn to the competition, and the company is no longer relevant. That’s why startup CEOs become increasingly stressed when their Product Engineering organisation slows down, or features decrease in quality. In the age of product-led growth, you can add as many salespeople as you want; if your product is inferior, you will lose. You can’t brute-(sales)force your way to success anymore.
That’s why I’ve seen an increasing number of startup CEOs request an audit of their Product Engineering organisation to assess its strengths and weaknesses and ensure that the company is positioned to meet its significant milestones, including successfully raising its next round of funding. Unlike due diligence, which investors typically commission to assess a potential investment opportunity, an audit commissioned by a CEO delves deeper into the operational aspects of the organisation. The idea here is less about identifying risks and more about identifying bottlenecks and recommending solutions to remove them.
As I’ve performed many of these audits over the past years, I thought it could be interesting to share my modus operandi. There are a few caveats, though, to my process. First, when auditing a startup, there’s often a lack of tangible metrics for a quantitative audit methodology. The approach described here is mainly qualitative. Second, auditing a startup can carry a high emotional weight, as it involves closely evaluating the contributions of founders and early team members who have invested considerable personal time and effort in building the organisation.
The Product Engineering system
My approach to auditing a startup's Product Engineering is grounded in systems thinking, viewing it as a comprehensive system with ideas as inputs and features and revenue as outputs.
This perspective is underpinned by a fundamental belief: an efficiently functioning Product Engineering system should excel in converting minimal effort into maximum revenue (aka margin). To do so, the system should select the best ideas and rapidly transform them into high-quality features that end-users will adopt.
The core of this system's success lies in its responsiveness to user engagement. When end-users actively utilise the new features, it indicates a successful alignment with their needs, leading to contract renewals, purchases of additional features, and referrals. If these features fail to resonate—due to irrelevance, poor user experience, or technical flaws—there's a risk of increased user frustration, which can escalate into churn or negatively impact referrals, thereby driving acquisition costs higher. Therefore, an efficient Product Engineering system must be proactive and agile, capable of swiftly identifying underperforming features and iterating them until they meet user satisfaction.
My audit methodology revolves around evaluating this efficiency. I scrutinise how well a company's Product Engineering system facilitates this cycle of idea selection, feature development, user adoption, and iterative improvement, ultimately determining its impact on its bottom line and market position. This approach measures current performance and gauges the system's potential for sustainable growth and adaptation in a rapidly evolving market landscape.
Part 1 - Organisational structure
Understanding the organisational structure of a startup's Product Engineering team is crucial, as it often reflects and influences the product's architecture, a phenomenon encapsulated by Conway's Law. According to Conway, a system mirrors the communication structures of the organisations that create it, implying that the organisational setup can significantly impact the product's efficiency and scalability. However, startups often grow rapidly without adequately considering their organisational structure. This lack of strategic planning can lead to inefficiencies and scaling challenges, as the ad hoc structures they stumble upon may not support sustainable growth. In my audit, I prioritise understanding how well the startup has thought through its organisational design. Here are the key aspects I tend to examine:
Team topology
When auditing team topologies, distinguishing between feature teams and component teams is essential. Feature teams are cross-functional, focusing on delivering complete product features end-to-end, fostering agility and user-centric development. In contrast, component teams specialise in specific system parts, emphasising deep technical expertise. Many startups start with component teams, mainly because developers like to divide teams into front-end and back-end. It is crucial to assess which model aligns better with the startup's goals and product strategy, as it impacts the speed and quality of feature development and overall product cohesion.
Managerial structure
In auditing the managerial structure, it’s essential to quickly identify if there are real managers or if those are individual contributors who also supervise team members. It’s common for startups to “reward” the best individual contributors by making them managers. But management is not a promotion; it’s a different job altogether.
Optimal configurations include full-time Managers supporting 6-8 Individual Contributors or 3-5 Leads. These managers should only have management responsibilities and make little to no individual contributions. Leads should supervise 1-4 Individual Contributors. Ideal team sizes are 6-8 members. A growth strategy involves expanding a team to 8-10 before splitting it into two smaller teams of 4-5, ensuring robustness without creating empty teams lacking immediate purpose or direction.
Senior-junior ratio
Auditing the senior-to-junior ratio involves evaluating the balance between experienced and less experienced Individual Contributors. While junior Engineers or Product Managers bring fresh perspectives and adaptability to the company's culture, an ideal ratio for Series A-C startups is approximately 3:1 in favour of senior team members.
Product Manager-Engineer ratio
Auditing the Product Manager to Engineering ratio is critical. If there are too many Product Managers, they will spend more time working on features that engineers won't be able to build because they're already busy building the previously specified features, creating frustration in the system. This is Parkinson's Law, which suggests that work expands to fill the time available for its completion. The feature engine will dry up if there aren’t enough Product Managers. An ideal ratio is typically one Product Manager for every five to eight engineers. Maintaining this ratio ensures effective collaboration and prevents workload bottlenecks, aligning feature planning with engineering capacity.
Part 2 - Product Engineering Value Chain
Auditing the Product Engineering Value Chain in a startup is akin to examining a supply chain, where the flow of work transitions from Idea to Specification, Development, Testing, and finally, Pushing to Production.
This process demands a critical evaluation to identify constraints and bottlenecks, drawing insights from Goldratt's Theory of Constraints. The goal is to determine the weakest links in this chain that restrict the system's overall throughput and to implement strategic improvements.
A primary focus area is the idea generation and selection process. Ideas are abundant and inexpensive to conceive but expensive to build. Therefore, it's imperative to have a rigorous process for triaging these ideas before they even reach the engineering analysis stage. This involves scrutinising who identifies these ideas and how they are added to the roadmap. A lack of selectivity at this stage can lead to resource drain and deviation from core business objectives.
The involvement of engineers in the specification phase is another critical area. Due to technical limitations or misunderstandings, excluding engineers from this phase often results in excessive back-and-forth between product management and engineering. Their early involvement ensures that specifications are realistic, feasible, and aligned with technical capabilities, reducing rework and delays.
Another aspect is the architectural design, which dictates whether multiple teams can work in parallel or must work sequentially. A well-designed architecture that supports parallel development can significantly accelerate the product development cycle and enhance team productivity. In contrast, a sequential requirement can become a significant bottleneck, especially in a rapidly scaling startup environment.
Quality Assurance processes also require thorough evaluation. An ideal scenario is where developers own their QA, fostering a sense of responsibility for the end product and avoiding the mindset of quality being someone else’s job. Relying solely on a dedicated QA team can lead to engineers disengaging from quality aspects. This shift in ownership ensures that quality is embedded in every development process rather than being an afterthought.
Lastly, the existence of a feedback loop post-production is crucial. It's not enough to push features to production; there needs to be a mechanism where product managers and engineers monitor and measure user engagement with new features.
This feedback is vital for making informed decisions about further enhancements or iterations. Without this loop, there's a risk of continued development of features that don't meet user needs or add value, leading to wasted resources and missed opportunities for improvement.
Part 3 - Interactions between Product Engineering and the other departments
The third and final part of the audit focuses on the interactions between Product Engineering and other departments within a startup. This part of the audit usually relies on “sentiment analysis” from team member’s interviews. While these interviews are less than perfect, getting an objective overview of what’s going on within the product engineering organisation and their interactions with other teams is possible. Ideally, it’s better to ask open questions and let the interviewees talk about their job, what works and what could work better.
While CEOs often express concerns about the speed of the product engineering system, an in-depth audit can reveal underlying issues beyond pace. Key areas of concern include trust, quality, and velocity, each affecting how product engineering collaborates with other teams:
Velocity
While the engineering team might be adept at swiftly addressing tickets and ad-hoc requests, this can sometimes be at the expense of core feature development. Such an imbalance can delay strategic initiatives, affecting the company's ability to meet long-term goals and respond to market demands.
Trust
If communication between tech and other departments is scrambled, it creates a lack of trust in the product engineering organisation. The product engineering organisation can become a "black box," inaccessible and misunderstood by other stakeholders, impeding collaborative efforts and shared problem-solving. This lack of transparency and mutual understanding hinders the overall efficiency and effectiveness of the organisation.
Quality
Defects in specifications, processes, and deployments don't just affect the tech team; they ripple across multiple departments, impacting overall productivity and the company's ability to deliver value to customers. This can lead to frustration, reduced cross-departmental cooperation, and a tarnished reputation internally and externally.
Team sentiment
Another crucial aspect to investigate is the self-perception of the product engineering team members. A team that is too content might indicate a lack of challenging work or lowered expectations. This balance is critical for fostering an environment encouraging innovation, ambition, and continuous growth. An effective audit should assess whether the product engineering team is not just satisfied but also motivated and pushed to reach their full potential, contributing to the overall progress and dynamism of the startup.
Conclusion
In conclusion, the role of a Product Engineering organisation in a software company is paramount, acting as the central engine driving the business. Any faltering pace or decline in feature quality can lead users to abandon the product, potentially rendering the company irrelevant in a highly competitive market. This underscores why startup CEOs are increasingly vigilant about the performance of their Product Engineering teams, often opting for an audit to identify and rectify any inefficiencies.
For those looking to implement my audit methodology internally, it offers a comprehensive approach to understanding and enhancing your Product Engineering operations. While it's true that this methodology leans more towards qualitative analysis, it is specifically designed to resonate with the unique nuances of startup environments. It considers not just the tangible metrics but also the intangible elements like the emotional investment of founders and the often anecdotal nature of data in such settings.
I genuinely hope you will find this methodology useful for your organisation!