Internal developer portals: a springboard to platform engineering success?
Platform Engineering Primer, Part 4. The fourth in a series of articles exploring how developer platforms benefit modern businesses.
Join me in examining the intricacies of platform engineering. Let's take it apart and find the effective strategies, the most practical ways of working and the technologies that fit your needs and circumstances. Above all, let’s explore how internal developer platforms are the gateway to modern enterprise success.
Let's face it, expecting developers to be experts in everything is unrealistic. That's why many engineering leaders envision a single, easy-to-use platform where everything works smoothly. It may sound like a tall order, but it's all about abstracting complex things so that developers can focus on creating quality software. From where I stand, it makes a lot of sense. I hope you'll join me as we chart a course towards the ultimate goal of platform engineering: a central hub for monitoring and managing the entire software development lifecycle.
These days, we are seeing a huge surge of interest in the adoption of internal developer portals (IDP) as a solution to improve the developer experience, scale existing knowledge and facilitate leadership in organisations. One of the most common pilot projects today is the introduction of an internal developer portal.
However, many organisations find it difficult to align the different dimensions, create the right contract between the platform and the portal, and then use that contract to create a successful portal. The resulting chaos and wasted effort are often enormous. In contrast, a portal, if done well, acts as a gateway through which your platform is embraced by the majority of its intended users.
As I mentioned in my first article, my goal has always been to broaden my horizons in the area of platform engineering and to glean wisdom from some of the brilliant minds in the industry. This article was inspired by a conversation I had with Zohar Einy, the CEO of Port, who's as close to the subject of this article as you can get.
As the person behind Port, the internal developer portals (IDP) product, he shared its most frequent use cases. He also discussed what companies want to achieve and the common pitfalls. Reading between the lines, you can discover all of this, along with my perspective.
"Everyone has a different and individual way of working, so we shouldn't provide a one-size-fits-all experience. Instead, we provide a toolset that allows you to create the experience that works for you and your organisation." Zohar Einy, CEO of Port
In this article, we’ll explore:
The potential value of adopting an internal developer portal in your organisation.
How companies approached building developer portals before the current IDP era.
The way internal developer portals (IDP) work from an architectural point of view.
What comes first, the platform or the portal?
Reduce workload by integrating your portal efforts and ongoing platform development.
The pitfalls to avoid when building or adopting IDP products.
What drives organisations to invest in developer portals?
We, the shapers of engineering effort in digitally intensive organisations, are now well educated. We know how important it is to make developers' jobs easier and how that can impact business results (Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations). The aim is to streamline tasks through code reuse, adherence to technical standards and the adoption of design guidelines. That's all well and good, but you know what? Developers still sometimes struggle to put it all together. This means that our ultimate goal may still be missed. Every decision on how developers dedicate their time in a non-functional world comes with trade-offs.
How can we help our people succeed? We can focus on engineering enablement as a strategy and scale our existing knowledge. Crucially, we also need to bring senior leadership closer to the work happening on the ground, so that they're aware of the challenges and friction points, and the opportunities to empower developers.
An internal developer portal is one of the keys to engineering enablement in organisations. It helps us make the shift from top-down to developer-centric. Whenever we need to connect the dots, a developer portal acts as the 'connective tissue'. It allows us to facilitate many processes, including:
Compliance and governance. All businesses want development teams to adhere to organisational standards. An example: coordinate code freezes from a single point.
Streamline feature deployment and service scaffolding. Of course, consistency means something different in each organisation. Portals give you the flexibility to create an experience that works for your organisation.
Simplify cloud and infrastructure as code. Teams often struggle in this area. Internal developer portals facilitate the inner source approach to eliminate duplication and encourage collaboration.
Improve incident management and on-call processes. When microservices impact each other, it's a challenge to triage incidents on time and escalate them to the right team. Having clarity about how systems interact and who owns what is essential.
Manage permissions to external or internal systems which usually end up as tickets. Portals allow you to manage in a more self-service way. Provide an approval flow or manage just-in-time access as needed.
Support other self-service capabilities, integration with existing CI/CD systems, triggering of custom actions or management of feature toggles.
Additional visibility into various aspects such as service catalogue, API versioning, team topologies or costs. An internal developer portal allows organisations to create scorecards or track productivity metrics such as DORA.
Centralised documentation so that developers don't have to search for information from multiple, unfamiliar sources.
This is by no means an exhaustive list. The number of processes routinely offered as self-service via internal developer portals will grow as the organisation and its developers grow.
What we used before the IDP era
The field has developed rapidly in recent years. Let's take a look back to see how far we've come.
Before the IDP era, organisations built custom web portals to abstract common operations such as onboarding, access management or provisioning new cloud accounts. Typically, each technology department created its version of the portal. And it was all built primarily to control rather than enable developers. I'm pretty sure you've seen something similar in your organisation. In most cases, the user experience wasn't great, it was slow and at the end of the day, it was just an abstraction of all the spaghetti of disparate systems and manual work underneath.
On top of that, we often heard that developers didn't want another "clicky thing", and that they preferred code-based interfaces. So it was hard to find the balance and make everyone happy. Fortunately, modern IDP solutions offer both intuitive GUIs and code-based interfaces such as SDKs, direct API access or CLI tools that don't derail efforts to improve automation across the board.
Put simply, the primary shortcomings of this legacy way of doing things were:
Lack of an API-first mindset when building these portals. It became increasingly difficult to streamline multiple processes together due to different API formats or the lack of an API format altogether.
Infrastructure engineers are not typically front-end experts. The result was often poorly designed web portals with non-intuitive interfaces.
Limited extensibility. Adding features and integrations was difficult and time-consuming. The high level of coupling and technical constraints significantly increased lead times and hindered overall progress.
Duplication of effort with multiple teams typically building their portals in slightly different ways.
Let's take a look at how this legacy way of doing things fits into a maturity model:
I see the move to powerful IDP solutions such as Backstage or Port as a logical evolution in response to the increasing complexity, integration requirements and efficiency demands of today's software development.
They aggregate information but often fail to integrate workflows or reduce developer friction. Developers still face the challenge of context-switching between an IDP and its underlying tools, which slows their workflow. That’s why we need to build a solid contract between internal developer portals and platforms which is outlined later in this article.
Do you think it draws a complete picture of how things have evolved? Have I overlooked something important?
How they work: developer portal architecture
Internal developer portals typically support a wide range of capabilities that you tailor to your operating model, tech stack, common practices and team topologies. Looking at this from a high-level perspective, the most notable areas are:
The core of the design is always around self-service. This is the central point where other internal or external capabilities interact. Depending on the use case it might be accessible via a simple web interface, direct API call or command line interface tooling. This mechanism is also responsible for the integration with your internal platform which I cover in the next section.
Besides that, IDP solutions typically provide:
Software catalog - a central place to store relevant information about software inventory, team structure or API schema and versioning.
Blueprints - which serve as a starting point for scaffolding an application of a piece of infrastructure.
Approval flows - provide a consistent and transparent way of keeping track and approving access to various cloud resources or external systems within the organisation.
Custom plugins - enable portal users to extend existing capabilities by implementing their integrations or workflows. It might be a low/no-code approach or a dedicated programming framework to use.
Insights - sometimes called scorecards provide real-time data about the current state of the services, their compliance status or other types of metrics.
To give you a more tangible example of how internal developer portals are built, let’s have a look at Port's high-level architecture. It’s a typical Software as a Service (SaaS) architecture with additional components which can be deployed in the customer environment. The product leverages AWS-managed services, as software is written using common programming languages such as JavaScript (Typescript and React), Python and Go.
AWS DocumentDB is used for storing the configuration of integrations such as a user's Ocean integrations and his dynamic webhook configurations.
AWS RDS is used primarily for RBAC information, the configured roles, and their permissions.
AWS Managed Streaming for Kafka is used to provide dedicated Kafka topics to users, which they can use to listen for changes in integration configurations and receive notifications about new self-service action requests.
All integrations written using the Ocean open-source integration framework are written in Python, some integrations such as Port's K8s exporter are written in Go, and the rest of the stack is written with NodeJS, Typescript and React.
The connectivity between SaaS and customer environments uses Rest API with JWT auth.
Port has an open-source integration framework called Ocean, all new integrations that Port publishes are open-source and use the Ocean framework as their core, you can learn more about Ocean here.
As you can see, building a portal is not a trivial task, it involves several components for data storage, streaming and even the need to create a custom framework to let users extend existing portal capabilities.
Platform or portal first?
Think of internal developer portals and platforms as two separate systems, each with its own life and different teams working on it. Deciding which one to implement first has significant implications for your platform strategy.
By introducing an internal developer portal first, you simply decide to start building the abstraction. You set the stage for self-service and automation. It gives you a lot of flexibility to model interactions and limits future problems associated with modifying some of the underlying solutions when they need an update to stay fit for purpose.
But it doesn't clean up the mess underneath. If you have some flaky managed services or processes that are mainly ticket-driven, the portal won't solve those problems.
In comparison, the introduction of a new platform is likely to require developers to adapt to a new operating model and migrate their services to take full advantage of the platform. However, developer portals offer the flexibility for developers to select only the processes they want to access in a self-service manner.
Let’s take a look at a simple comparison of what each approach provides.
What's my opinion? In most cases, building and operating a stable, scalable and efficient platform infrastructure is a key starting point. This unlocks the basic capabilities that can later be exposed in a self-service manner via an internal developer portal. Starting with the platform creates an interesting opportunity to partner with developers and observe how they actually do their work. So we can understand friction points, eliminate them and bring more standardisation across the stack, and then build a portal on top of that knowledge. Otherwise, with portal first, we’d have to experiment with self-service interfaces, revert what didn’t work out and constantly change how developers work until it stabilises. And at the end of the day, this is a more costly approach. Moreover, it is important to have a clear vision so that the two initiatives eventually converge. In some rare cases, if the teams share a vision and have a collaborative nature, it is possible to develop both in parallel.
There are many circumstances in which you probably don't need a portal right away. This is a nuanced question, so I decided to put it to other people who are also closely involved in platform engineering.
Here is a selection of the views expressed in the conversation thread shown above:
“Platform first for sure. You want to add value to teams first before you have to add a portal experience on top”
“You can have a platform without a portal. A portal without a platform makes no sense.”
“Always the correct answer: “it depends”. But from my point of view, if the platform is a product, I would include the portal as part of the MVP (2-3 services addressing common problems + portal + implementation layer).”
“Platform first, get the MVP and buy-in and then use that experience to build the portal.”
Integrating your platform with developer portals
Looked at holistically, all efforts to build platforms or portals aim to solve developers' problems. Portals create a gateway to platforms and platforms provide a solid foundation for portals. This combination brings value.
When integrating the two, consider the following aspects:
1. Level of coupling. This refers to how the portal and platform work together and the degree of interdependence between them. A reasonable level of coupling improves maintainability and enables the portal and platform to scale company-wide. It also allows them to be developed and tested in isolation.
2. Level of ownership. This minimises potential confusion over who owns the various services, automation and interfaces. A clear level of ownership is shown below.
3. Level of abstraction. Identifies the desired operating model for developers. For more mature organisations it’s better to stay with open standards in technology, while less mature or less engineering-focused benefit more from a high level of abstraction.
Now let's look at common portal-platform integration patterns from a more technical perspective.
Declarative configuration in git is probably the most common pattern used in custom-built IDP solutions and is relatively easy to implement. It’s also often integrated with a command line interface (CLI) tooling which supports additional validation and automation techniques. For developers to prefer code integrations instead of web interfaces this is the best approach.
Agent-based is commonly used in more advanced IDP solutions and especially as part of SaaS products that provide access to real-time data, as well as a number of existing integrations with 3rd systems.
Direct API integration is a traditional way of integrating systems. It’s flexible but requires additional development efforts.
All of these three integration patterns can be used interchangeably depending on the use case.
Some common pitfalls
Having seen quite a few successful and less successful attempts at building internal portals for developers, here are some common pitfalls I’d like to share with you.
Approaching internal developer portals in a vertical way. This often leads to building multiple portals which provide a limited set of capabilities, are inconsistent and make it difficult to automate processes end-to-end. The full promise of IDP is to have a single pane of glass for your service catalogue, costs, resource management, blueprints and more.
Providing just a static view instead of automatically collecting the live data across software and resource catalog. You should be also able to perform actions from there like: spawn new microservices and create new environments. Otherwise, it makes no difference than another Confluence site which requires a lot of admin work to keep up to date.
Not having a buy-in from developers. Where often the development of IDP is driven top-down without input from the developer community. It might lead to a mismatch of use cases and actual needs.
Lack of shared contribution leads to capped innovation and growth. It’s not solely an infrastructure project.
Not investing in the right design which is user-friendly. In other words, building an internal portal by the infrastructure folks alone is not gonna work out. You need someone with design skills.
Abandonware portal which is not actively developed once the initial set of features has been delivered. This especially applies to custom-made portals which don’t have a dedicated team to maintain it.
There are many views on when and how to introduce an internal developer portal to an organisation. Let's draw upon my conversation with Zohar once more, to better understand his perspective:
“Connecting and bringing live data, which is reconciled with the source of truth, is crucial for the portal. Without it, you end up with just another, maybe prettier, documentation site.” Zohar Einy, CEO of Port.
We also touched on some of the challenges they face with their IDP product. One of them is meeting customer demand, prioritising which features to build into a portal is a real challenge. Especially when it comes to integrations, where they have a dedicated integrations team to deliver value quickly. When you build a portal in-house, developers ask for more, and that is when it starts to break.
Let's leave it there for now
The title of this article suggests that portals propel progress towards platform engineering goals, much like a springboard propels a diver into the air. But now we can see that there's more to portal implementation than a simple starting point or driver for platform engineering success.
As organisations begin to realise the value of internal developer portals, the synergy between the portal as a gateway and the platform becomes critical. However, for organisations that have both platform engineering and internal developer portal initiatives, the result is often two separate products with a common goal and potentially overlapping functionality that must make trade-offs to co-exist.
It is important to recognise that internal developer portals are becoming a part of the platform engineering discipline. These two efforts need to be properly aligned and mutually supportive to improve the chances of achieving greater developer autonomy and productivity.
Now that you've read this article, I hope you have a more complete view of the platform engineering landscape, its pitfalls and advantages. Everything in the above article is based on my own experience and from my point of view, which is why I want to ask you:
What's your perspective on platform engineering?
Do you have any interesting real-life stories to share?
What new dimensions did you discover while reading?
Feel free to write to me directly or leave a comment here. I will make sure all your comments and questions receive a thorough response.
Read previous articles: