This article is aimed at test analysts and other stakeholders who engage in software quality assurance in a managerial capacity. The article discusses some of the unique challenges and opportunities associated with testing an Internal Developer Platform product like SkyU and shares a few reasons why buying one is better than building one in-house.
The International Software Testing Qualifications Board defines testing as “a set of activities to discover defects and evaluate the quality of software artifacts” [1]. Typical software testing activities include activities such as analysing requirements, designing test cases, executing test cases, and comparing expected results with the actual results to identify defects. Based on criteria such as the product architecture, technologies, and business requirements, these testing activities' effort, complexity, and risk tolerance can vary greatly.
To understand this idea, imagine how a product team developing a food order management system for the internal use of an organization might settle for simpler less effortful solutions with higher risks related to the quality of their product compared to a mission-critical food order management system that is built to cater to an entire nation which would demand a much lower tolerance of risk at higher complexity and effort. SkyU, as a PaaS product open to organizations with product development aspirations on both ends of this spectrum, the SkyU QA team has to ensure the product meets the quality assurance expectations of both mission critical software projects as well as software projects catering to trivial business needs.
Challenges and Opportunities Testers of Platform Engineering Products Can Expect
The challenges of testing a platform engineering product like SkyU can be categorized into,
- Challenges that arise from the large number of 3rd party services and technologies that need to be bundled in.
- Challenges associated with conducting manual and automated end-to-end testing.
Though internal developer platforms are built specifically to meet an organization's needs, with an OOTB platform engineering product like SkyU, you can't offer a narrow set of features and configurations. This results in the need to support a large number of third party services such as Github Actions, Docker Hub, AWS EKS, and ChatGPT, as well as technologies like Docker, Kubernetes, Helm, and Kustomize. These services and technologies need to work seamlessly across the various layers that make up the product. From a quality assurance perspective, the interconnectedness of these components increases testing activities.
When it comes time to design test cases, a tester will analyze the functionality offered by these 3rd party services and identify how it relates to the functionality offered to the end user. The tester will then employ test design techniques such as pairwise testing, decision tables, state transfer diagrams, and his or her own experience to design the optimal number of test cases to cover the feature. A pain point unique to IDP products to this end is the identification of all the expected results for a given test case.
For example, if a new Cloud CI service is being introduced as a feature (say BitBucket), when creating test cases, a tester may need to identify the expected results to look for in the GUI under the ‘Developer Control Plan’, ‘Integration’, and ‘Delivery Plane’ and the ‘Monitoring Plane sections’. The tester will then need to identify any expected results related to the database that is only accessible through the API layer. Finally, the tester will need to look at the expected results that need to be accessed in the 3rd party service, such as the commit history or pipeline information.
Another pain point associated with testing IDP products is the complexity involved in creating end-to-end test automation scripts. The automation scripts require adaptation to interface across multiple services, architectures, and interface types while maintaining synchronization between the scripts and the system under test. This is to ensure a reliable test execution.
However, it's not all pain for the tester, as Churchill is known to have once said “A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty”, the very challenges associated with testing an IDP can be the strengths that can propel the careers of testers of an IDP product past that of their peers in this day and age of AI prompt based software engineering. As testers work their way into the innards of the product, they will get opportunities to up their DevOps skills and knowledge which is still highly sought after. Coupled with their experience in quality assurance, they will be able to uniquely position themselves in the job market.
The benefit of going with platform engineering products on an organisation's software quality assurance function
One of the main selling points of going with a platform engineering product like SkyU, as opposed to building a platform in-house, is the lowered cost of ownership. One reason why this is a valid point is the quality assurance-related effort that goes into the delivery of an IDP product. Organisations can piggyback on the process of testing which the vendor has already put into the product. They can benefit from the feedback cycle from other users who’ve used it and the community as a whole.
Ideally, the IDP product vendor should ensure the quality of the three lower levels of the test pyramid. That is the component level testing through developer unit tests; integration level testing through component integration testing and API testing and system level testing through end-to-end testing. Meaning, organisations can focus on the different acceptance tests such as operational acceptance, regulatory acceptance, and user acceptance. All of this will lead to a cheaper and faster ownership of the IDP.
In summary, this post aimed to provide an overview of the challenges and opportunities that IDP product testers may encounter. It also highlighted the benefits organizations can gain by purchasing an IDP rather than building one in-house, particularly in terms of the advantages it offers to the quality assurance function and the impact on the bottom line.
References
[1] - https://www.istqb.org/certifications/certified-tester-foundation-level
[2] - https://qacraft.com/levels-of-testing-in-software-testing
