Loading

Testing Software with Limited Resources When you experience tight budgets and project cuts, business leaders must balance user expectations with what their teams can deliver feasibly.

Today's fast-paced business world demands quick delivery of new competitive applications, re-platforming of current business-critical applications, and the addition of new product features to current applications. Leaders are faced with balancing the demands of quality software delivery and end-user satisfaction with cost control and limited resources.

Although all team members are important in a software project—including the product owner, scrum master, business analyst, UX designer, software engineers, and test engineers—testing activities sometimes are reduced or cut out of project budgets. Other times, limited testing resources cannot keep up with the pace of development activities, making testing a bottleneck.

Importance of software testing

The most mature organizations, however, see testing as an integral part of software delivery. According to the World Quality Report 2019-2020, executive management view testing as very important in:

  • Contributing to business growth and business outcomes
  • Detecting software defects before go-live
  • Ensuring end-user satisfaction
  • Increasing quality awareness among all disciplines

Testing with Limited Resources: A Step-by-Step Guide

Testing is a risk mitigation plan for software development projects; it helps to reduce the risk of software defects, poor software performance, and a sub-par customer experience. Two goals of testing include:

  • Continuous validation of the quality of the application's functionality, workflows, data integrations, security, and performance to provide visibility into the application's quality
  • Creation of tests that are assets with high reusability both during the project and for future releases of the product

In times when your business is faced with delivering software projects with limited testing resources and/or skills, there are steps you can take to deliver high-quality, tested software. These are outlined below.

Step 1 | Start with a test strategy

We inherently understand the importance of a strategy before starting a major undertaking. We plan our vacations with an itinerary, we draw a blueprint before building a house, and we architect our software before writing lines of code. Similarly, we need a strategy before testing.

A test strategy clearly defines the who, what, when, how and why for all project testing activities in order to maximize limited resources utilization, maximize testing coverage, and reduce production issues. It gets the entire project team on the same page regarding testing.

Whether the test strategy is a formal document or a discussion with documented notes, it should cover key items such as testing scope, test cycles definitions, the testing approach, and what resources will be used.

Key Items in Your Software Testing Strategy

  • Testing Scope: What needs to be tested (functions, features, workflows, integrations, etc.) and why? What impact do software changes or additions have on the existing software or integrations that should also be tested?
  • Test Cycles Definition: When will the items in scope be tested for early detection of issues and to avoid redundant testing? Test cycles include unit tests, data validation tests, API services tests, functional tests, end-to-end workflow tests, regression tests, performance tests, accessibility tests, device compatibility tests, security tests, and user acceptance tests.
  • Testing Approach: How will the items in scope be efficiently tested within the project as well as for future reusability? Which tests will be manually performed or automated? Are the tests at the API or UI layer? Which test environments are available? How is test data created and managed? How will the test cycles be performed within a sprint or across sprints? How will defects be reported and managed? What tools will be used? And how will the test repository be organized?
  • Testing Resources: The skills required and who will perform testing are driven by the test cycles identified for the project. Resources may include test engineers, software engineers, and business representatives using the appropriate testing processes and tools.

If you are about to start a new software development project, now is the time to prepare a test strategy. Ideally, the project team should agree to the strategy before the start of Sprint 1. If your project is already underway, it’s never too late to put a strategy in place as it will help to save time and cost in the long run.

Step 2 | Take an 'All Hands on Deck' Approach for Resources

The ideal situation is to assign professional testers to a project to focus specifically on testing because they bring a testing mindset and knowledge of appropriate processes and tools. That said, if you are unable to reassign current testers, hire new employees, or use consultants to fill your project's testing resource needs, do not skip testing! Consider distributing testing activities within your project team based on required skills. And while the all-hands-on-deck approach is a plausible temporary fix to ensure software is adequately tested and usable, be aware; there are drawbacks.

What to do when you don’t have professional testers

For when you do not have a team of testers, consider other roles that make for a best fit. For instance, if a test cycle is functional UI tests performed manually, business reps with domain knowledge can perform testing using your organization’s defined testing processes and tools. If the test cycle is API services tests or automated UI tests, software engineers can develop and execute these tests using the approved tools and frameworks; furthermore, software engineers can build unit tests to cover key functionality and business rules, and develop, run, and analyze performance tests. Finally, business reps should be able to cover end-to-end workflow testing through a planned and orchestrated user acceptance test.

A company sells products through an e-commerce site where customers can order directly. Additionally, the sales department sells to corporations, entering these orders into the system and performing an end-to-end workflow manually. At this same company, the product owner for the ordering application asks the IT department for a new functionality that clones an order. The product owner gives the specs to IT to develop, but it still needs to be tested. The salespersons become the testers themselves as they enter orders, testing the new cloning function prior to rolling out to customers.

Prerequisites for Testing

There are a few prerequisites to using this distributed responsibility for testing, when you can't bring onboard more testers to meet your project's demand.

  • A QA/Test Lead is assigned to plan and coordinate testing activities across the project team; the Lead reports testing progress and metrics for visibility into the software’s quality and coaches project team members on testing practices.
  • There is an agreed to test strategy for the project as previously described.
  • There are processes and tools for test management and defect management. Project team members performing the testing role are trained on these processes and tools.
  • There is a framework(s) in place for test automation at the UI and API layers.
  • Testing is fully integrated within the Agile or Scrum software development process.

Although this resource approach will overcome the shortage of testers for a project while still getting the software tested, it will not save project time or cost. Software engineers, for instance, perform testing at a higher cost than a test engineer. Plus, the time that a software engineer spends on testing could instead be used to write additional code for functions/features to progress delivery. Similarly, business reps assigned to testing activities are removed from their day-to-day business jobs and must be replaced by someone else temporarily at an added cost to the organization.

Still, this approach fosters the philosophy that everyone within the project team is responsible for quality software. Quality is implicit in everything that the project team does and is not a separate activity within the software development process. Also note that this approach does require a QA/test lead to advocate, govern and coach testing activities. The QA/test lead also coordinates activities to avoid duplicate testing efforts across the various team members and to optimize test coverage with limited resources.

Step 3 | Put Repeatable Testing Processes in Place

Before you get to the point that testing demand exceeds testing resources, you should implement repeatable testing processes with supporting tools. This allows you to easily use the 'all hands on deck' resource approach while ensuring the consistency and reliability of testing activities.

Testing processes and tools should follow testing best practices within Agile or Scrum software development, including ‘shift left’ and ‘continuous testing’ practices.

  • Sprint Planning: There should be a reliable method for estimating the testing effort for a user story. This includes the time to test the user story (either manual and/or automated) and to regression test existing components of the software impacted by the user story. Additionally, there should be consistent criteria used to evaluate the completeness and testability of the user story and its acceptance criteria.
  • Test Case Design: Whether the test is manual or automated, put guidelines in place to foster proper test design for coverage, stability and maintainability. This includes a modular design for functional test cases to serve as building blocks to string together for end-to-end workflow testing. Establish standards for test naming conventions, granularity of test steps, and data-driven approaches to handle test conditions through data combinations within a single test. It’s best to model a few test cases to serve as examples of test case design best practices.
  • Test Repository Organization: A test repository is essential for managing and maintaining test assets. There are various test repository tools in the marketplace, both commercial tools such as Microsoft’s ADO and Tricentis qTest, plus plug-ins to the popular Atlassian JIRA development tool such as Xray and Zephyr. Keeping test cases organized by aligning to the functions/features of the software will facilitate their future usage and expansion when changes are made to the software. Avoid organizing test cases by project or sprint.
  • Defect Management: Defects discovered during testing must be reported so that a developer can easily reproducing the defect and fix it. Complete information about the defect is also needed for the tester who will retest it, since it may not be the same person who originally reported it. Similar to test cases, there should be naming standards and guidelines for the information captured and reported.
Step 4 | Integrate Test Environments and Test Data in an Ecosystem

Today’s software is an ecosystem of interconnected software for end-to-end business workflows. For example, a supply chain can cross multiple applications from a CRM and order entry to order fulfillment, billing and financial reporting. Besides testing individual functions/features within specific software, end-to-end workflows across applications must be tested to ensure the integrity of business operations.

The challenges of end-to-end testing

End-to-end testing is challenging due to inadequate test environments and test data. According to the World Quality Report 2019-2020, 56% of the respondents stated they lacked appropriate test environments and data. Without proper test environments management (TEM) and test data management (TDM), testing resources are quickly consumed by environment and data issues. Less time is available to plan and execute tests, either manual or automated.

Without proper test environments management and test data management, your limited testing resources can be consumed by other issues.

Test environments must be available and properly configured to synchronize across multiple inter-connected applications. This includes the version of the software (production, upgrade, or sprint version) as well as the predictability of the data (databases and transactions) in the environment.

For example, consider an upgrade to a CRM application that needs to test with the production version of an order entry application or vice versa. Services virtualization can be used to simulate these software connections without testing with the software itself. Multiple test environments may be needed to avoid environment contentions with the various test cycles such as functional tests, end-to-end workflow tests, regression tests, performance tests, user acceptance tests, etc. A cloud-based approach to test environments can address contention by having separate environments that can quickly be made available and eliminated when no longer in use.

Test Data Management

Equally important is test data management (TDM) to create and maintain real-time test data that mimics actual production data, both transactions and databases. Each test cycle must have predictable and reliable data for successful test execution. For example, when testing an order-through-fulfillment workflow, each order transaction must have inventory on hand within the database to fulfill the order. If multiple tests are using the same order item, the on-hand inventory will be depleted, and the test will fail due to improper data set-up and not an actual software defect.

Proactively investing in TEM and TDM solutions, if you have integrated applications, will save you time and effort when you have a software project to test with limited testing resources. Your testing resources can focus instead on planning, designing and executing tests to provide visibility into your software’s quality.

What we've learned: Be prepared ahead of time

There are times when you have more software projects to test than you have testing resources. If you don’t have the option to onboard additional testers, there is an approach as outlined above to get the software tested in an effective and efficient manner, and it's best to be prepared for workload peaks before they happen.

A test architect with expertise in testing processes and tools can assist in implementing best practices and proactively resolving TEM and TDM challenges for your enterprise. With testing practices in place along with a project-specific test strategy, you’ll have flexibility in ‘who’ does the testing. Plus, with a strong QA/test assigned to lead the software project and coordinate an ‘all hands-on deck’ approach, you will be positioned to achieve proper testing under resource constraints.

What current challenges do your development and testing teams face? Share with us by sending a note to the author, Nancy Kastl, Ex. Director, Testing Services, SPR.
Nancy Kastl

Looking for more on how to test software and applications in today's environment? Visit The Lumen, SPR's digital magazine featuring thought leadership, insights, and trends.

Credits:

Created with images by Priscilla Du Preez - "untitled image" • Nicholas Kwok - "greed." • Startaê Team - "📸 @mariogogh" • Geetanjal Khanna - "untitled image" • Roberto Nickson - "Remote Work" • Eleventh Wave - "untitled image" • Marcos Paulo Prado - "Writer at Coffee shop - Toronto - All pictures edited with my presets that you can find on my website in BIO" • Erol Ahmed - "The Freemason building in Washington, D.C., houses a large library — open to the public. This was a card catalog I saw inside their collections." • Markus Spiske - "untitled image" • Markus Spiske - "untitled image" • David Werbrouck - "Somewhere at Museu del Disseny de Barcelona."