Phenquestions
Unit Testing
Unit Testing is testing done on an individual function, class, or module independently than testing a fully working software. Using a framework for unit testing, the programmer can create test cases with input and expected output. When having hundreds, thousands, or tens of thousands of unit test cases for a large software project ensures that all the individual units work as expected as you continue to change the code. When changing a unit that has test cases, the test cases for that module should be studied and determine if new test cases are needed, the output has changed, or the current test cases can be removed as no longer relevant. Creating a large volume of unit tests is the easiest way to achieve high test case coverage for a software code base, but will not ensure that the final product is working as a system as expected.
Functional Testing
Functional testing is the most common form of testing. When people refer to software testing without much detail, they often mean functional testing. Functional testing will check the primary functions of the software work as expected. A test plan could be written to describe all the functional test cases that will be tested, which corresponds to the major features and capabilities of the software. Primary functionality testing will be “happy path” testing, which does not try to break the software or use it in any challenging scenarios. This should be the absolute bare minimum of testing for any software project.
Integration Testing
After unit testing and functional testing, there may be several modules or the entire system that has not yet been tested as a whole. Or there might be components that are largely independent but occasionally used together. Any time components or modules are tested independently but not as a whole system, then integration testing should be performed to validate the components function together as a working system according to user requirements and expectations.
Stress Testing
Think about stress testing like you are testing a space shuttle or airplane. What does it mean to put your software or system under “STRESS”? Stress is nothing more than an intense load of a specific type that is most likely to break your system. This could be similar to “Load Testing” in the sense of putting your system under high concurrency with many users accessing the system. But stressing a system could happen on other vectors too. For example, running firmware on a hardware component when the hardware has had physical deterioration and is operating in degraded mode. Stress is unique to all types of software, and systems and designing stress tests should be under the consideration of what natural or unnatural causes are most likely to stress your software or system.
Load Testing
Load testing is a specific type of stress testing, as discussed above, whereby large numbers of concurrent user connections and accesses are automated to generate the simulation of the effect of a large number of authentic users accessing your software system at the same time. The goal is to find out how many users can access your system at the same time without your software system breaking. If your system can easily handle normal traffic of 10,000 users, what will happen if your website or software goes viral and obtains 1 million users? Will this unexpected “LOAD” break your website or system? Load testing will simulate this, so you are comfortable with the future increase in users because you know your system can handle the increased load.
Performance Testing
People can become utterly frustrated and despair when the software is not meeting their performance requirements. Performance, generally, means how quickly important functions can be completed. The more complex and dynamic the functions are available in a system, the more important and non-obvious it becomes to test its performance, let's take a basic example, Windows or Linux Operating system. An operating system is a highly complex software product, and doing performance testing on its system could involve the speed and timing of functions such as Bootup, installing an app, searching for a file, running computations on a GPU, and/or any other of the millions of actions that can be performed. Care should be taken when selecting the performance test cases, to ensure the important and likely to malfunction performance features tested.
Scalability Testing
Testing on your laptop is good, but not really good enough when you are building a social network, an email system, or supercomputer software. When your software is meant to be deployed on 1000 servers, all functioning in unison, then the testing you do locally on one system will not uncover the bugs that occur when the software is deployed “At Scale” on hundreds of thousands of instances. In reality, your testing will likely never be able to run at the full scale before releasing to production because it would be way too expensive and not practical to build a test system with 1000 servers costing millions of dollars. Therefore, scalability testing is done on multiple servers, but usually not the full number of production servers to try and uncover some of the defects that might be found as your systems are used on bigger infrastructure.
Static Analysis Testing
Static analysis is testing that is done by inspecting the software code without actually running it. To do static analysis, generally, you would use a tool, there are many, one famous tool is Coverity. Static analysis is easy to run before releasing your software and can find many quality problems in your code that can be fixed before you release. Memory errors, data type handling errors, null pointer dereferences, uninitialized variables, and many more defects can be found. Languages like C and C++ greatly benefit from static analysis because the languages provide great freedom to programmers in exchange for great power, but this also can create great bugs and mistakes that can be found using static analysis testing.
Fault Injection Testing
Some error conditions are very difficult to simulate or trigger, therefore the software can be designed to artificially inject a problem or fault into the system without the defect naturally occurring. The purpose of fault injection testing is to see how the software handles these unexpected faults. Does it gracefully respond to the situation, does it crash, or does it produce unexpected and unpredictable problematic results? For example, let's say we have a banking system, and there is a module to transfer funds internally from ACCOUNT A to ACCOUNT B. However, this transfer operation is only called after the system has already verified that these accounts existed before calling the transfer operation. Even though we assume that both accounts do exist, the transfer operation has a failure case where one target or source account does not exist, and that it can throw an error. Because in normal circumstances we never get this error due to pre-testing of inputs, so to verify the system behavior when the transfer fails due to a non-existent account, we inject a fake error into the system that returns a non-existent account for a transfer and test how the rest of the system responds in that case. It is very important that the fault injection code is only available in testing scenarios and not released to production, where it could create havoc.
Memory Overrun Testing
When using languages like C or C++, the programmer has a great responsibility to directly address memory, and this can cause fatal bugs in software if mistakes are made. For example, if a pointer is null and dereferenced, the software will crash. If memory is allocated to an object and then a string is copied over the memory space of the object, referencing the object can cause a crash or even unspecified wrong behavior. Therefore, it's critical to use a tool to try and catch memory access errors in software that uses languages like C or C++, which could have these potential problems. Tools that can do this type of testing include Open Source Valgrind or proprietary tools like PurifyPlus. These tools can save the day when it's not clear why the software is crashing or misbehaving and directly point to the location in the code that has the bug. Awesome, right?
Boundary Case Testing
It is easy to make errors in coding when you are on what is called a boundary. For example, a bank automated teller machine says you can withdraw a maximum of $300. So, imagine the coder wrote the following code naturally when building this requirement:
If (amt < 300)startWithdrawl()
else
error(“You can withdraw %s”, amt);
Can you spot the error? The user who tries to withdraw $300 will receive an error because it is not less than $300. This is a bug. Therefore, boundary testing is done naturally. Requirement boundaries then ensure that on both sides of the boundary and the boundary, the software is functioning properly.
Fuzz Testing
High-speed generation of input to software can produce as many possible input combinations, even if those input combinations are total nonsense and would never be supplied by a real user. This type of fuzz testing can find bugs and security vulnerabilities not found through other means because of the high volume of inputs and scenarios tested rapidly without manual test case generation.
Exploratory Testing
Close your eyes and visualize what the word “Explore” means. You are observing and probing a system in order to find out how it truly functions. Imagine you receive a new desk chair in mail order, and it has 28 parts all in separate plastic bags with no instructions. You must explore your new arrival to figure out how it functions and how it is put together. With this spirit, you can become an exploratory tester. You will not have a well-defined test plan of test cases. You will explore and probe your software looking for things that make you say the wonderful word: “INTERESTING!”. Upon learning, you probe further and find ways to break the software that the designers never thought of, and then deliver a report that details numerous bad assumptions, faults, and risks in the software. Learn more about this in the book called Explore It.
Penetration Testing
In the world of software security, penetration testing is one of the primary means of testing. All systems, whether biological, physical, or software have borders and boundaries. These boundaries are meant to allow only specific messages, people, or components to enter the system. More concretely, let's consider an online banking system that uses user authentication to enter the site. If the site can be hacked and entered into the backend without proper authentication that would be a penetration, which needs to be protected against. The goal of penetration testing is to use known and experimental techniques to bypass the normal security boundary of a software system or website. Penetration testing often involves checking all the ports that are listening and trying to enter a system via an open port. Other common techniques include SQL injection or password cracking.
Regression Testing
After you have working software that is deployed in the field, it is critical to prevent introducing bugs into functionality that was already working. The purpose of software development is to increase the capability of your product, introduce bugs, or cause old functionality to stop working, which is called a REGRESSION. Regression is a bug or defect that was introduced when previously the capability was working as expected. Nothing can ruin the reputation of your software or brand faster than introducing regression bugs into your software and having real users find these bugs after a release.
Regression testing cases and test plans should be built around the core functionality that needs to continue working to ensure that users have a good experience with your application. All of the core functions of your software that users expect to work in a certain way should have a regression test case that can be executed to prevent the functionality from breaking on a new release. This could be anywhere from 50 to 50,000 test cases that cover the core functionality of your software or application.
Source Code Bisection Testing
A bug was introduced in the software, but it is not obvious which version of the release introduced the new bug. Imagine that there were 50 software commits from the last known time the software was working without the bug, until now when…
Localization Testing
Imagine a weather application that shows the current and projected weather in your location, as well as a description of the weather conditions. The first part of localization testing is to ensure that the correct language, alphabet, and characters are displayed properly, depending on the geolocation of the user. The app in the United Kingdom should be displayed in English with Latin characters; the same App in China should be displayed in Chinese characters in the Chinese language. More elaborate localization testing done, the wider range of people from different geolocations will interface with the application.
Accessibility Testing
Some of the citizens in our community have disabilities, and therefore, may have trouble using the software being created, so accessibility testing is done to ensure that populations with disabilities can still access the functionality of the system. For example, if we assume that 1% of the population is color blind, and our software interface assumes that users can distinguish between Red and Green but those color blind individuals CANNOT tell the difference. Therefore, a well-software interface will have additional cues beyond the color to indicate meaning. Other scenarios besides color blindness testing would also be included in software accessibility testing, such as full visual blindness, deafness, and many other scenarios. A good software product should be accessible by a maximum percentage of potential users.
Upgrade Testing
Simple apps on a phone, operating systems like Ubuntu, Windows, or Linux Mint, and software that runs nuclear submarines need frequent upgrades. The process of the upgrade itself could introduce bugs and defects that would not exist on a fresh install because the state of the environment was different and the process of introducing the new software on top of the old could have introduced bugs. Let's take a simple example, we have a laptop running Ubuntu 18.04, and we want to upgrade to Ubuntu 20.04. This is a different process of installing the operating system than directly cleaning the hard drive and installing Ubuntu 20.04. Therefore, after the software is installed or any of its derivative functions, it might not be working 100% as expected or the same as when the software was freshly installed. So, we should first consider testing the upgrade itself under many different cases and scenarios to ensure that the upgrade works to completion. And then, we must also consider testing the actual system post upgrade to ensure that the software was laid down and functioning as expected. We would not repeat all test cases of a freshly installed system, which would be a waste of time, but we will think carefully with our knowledge of the system of what COULD break during an upgrade and strategically add test cases for those functions.
Black Box & White Box Testing
Black box and white box are less specific test methodologies and more of categorizations types of testing. Essentially, black box testing, which assumes that the tester does not know anything about the inner working of the software and builds a test plan and test cases that just look at the system from the outside to verify its function. White box testing is done by software architects that understand the internal workings of a software system and design the cases with knowledge of what could, would, should, and be likely to break. Both black and white box testing are likely to find different types of defects.
Blogs and Articles on Software Testing
Software testing is a dynamic field, and many interesting publications and articles that update the community on state-of-the-art thinking about software testing. We all can benefit from this knowledge. Here is a sample of interesting articles from different blog sources you might want to follow:
- 7 Tips to Follow before Testing without Requirements; http://www.testingjournals.com/
- 60 Best Automation Testing Tools: The Ultimate List Guide; https://testguild.com/
- Open Source Database Testing Tools; https://www.softwaretestingmagazine.com/
- 100 Percent Unit Test Coverage Is Not Enough; https://www.stickyminds.com/
- Flaky Tests at Google and How We Mitigate Them; https://testing.googleblog.com/
- What is Regression Testing? ; https://test.io/blog/
- 27 Best Chrome Extensions For Developers In 2020; https://www.lambdatest.com/
- 5 key software testing steps every engineer should perform; https://techbeacon.com/
Products for Software Testing
The majority of valuable testing tasks can be automated, so it should be no surprise that using tools and products to perform the myriad tasks of software quality assurance is a good idea. Below we will list some important and highly valuable software tools for software testing that you can explore and see if they can help.
JUnit
For testing Java-based software, JUnit provides a comprehensive test suite for unit and functional testing of the code that is friendly to the Java environment.
Selenium
For testing web applications, Selenium provides the ability to automate interactions with web browsers, including cross-browser compatibility testing. This is a premier testing infrastructure for web testing automation.
Cucumber
A behavior-driven testing framework allows business users, product managers, and developers to explain the expected functionality in natural language and then define that behavior in test cases. This makes more readable test cases and clear mapping to expected user functionality.
Purify
Find memory leaks and memory corruptions at run time by executing your software with the Purify Plus instrumentation embedded that tracks your memory usage and points out errors in your code that are not easy to find without instrumentation.
Valgrind
Open-source tools that will execute your software and allow you to interact with it while pointing out an error report of coding errors such as memory leaks and corruptions. No need to recompile or add instrumentation into the compilation process as Valgrind has the intelligence to dynamically understand your machine code and inject instrumentation seamlessly to find coding errors and help you improve your code.
Coverity
Static analysis tool that will find coding errors in your software before you even compile and run your code. Coverity can find security vulnerabilities, violations of coding conventions, as well as bugs and defects that your compiler will not find. Dead code can be found, uninitialized variables, and thousands of other defect types. It's vital to clean your code with static analysis before releasing it to production.
JMeter
An open-source framework for performance testing oriented to Java-based developers, hence the J in the name. Website testing is one of the main use cases for JMeter in addition to performance testing of databases, mail systems, and many other server-based applications.
Metasploit
For security and penetration testing, Metasploit is a generic framework with thousands of features and capabilities. Use the interaction console to access pre-coded exploits and try to verify the security of your application.
Academic Research on Software Testing
- University of Sheffield Testing Research Group
- University of Kentucky Software Verification and Validation Lab
- North Dakota State University Software Testing Research Group
- System Testing Intelligent Lab; Czech Technical University Prague
- NASA: Jon McBride Software Testing and Research (JSTAR)
- McMaster University; Software Quality Research Laboratory
- Ontario Tech University; Software Quality Research Lab
- The University of Texas @ Dallas; STQA
Conclusion
Software's role in society continues to grow, and at the same time, the world's software becomes more complex. For the world to function, we must have methods and strategies for testing and validating the software we create by performing the functions it is intended to perform. For each complex software system, a testing strategy and testing plan should be in place to continue to validate the functionality of the software as they continue to get better and provide its function.
