The more experience you have in software development, the more you will encounter specific patterns of code. After 2, 5, 10, and 30 years of software development, there will be certain lines of code you will have written hundreds or thousands of times. In some ways, they are part of your signature but in other ways, your mind simply gravitates to them and those specific lines of code can differ between two persons. Your experience with certain code expressions can be so well established that your ability to apply them well can shift into a highly competent instinctive usage.
The problem though is while you will undoubtedly excel in certain code constructs, software is not about any one code statement. Code statements integrate and form a whole. The quality of the whole is impacted by the quality of that integration. As code executes receiving and delivering data from one statement to another, the program migrates from one state to another. That state shift represented by correctly encoded data with correct values for the represented state has to be confirmed. That is called testing and takes many forms. I am going to emphasize a software developer centric form that should be an automatic approach for everyone who writes code. That does not guarantee 100% correct programs but it does significantly reduce the occurrence of inoperable programs or corrupt data.
Single Line Check
A program consists of single lines of code operating together to transform data. You can test each and every one of them but often not all at once. A change in one area of a program can and often has an impact on the behavior and data in another area of the program. You need will want to know that any modifications you made to the program are not mistakes. If any mistakes exist, testing is the #1 way to identify them so you can correct them.
Acutely testing a few lines of code may fall under the category most would call unit testing. Unit testing is useful in two cases. The initial stages of a new program consisting of a few lines of code. Changes you have made to code in both new and established programs. In both cases, you need to confirm code modifications and their impact to the program.
Testing a single line is very straightforward. You set a debugger’s break point on or just before or just after the line you want to test. Observe the state of the variables near, at, or just after the line of code you intend to write. You may encounter a variety of conditions that inform your acceptance of that line of code (or other lines involved). Those conditions could be invalid access (segmentation faults, null reference, and other resource access rejections) and data values (variables, files, graphics, output, and table columns) you provable know are incorrect in relation to intent of the specified code statements that precede the observed line of code.
Both command-line and user interface programs start in exactly one spot. C++ programs define a main function. Testing will usually begin in this main function. As other functions near the main function are introduced you will want to test them quite rigorously. The code structure and statements within a main method often stabilizes faster than other parts of a new program. After you have tested it thoroughly, it can be one of the few areas you will need to revisit again short of code modifications.
Unless the main method is boilerplate, it will need to be tested very carefully. Additional functions you introduce will require your full attention. A function is not just a group of software statements that work together to return data, modify data, or facilitate I/O. A function has parts such as entrance, exit, preconditions, post conditions, a possible preamble, possible return statement(s), and one or more statements that define the consummate body of the function. Conduct a test by working through the parts of a function to ensure each part operates properly in support of the whole function. Well written functions typically fulfill a single task. Examples include opening a file, changing specific text in a file, or closing a file. Those operations can be addressed by individual functions that you should test to ensure that the operations are carried out correctly and consistently.
Some functions you write will eventually work properly every time they are used. After you rigorously test them, such functions can often be relied upon to support other functions in a program. That is essentially how you build up a good program. One solid, well-tested function at a time. Stabilize functions through a combination of solid unit tests applied to single lines in functions and confirmation of functions in various scenarios. Beyond a certain point in the development of a program, the majority of functions should be close to ideal in terms of stability and reliability. The earliest, well-tested, proven functions in turn become the ingredients used to build further functionality in a program.
Functions reside in files and may optionally be enclosed in scoping mechanisms such as namespaces or classes. Several functions may be organized in this way to form a collective unit of functionality. A group of functions organized this way often work together on data to produce an overall result. In other cases, the functions are organized in topical groups that make locating needed functionality easier. When functions work collectively, the stakes are higher. Multiple functions working together can increase the number of scenarios you have to test to ensure collective results are correct.
A module will have a specific interface. Functions in a module may cooperatively modify data retained in the module that is accessed by one or more of those functions. The number of permutations you have to test can be quite high based on the number and type of functions and the number and type of data points accessed and modified by multiple functions. Focused, simpler modules can improve on this situation greatly.
Whether simple or complex, modules you introduce will need to be exercised regularly. The testing approach consists of selective single line and function checks in various overlapping contexts. Testing includes the contexts of one or more modules overlapped with those of initiating functions. Basically, a given module may not be accessed everywhere but instead are accessed from a few functions. Those functions are the starting points at which a module may be accessed. Testing would include determining if the module and any data from the module are accessed and referenced appropriately.
Overall Application Check
The next stage in this process is to observe the overall condition of the program itself. How are the inputs to the program handled and how well? How are the outputs expressed and how well? How well are the transitions from one state of the program to another handled? The emphasis is on determining if the program meets overall goals as well as demonstrates evidence of malformed output in all areas where output occurs or is possible.
Output refers to but is not limited to one or more of the following:
- General response text in command-line programs
- General visual updates in GUI programs
- Error and/or status response codes
- Values returned from functions
- Variables updated within the program
- Data applied to files including file attributes
- Database tables and any modified database attributes
- Local and network folder contents and any modified system or server environment attributes
At an application level, tests are geared to produce all output specified for the program to achieve complete test coverage. Correct output is evidence of a correctly specified and implemented program. The process represented and facilitated by the program works across modules, functions, and involved sources of data across diverse contexts within the program. The test establishes if their combined operation achieves the results intended.
Testing a program starts with launching it. You can type the name of the program at the command-line and see output on the command-line if the program is quintessentially a command-line program. This is also the case if a GUI program also emits to the command-line. If the command-line program is launched from a visual program, a desktop, or other visual environment, then you may be unconcerned with any command-line output if the program does all output exclusively to files, databases, network locations, and other destinations you can verify that does not involve a command-line read out. GUI programs can be launched by double-clicking an icon, tapping an icon, activate by voice, or otherwise typing its name on a command-line and launching it from there.
The final stage and main part of this article concerns how you would generally test a command-line program. Run the program multiple times observing its behavior and output across multiple executions of the program. Each time you change any code in the program, you will run and use the program several times. On each run, you verify proper program function overall with additional focus on the area where the change occurred. Executing the program through many rounds of tests are not limited to software changes.
Other changes will often impact the quality of the program. They include database changes; file format changes; network configuration changes; and modifications to the environment in which a program runs. That can include operating system updates; hardware driver changes; and other changes to desktop, mobile, server, and embedded hardware and operating system configurations. Changes in any of these areas may not negatively impact the program, but it is best not to guess but instead you should test. Only by testing the software multiple times following such changes can you confirm the functional integrity of the program.
A command-line program can be called up from a command-line script file. That allows you to type the command to launch the program in the same manner as when you type it on the command-line. The difference is you can place multiple lines in a script file to enact each individual run of the program. This greatly reduces your typing on the command-line and allows you to coordinate tests in a more systematic way. A script allows you to specify any parameters or preconditions in terms of the command-line environment as well as file, database, security, or network prerequisites before the program is invoked. Running the program with any prerequisites, parameters, or logging setups occurs consistently when ran from a script. You execute the script and the sequence of actions that are specified in the script to call the program in as many times you specified the program be called occurs in sequence.
As described in the previous sections, testing can proceed through 4 levels. A single line test executes either organically defined, structured, or automated unit tests to identify defects in the program. A function check observes the effects of functions to identify defects in their specification related to their effects. Module checks are the same as function checks in purpose. They involve multiple functions as a topical unit applied to data organized within that unit expressed in a concrete implementation file. Overall application check identifies defective quality in the program as a whole in sequence graphs that cut across functions, modules, contexts, topics, and various I/O.
All programs that can be observed by a person will have a subjective aspect even if that is only related to how that program is represented. More substantively, that will include how parameters to a command-line are defined, used, addressed in error and status messages. The output from the program will also factor into subjective judgments regarding the qualities of the program. If preferences for how the program represents process apart from correct implementation exists then that would be a distinct area to test. How well does the program meet the superficial or subjective considerations of either an actual third-party (or yourself) or a well-defined persona? You can test for that.
Some programs or some parts of some programs could accept inputs in no specific sequence. You or those who work with the program may have a preference as to the ideal sequence of inputs. Sure, there could be technical considerations that narrow the set of practical input sequences, but you may be working with a program in which a preferred order of inputs is expected. The same for output. If that applies, again, that can be part of your test.
Subjective criteria for the program may or may not exist but technical criteria will definitely apply. The program must meet the requirements for the human situation. The human need and intent is the #1 purpose for the existence of the program. That purpose has to coincide with technical conditions. A program has to meet the human requirements to be considered successful. Many programs can have many users or continue to operate for many years but still fail actual technical or technical-subjective criteria. I will not touch on technical-subjective criteria. That is a huge topic covered in part or whole in books such as Clean Code, Code Complete, Refactoring, and many others. A huge topic.
A few of the technical areas you can test for will either directly support the main purpose of the program; facilitate conditions that allow the program to work according to its purpose; or otherwise address lateral criteria. Additional criteria could include maintaining or raising the professional standard of the software, its best practices, or proactively addressing security, reliability, and efficiency. Regarding technical areas to test, observe, and improve, you will want to be aware of a few areas.
Policy Check – App stores, mobile environments, desktops, servers, and cloud environments always has a policy in effect. The policy are constraints on what a program can do. An example is when a program needs more permissions in Microsoft Windows which may trigger a UAC prompt. UAC may take effect and require the end-user approve the operation the program is about to do. You can of course loosen the policy on some versions of Microsoft Windows to reduce or increase UAC. Another example is when you install some mobile apps that are governed by a user data sharing policy (i.e. contact, credit card info, and location). A server environment may impose a policy that disallows certain types of network connections and a program that attempts to work in a manner contrary to policy may encounter an error. Numerous policies may be encoded into the environments programs operate in and many operating systems will have default policies that can be adjusted by either administrators, updates, or end-users. You will want to test that the program works according to certain system requirements.
Interface I/O Check – Generically cast, a file system, a network port, a function’s signature, a graphics card, and an API are all interfaces. A given interface will accept input, provide discernible output, or both. Your test includes checking use of interfaces both implemented by you and that are external. That includes evaluating an input’s compliance with an interface. Review output to determine if it is complete, gives any indication of defective input, functionality, process, or data used to create it.
Network I/O Check – Requirements exist on network interfaces. A given network interface may include security, sequence, capacity, and encoding requirements in addition to many others. When testing a program that accesses and uses one or more network interfaces, you will attempt to establish that the interface is used properly and potential issues mitigated.
File Quality Check – Not all programs read or create files but many programs do including many command-line programs. An output file created by the program may consist of nothing more than a log of operations. A program may also a read file it did not create including a file from the Internet. The objective here is to ensure the input file is correct either through appropriate checks by the program itself or as a standing prerequisite that applies in part or whole in a program. During testing, files created or modified by the program will need to be inspected in detail to ensure they are accurate. One or more problems could lead to an incorrect file. Errors in the software code; improper source data; poorly drafted requirements; file system issues; and other technical or process issues. All issues can be identified through testing but careful, systematic testing is required before correction can be attempted.
Performance – Program duration in whole or in part will factor in testing. The performance may be acceptable or accepted. If the performance is acceptable, then the duration of the program’s running time has to factor into all other test areas. This impacts how many tests can be done and can determine the extent of testing in one or more areas. When performance is not acceptable or there is a preference to improve it, then you will want to apply various strategies to measure performance, identify bottlenecks, and update the software code to a more efficient level. A good book that covers this for C++ goes by the title, Optimized C++. Usually, performance concerns certain statements or functions taking too long, but in rare cases, it can also involve those same things going too fast.
Memory Use – A program may ideally use less memory but choices in data representation, implementation time, operating environment, and others may lead to memory use that is either satisfactory or a settled matter. Testing may not serve to change or even observe changes in memory use but the amount of memory used by the program, like performance factors, can affect how you test, how long you test, and how often. When your objective is to change the program’s memory use, various tools can be used to measure memory (i.e. valgrind), identify the main contributors to the aspects of memory you want to better understand and apply differently.
Programs exist that are designed to test other programs. Test simulation programs can range between simple to complex with many tools between these extremes. A test automation program can include the ability to imitate mouse clicks, touch screen taps, or voice commands and do them in the same sequence each time test is executed. They can be very useful in ensuring a sequence of inputs are applied consistently in between code changes.
Another form of automated testing includes instrumenting software code so parts of it can be interpreted by an automated testing process. Automated tests of this form can systematically report on quality defects in the software code. The point of this article is not to describe automated tests in detail as many volumes are written on this topic alone. Rather, you should be aware of them as the contents of this article still applies. If you are interested in more details about formalized testing of this nature, you may want to look into test driven development.
I listed test planning last, not because it is unimportant, but to emphasize the earlier test processes. With or without an official plan, you should develop the instinct to conduct single line, function, module, and overall application tests. Those are tests you should do automatically after you write code either in the manner described in this article, or represented in books and articles on testing, and/or through fully automated testing. The way you enact testing processes will involve some form of iterative running of the application.
You should have the muscle memory to walk through the software through continuous runs as a program. When you do this, you can be sure you have at least reviewed the status of the latest code revisions you have put in place. You may not like what you see nor be in a situation to immediately do anything about flaws that emerge, but you will be aware of how well the program runs at the current revision level. Again, some minimal testing should be your natural response following code changes that compiles successfully.
A plan is often more reliable when it is written down. However, testing does not always have to be a formalized activity. A written, structured plan is an option you can apply to achieve consistent test and quality assurance processes. Later, you may want a documented test plan that steps through the various tests you are to conduct. That means breaking down the various test categories presented in this article with greater detail. Many strategies exist to define detailed test plans, but one that I find quite effective is the use of what if questions. Common what if questions are a good starting point and will define the bulk of your test plan. Also stretch a bit and ask absurd what if questions as well.
The format of a test plan will vary. Checklists work very well and is a great way to achieve a highly consistent test process. A checklist itself can vary. Some checklists simply list activities that are to occur and you check off which activities occurred. That can be the simplest checklist. Another type involves yes, no, and inconclusive columns. The first type of check list ensures you consistently do the same activities with less emphasis on tracking the results of individual activities. They are geared towards an overall sense of the process. The second type of check list is designed to guide follow-up efforts so you not only get test coverage but identify which spots to focus additional effort.
Other test plans will involve people, resources, dependencies, and can blend into project management practice. This article is not an introduction of those concerns as they are more extensively covered in books, journals, and web articles elsewhere. However, it is good to apprise yourself of more detailed test planning as it can help you further refine and focus your individual test planning activities. You may also want to look at the overall concept of software testing. Also, Steve Baker has a good, short answer on Quora.com on the 4 types of errors in the C programming language.
The next article in this series covers a type of testing that is the most effective approach of all. Although less structured, it nonetheless uncovers problems in functionality, technical implementation, and situation alignment that routine test processes may not uncover.