Each of us does our job to the best of our ability, regardless of which point in the software development process it involves. We try to make sure that our work adds value and is the link that allows us to produce a product that meets the requirements and specifications. However, trying alone may not be enough to achieve the goal, and that is what this article will be about.
While working as a junior tester, I experienced the perception that I was a debugger, a tool that programmers use to thoroughly check the performance of code and possibly improve it. At first, this bothered me little, as my awareness of the testing process was low. Over time, as I began to take an active role in producing test documentation, planning tests, I came to understand that this process does not end with just executing planned test cases. Updating the test documentation started to become difficult because of the information that was flowing from the customer, and I was being bypassed. The omission of me --- the person responsible for quality control of the product --- from the information chain had the effect that I had no idea about changes in key functionalities and other findings relevant to my work.
At first, my reaction to the situation was purely human feelings like anger and frustration. After a while, however, I began to wonder and dig deeper into the subject, trying to get an answer to the question, why is this happening? Why do I find out about the changes after the fact, when it is already "mustard after dinner", that is, in the reality of my work: a request with a status of "For review" will be assigned to me? Analyzing the situation, I began to understand that this is not due to ill will, but perhaps to a misunderstanding, an ignorance of what quality assurance is. I decided to change this in order to improve the development process, to improve the QA process and to raise awareness of QA throughout the team. In order to make sure that my thesis --- the team does not understand what QA is, is correct --- we conducted a survey among those involved in software delivery. We will present the results of the survey along with its analysis later in this publication, but before that happens, the co-author of this article --- Karolina Zmitrowicz, will explain the very concept of quality assurance.
Quality assurance --- what is it, what is it not?
Based on observations of the IT market and the "collective knowledge" of the IT community, one can come to the conclusion that the concept of quality assurance is mostly misinterpreted. Both recruiters and IT employees themselves (often just in QA-related positions) have a misconception about the nature, tasks and goals of the QA process. In many cases, in many companies, QA is equated with product testing and treated as synonymous with testing. This is a misconception of the state of affairs.
Testing and QA are undoubtedly related to the topic of quality, but each in a slightly different way. When wishing to explain the concept of quality assurance, it is necessary to look at the issue in a slightly broader perspective and begin by introducing the concepts of quality and quality management.
There are many definitions of quality --- from those referring to compliance with requirements, to definitions focusing on the market value of a product (https://www.jakosc.biz/definicje-jakosci/) The very concept of quality has been the subject of many analyses, among which I would particularly like to highlight the item "Doctrine of Quality" by Andrzej Blikle and a shorter, more comprehensive analysis presented by the co-author of this article in the publication "In Search of Lost Quality". Based on the analysis of the market, the behavior of consumers of products and services, the author's experience and the compilation of existing sources related to the field of quality, the latter publication proposes the following definition of quality:
Quality --- the degree to which a proposed solution delivers value to specific stakeholders in their business, technological and cultural environment.
This definition emphasizes the delivery of value --- that is, real, measurable benefits (not necessarily financial) --- to a specific audience. It is the value that is the determinant of quality --- not, as those involved in testing happen to assume, the test cases performed or not performed and the number of defects. Information about tests or defects, of course, helps to assess the level of value delivered and confront it with the desired presumed state of affairs, but it does not in itself inform about quality. Having already known the concept of quality, let's look at the topic of quality management. What is quality management? A concept that many people have heard of, but relatively few understand, and even fewer --- are able to apply to project realities? Quality management is a certain approach to the management of a manufacturing endeavor, the primary goal of which is to introduce methods and means to deliver products and services of a certain (declared) quality. The prerequisite for enabling the implementation of quality management is, of course, to understand and define the very concept of quality --- one can refer here to words attributed to Deming: "If you can't measure something, you can't manage it." The entry point for talking about quality management, then, is the very definition of quality. The quality management process itself involves at least several elements. According to the ISO 9001 standard for quality management systems, quality management is assumed to include four main elements:
- quality planning,
- quality assurance,
- quality control,
- quality improvement.
Even a preliminary glance at the above-mentioned processes allows us to place the subject of this article --- the concept of quality assurance --- in a broader perspective in some way.
The first element of the quality management process is quality planning. This is one of the key planning processes carried out during any project and should be done in parallel with the development of the project plan. In the most general terms, quality planning can be explained as the definition or selection of quality standards and requirements that the outcome of a project (e.g., an IT product) must have. Thus, the quality plan defines not only the desired definition of the quality of the created product, but also all activities, techniques, methods, tools necessary to achieve this quality.
The quality plan is implemented through quality assurance activities, understood as a set of systematic actions to make sure that we meet the specified quality requirements. Quality assurance is often explained as activities that enable quality to be "built in" to a product or process.
Another quality management process --- quality control --- makes it possible to systematically monitor the results of project work to determine whether specified quality requirements have been achieved. Thus, an element of quality control is testing, understood as checking the compliance of the results of work with the adopted assumptions (e.g. requirements). In case of deviations from the desired state, corrective actions are taken (e.g. repair of reported defects).
The last process related to quality management is quality improvement. It is worth paying special attention to this issue, as it is quite often overlooked in IT ventures. Quality improvement is an obligatory element of mature quality management --- as it enables continuous improvement of the organization's processes and thus continuous improvement of quality, both of the processes and of the products created. Mere testing or even quality assurance is not enough to comprehensively increase the quality of products created in the organization --- as it focuses on checking the current level of quality and its possible improvement through repairs. It does not give direct information about the sources of problems and possible improvements, the implementation of which would avoid similar mistakes in the future, in other words: in most cases it does not support the improvement of manufacturing processes, but only eliminates defects in the product once.
From the point of view of the maturity of an organization wishing to deliver value to stakeholders (including customers, users) in a predictable, controlled manner, it is important to implement a comprehensive quality management process.
Survey results
The survey included 131 people who work in the IT industry in software development positions. This number includes 35 programmers, 6 project managers, 79 testers and 11 people in positions that do not correspond to the above groups. The survey itself consisted of 10 questions, 8 of which helped verify respondents' knowledge. The remaining 2 were metrics --- position held and seniority worked in that position. Most of the questions were multiple choice questions. Due to the small group of respondents, we did not delve into analyzing the results by seniority. Such an exploration would only make sense when there were a minimum of 100 responses for each group. Only such a sample would allow us to obtain reliable results in terms of seniority.
Analysis of the results
The first question was: how do you understand the concept of quality assurance (QA)?
The chart above shows that 71% of respondents (without distinguishing between position and seniority) understand quality assurance as "All processes, techniques and methods aimed at making sure that the manufacturing process and the product itself meet a minimum predetermined level of quality." At the same time, 60% of respondents indicated "Testing of the product or part of the product (module)" and 67% "All testing activities performed in connection with making sure that the manufactured product will be of the highest quality," which is a significant narrowing of the view of the quality assurance process, since these responses refer only to the product testing part of the process. 48.9% of respondents agreed with the statement that quality assurance is "Creating test documentation and performing tests." The least frequently selected responses were: "Design documentation reviews" (42%) and "Don't know" (2.3%).
Another look at the responses regarding question one is the classification of respondents by job position.
Analyzing the chart, it can be seen that most groups, with the exception of testers, answered that quality assurance is "Testing the product or its part (module)." The second most frequently selected answer was "All processes, techniques and methods to make sure that the manufacturing process and the product itself meet a minimum predetermined level of quality." The group of project managers, on the other hand, considers quality assurance to be "Designing and producing a product that conforms to specified quality requirements."
The question in the survey provided an opportunity to enter one's own answer. Two project managers with more than 7 years of experience took advantage of this opportunity and answered as follows: "Along with documentation reviews --- code review, test review" and "Self-monitoring tests for more advanced projects" (original spelling). Both answers are a narrowing view of the understanding of the QA process, because they refer strictly to testing activities and do not go beyond this zone. One tester with seniority of 1-3 years gave the answer: "Looking for potential hazards that may affect the product and its quality" (original spelling), which is an accurate comment, but the listed activities should be sought in the answer "All processes, techniques and methods aimed at making sure that the manufacturing process and the product itself meet a minimum predetermined level of quality." In the group of other posts, there was a response from a person with 5-7 years of experience: "Marked answer, however, I do not agree with settling for a minimum level of quality" (original spelling). The quality assurance process must assume some minimum (the minimum level of quality acceptable to stakeholders) that we must achieve. If there is opportunity, budget and time, one should strive to provide as high a level of quality as possible, which is not always easy.
Another question in the survey was: what is the cost of quality? Respondents on this question were overwhelmingly adamant that quality is "the sum of all operating costs associated with achieving quality" (94.7%). In contrast, less than 5% of respondents admitted that they did not know what quality is. In contrast, only 0.8% of respondents believe that quality is a cost of testing only.
Let's now look at the distribution of responses in relation to the position taken
The most decisive groups that unanimously chose the answer "The sum of all operating costs associated with achieving quality" were project managers and respondents falling into the "other positions" group. Programmers have the greatest doubts, answering in the largest number that they do not know the answer to this question (14.3%), but 85.7% of programmers answered indicated the correct answer. Testers, despite being part of the testing process themselves had doubts. However, the vast majority, 97.5%, knew that "the sum of all operating costs associated with achieving quality" was the correct answer. However, 1.3% of testers' responses thought it was the cost of testing, and the same proportion admitted they did not know.
Another question asked about determining the amount of quality assurance costs.
The majority of respondents (69.5%) answered that the cost of quality is "dependent on the stage of the project in which quality assurance activities are introduced." The next most frequently selected answer was "don't know" (9.9%). In third place, in terms of choices, is "Usually too high in relation to the project budget" (7.6%). This is interesting, because one can guess where this belief among respondents comes from. Perhaps the projects they work on do not include quality assurance activities in the estimates and budget. It is also worth noting another answer that received 5.3% of responses --- "Small for smaller projects, high for complex software." The cost of quality is always built into the cost of the project. The cost of quality can vary mainly depending on the complexity of the project, rather than its size. If the quality assurance process is well structured and embedded in the development process, the cost of quality should not be a surprise in any way. However, 4.6% of respondents believe that the cost of quality is "impossible to determine at the project stage." In contrast, 2.3% of respondents maintain that the cost of quality assurance is "disproportionately high in relation to the results that can be achieved." Let's take another look at the responses of each group.
At a glance, we can see that testers focused mainly (81%) on the answer "Depends on the stage of the project in which quality assurance activities are introduced," followed by "usually too high in relation to the project budget," with a much smaller percentage reaching 7.6%. 3.8% of testers described the cost as "impossible to determine at the project stage," and the same number admitted that they "don't know." The remaining groups also overwhelmingly favored the answer "Depends on the stage of the project in which quality assurance activities are introduced." --- programmers 48.6%, project managers 50%, other positions 63.6%. Programmers also admitted that they do not know (28.6%) what the cost of quality assurance is. Three groups: programmers, project managers and, interestingly, testers chose the answer that "usually too high in relation to the project budget." The percentage distribution for this answer in turn: 8.6%, 16.7%, 7.6%.
5.7% of programmers and 9.1% of positions not falling into either group responded that the amount of QA costs is "disproportionately high in relation to achievable results." The remaining positions and project managers in nearly ⅕ of their responses considered the cost to be "impossible to determine at the project implementation stage." A small (3.8%) percentage of testers took a similar view. Additional responses included a statement from a test manager with more than 7 years of experience, who wrote: "Depending on the complexity of the project and its possible maintenance later. The scope of QA in the project should be estimated and taken into account in the project planning phase." (original spelling). The second author's response was from a tester with between one and three years of seniority: "Depends on the project and approach, but with a good approach it is a big cost, but still low compared to the costs we will incur by not ensuring quality (financial penalties, loss of customers, losses due to giving away too many rewards, etc.)." (original spelling).
The next question was: what is the role of the QA team in the project team? And here is how the respondents answered:
Respondents believe that the primary role of the QA team on the project team is "to provide techniques, methods and processes to achieve the desired level of quality" (80.2%). The next most frequently selected responses: "Minimizing the risk of project failure" and "Quality control of project work," garnered 58.8%. 48.9% of respondents believe that the QA team's task is to "support the manufacturing process," and 45% to "improve the manufacturing process." 5.3% of respondents opted for "management of managerial processes." The smallest percentage, less than one percent, admit they do not know what role the QA team plays.
Below is a percentage breakdown of responses by group.
Programmers put 68.6 percent on the answer hitting the definition of the QA team's role, which was "to provide techniques, methods, processes to achieve the desired level of quality." The next most common answer was "quality control of project work," which already defines only part of the quality assurance process --- testing. 65.7% of the surveyed programmers put this answer, followed by the answer "Minimize the risk of project failure." Interestingly, 37.1% of developers answered that QA's role is to "improve the development process" and 28.6% "support the development process." The QA team should improve the quality assurance process, which can translate into manufacturing process improvement, but the QA team itself does not improve manufacturing processes.
For project managers, the most important task of QA is to "minimize the risk of project failure" (83.3%). The next two responses scored 66.7%, and they are "Support the manufacturing process" and "Ensure techniques, methods and processes to achieve the desired level of quality." Half of the surveyed project managers believe that the role of the quality assurance team is "Process improvement," and an equal number of respondents in this group believe that the role is "Quality control of project work." A surprisingly large percentage of respondents answered that the role of QA is "managerial process management" (16.7%).
Testers overwhelmingly specified that the role of QA is "to provide techniques, methods, processes to achieve the desired level of quality." Testers also consider "supporting the manufacturing process" (58.2%), "minimizing the risk of project failure" (57%) and "quality control of project work" (55.7%) to be part of the QA team's role. The first two cannot be disagreed with, but the last one only talks about conducting test work. Testers themselves put "improving the manufacturing process" (49.4%) in nearly half of the votes. This is an unexpectedly high percentage. Other positions, not included in any of the groups, as much as 90.9% of the responses put their emphasis on "Ensuring techniques, methods, processes to achieve the desired level of quality." Another was "Minimize risks associated with project failure" (72.7%). High on the list was the answer "Quality control of design work" (63.6%), followed with the same score of 36.4% by "support of the manufacturing process" and "improvement of the manufacturing process." The last listed answer with a score of 27.3% was "management of managerial processes." According to one project manager with more than 7 years of experience in this position, "Good quality translates into a big bugfixing cost later" (original spelling). The statement was in response to a question about the role of the quality assurance team. Another statement was included. This time of a tester with seniority between 3 and 5 years: "verification and monitoring of task performance, predicting possible error locations, exploring the application for possible end-user actions in search of vulnerabilities and errors" (original spelling).
Another question in the survey referred to QA tasks. Respondents were asked to list them. The chart summarized all responses without distinguishing between groups.
The QA team's tasks also oscillate around testing itself. The most popular answer, which received 87%, was "test planning, selection of test conditions, test supervision, design and execution of test cases," followed by "Commissioning and verification of product performance" (72.5%), followed by "software defect reporting" (69.5%), "process and product monitoring" (68.7%). 64.9% believe that QA's task is automation, 63.4% "selection of metrics to evaluate product and process quality." Only at this point are responses that say what QA is about, namely "developing standards to ensure quality in the project/organization" (62.6%), "building pro-quality awareness in the team" (61.8%). Slightly more than half (55% and 51.1%) considered "reporting quality level information to stakeholders" and "developing a quality strategy for the project or organization" to be the responsibility of the quality assurance team. Only 38.2% considered "using lessons learned and experience to improve manufacturing processes" as a task that QA could perform. Interestingly, ⅕ of respondents see the QA team as responsible for "sourcing and analyzing stakeholder requirements." Marginal answers, but ones that also appear on the chart, are "debugging" (11.5%), "scheduling project work" (8.4%), and "implementing training for system users" (7.6%). 2.3% of respondents answered that they were not familiar with QA tasks.
The top three groups (programmers 82.9%, project managers 100%, testers 89.9%) believe that the main task of QA is "test planning, selection of test conditions, test supervision, design and execution of test cases." Other positions put 72.7% on this answer. Project managers put "development of standards to ensure quality in the project/organization" on par with the aforementioned answer (100%). However, this answer was given by only 64.6% of testers and 45.7% of developers. Project managers scored high on "Launching and verifying product performance," "Monitoring process and product quality," and "reporting defects in software." All three received 83.3% each. Testers also found the answers "Launching and verifying product operation" (74.7%), "Reporting defects in software" (74.7%) to be extremely important. Meanwhile, the third answer ("Monitoring process and product quality") has 68.4% of responses, on par with "Test automation." For the remaining positions, on par are "Building pro-quality awareness," "Test planning, selecting test conditions, test supervision, designing and executing test cases," and "Launching and verifying product performance" (72.7%). Programmers and project managers themselves do not see QA tasks as "building pro-quality awareness." This response is with programmers at 45.7% and with project managers at 33.3%. Similarly, "using experience and lessons learned to improve manufacturing processes" ranks for both groups mentioned above.
The next question posed to respondents was: What should the QA team be informed about?
The most frequently selected answers were, in turn, "about changes in requirements" (90.8%), "about changes in the scope of the project" (90.1%) and "about new versions of applications delivered for testing" (84%) . Next in line, still frequently selected, were the options: "about product version release plans" (78.6%), "about the status of problem (defect) resolution" (77, 9%) and "about external changes affecting the perception of product quality (e.g., regulatory change)" (74%). The least frequently indicated responses were: "about the content of the contract" (14.5%) and "about the cost of the project" (7.6%). 3.8% of respondents answered that they did not know what to inform the quality assurance team about.
All groups felt, albeit in different percentages, that QA should be informed about "changes in requirements" and "changes in project scope." To these responses for project managers should be added "about new versions of the application delivered for testing," "about the status of development work," and "about external changes affecting the perception of product quality (e.g., regulatory change)." All of these responses are at 83.3% of responses. Other responses from project managers remain at 66.7%. The exception is "about the content of the customer contract," where the response percentage is at 16.7%. The other groups responded more variably, although in relative agreement. Testers and respondents in other positions agreed (72.7%) that QA should be informed about "schedule changes," where programmers only half (51.4%) answered this question. The response "about the status of development work" for the other positions also ranked at 72.7%, with testers at 73.4% and programmers at 48.6%. Testers and other positions ranked very similarly, with a difference of one hundredth, in favor of "about the status of analytical work" (54.4%, 54.5%). This answer would be worth detailing to find out what specific information testers expect from business analysts. Programmers, on the other hand, did not find this information as significant --- a percentage of 31.4% in this group. The least frequently selected answers for all groups were: "about the content of the contract with the client", "about the cost of the project". 11.4% of developers and 1.3% of testers answered that they did not know what QA staff should be informed about. In this question, there was an additional answer from one tester with seniority between 3 and 5 years. It reads as follows: "The QA team should have the best information about current requirements to ensure that the product is responsive to needs and that it is of the highest possible quality relative to those needs" (original spelling).
The last question was an open-ended question and asked: what value does the QA team bring to the project/organization? Only about half of the respondents (47%) answered this question. A sizable portion of them referred strictly to testing itself. Here are some of them (original spelling): "We test the product, determine if there are bugs, possibly suggest what could be improved," "The more often a new version is released, the less likely it is to introduce bugs not directly related to the scope of changes in the release," "Detects software defects before it is released to production," "Product quality information and bug list." There were also other voices (original spelling): "Builds pro-quality awareness in the organization, supports the production of a better product, reduces the risk of producing a defective product, improves processes, indirectly influences greater customer satisfaction ", "The QA team brings incredible value to the project --- stands for quality, creates an atmosphere of respect for the product, the customer and the work of the entire team involved in the project.", "Reduces the likelihood of project failure; helps reduce costs due to project errors (to the greater extent the earlier it is included in the project work).", "Increases the responsibility for quality throughout the team/organization.". Most of the statements that did not give emphasis to testing itself came from testers, and they were also more likely to answer this question.
Finally, we want to show what percentage of respondents have ever worked with people in a QA position.
Conclusions
At the beginning of the article a thesis was stated. Let me reiterate it at this point --- the team does not understand what quality assurance is. The results of the survey show that although teams rather understand what the QA process is for, they overlook the components of the process and look at QA itself as quality assurance, which is wrong. Often, testers themselves do not recognize the breadth of the QA process, although it is from them that one would expect the most extensive knowledge of the subject.
The biggest surprise is undoubtedly that project teams see the QA person as having many functions that the QA person does not represent. Overlooked in all this is the building of pro-quality awareness, the improvement of the quality assurance process, which naturally forces the organization to introduce standards, and it is this action that translates into improvements in the manufacturing process.
The only solution to this state of affairs is to make the team aware of quality assurance, to take care of communication in teams. A good starting point for a better understanding of the roles and tasks of those in quality assurance positions, as well as the rest of the team, can be the creation of a responsibility matrix, which clearly and precisely defines the positions in the organization and their associated responsibilities, and thus dispels any doubts, drawing a picture of the quality assurance department for all employees.
Going back to the words that were written at the very beginning, just trying to make our work give value is still not enough. It is important to look at the processes as a whole, and the team's responsibilities should be clearly outlined so that there is no doubt or guesswork. Otherwise, responsibilities will become blurred and processes will not be improved.
