Thinking about replacing an existing automation tool in your organization or introduce a new one?
Do not know how to approach the subject and how to start the POC process?
Are you afraid of ambiguous results due to the Proof of Concept (POC) process that will leave a place for doubt?
Let’s see together how to lead the POC process in a way that will not leave a place for doubt and how to choose the right automation tool for your organization.
Replacing the tool or implementing the new tool from scratch in the organization is a process required to be well planned, with the definition of process goals and what should be achieved in choosing the new tool compared to the old one.
The POC process will include a clear definition of the criteria by which the tool is measured.
Criteria should be approved in advance by the relevant stakeholders in a project(beyond the testers other teams can use the tool as well).
Presenting criteria to the additional stakeholders, have another aspect that can be earned and is a kind of”commitment” of additional teams to the issue of tool replacement and future use.
“Commitment” of development teams can contribute quite a bit to the success of implementing the tool in the organization.
So let’s see how to lead the POC process correctly?
Before starting the POC it is very important to understand why you and your team want to go with another tool? What are the pros and cons that affect the work of the team on a daily level in the existing tool?
Try to discuss with a team and understand why you and your team want to replace the tool.
In one of the projects I have heard over time from the team the various complaints about the tool, but not something concrete, so I held a meeting to discuss it. My goal was to understand if the complaints related to real technical gaps in a tool.
The team was asked to bring the pros and cons of the tool to affect the daily automation testers’ work.
We have seen a team that spends quite a bit of time on the maintenance of automation tests via the tool, some of the functions do not exist in the tool despite the expectation of the testers, all this alongside the high cost of the tool.
As a result, were decided that we want to go to the POC process and by its results understand whether we are ready to replace the tool or not.
Let’s see together the required steps to select the automation tool:
1. After understanding the “why” the POC process should be planned as part of the version roadmap, set a reasonable time as part of version development for the execution of POC by relevant staff members.
2. Choose 3 tools in the market for decision. How do choose the three tools for the POC process in the tools market?
You can consult with other test managers, check with other colleges via QA groups managed in social media, try to explore the “territory of the market” to understand what tool might suit your organization.
3. Set up the end-to-end system flow you want to test, it could be more than one end-to-end flow.
In my opinion, it is important not to choose a very complex end-to-end but make sure it covers the main functionality of the system. The goal here is to see how the tool handles the end-to-end process of the product.
4. Define criteria for the automation tool. Important to choose criteria worthwhile for your product.
See example for criteria table implemented for one of the products:
With starting the POC process, we selected criteria for a user interface automation tool (GUI). You can see that we defined in advance priority for each criterion for the tool, thinking what the need for the product and what is important to us as an automation testing team.
Certainly, criteria priority may vary according to the product needs.
| Criteria | Priority | Tool #1 | Tool#2 |
| Cross Browsers Testing | High | ||
| Supported Dev Languages | High | ||
| Supported Test Framework High | High | ||
| Setup and Execution- CI/CD | High | ||
| Integrations | High | ||
| Code oriented | High-Medium | ||
| Version Control Supported | High | ||
| Easy Maintained | High | ||
| Debugging and Logging | High | ||
| Easy to write | High | ||
| Reporting Capability | High | ||
| Maturity, Documentation, Support | High | ||
| Cost-Efficient | High | ||
| Speed | High | ||
| Wait for Element | Medium | ||
| Video and Screenshot support | High | ||
| Parallel Execution | High | ||
| Remote Execution | High | ||
| Manage Cookies | Medium | ||
| Repeatable, Reliable | High | ||
| Email Notifications (Custom email message received if tests are passed successfully/ failed/or any network failure) | High | ||
| Dev Engineer Skill Level | High | ||
| Installation Complexity (both in time and in resources | Medium | ||
| Load/Performance | High |
You can see that we have defined a variety of criteria, some of which are very technical and some related to the costs that the chosen tool may bring with it, and yes it is even worthwhile to introduce criteria such as the training cost of the staff in the organization.
It is important to think about all the aspects during the planning POC process which may be relevant, it will help to get a complete picture.
Afterward, it is recommended to hold a meeting with relevant stakeholders and present to them the selected criteria, during the meeting over the meaning of each of them. It is important to hear all opinions. The criteria could be updated after the meeting according to meeting conclusions.
An additional goal of the meeting to agree on which end-to-end flow needs to be selected for POC. This way you can be sure that you are on the right path before the start of the POC and you can earn additional “commitment” to the process by stakeholders.
5. Starting the POC process.
- The first stage-according to the criteria we have defined previously starts to learn the tools, some of the information can be found online and some via work with the tool (installation, writing scenarios, attempting to execute reports, etc.)
- The second stage is to automate an end-to-end flow for each of the tools. During the process, the test manager is advised to follow up closely on POC progress and make progress status once every two three days and see if progress according to a pre-defined plan.
6. Schedule an internal team meeting to present results and answer questions about the tools and their implementation.
7. Schedule a concluding meeting with relevant stakeholders to view POC results, answer the questions and decide about final tool selection.
To sum up:
The POC process can be challenging, yet a process teaches breaks routine and feels like lifting something new with anticipation for better results. It is important to lead a process in an orderly manner and predefined as well as involving all stakeholders in order ensure objective and unquestionable results.
Is this post helpful for you? Share your thoughts in comments on how it helped you.
Join to Productive Hut Family and be part of a testing community!
No Comments