As a SDET, my question is - does anyone uses tools such as this in professional work? If yes, what is the use case?
IME, creating and developing an automated test framework is 1% trying to find the best locator, and 99% maintenance of tests that use dynamic, nondeterministic flows, fighting with backend issues/flakiness etc. I find Codegen to be OK-ish for only the simplest/worthless test paths.
Playwright has some pretty good strategies for selection using positional locators[0] which generally feel pretty resilient. Not quite as ergonomic as Taiko[1], but still really good.
Both have good heuristics for auto-waiting that simplify interactions and make them "faster" because there's not as much wasted time doing manual waits.
I particularly liked Taiko + Gauge[2], but Playwright can also be plugged into Gauge and the ergonomics are really nice because then you can write "plain English" test scripts that translate into system actions.
> creating and developing an automated test framework is 1% trying to find the best locator, and 99% maintenance of tests that use dynamic.
This is completely true. Quality e2e tests that have good setup and cleanup and aren't flaky take focused dev work. I find myself regularly using the locator finder tool in playwright UI/debugger but I've never found any real value in using the test generator.
I could see it being valuable in a really small subset of exceedingly simple UI, probably personal-project level but even those tend to grow too complex for this.
I tend to agree. An example of how it can be net-zero gain is in this screenshot[1]. To have something maintainable, you have to do your own refactoring pass on the generated code. At that point, it's not much effort to write the test yourself than it is to refactor all of the redundancies.
I do use Playwright’s codegen a lot and I have found it most valuable when I mentally predict what it should generate and then look to see how it differs. This often highlights incorrect assumptions. I would say approximately 1% of the generated code ends up in the tests, but about 30% of the generated code informs how the tests are written.
Yes, my company uses them to verify that nothing serious breaks as a result of deployment to staging. Of course we have unit and integration tests too, but this is used as a last line of defense against breaking things entirely.
It cries wolf a lot, but it's also saved our ass several times. So we run 3 attempts per test and if all 3 fail, sound the alarm.
My org uses codegen as a starting point for one of our test layers.
It works for us probably because we sidestep the pain points you list - the environments we run in are pristine complete copies of known datasets, we remove as many sources of randomness as possible, and our environment flakiness level is very low.
They still break but usually because the locators in use have been chosen poorly (or we've made planned changes to a page/component)
We're a web based b2b saas that runs an instance of the entire environment for each of our customers. Our non prod setup consists of a bajillion static test environments but more importantly we use testcontainers to spin up the transient test environments from database snapshots. Using the recorder on the static environments (before the transient ones existed) _was_ a pain
We use Playwright and the Page Object Model approach to test our applications because that approach is more maintainable for larger applications. Automated E2E testing is something all developers are expected to do, not just an "SDET" engineer or QA engineer role. While we take the POM approach, I frequently use Playwright's recorder to help me get selectors and interactions with the DOM needed to verify a captured feature. Next, I take the code generated and incorporate it into object models. Adding an automation attribute to all of your elements that are both unique and stable helps go a long way to make E2E tests enjoyable.
> Adding an automation attribute to all of your elements that are both unique and stable helps go a long way to make E2E tests enjoyable.
First of all maybe it wasn't clear from my comment but I use (and like it a lot) Playwright on a daily basis.
Second - did you mean using a testid for example? Because we specifically avoid using special attributes for this purpose and instead follow best practices as detailed here: https://testing-library.com/docs/queries/about#priority (main reason being: killing two birds with one stone by making sure your page is accessible).
Imho IDs are more explicit, and a bless when working with large codebase apps, when trying to finding the related code to an element just grab the id and search for it.
SDET here, and no, I don't use these tools. Hammering out the actual test code is the easiest (and most fun!) part of the job for me, tools like this one actually make that experience painful and slower. :(
It works quite decently. I've noticed a nice side-effect of setting up playwright tests is that it helps catch accessibility issues, like interactive elements not being properly labeled as such.
I sometimes use the generator to get an idea of what selectors I might want to use and then extract the relevant bits to page objects. The risk is that not everyone knows that recorded tests are a starting point and not the destination.
If you are looking for something that generates selectors for you but also gives you some control and granularity, I made a free chrome extension called automize
IME, creating and developing an automated test framework is 1% trying to find the best locator, and 99% maintenance of tests that use dynamic, nondeterministic flows, fighting with backend issues/flakiness etc. I find Codegen to be OK-ish for only the simplest/worthless test paths.