I am less than lukewarm on those sort of "Natural-Language-Like" DSLs that seem to mostly come out of the QA space.
When I worked together with a QA team for automated testing, they used Cucumber, which uses Gherkin as an english-but-not-really DSL. Now the idea is obviously, that someone who doesn't know how to code can still describe and read test scenarios. But what ended up happening eventually is that I spent a lot of time writing code to support different Gherkin expressions, but then had to take over writing the Gherkin expressions as well, because even though Gherkin -looks- like english, it still behaves like a programming language. In the end, it would've been much easier for me to just write the tests in a programming language from the start, instead of using an abstraction layer that turned out to serve no purpose.
What I took away from that frustrating experience is that a lot of people think that learning to code is learning a syntax of a programming language, and if you could just do away with that pesky syntax, everybody could code. But, obviously, the syntax ist just the initial barrier, and actually coding is crystallizing and formulating intent - something which programming languages are much better at than natural languages.
But for some reason, this low/no-code idea never seems to die down, especially in some spaces, and I have the feeling I'm just missing something here.
Yeah, I share the sentiment. What it boils down to is that you need to write the test two times. The supposed feature that once written the keywords can be reused is mostly a joke. There will be few heavily-used keywords, and a really long tail of one-time keywords, which is a pain.
I see a single advantage to these frameworks though. They force you to write a test in a specific way that looks like a real-world scenario with clear delineations between setup stages, assertions, and so on. Not that it's impossible to achieve with regular programming language, obviously.
I like using catch2 (c++ test framework). the keywords are only reused when they group together in the code.
GIVEN("Some setup") {
// code to create the setup
WHEN("some use of the setup") {
// do something
}
WHEN("something else"){
// do something unrelated to the previous when
}
}
> The supposed feature that once written the keywords can be reused is mostly a joke.
I've seen well maintained and designed cucumber / rspec / robotframework "code" and i've seen some really horrible shit. Keyword re-use is really a joke if you are not familiar with any good practices to actual software development. Granted, in "low code" scenarios, the worst selling point is "you don't need to know how to write code" - which imho is bullshit and from my pov, statements like i quoted echoes that.
Disclaimer; i partake and contribute to RF ecosystem. And i do prefer to write my tests or process automation with it even thought i can write code in multiple languages..
From my POV, the idea of Natural-Language-Like tests is it is theoretically more inclusive, can avoid duplication, and can create a more abstract description.
Granted, I've had middling success convincing non technical people to write or even look at tests, but I think it's irresponsible not to try to create this bridge, depending on the audience of the code. Because very few projects are well documented, and this syntax can act as documentation. In my current work, I am using BDD style to create high level flows, where screens are automatically captured, which is useful both for verification and generated documentation, especially when versioned with code.
It can avoid duplication (and its pitfalls), because in unit style tests, there is a text description along with the code, which can drift away from being accurate. If the test is the description, that won't happen.
It creates a more abstract model, because the tests are forced to a natural language, which will tend ot make it less implementation dependant, so the implementation can be swapped out. I've had good success with this approach, seamlessly migrating a complex workflow from one backend to another, and the tests were kinda pricesless.
I also find keyword and flow re-use to be quite high, and it's great how this approach means the more tests you write, the less low level, implementation specific code and scenarios you need to write.
I will never understand the reason for Cucumber to exist. As you say, it just introduces unnecessary complexity for no reason.
If you can't code, you shouldn't be writing automated tests. Most of the "benefits" of Cucumber can be gained by using a good reporting framework. I like to use Serenity as I love its reports.
> I will never understand the reason for Cucumber to exist
In a previous company I had one use-case that I thought cucumber helped out a lot at.
We were writing software that would be vigourously tested and approved by a customer who didn't have any technical knowledge. Their initial plan was the "Excel sheet of death" test case management system with really fuzzily named tests.
I convinced them to write Cucumber tests, which turned out to be around 100, using the very explit syntax that then got called stupid by the cucumber devs and moved to the "training-wheels" gem. I never expressed what we were doing as cucumber tests. I just told them "Write them like this and it is easy to understand" and they agreed and did so, then they were signed off by our project team and them.
After that I then automated them our side so that we could run them quickly and easily in our pipeline while the customer could still use the tests for their tedious manual user acceptance testing at the end.
It worked out quite well. The customer never knew we automated the tests our side, they were just thrilled that something like 99% tests passed first-time and the 1% that didn't were down to interpretation differences.
Congratulations! I think this is actually the first time I've seen a use for Cucumber. Obviously, it would have been better if you didn't have to do this but I've spent enough time as a consultant to know when to just work around the customers.
The fact that gherkin can be be turned into an automated test is a side effect. The real purpose is to get non-programmers to put their requirements in a rigorous form so that programs can implement the spec and know it will do what is intended.
I have never seen it work in practice, but that what it is supposed to be about. If your managers are not looking at the gherkin and making suggestions then there is no point in doing it.
I'm a big fan of the Gherkin syntax and the output of my Example Mapping sessions is generally a big pile of them.
Automated tests are then written based on these but not "using" these. By not "using" the Gherkin scenarios for our tests we can be looser with the language and not have to worry about RegEx.
If a manager, or other non-coder, wants to see the tests, they can look at the very pretty report that was generated.
I can count on zero fingers the number of times I've worked with a non-technical QA person who's written tests, let alone benefitting from cucumber. As the GP says, the rest of the time is just wasted maintaining a pseudo-natural language programming language.
Unfortunately, I'd need a few more fingers. Not too many though which makes that argument for Cucumber fairly moot. I did work somewhere where each team had 2 testers. One to write the cucumber and one to write the glue code for it. It was about as successful as you could imagine.
I'd also say "non-coding" rather than "non-technical". Coding is just one aspect of being "technical".
There is benefit in making the tests easy to read but this is better done in actual code, e.g. the screenplay pattern.
My experience with trying to use Cucumber for tests pretty much shown me that it doesn't work there. But one thing that I played a bit and seemed that it might work was using Cucumber for writing requirements. At least for when the requirements are written by not technical people, it could give them better structure and for me as a tester make it easier to spot possible problems in them. But I haven't had to chance to try this more, so it might up end not working just like tests themselves...
yeah we had cucumber at a company I worked for. it was touted that PMs and other non-coding staff could write tests and how great that would be. After we switched tests to the Golang cucumber implementation, exactly no one outside of Dev team wrote any tests. meanwhile Devs have to use this awful new framework everytime we make a change. burn it with fire I say
"Not being able to code" is not black and white. Many can follow basic steps, as long as it gets not too complex. Thus "Low code" tools like Powerautomate,selenium ide, ahk or UIvision are great for people to do test automation. They are a quick way to create an automation. And many users learn/start to code this way.
QA should be writing the underlying code along with the BDD-style tests, using something like a Page Object Model design, with BDD-wrappers on top mostly for non-technical analysts like PMs and BAs to understand coverage and add test cases. I think if you're doing it for anyone else's benefit it falls flat. If it's a no-code solution for QA then it becomes a drag and you should hire an SDET to manage it (and train the other staff to write the underlying code too). If BA/PM is not involved in test design it's a pointless wrapper, I agree. Personally I'm not a fan and spend my time focusing on better reporting so people can see what's tested & if they want an additional case I can just add it myself.
The problem is there are three to 4 veins of QA. SDET (largely automators, tend to scale and manage the automation), analysts, who go over test results and pull out trends, manual testers/non-automators, because nothing brings out the bugs/bad assumptions like putting a non-techie in front of something, and then... well, I'd lump software maintainers/DevOps into it too.
I'm not impressed at all by Gherkin, BDD, or Cucumber. Never once has any business person gone "oooh, let me read the tests!", And in the rare circumstance they do, what you'll find is a secondary grammar develops where particular step implementations gain loaded significance, far and above the natural language implementation of a step, making things complicated.
At this point, I just tell people to keep notes, make checklists, get familiar with the data model for test data authoring, then go eith God, and always put the User first.
Mileage varies greatly, but there's a culture problrm in a lot of places where "the fiture must go out" always takes precedence over "the spec must be right".
And no one ever wants to rip apart testing tools to figure out how they work.
I think it all boils down to execution and the company’s mindset and skillset.
I was also a Quality Engineer for a time, and one of our bigger projects was to implement precisely a Natural Language system or something so that PMs or whoever wanted to “live in the future” could write their tests directly in the User Story.
What we did was create a large set of commonly used “phrases” to interact with systems, be it UI or APIs, which we coded the actual test framework for, document them for anyone wanting to write User Stories (in Rally platform, like JIRA), the instructions on what to do if a phrase didn’t exist for their case, etc. It took us 6 focused months, but we delivered, and people were happy. Of course, we are talking about technical PMs, PMs that met with designers and set standard guidelines, PMs that required API documentation and read it, and the like; just not “coding much” PMs.
Albeit this being a company where we already wrote automation tests right next to the developer’s timeframe for their own coding, where a feature got both code/tests right around the same time, 99% of the time, so it was easier for people to take advantage of features instead of shallowly complaining.
So, yes, I concur with you that probably your case would be better if everyone were a developer, I don’t really see the need for those kind of DSL, but if you reeeeeeally need to have other people intervene, and want them to focus more on their end of the bargain (not coding), I think these DSLs have their place: Assuming the whole company has the stomach for it.
How is it based on anything in Excel? There's nothing about test plans in RF. Test plans are about schedules, strategy, deliverables, roles etc. And those are typically garbage. Absolutely nothing related to RF.
Might be difference in local terminology, but by "test plans" I mean spreadsheets, typically in Excel (sometimes printed and bound), that described step by step every test for manual tester to follow.
And that's how a lot of testing is done in various companies, especially where manual testing is still common, at least in Poland.
I presume low code/no code comes from the simplest notion of programming language, which is a medium between human and machine. And the easier it is to read and write means the "better" it is. Therefore pitching it to a non-technical or even semi-technical people is as simple as teaching people to write code at surface level.
However what people don't understand deeply is that any language is a set of symbols and rules requiring a certain amount of cognitive space and a certain time to absorb it. The more ambiguous the symbols are, the closer the growth of the number of the rules to exponential. This is a fundamental point that happens in real-life but is rarely understood, moreover to programmers, surprisingly.
I believe a proper low-code/no-code should have natural language processing capability and understanding context, not just providing syntactical aliases.
I work in an environment which has dozens of products, very significant complexity and legislative needs. Cucumber works very well for us because we use it for specifying outcomes. There's way to much Gherkin out there which tries to use it for describing a journey (I click this, then this, and this....)
It works really well for us to get a shared understanding between a non-technical business, devs and QA.
That said, we then hand it the Gherkin over to the SDETs who fettle it according to our house style (which is strict and must pass enforced linting) and it does take gardening as the product matures. It's our SDETs who write the test code: we wouldn't dream of letting a non-technical person actually drive the test logic itself.
I’ve spent a fair number of years doing corporate cucumber training.
At every client I emphatically insist that if they are not using Gherkin as a “Rosetta stone” for the business folks then they’re wasting their time with cucumber. Don’t go to the trouble of writing gherkin if you’re not using it as living documentation. Save yourself loads of trouble and just use an ordinary test framework.
What you describe may apply more to writing code, rather than reading code. Of course, knowing syntax is not sufficient for programming, but if you want just to review existing code, (unfamiliar) syntax may be an obstacle to non-programmer domain experts.
I've read something along these lines about the choice of OCaml at Jane Street, where they argued that their investor-types are able to proof-read new code, despite not being programmers, because the language "reads like english", similar to SQL or COBOL.
Can you really effectively proof read if you're just guessing about the underlying semantics? If the goal is just to give investors a false sense of security then I suppose it achieves that
When I worked together with a QA team for automated testing, they used Cucumber, which uses Gherkin as an english-but-not-really DSL. Now the idea is obviously, that someone who doesn't know how to code can still describe and read test scenarios. But what ended up happening eventually is that I spent a lot of time writing code to support different Gherkin expressions, but then had to take over writing the Gherkin expressions as well, because even though Gherkin -looks- like english, it still behaves like a programming language. In the end, it would've been much easier for me to just write the tests in a programming language from the start, instead of using an abstraction layer that turned out to serve no purpose.
What I took away from that frustrating experience is that a lot of people think that learning to code is learning a syntax of a programming language, and if you could just do away with that pesky syntax, everybody could code. But, obviously, the syntax ist just the initial barrier, and actually coding is crystallizing and formulating intent - something which programming languages are much better at than natural languages.
But for some reason, this low/no-code idea never seems to die down, especially in some spaces, and I have the feeling I'm just missing something here.