“Made simple”. Well...I tried it and it’s by far the most complex, sprawling, cluttered, “everything under the sun” testing framework I’ve ever encountered.
I kinda liked the premise, but take “simple” with a truck load of salt.
Further it's not helped by the fact that author's attitude is that of a whiny baby[1]. His entire strategy of publicising his work is taking a huge shit on the current industry wide standards[2][3]. Also see his reaction to this post itself[4].
I can't help but note that Intuit, who profits off of complexifying the "simple" tax code, cannot see the complexity in their "simple" test automation framework. Conway's Law strikes again.
I've recently used Karate for contract testing and I have mixed feelings about it. In my case it served me reasonably well. Usually my go-to choice for contracts is "REST Assured", but in that recent project I had more complex JSONs and I had to write rather lengthy validation rules manually. I felt that rest assured's java-based syntax could quickly turn against me so I was looking for an alternative and someone suggested Karate.
Pros:
* the resulting code was relatively short
* once we got used to Karate the relatively complex tests were quick to write
* the whole framework is really flexible, you can describe complex json structures using just DSL and you can always call a Java method if you are missing something.
Cons
* some of the commenters here already noticed a lengthy README.md - well, you need to read this whole file before starting any work. The framework is reasonably well documented, but you need to ingest the documentation as a whole before attempting to write a test. You cannot just search for "how can I do ___ in Karate", I mean you can try, but you are unlikely to find an answer this way.
* entry point while short is rather steep. You need about 2-3 days before you begin to understand what's going on in the code
* the code is unreadable for bystanders. Good luck with understanding "#[] #(^SOMEVAR)" or "#[_ > 0]"
* the architecture: the whole thing runs on Maven/Java. Java interprets Karate DSL, which can contain Javascript(!) which can call Java again! Yes, you can make a chain of calls Java->DSL->JS->Java! To be fair, it makes the whole framework really flexible, but still, wtf.
Will I ever use it again? Maybe. I'd say it's specific tool for rather specific tasks. It can serve certain purposes quite well. But if you are wondering what framework you should use in your next project - there are much safer choices out there.
Gherkin is supposed to be for expressing human problems in a human-readable form, that also happens to be useful for computers. Using it to specify technical details of APIs (as opposed to, I dunno, writing some code to hit those APIs) seems like a really weird choice.
Their first example for BDD is just painful to look at.
Transforming the "business" language into a "developer" language so that "there is no glue code" is missing the point of BDD.
Imagine putting that example in front of a user and asking: "Is this what you want?". Even if the end user is the developer using your API it makes little sense.
As usual, I mat be missing something here - but it seems it's basically moving the "glue code" to the Gherkin level, making using the Gherkin useless.
given that some test people might be reading these comments - is anyone aware of a configuration for test frameworks that:
1. lets users opt code out of the testing footprint (I know this is a given), but also
2. monitors opted-out code for being called a certain number of times, therefore meaning it's depended upon, therefore meaning it SHOULD be tested, and gets flagged?
I feel like this would be the way to go to minimize non-valuable testing while allowing you to discover code thats becoming more leveraged and therefore more critical to test.
1. In any testing framework I've used, this is (as you said) a given. The exact mechanism you have for slicing and dicing your tests varies, but this is usually well supported IMO.
2. This feels like the job of a code coverage tool to me (and it's something I would like as well), but most coverage tools I've seen don't quite have the granularity you're looking for. In certain ecosystems you could hack something together pretty quickly (e.g. the `trace` module in python).
Correct me if I'm wrong, but this idea sounds like you'd like to bias your tests away from small units and toward integration tests (and a similar tool could be useful for dead code elimination and a few other tasks). I know they're not a panacea in general, but in your personal experience what problems have you found in fine-grained tests that outweigh their benefits?
I'm a believer in "right-sized" testing, meaning balancing the cost of test creation and updating against the value the tests provide. I believe tests provide the greatest value when they are testing:
* public methods
* critical business logic
* highly reused core logic
in other words, "code for others".
For code that's used for one specific page render or something similarly isolated, I think it's more questionable whether to bother going for 100% code coverage.
The latter category is where I'd like to "trust but verify" with devs permitting them to opt out of unit tests where they believe the code isn't critical, but have some confidence that we would catch code if it became more central and then enforce testing requirements.
I could see something like this being extremely useful when working with legacy code. In the book "Continuous Delivery (Humble, Farley)" they suggest when working in legacy code without tests, to begin adding testing to systems starting only with the parts you changed. If this is hard to determine, a test feature that shows unit dependencies could be really helpful.
Choosing java as the runtime for the framework is laughable when you want people not to care about the technology that the binary is running on. It requires you to install the java runtime, or run a bloated container, just to run the api testing.
If you want the runtime to be invisible, don't ask me to install a runtime.
I kinda liked the premise, but take “simple” with a truck load of salt.