Fail scenario step if connected spec operation/response not found

idea

(Thomas Pytleski) #1

Summary

If a specification is connected to a Scenario Collection, and contract assertions fail to run for whatever reason for a HTTP Step, then fail the step.

Add option to disable this with a collection level setting or a step level setting.

Benefits

  • Catches OpenAPI paths/operations/responses that are not present (helps to test for completeness of specification).
  • Makes your tests more resilient. Example, if somebody removes an endpoint from the OpenAPI specification, rather than simply not perform contract testing (what happens now), an error would be thrown in any tests that try to send requests to that endpoint.

Thoughts? @ntiss


(Kmeister) #2

Per this second point:

Catches OpenAPI paths/operations/responses that are not present (helps to test for completeness of specification).

This is a good idea in practice, but sometimes a test for the following:


Doesn’t correctly get identified as being from the design doc, even though it is present.

I’d be happy for this feature if there were no false-positives.


(Nicolas Tisserand) #3

Hi @bear,

The benefits seem interesting, it’s very useful to alert the users about something that has changed. But this idea needs perhaps more reflexion. It’s not completely clear in my mind. Here are some ideas :

For example, if somebody deletes an endpoint, the server will probably be updated with the new release of the API. Thus, not only the contract testing will fail but also the response code assertion. So, the step fails, it’s normal.
On the other hand, if the server is not updated for some reason, the request will succeed, and I think that the contract testing is more likely a “Warning” but this level doesn’t exist in Prism (for the moment ?).
In my Jenkins plugin, I’ve decided that a step without any assertion is a “Warning” because it’s curious to request something without taking care of the result. It’s my invention.

From another point of view, you could consider contract testing like any other assertion in the “Test” tab.
This way, an error will be thrown automatically. Moreover, the user could add or remove the assertion if he wants.

Your suggestion of having an option at the collection or step level looks dangerous. It can lead the collection to be silent for all steps if the user always activates it. In this case, the regression tests become less pertinent and not able to ensure that the API is still compliant (yes, like now).
Moreover, having the option at collection level is far from the step screen. So, not really user-friendly

As a conclusion, interesting idea, lots of things to discuss on.