Stoplight integration with Jenkins


(Nicolas Tisserand) #1

Hello,

I’m pleased to share with you my Jenkins plugin for Stoplight.
This plugin is published on the Jenkins Github : https://github.com/jenkinsci/stoplightio-report-plugin

Summary :

As you know, it is possible to launch a scenario or collection either in commandline or by requesting the prism url. As result, you’ll get the console log (or a JSON file).
My plugin anlyzes these outputs to extract the useful data and show them in the Jenkins UI by providing graphs and dashboard-like directly available from the Jenkins job.

How to install it
The plugin is referenced in the Jenkins Update Centre, so :

  • Log in Jenkins as administrator

  • In the left menu, choose “Manage Jenkins” and then “Manage Plugin”, tab “Avalaible” and search for “Stoplight Report” :

  • Check the box and complete the installation (don’t forget to restart Jenkins)

How to use it

  • Go to your Jenkins job configuration

  • In “Build” section, add a “Execute shell” or “Execute Windows batch command” depending on your OS.

  • Run the scenario by using the commandline or the url provided by Stoplight. For example :


    Of course, if you choose the commandline, please ensure that prism or prism.exe binary is available on your Jenkins server with the execution rights. (You can even install it with curl or wget in another “shell” or “windows batch”).

  • Just below, add now a post-build action. Search for “Publish Stoplight Report” in the dropdown list and configure it :
    image

  • Build your job and enjoy the reports from the Jenkins menu entries :

Trend graph on the job main page :

Builds History :


(The history depth depends on the “Discard old builds” option of your job)

Details for a specific build :

To go further
This plugin is also compliant with the new Jenkins pipeline feature :
You can use the Snipper Generator or this fragment :

...
...
post {
    always {
        // Choose one of these options

        //publishStoplight consoleOrFile: 'console'
        publishStoplight consoleOrFile: 'file', resultFile: "${env.WORKSPACE}/prism.log"   // Use double-quote to have env variables replacement
    }
}        

Please feel free to use, share, and give feedback here !

Regards.
Nicolas.


Fail scenario step if connected spec operation/response not found
(Kmeister) #2

Nice integration. One feature that would be nice is if the parser/html reporter is moving to its own repo, so other build systems would take advantage.

It looks like you read the text log file, did you look at the prisim --reporters flag? I see it mentioned in the prism docs.


(Nicolas Tisserand) #3

Hi @kmeister ,

What do you mean exactly by moving to its own repo ? Do you want that the parser have its own Github repo ?
If yes, it’s not planned to extract the parser for the moment. At the beginning, I wanted only to support Jenkins for my company.
If you want the parser, it’s developed in Java. Just pick it up from here (and take the package model. If you look in the plugin sources, you’ll see that Jenkins code is very rare (only 5 classes).
Let me know if I’ve not correctly understood what you mean.

Regarding to the prism --reporters, it’s relative to the test option and it’s not supported anymore by recent prism release (>= 2.x.x). Moreover, I don’t really understand the usage of this old command.
Hey @bear and @robbins, it’s perhaps time to update this page : https://help.stoplight.io/prism/getting-started/commands (and also other docs about prism) :wink:


(Amal Jayaraj) #4

Hi @ntiss,
Thanks for the plugin. The reports look really nice and readable.
Can you please explain how you are able to generate a prism log so that it can be passed to the plugin? Currently, I tried with the publishStoplight consoleOrFile: 'console' option and it gives mixed results. Even though I have 41 tests running the plugin shows random data. Running the same collection again provides a different data.
I am not aware of any command to generate a prism log which I can pass to the plugin to publish results.
I am attaching a screenshot of the random data shown by the plugin here.


(Nicolas Tisserand) #5

Hi @amal_jayaraj

Indeed, your report looks weird.
By choosing “console”, the plugin parses the Jenkins console with regexps. This parsing can fail if there are lines before or after prism execution that are looking like prism result : regexp catches all.

As a workaround, you can redirect the prism output into a file, like this :

prism conduct "https://exporter.stoplight.io/XXXXXX/scenarios.yml" --env "host=http://....." 2>&1 | tee ${WORKSPACE}/prism.log ; test ${PIPESTATUS[0]} -eq 0
(the test ${PIPESTATUS[0]} -eq 0 is here to return the exit code of prism and not the one of tee)

Then you can run the plugin against the prism.log file :

publishStoplight consoleOrFile: 'file', resultFile: "${env.WORKSPACE}/prism.log"

Let me know if it works for you.
If not, I propose you to send me your Jenkins console output in PM on this forum (and if it doesn’t contains any sensitive data). Can you also precise which version of prism you are running.

Regards.


(Amal Jayaraj) #6

Hi @ntiss,

Thanks for that! Your command worked. The published result has become consistent and is displaying the correct values.
My Prism version: v2.0.10

Cheers!
Amal


(Nicolas Tisserand) #7

Fine, thanks for the feedback.


(Amal Jayaraj) #8

Hi @ntiss I am have multiple tests to run within a single Jenkins build job. I am running prism conduct with different scenario files sequentially in a Jenkins job. Now, is there any way the results can be grouped together and displayed or any way to have different reports for each of the prism runs executed from Jenkins?


(Amal Jayaraj) #9

Hi @ntiss Do you have any update on my query?


(Nicolas Tisserand) #10

Hi @amal_jayaraj, sorry for the delay.

So, the main constraint is Jenkins : the plugin is coupled to its current job. You can have only one plugin integration per job.

BUT, what could work, is to concat all prism output in a single file and configure the path in the plugin parameters.
For example :
prism conduct http://yourScen1 2>&1 | tee ${WORKSPACE}/prism.log ; test ${PIPESTATUS[0]} -eq 0
prism conduct http://yourScen2 2>&1 | tee -a ${WORKSPACE}/prism.log ; test ${PIPESTATUS[0]} -eq 0
prism conduct http://yourScen3 2>&1 | tee -a ${WORKSPACE}/prism.log ; test ${PIPESTATUS[0]} -eq 0

and in the plugin configuration, use “Parse a file” with this path : ${WORKSPACE}/prism.log

Let me know if it works.

Note : the commandline with test ${PIPESTATUS[0]} preserves the exit code from prism, and not the one from tee.


(Amal Jayaraj) #11

Thanks for your reply @ntiss.
I tried your suggestion in my Jenkins job which is running on a Linux AWS instance. I am running the prism conduct command within a docker container having prism installed.

sh """prism conduct '${WORKSPACE}/api_tests/PM_Users.scenarios.yml' --env 'something' --env 'something' 2>&1 | tee ${WORKSPACE}/prism.log; test ${PIPESTATUS[0]} -eq 0"""

And then publishing the report

publishStoplight consoleOrFile: 'file', resultFile: "${WORKSPACE}/prism.log"

But it looks like the log file is not getting written as Jenkins console throws this error

Running PublishStoplight step.
Mode : file
ResultFile : /home/ec2-user/workspace/mybuildname/prism.log
The file /home/ec2-user/workspace/mybuildname/prism.log doesn't exists

I had previously tried the command you suggested in Jenkins on a windows machine and it worked fine. Is there any change which needs to be made to run this command in a unix environment?


(Nicolas Tisserand) #12

In Jenkins pipeline, try using this :

publishStoplight consoleOrFile: 'file', resultFile: "${env.WORKSPACE}/prism.log"

(Use double-quote to have env variables replacement)

Also, please check in the Jenkins UI that the file really exists, in the right place.
You can also try to “hardcode” the path by using “/home/ec2-user/workspace/mybuildname/prism.log” directly in the configuration


(Amal Jayaraj) #13

Hi @ntiss,

The log file was not getting created in the 1st place. When I removed the test ${PIPESTATUS[0]} -eq 0 command from the pipeline, the prism.log file got created.

The below command is creating the log file:

sh """prism conduct '${WORKSPACE}/api_tests/PM_Users.scenarios.yml' --env 'something' --env 'something' 2>&1 | tee ${WORKSPACE}/prism.log;"""

Now the issue is with publishStoplight step. Even though the log file is present in the expected location the publish step is throwing error.
[Pipeline] echo
Publishing test results
[Pipeline] step
The file /home/ec2-user/workspace/connectedsite-profilemanager-test/prism.log doesn't exists

But the actual file exist in the expected location
[connectedsite-profilemanager-test] Running shell script
+ find / -name prism.log
/home/ec2-user/workspace/connectedsite-profilemanager-test/prism.log

I tried both the Jenkins declarative and scripted pipeline for the publish step and even hardcoded the log file path.

Scripted pipeline:
step([$class: 'StoplightReportPublisher', consoleOrFile: 'file', resultFile: '/home/ec2-user/workspace/connectedsite-profilemanager-test/prism.log'

Declarative pipeline:
publishStoplight consoleOrFile: 'file', resultFile: '/home/ec2-user/workspace/connectedsite-profilemanager-test/prism.log'

Both of the above method is not working.


(Nicolas Tisserand) #14

Hi @amal_jayaraj,

I’m a bit stucked with your problem. I don’t manage to reproduce this behaviour on my side.
It shouldn’t be the reason but are you using a Jenkins instance with slave nodes ?


(Amal Jayaraj) #15

Hi @ntiss,

The builds are run on slave Jenkins nodes. Both the master and slave are Linux instances run in AWS infrastructure. Please let me know if you need any more information/help to debug this issue. We are currently not publishing the results in our build due to this issue.


(Nicolas Tisserand) #16

And which version of Jenkins is running ?


(Amal Jayaraj) #17

It is running Jenkins ver. 2.68