Get results for all tests when running folderId on CLI

Get Support
1

When I use
apidog run --access-token XXX -f 1076345 -n 1 -r html,cli,junit
the json filer generated only contains one of the test scenarios executed, not all, nor I have one per test scenario.
Can you please share how can I get all tests for all scenarios when using the folderId as input?

2

I was able to get 2 files now:
apidog run --access-token XXX -f 1076345 -n 1 -r html,cli,junit --out-file {SCENARIO_NAME}

but I still get less information on the junit report than the total number of tests I have under the folder.

The Junit results have all tests name as “validate output” and I can only see 7 of them, while on the 2 test suites I have 21 tests in total with different names.
Is that a known issue with the Junit extractor?

3

Hello,

For the command -f {folderId}, the final HTML and JSON formats will only output one report file, which contains the test results for all scenarios in the specified folder. {SCENARIO_NAME} will be replaced with NULL, so we recommend using {FOLDER_NAME} instead.

Regarding the JUnit report, it should generate reports based on the number of test scenarios. In our tests, the {SCENARIO_NAME} in the JUnit report is correctly replaced with the scenario names, and we haven’t been able to reproduce the issue you described.

Could you please provide:

  1. Your Apidog version (check via apidog -v)?
  2. Screenshots of the relevant test scenario folder and exported test reports?

This will help us investigate further.

Thanks!

4

I think there are some major differences when running from cli and when I use the application.

  • when I run the test scenario on application, each API request is validated against the schema and each request has description. When one of the requests doesn’t match the schema it is marked as failed.
  • When I run the CLI the description for each request is not available on the test results and the schema response is validated but test is not marked as failed. Also the junit only has results for requests that have post processor defined

Here is my test suite for positive scenario for a service and all endpoints:

image
image1152×757 124 KB

The test failure is due to schema validation failure.

Here is what I get from running it on CLI:

image
image857×696 33.3 KB

and here is what Azure-pipelines reports when reading the junit generated:
image

Here is what I see on Junit report:

image
image1451×282 69.5 KB

So, all requests without post processor have 0 tests even if they had schema validation.
Also notice that test case names are marked as post processing rule name applied, not the request description…

So, when I look the results of the test on aure devops it looks like I only run one test and all passed, instead of running 6 tests and one failed

5

Regarding the differences you mentioned between CLI and application execution, from your screenshots, both show 1 failed step. Could you clarify where exactly you see the differences?

Also, we don’t validate request data models automatically (you’d need to add custom assertions for that) - we only support automatic validation of response data structures.

6

The problem is on the junit generation. The CLI report says http requests 6 and 1 failed, but junit file says 1 test and 0 failures which is what the ccli report says under assertions.
So, what it looks like is that the junit shows Assertion results which will not capture all failures on tests.

7

Understood. We will fix the content of the exported JUnit report in a future update.