CLI tests yields different results than app

Running a test folder with same parameters, same environment yields different results than on the app or web version. CLI seems to behave differently.

The error that’s showing is a schema mismatch, this is not showing on the app or web runs. Is this something I’m missing? Or does it take some time to “refresh” the schemas?

Yes, if you’ve updated the endpoint spec or schema, the CLI data may take some time to sync. Please try running it again now.

I am also experiencing this. It does not resolve itself after waiting

We have a github action running once a day, if it might take time to sync, most likely it’ll give us false negatives.

Last saved was over a day ago, and I’m still seing errors that I’m not seeing on the app or web

This appears to be a potential bug. Thanks for reporting - we’ll investigate and fix this issue as soon as possible.

Thank you!

Hi, we haven’t been able to reproduce this issue yet. If the problem persists, please contact us and provide screenshots showing the assertion differences between the app/web and CLI versions, along with the corresponding response definitions.