No longer fail runs based on recording limit

[tennisgent]

I can't see any explanation for this behavior here so I'm going to assume this is a bug or a feature request for something that may have been overlooked.

Current behavior:

Once I've reached my recording limits for a month (on the free tier), I get this error:

It looks like this is your first time using Cypress: 3.2.0

[16:10:43]  Verifying Cypress can run /root/.cache/Cypress/3.2.0/Cypress [started]
[16:10:45]  Verifying Cypress can run /root/.cache/Cypress/3.2.0/Cypress [completed]

Opening Cypress...
You've exceeded the limit of private test recordings under your free plan this month. The limit is 500 private test recordings.

To continue recording tests this month you must upgrade your account. Please visit your billing to upgrade to another billing plan.

https://on.cypress.io/dashboard/organizations/<org_id>/billing
error Command failed with exit code 1.

Desired behavior:

I'd prefer that you let my tests still run and execute and just print a warning that the tests aren't being recorded. That way I can continue to run the tests with --record in CI all month long and just take advantage of the recordings whenever I am within my limit for that month.

I can't update my CI scripts to know when I'm within the limit or when I'm not so I will always run the risk of getting failing CI runs because I might cross the limit threshold. The only safe thing would be for me to not ever record runs in CI, but I would prefer not to do that. I love your dashboard and I would love to take advantage of it as much as possible.

I understand I'm on the free tier and I'm not exactly your target customer so I understand if you chose to stick with test failures as some way of pushing users to the paid tier. But I don't have the money for the paid tier on this small side project so I would be forced to stop using the dashboard all together.

Versions

Cypress 3.2.0

Thanks in advance

[jennifer-shehane]

@tennisgent Tests that exceed the test recordings limit on the free plan do not record. If you are within the limit at the beginning of the cypress run command, the recordings should continue as normal, even if you exceed the test limit, but upon exceeding it, the next cypress run command will no longer run the tests in CI and instead show this error.

This only happens on the free plan. The test recordings are always recorded on any of our paid plans or our OSS plan, which offers unlimited test recordings for open source projects.

To continue running your test suite, you can upgrade to a paid plan or apply for an OSS plan (if eligible) or remove the --record flag from cypress run.

This is intended behavior that we don't plan to change at the moment.

Thank you for the feedback. I will forward this along to our team. We're always looking for ways to improve our pricing offerings and your feedback is a big part of that evaluation.

[brian-mann]

@tennisgent what you're asking for can actually be achieved today using the module API programmatically: https://on.cypress.io/module-api

Here's some sample code...

const cypress = require('cypress')

const didRunError = (results) => {
  return results.message && results.message.includes('Could not find Cypress test run results')
}

// run using our module API
cypress.run({ record: true })
.then((results = {}) => {
  if (didRunError(results)) {
    return cypress.run({ record: false })
  }
})

The code in the module API that specifically handles this is here: https://github.com/cypress-io/cypress/blob/master/cli%2Flib%2Fcypress.js#L31

Under the hood - if the run errors out before running tests it will not generate any test results, and therefore we resolve with the object as described above.

However, I've opened a new issue to make this a little easier - for instance we should probably capture the underlying error reason why Cypress abandoned the run, and then we should probably reject the promise under the hood rather than resolve it with an arbitrary object.

[ghost]

@brian-mann Thanks for that! I will write a little node script for our CI runner that uses that module API. That seems like it would be pretty simple. Much appreciated.

[edelgado]

Thanks @brian-mann, I was able to make our CI runs less fragile by using your suggested approach. Nonetheless, I wish the default behavior was for the tests to run, and just skip the recording part.

I understand the motivation behind the behavior, but maybe we can be alerted of having reached the limit via email or some other avenue?

[jennifer-shehane]

Hey @edelgado, Cypress does send an email when the test recordings limit is reached to all admins and owners of the organization. We also display a warning banner within the Dashboard when viewing the project and the organization's billing.

Admins and owners of the organization can see the amount of test recordings used per month by visiting the organizations 'Billing & Usage' page at any time.

We're definitely open to improving this experience for everyone though and welcome any more suggestions. cc @cypress-io/services

[edelgado]

Hi @jennifer-shehane. Thanks for your reply. The point I'm trying to make is that stopping CI runs because the recording limit is too disruptive.

Notifying folks via email / dashboard messages is great! I did see the message on the dashboard, but I did not received an email, btw.

My suggestion is to keep alerting via email and the dashboard, but to still run the tests, albeit without recording it.

Thank you!

[ghost]

@jennifer-shehane For what it's worth, I never received any emails about any limits either. I have just gone back through my email to verify and there's nothing there, even in my spam folder.

[jennifer-shehane]

@edelgado Thanks for the feedback.

The emails would have been sent to the email that is set on your Cypress account shown at https://dashboard.cypress.io/#/profile

[edelgado]

@jennifer-shehane you are welcome. Thanks for listening.

PS: Besides the point, but I didn't get an email to the email address that shows in my dashboard profile.

[dotherightthing]

I find the language here confusing and agree with the others that the tests should run irrespective of whether the video is stored / sent to https://dashboard.cypress.io. Running tests and recording them are two separate things. I like the videos but I can live without them, but Cypress has zero value if I can't see my test results in the Terminal.

[edelgado]

@dotherightthing I ended up coming up with this script. It does the check Brian suggested, passes params and returns the correct error codes:

https://gist.github.com/edelgado/9dbceac4e275c0579fe91170ecc35645

[thomasboyt]

I find this to be a total dark pattern - it seems like the only reason this is done is to convince people to convert to paid customers to unbreak their build. A warning is fine, even a loud and pronounced one, but to completely exit tests if you run out of recordings is absurd. I should not have to commit changes to my CI config because I ran out of recordings.

I'm pretty disappointed, since I use Cypress at work and was excited to use it on a side project. I enabled dashboard recordings since I usually stay under 500 tests a month, and figured there wouldn't be any harm in hitting the limit anyways - and now I see this is not the case :(

Since Cypress doesn't have any plan between 0 and 100 dollars a month, and my side project is (a) noncommercial and (b) not under an OSI-approved license, there's not really any path forward here other than either removing Cypress recordings or using one of these wrapper scripts. I get not wanting to give stuff away for free, and wouldn't have even minded if y'all just didn't have a free plan, or only had a timed trial, but I think breaking builds when the free limits are hit is the worst imaginable experience I could have with Cypress.

[malimccalla]

Are there any updates on this? I find it hard to believe that this behaviour has been decided on by anyone other than a money person 😔, a developer would design it in way that allows the tests to run without recording them.

I agree with others in this thread that it should just warn that tests are not going to be recorded and link to an upgrade link. At the end of the day my tests need to pass if they are recorded or not and for my current projects budget this means removing the --record flag instead of upgrading.

As a side note (and some customer feedback), I'm actually more likely to upgrade to a paid plan if I keep seeing a warning in my tests than if I were to remove the --recordall together and never put it back in due to this restriction. I'm even happy for the GitHub integration to show me the warning and prompt me to upgrade in my PRs once I've hit the limit... just let my tests run please 🙏.

[jennifer-shehane]

The original intention of this behavior was to be completely clear about the outcome of the run when test recordings are reached.

We do not maintain the records of the test runs at all once the limit is reached for free plans.

At the time we decided it would be better to exit the run and be clear that no recordings will be saved as opposed to running the run and giving a false impression of the recordings being saved only for the developer to go to the Dashboard and find their recordings do not exist. Our impression at the time being that it's very easy for developers to not read stdout during CI runs and therefore not see any warnings about not recording.

We do read the feedback from this thread and we have an ever evolving product, so please feel free to express your thoughts on this behavior for us to use to evaluate any future decisions. I just wanted to express our original intention.

[breart]

Let me join your friendly talk :)

From what I've read above, it does not seem like you listen to your customers (which actually works against you, because the worst customer — is an annoyed customer). Instead of trying to find a common solution, you just ignore everyone here.

If you don't want to change the default behavior — it's fine, but at least add an option to override this behavior. I don't want to remove --record and commit this change every time just because of your bad UX. I like your product, but this solution can be improved in favor of your customers.

You could add an option like this:

cypress run --record --key ... --mute-record-failures

And everyone here will be happy 😃

[tommueller]

I was also quite surprised to realize this is how Cypress handles reaching the free-quota. I would also strongly suggest to change this. Just show a note ("Plan limit reached, these tests will no be recorded") and run the tests nonetheless.

Also I do not think that this will influence the developers experience in any way. You get an email, you get a banner on the dashboard and a message in the CI. If that is not enough to clarify the situation, then I think someone chose the wrong job ;)

[mattiasw]

The current behavior actually forced us to never really try the recording functionality for any extended time. I hadn't really looked into recordings until we wanted to run parallel Cypress tests. It worked fine, until even our small team reached the recording limit within only a couple of days. I don't think the workarounds posted earlier will work for parallel runs so I wrote our own parallelization script which seems to work fine and we will probably not look at the dashboard again for a while.

I think there are two issues here: 1) failing the tests when the recording limit is reached (add an option at least), 2) too few recordings per month to actually have time to see the value and/or too high price for the first level.

Otherwise, Cypress is awesome! :-)

[jennifer-shehane]

After reading feedback provided from this thread and discussing requirements needed to implement this change with our team, we have decided to change how we handle recording limits being reached for free tier plans - to no longer fail and exit the run. The new behavior will reflect how the recording limit is handled for paid plans, the run will exit normally and viewing data will be limited within the Dashboard views.

We don’t have a clear estimate on when this will be delivered, but we wanted to update everyone to know that there is work planned and to reopen the issue.

Thank you all for providing feedback. We welcome criticism on the product decisions we make and want you all to be successful in testing with Cypress.

[jennifer-shehane]

Recorded runs that reach the limit within a free tier will no longer exit the run. When the test recording limit is reached - a warning message will display within the stdout like shown below, then continue on to running and recording the run. Admins and owners of the organization will also get an email warning them of the overage.

We will probably tweak this warning message to be a little nicer in a future versioned release of the Test Runner.

Upon going to view the data within the Dashboard, the run data will be hidden and parallelization will be disabled until the organization upgrades to a paid plan. This is the exact same behavior as any other paid plan that reaches their test recording limit.

Thanks everyone for your feedback! If you encounter any issues with recording in the free tier, please make sure to open a new issue detailing the problem.

Closing this issue as resolved.

[tennisgent]

@jennifer-shehane Thank you for addressing this issue. We really appreciate the ability to use recordings on the free tier. 🎉 👏

[TirathSojitra]

@jennifer-shehane It is mentioned that parallelization is available in the free tier plan of the cypress dashboard. However, when I run in parallel 4 threads, the tests are getting executed in one thread only. Am I missing anything?

[hckhanh]

I doubt the parallelization will not run if the account exceeded the free tier, which means it is the paid feature.

[hckhanh]

I get the same issue