Over the last year at SingleStore, we've migrated from Jest and React Testing Library (RTL) to Cypress Component Testing (CCT) for UI integration tests.
In this article, we'll dive into the why and the how. We'll also reflect a bit on the migration, and how it has worked out for us in the end (it's been about five months since we finished this work). In doing so, we hope others can get insight into how we approached the issues we had. However, it’s important to note that different developers might have different experiences with the same tools.
As such, we're not recommending to anyone that CCT is a direct replacement to RTL, Jest and JSDOM and that you should migrate straight ahead! We encourage everyone to follow their own investigative path on these things and to make the choices they think are right for them.
We've also written about our broader history with UI testing at SingleStore before.
Introduction ("The Why")
For the UI of our Customer Portal, we have unit tests, integration tests and end-to-end tests. Our unit tests were very stable (Jest), and our end-to-end tests were also quite stable (they're written using Cypress). However, the UI integration tests which spin up the entire frontend application but mock the backend/API were very flaky. This is very strange since there is no network on these tests — which is a very common cause of flakiness in tests.
Moreover, and perhaps most importantly, debugging these tests was very hard. With RTL, which recommends using Jest, all the rendering is simulated using JSDOM (which renders your application's React tree in JS objects). So, there's really no browser engine running. This means that CSS is not in consideration for your tests. If there's an element that's not visible because of a styling issue, the test won't catch that. Instead, a button that was supposed to be hidden will indeed be clickable in the test.
This isn't a huge deal for catching bugs but it is actually a very big deal for debugging tests. If we can't easily visually inspect the state of a test, it's very hard to debug a failing test. Slowness also became a problem for our test suite using Jest and JSDOM. As more and more tests were added Jest became slow, which seems to be a common issue.
As for the flakiness of these tests, it remains mostly a mystery to this day (and actually, we still have a lot of unexplained flakiness in our migrated CCT integration tests today). We have narrowed down some possible sources of flakiness in the meantime however. A big one is improperly written tests. For example, a test could become flaky if user actions are dispatched immediately before a component re-renders, which causes a stale reference to a DOM element to happen sometimes. Another example of an improperly written test would be a missing API mock. It is easy to miss when this happens, and it can cause test failures that are difficult to diagnose using the output from RTL.
In addition to this, we were using msw.js to mock the network in our RTL tests (and continue to use it with CCT today), and we believe that it might have been a source of the flakiness (most errors we saw were related to network calls). At the same time, we found some interesting Jest issues related to MSW mocked requests bleeding between tests while running the tests in parallel. We decided to move to Cypress before we had the chance/time to get into the root of the problem.
After spending a lot of time investigating all of the flakiness issues, we got to a point where we decided that it wasn't worth it to spend any more time on it. Instead, we decided to look for alternatives to our problems. We knew Cypress had recently launched an Alpha version of "CCT" which allowed us to render individual components — instead of running an app build and accessing pages by URL. To do this, CCT uses webpack dev server to compile a dev build of our app. This can make startup times slower than with RTL, but also allows for hot module reloading and visual feedback that we did not get before.
Besides looking at CCT, we also thought of some other options:
- Playwright – did not support component testing at the time, but does now (something we’re keeping in mind).
- Testcafe – we never got to try it.
- Pleasantest — we never got to try it.
So, we ended up going with CCT mainly because of the familiarity we already had with it. As I mentioned before, our end-to-end tests were already written with Cypress. So, it made sense to keep our list of tools smaller and standardize around Cypress for both integration and end-to-end tests.
The Migration to CCT and How We Use CCT ("The How")
The vast majority of our integration tests render the entire application and test various user flows. These are long tests, which give us a lot of protection against bugs, but they are also harder to migrate since you have to migrate large chunks at once. However, little by little we did chip away at this. So far, we are down to two RTL test files and have roughly 70 CCT test files.
The main technique we used was that for any test which was flaky (there were a lot of them), we'd convert them to CCT instead of trying to fix the underlying flakiness. But more importantly, some members of our team really crushed this work faster than we expected. We also encouraged test conversion if new tests or test modifications were needed on existing RTL test files.
The first bit of work we had to do was actually set up CCT. To run tests in CI, we're just using the same image that we use for end-to-end tests which is an extension of cypress/included. So, the CI job for running the tests is fairly straightforward.
- We start by installing our dependencies with
CYPRESS_INSTALL_BINARY=0 pnpm install --no-verify-store-integrity --frozen-lockfile -r. We skip installing the Cypress binary since that's already included in the Docker image. - Then, if we're running on the main development branch, we setup recording with Cypress Dashboard. We do this by appending
--record --key ${CYPRESS_DASHBOARD_RECORD_KEY_CCT}to the last command. - Then we initiate msw with
npx msw init. - And the last command is to run the CCT tests:
npx cypress run-ct ${OPTIONAL_PARAMS}.
(We always record videos and screenshots of all of our tests which we keep for 15 days.)
One important thing to mention is that we fundamentally believe in testing the application in the same way that our users interact with it. To that effect, we're using @testing-library/cypress and @testing-library/user-event so that we get the benefits of RTL's API which discourages testing implementation details (and also testing applications in ways that also help ensure their accessibility).
We've also been able to build entire UI features with mocked backend APIs very easily. We can build entire pages and even flows where all the API is mocked while the API changes for a given feature are still being implemented. This is a feature that has resulted in a huge amount of positive feedback from the team and improved development velocity.
Looking back on all of this ("The reflection")
So it's been slightly over a year since we've started this entire process and looking back on it, it's basically night and day for us. The experience of writing and debugging tests with CCT is simply amazing compared to RTL. Moreover, onboarding new engineers is easier since we only have 1 tool to learn. As for performance, it's possible that CCT tests are slightly slower to execute but we don't notice it at the moment. Finally, our tests are more realistic since we can actually detect issues caused by CSS.
Another advantage of Cypress, in our view, is the API which gives it another real edge in terms of developer experience. With Cypress, it's much easier to wait for things to happen (it naturally waits for things instead of forcing us to choose between getBy/findBy or using waitFor). Also, we don't have as many issues with doing things async vs sync (e.g., not wrapped in act(...)” warnings). The error messages from Cypress are also generally much more complete and easier to read.
We still have flakiness in our integration tests which we cannot really explain, but it's much less common than with RTL. We've spent a lot of cycles on this in the last few months but still haven't reached any conclusions. Our latest investigations lead us to believe that it might be related to Apollo Client. That is because we see on and off network errors both with msw.js for mocking the network layer as well as when using cy.intercept. We also still haven't parallelized the execution of our CCT tests, but as the number of these tests grows it's something we'll have to look at.
So all in all, we've learnt a ton during this process and the migration has been a success. If you're interested in this sort of work, reach out!
