How to Clean Test Data From End-to-End Tests

When you are creating an end-to-end (e2e) test with Cypress or Puppeteer, one plaguing question always comes up: How do you clean up the data?

There are several different strategies I've used.

• Drop the Database

  This is where you run a script to drop the data and schema of a database, typically DROP DATABASE posts;. Before running the e2e tests a cleanup script will be run and the database will need to be created anew.

• Truncate the Tables

  This only removes the data within the tables. Generally, I've seen the test process signal to the application to clear the tables. Confession: I've implement HTTP endpoints to truncate (yikes!) only loaded into the application in a testing state. Configured or not, having any behavior in app that deletes all your data is a scary prospect.

• Transactional

  This one is rarer, but it can exist when the e2e process can share a connection with the application. At the beginning of the test, a transaction is started and at the end of the test the transaction is rolled back. Again, if the e2e test is in a separate process this one isn't an option.

After much trial and error, I've found the one that works the best to manage your data is: just leave it. Let the data live after the test runs.

The biggest downside to cleaning the data is that it lets the developer test the system when it's a blank slate. Hopefully a production system is only a blank slate for a moment and then people are using ever after. End-to-end tests should be written so that you can get the most signal out of them as possible. When tests run in a system with existing data, those tests are running in an environment that is more similar to production.

How do we do this?

Let's use the classic blog application example project. We are testing that we can create a new blog post and then we see it in the list of blog posts. A Cypress for this might look like:

cy.click("Create New Post")

cy.findByLabelText("Title")
  .type("My Test Title");

cy.findByLabelText("Content").type("Hello World!");

cy.get("Create Post").click();

cy.findByRole("alert").should("have.text", "Post created!");

cy.get("Posts").click();
cy.findAllByRole("listItem")
  .get(0)
  .should("have.text", "My Test Title")
  .should("have.text", "Hello World!");

Now if we run this test twice, we will encounter an error because our system doesn't allow blog posts to have the same name. To get by this error let's do what our user would do: Let's make a unique name. For our test, we can generate a random number and append that to our post name. We will also need to reference that when we make our assertion.

const seed = Math.random();
cy.findByLabelText("Title")
.type(`My Test Title ${seed}`);

//...

cy.findAllByRole("listItem")
.get(0)
.should("have.text", `My Test Title ${seed}`)
.should("have.text", "Hello World!");

Now our test can run an infinite number of times! Our previous data can exist side by side with our prior data.

Other Benefits

Now, not only can we run our tests over and over, but we get several other benefits as well.

1. Our tests operate more like a user would in a production environment.

   We would never reset our database in production! Why would we do it in our tests then?

   Sometimes when there's a bunch of test data in our system, it's difficult to navigate, select the right thing, wait long enough, etc. What is your test telling you? Is it plausible that a user would experience the same confusion?

2. You can run these test in other environments

   Since our tests don't rely on any special connections to the database or things enabled in our app, we can run these tests in a staging, or if we are feeling bold, production. For a run against production, I'd try to identify ways to hide the produced data from actual users.

3. You use your application when it has a bunch of data

   If you run these tests often, you'll end up with a bunch of data. This will cause you, the developer, to have to use the app when it has a lot of data. You will feel where performance slows down. You'll encounter things (before users do!) that the designers and product managers hadn't thought of.

CI

What about CI? Often the database is created each time our tests run.

I haven't used a CI env that has a persistent database. You could try to set one up and see how it goes! However, I don't think that this is a big problem. I think the major benefit is that the developer experiences the friction that occurs when there's a lot of data in the system.

In Conclusion

Ultimately we want to our end-to-end tests operate as close to what our users will experience as possible. When we let our test data accumulate we get closer to the real thing.