Snapshot Testing

What is Snapshot Testing?

Snapshot testing is an extension of usual expected-vs-actual assertion where the following conditions apply:

1. The expected object (or "snapshot") is too large to be contained in a few lines of code (e.g. a web page HTML content)
2. The snapshot should be tolerant to some degree of data variation (e.g. date and time of test execution)
3. (Optional) There is an automated way to update the snapshot based on the state of the actual object at any point of time

In the case of TEAMMATES, snapshot testing is useful for the following comparisons:

1. Expected web page content/DOM structure vs. Actual rendered web page
2. Expected email content vs. Actual email generated by the system
3. Expected CSV content vs. Actual CSV generated by the system

How does Snapshot Testing work?

Snapshots testing can run in two modes, namely, verification mode (default) and auto-update mode.

In verification mode, snapshot testing is essentially the same as a normal expected-vs-actual assertions, where the actual object is compared against the snapshot.

Auto-update mode is the flagship feature of snapshot testing. Essentially, this reverses the process of testing, i.e. using the actual generated object to overwrite the snapshot provided in the test. To remove redundancy, even if auto-update mode is enabled, this overwriting procedure only happens when a test fails during the test run.

For such testing method to be effective, before the changes are committed, a manual verification by the developer is mandatory to ensure only the intended changes have occurred.

How do we use Snapshot Testing?

For the web page comparison and CSV content generation, the tests are done in the frontend using Jest. Jest has native support for snapshot testing (in fact, that is where the name is obtained from!). Auto-update mode is activated by pressing u when running Jest under watch mode. More details on updating snapshots in the frontend tests can be found here.

For email generation, the tests are done in the backend. Auto-update mode is activated by setting the value of test.snapshot.update to true in test.properties.

When do we use Snapshot Testing?

Snapshot testing is typically used in the following two situations:

1. To create a new source file for a (new) HTML comparison test, email content test, or CSV file content test.
2. To update existing source files to reflect intended changes to the UI of the web pages, the email content, or the CSV file content.

The following example describes the behaviour of snapshot testing and how it can be used in practice. Let us consider the case where the following line of test code is executed:

expect(fixture).toMatchSnapshot();

    

    


    
    

Here are three possible situations and the corresponding behaviours of snapshot testing when the test is executed with auto-update mode enabled:

1. If the snapshot exists and has the correct content, no updates to the source file will be observed.

2. If the snapshot exists but has the wrong content, it will be updated with the correct content. Subsequently, the test case will pass subsequent test runs with/without auto-update mode enabled.

3. If the snapshot does not exist, it will be created with the given name (in Jest it is auto-generated) AND with the correct content. Subsequently, the test case will pass subsequent test runs with/without auto-update mode enabled.

The same idea applies to email content test:

EmailChecker.verifyEmailContent(email, recipient, subject, "/studentCourseJoinEmail.html");

    

    


    
    

When do we NOT use Snapshot Testing?

Snapshot testing is useful for regression testing, e.g. if a change to frontend logic caused the rendered HTML to change, snapshot testing will reflect those changes and it can be checked whether those changes are intended or not.

However, snapshot testing should NOT be used to check how a rendered HTML looks like before/during/after interactions, e.g.:

1. Take one snapshot at the initial phase
2. Perform a 'click' on a selected component of the page
3. Take another snapshot afterwards

Such (mis-)usage of snapshot testing will erode the value of test as the difference between the two snapshots cannot be traced.

In such a case, normal assertions should be used, e.g. by checking the component's internal data before and after the 'click'.

Best Practices

• Snapshot testing complements unit tests, but does not replace them.
• Modify snapshot objects with auto-update mode: Avoid creating or modifying snapshot objects manually. Use auto-update mode even for seemingly trivial changes. The generated snapshot may not reflect the actual object identically to achieve cross-compatibility (e.g. white space standardization).
• Once the necessary changes have been made, confirm that all the changes are EXPECTED and disable the auto-update mode, before committing the changes.
• Run tests without auto-update mode: Running any snapshot test with auto-update mode enabled can lead to false positive results since comparison failures, if any, are suppressed. After all the necessary changes have been made, run the test suite WITHOUT auto-update mode enabled to truly test the system.
• In general, only the lines that are modified should be changed. However, since there are some forms of standardization such as white spacing, sometimes multiple (seemingly unrelated) lines may be affected due to changes in the indentation, and as such it is not a cause for concern. An example of this is shown below:

<div>
  <span>
    <img>
  </span>
</div>

    

    


    
    

being changed to

<div>
  <div>
    <span>
      <img>
    </span>
  </div>
</div>