preview

Analysis Of Squarespace's Product Addictions

Good Essays
Open Document

Over the last four years, Squarespace’s product offerings have expanded significantly. To keep up with Squarespace's growing platform, we needed to scale our functional test coverage. We created Firepit, a user interface (UI) for configuring and triggering tests. Firepit empowers engineers to easily run custom test suites in Squarespace's deployment environments. The foundation of our testing philosophy is straightforward: write tests that are simple, readable, and stable. Automated screenshot testing is notoriously unstable. To mitigate that instability, our tests carefully place screenshots to minimize the usage of DOM assertions. Using lean DOM assertions is essential for keeping our tests simple and scalable, allowing few points …show more content…

This builds trust in the validity of a failure alert. In July 2017, an individual test in our smoke suite had a per run pass rating of 99.49% (excluding true failures). In a 50-test session, that means the compound probability of a session reporting a false failure would be 23.4%. To reduce this rate of spurious failure, we will run a failed test three additional times. A test result is recognized as a true failure only if the test fails on all four runs. This configuration reflects our focus on detecting severe breaking changes and helps foster trust in the validity of failure alerts. Operational impact Firepit has had a significant operational impact at Squarespace. With automated functional testing, we can implement continuous deployment without manual QA for day-to-day releases. For a release to successfully build in our staging environment, it must first pass a series of unit, integration, and API tests. Finally, it has to pass a smoke suite of around 50 Firepit tests. If the build fails any of those tests, the release is designated as a failed build within Bamboo and is not promoted to production. In addition to the build failing within Bamboo, a Slackbot notification is sent to our #engineering Slack channel with the failed test titles, links to the test run page in Firepit, and alerts those who committed code to that release. We execute tests on seven virtual machines (VMs). Each VM runs eight docker

Get Access
Related
Get Access