It's a complex task to stand up multiple pipelines of multiple apps for automated animation creation. And, as any software engineer knows, sometimes making the smallest change in one part of your complex software system can wreak unexpected havoc in another.
To protect our API users from unexpected breakages, we've created a fully automated test suite to run against the testing/pre-release areas of our API. I'm a firm believer in the idea that any web-based service API should have a web page that tests itself. Before we publish any updates to our API from our test staging areas, this test suite executes all core features of our API so we're sure we haven't broken anything - or 'introduced regressions' in programmer-speak. We also add new tests for any broadly-reaching features of the API, and sometimes even for individual animations that are particularly important to us. Our test suite currently executes over 300 tests in about 3 minutes. Keeping that time low is important; tests suites should be a minimal barrier to the developer release cycle. Our test page also supports arbitrary degrees of multi-threaded requests, so we can measure the total throughput of the pipelines in our API.
The test page has varying degrees of local test verification. In some cases the test page's JavaScript itself is able to verify the integrity of the data returned by our API. In other cases, test verification is best performed by supporting service calls on the API, such as perceptual and absolute image comparisons.*
Also the generated images from the test run are downloaded for display after the test suite completes. Helpfully, the page also supports side by side comparison for any tests that failed. If the images are different due to intended changes, or image comparison sensitivity is too high, the test result images can be added to the canonical library quickly and efficiently, or fully replaced.
Here's a link screen capture movie of the test page in action:
https://www.youtube.com/watch?v=VRhvoSdD6y8 * We're considering a new feature to our service; a public facing image comparison service. Visually and statistically differentiating images efficiently as supported by a web service would greatly enhance our test page, and would be helpful to may other software engineers and image comparison folks as well.
Comentarios