title: [[WWDC 2020]]: Get Your Test Results Faster --- title: [[WWDC 2020]]: Get Your Test Results Faster --- - **Metadata**: - Date: [[June 24th, 2020]] - Conference: [[WWDC 2020]] - Speaker(s): [[Sean Olszewski]] - Video: https://developer.apple.com/videos/play/wwdc2020/10221/ - Slides: - Tags: [[Talks]] - [[Testing Feedback Loop]] - Write tests - Run tests - Interpret results - Write more tests - Or move on - Repeat - Ensuring you always get feedback - Get even faster feedback - A Real World Example - It’s Monday - CI Tests from Friday never finished - [[[[Xcode]] 12]] - Reasons why tests could hang - Deadlock - Extremely slow progress - Bad timeout values - Execution Time Allowance - Enforeces time lmiit - Xcode will… - Capture spindump - Fail the tests - Restart the test runner so other tests can finish - By default 10 minutes - If you need more time for all tests, you can edit it - `XCTestCase.executionTimeAllowed` - Rounds to the nearest minute - Test Plan - Edit Test Plan - Enable Test Timeouts - Re-run tests - This will generate a new report in the navigator - Shows a failure reason that explains it went pass the time allowance - Double click to open a `spindump` - Preamble - Series of stack traces - Precendence - `executionTimeAllowance` - `[[xcodebuild]]` option - Test Plan setting - System default - What happens if a test asks for unlimited time? - Set a maximum via Xcode Settings or `xcodebuild` option - Recommendations for Execution Time Allowance - Use `executionTimeAllowance` - Use `[[XCTest]]` performance APIs - Use Instruments to debug - See [[[[[[WWDC]] 2018]]: Getting Started with Instruments]] - Running tests on Multiple Devices - Non-distributed testing - Serially running tests on a single run destination - Parallel Distributed Testing - Multiple Devices, each run one Test Class - Allocation is non-deterministic - Supports [[Unit Tests]] and [[UI Tests]] on iOS and only Unit Tests on macOS - Now supports tvOS via `xcodebuild` - `-parallel-test-among-destination` - 30% speedup with only 2 devices - Recommendations - Identical devices and OS versions per test suite run - If you can’t, prefer distributing logic tests only, not UI related tests - Intentionally use Parallel Destination testing if you want to test against more OS versions - See [[[[[[WWDC]] 2018]]: What’s New in Testing]]