Test Experience Share: We should consider the impact of the hardware improvement to the test code

This experience is achieved during I investigating some job fails in the Alchemy inputs testing.

The test:

n  The test purpose:

1.         Verify input characters to an Alchemy TextBox control in Windows CE 7

n  The test process:

1.         Add a “TextChangedEventHandler” to the Alchemy TextBox

2.         Input characters to the Alchemy TextBox via key_event (This is just an OS behavior)

3.         Monitor the Alchemy TextBox during the input, and update a specified flag (This will handle via the Alchemy engine, the xamlruntime.dll)

4.         Input done

5.         Check the flag to verify the test result

n  The results:

1.         After the hardware improvement (e.g. from CEPC to Tegra2, which is a dualcore system), the test fail

n  The reason:

When the hardware is improved, all the test steps are become faster than before, but the step 3 is not increased as much as step 2.

So there will create some delays, which means thought the OS has done the input and is going to verify the result, the Alchemy is still performing the event handling, this is the issue.

The summarization:

Hardware improvement may bring asynchronous increase to the test steps, which means different delays, so we need to consider this asynchronous during the test solution design.