How to Use Skallop Fixtures
When to use it
When the System Under Test (SUT) needs to be set up and torn down in a broad and robust manner separating code managing the SUT from code testing the SUT.
More specifically if you need to run tests requiring the telescope to be automatically maintained ON/OFF, and/or subarrays to be in an IDLE/READY state; and for which the configuration settings necessary to get to that state is secondary to testing the behaviour in that state.
A large SUT with many moving “asynchronous” parts can create a large set of failure conditions requiring extensive code dealing with “dirty” states which need to be cleared during tearing down.
The skallop fixtures are in essence there to create a set of “context aware” objects, placed in a specific state and returned to that same state when the test has finished (tear down). Before doing so it checks the readiness of the system to be taken to that state and attempt to handle an unready SUT through various controls and checks.
Thus the first question to answer when selecting a skallop fixture is to determine what the given state of the SUT must be in, before the test should exercise the SUT. The table below lists the main set of fixtures and the corresponding state they are responsible for.
Fixture Name |
State |
Dependency |
Object |
---|---|---|---|
telescope_context |
Telescope OFF/ON |
None (base) |
|
running_telescope |
Telescope ON |
telescope_context |
|
standby_telescope |
Telescope OFF |
telescope_context |
|
allocated_subarray |
Subarray IDLE |
running_telescope |
|
configured_subarray |
Subarray READY |
allocated_subarray |
|
What you need before starting
Skallop package installed. The package will automatically install these fixtures.
Setting up your environment
When the skallop fixtures performs “readiness” checks and failure handling, it does so based on knowing the specific ska telescope environment the SUT is a part of.
Configuring the telescope type
The skallop package automatically selects the correct type of telescope based on user defined configuration. To set this from a host env variable, either one of the following variable names must be used:
Var Name |
Value meaning SKA Low |
Value meaning SKA Mid |
---|---|---|
SKA_TELESCOPE |
SKA-low |
SKA-Mid |
TEL |
low |
mid |
Note that the env var TEL will be set when you setup remote connections using the setenv.sh command (see Setting ENV variables using script. )
Todo
Section describing how to change scope of readiness checks once merge for feature into master done.
Using skallop fixtures
Environment Values
By default, the fixtures dealing with the overall state of the telescope will maintain the telescope in an “operational” state (e.g. ON) for the duration of the entire test session. This saves time during a test session in which each test requires an operational telescope, but may lead to time wasted when the tests themselves are about setting the telescope parts in an operational state.
Therefore this behaviour can be “opt outed” by setting the environment value DISABLE_MAINTAIN_ON to True.
Implicit Usage
The simplest way to use fixtures is to just have the necessary fixture referenced by pytest:
@pytest.mark.usefixtures('configured_subarray')
def test_scan():
my_own_testing_code_to_run_a_scan()
my_own_testing_code_to_check_results()
Not only will the configured_subarray be used to set up and tear down the SUT for you implicitly, the dependent fixtures needed to realise a configured subarray will also perform their set up and tear downs. Each dependency on a skallop fixture will cause the setting up and tearing down to be placed on a stack as illustrated on the diagram below:
However with each layer setting up your SUT behind the scenes, more and more configuration assumptions will be made, creating potential hidden dependencies on your test working correctly. The following key configuration settings are fixed for corresponding fixture:
Fixture |
Settings |
---|---|
running_telescope |
maintained on after test (unless overridden) |
standby_telescope |
switched back on after test (unless overridden) |
allocated_subarray |
subarray id = 1; receptor ids = 1 and 2 (for ska mid); resource configuration = |
configured_subarray |
subarray id = 1; receptor ids = 1 and 2 (for ska mid); resource configuration = scan configuration = |
Explicit Usage - factory_functions
Sometimes you want to have more direct control over the input parameters used for creating the given fixture. In order to use you can substitute the actual fixture with the factory function used to create the fixture. Just prepend the suffix factory_ in front of the needed fixture. This factory function is itself a fixture which either uses the default input arguments for creating the fixture or those injected by the user.
for example: .. code-block:: python
@given(“An allocated subarray with id {subarray_id}”, target_fixture=allocated_subarray) def an_allocated_subarray(
subarray_allocation_spec: fxt_types.subarray_allocation_spec, factory_allocated_subarray: fxt_types.factory_allocated_subarray
- ):
subarray_allocation_spec.subarray_id = subarray_id # we inject a manipulated subarray_allocation_spec as input argument return factory_allocated_subarray(subarray_allocation_spec=subarray_allocation_spec)
Explicit Usage - StackableContext
In order to have more direct control over the setup and teardown configuration the corresponding fixture objects can be used directly in your test code.
In essence your code takes responsibility for the configuration and control parts that was usually done by the fixtures themselves. In order to understand how to use these objects the concept of a StackableContext needs to be understood first.
The fixture objects enable a user to load predefined tear down and setup code as a contextmanager arguments.
For example when running on pytest the following my_test()
function…
@contextmanager
def my_cm():
setup()
yield
teardown()
def my_test(context: StackableContext):
context.push_context_onto_test(my_cm())
… pytest will cause the setup()
function to be called immediately and the teardown()
only when the test is finished.
Pushing a context manager will result in tear downs loaded in a stackable queue that will be called in a FILO
order. This ensures your state will be removed in a layered fashion in the same reverse order in which yuu have
affected it.
The objects provided to you from the fixtures makes use of this mechanism when you are making a setup call ensuring the correct tear down always goes together with its corresponding setup. The user does therefore not have to concern the testing code with teardown when calling setup commands on the fixture object.
In essence, using fixtures explicitly requires two steps:
Determine and define Configuration: settings determining how the setup should be conducted. These variables will be used in subsequent setup call as parameters.
Setup: configures or changes SUT state with correct tearing down loaded onto pytest
Each fixture object has the corresponding setup call and input arguments as defined by the method call’s signature.
Fixture |
Object |
Setup call |
---|---|---|
running_telescope |
|
|
standby_telescope |
|
|
allocated_subarray |
|
|
configured_subarray |
|
|
Note
The explicit use of fixtures requires you to ‘override’ the existing fixture; i.e. the dependency listed by the function gets a different object injected than would have been the case without explicit fixture manipulation. It is therefore important that the names of your fixtures are correct.
The example below shows how a tester sets up a configured subarray by explicitly calling the
subarray setup commands (telescope fixtures are still implicit).
Note the use of the namespaced object fxt_types
to explicitly get the correct fixture type resulting from injection.
# overrides allocated_subarray
# note the return statement as the result of the
# allocate command return a subarray_context object
@pytest.fixture(name="allocated_subarray")
def fxt_allocated_subarray(
running_telescope: fxt_types.running_telescope, exec_settings
) -> fxt_types.allocated_subarray:
subarray_configuration = get_my_subarray_configuration(id=1)
return running_telescope.allocate_a_subarray(
subarray_configuration.id,
subarray_configuration.receptors,
subarray_configuration.sb_config,
exec_settings,
subarray_configuration.composition,
)
# overrides configured_subarray
# note the return statement as the result of the configure
# command returns the same object after being affected
@pytest.fixture(name="configured_subarray")
def fxt_configured_subarray(
allocated_subarray: fxt_types.allocated_subarray, exec_settings
) -> fxt_types.configured_subarray:
subarray_configuration = get_my_subarray_scan_configuration(id=1)
return allocated_subarray.configure(
subarray_configuration.configuration,
subarray_configuration.duration,
exec_settings,
)
@pytest.mark.usefixtures("configured_subarray")
def test(configured_subarray: fxt_types.configured_subarray):
set_up_my_test()
exercise_my_test()
check_my_test()
Using Fixture objects automatic teardown
Often the exercising of a test has a known change in state and therefore a standard teardown to go with it. The tester can therefore make use of the fixture object’s api to set specific tear downs at points just before the test will be exercised.
The list of api methods to use for setting up your tear down for the corresponding fixture object is shown in the table below:
Type of Test |
Fixture |
Object |
tear down |
---|---|---|---|
switch on telescope |
standby_telescope |
|
|
allocate a subarray |
running_telescope |
|
|
configure a subarray |
allocated_subarray |
|
|
run a scan on a subarray |
configured_subarray |
|
|
The examples below shows how each test sets a specific teardown to go with it. Note the tester is only required to set the SUT back to the state it received the fixture in. The fixture itself will take care of tearing down itself afterwards.
def test_set_to_running(standby_telescope: fxt_types.standby_telescope, exec_settings):
set_up_my_test_for_setting_it_to_running()
# sets a teardown to switch telescope off at the end
standby_telescope.switch_off_after_test(exec_settings)
exercise_my_test_startup()
check_my_test()
# switch off will happen automatically
def test_allocate_subarray(
running_telescope: fxt_types.running_telescope, exec_settings
):
configuration = set_up_my_test_for_allocating_a_subarray()
# sets a teardown to release subarray
running_telescope.release_subarray_when_finished(
configuration.subarray_id, configuration.receptors, exec_settings
)
exercise_my_test_allocate(configuration)
check_my_test()
# subarray release will happen automatically
def test_configure_subarray(
allocated_subarray: fxt_types.allocated_subarray, exec_settings
):
configuration = set_up_my_test_for_configuring_a_subarray()
# sets a teardown to clear subarray configuration (take to state IDLE)
allocated_subarray.clear_configuration_when_finished(exec_settings)
exercise_my_test_configure(configuration)
check_my_test()
# subarray release will happen automatically
def test_scan_subarray(
configured_subarray: fxt_types.configured_subarray, exec_settings
):
configuration = set_up_my_test_for_configuring_a_subarray()
# sets a teardown to check configuration (no waiting)
configured_subarray.check_configuration_when_finished(exec_settings)
exercise_my_test_scan(configuration)
# assumes waiting for scan to complete happens
check_my_test()
# subarray will automatically check scanning completed correctly