API Reference¶

Core¶

chrysalis.get_knowledge_base() → KnowledgeBase | None¶: Fetch the knowledge base of the current session.

chrysalis.register(transformation: Callable[[T], T], invariant: Callable[[R, R], bool]) → None¶: Register a metamorphic relation into the current knowledge base.

chrysalis.reset_knowledge_base() → None¶: Initialize a new knowledge base for the module.

chrysalis.run(sut: Callable[[T], R], input_data: list[T], search_strategy: SearchStrategy = SearchStrategy.RANDOM, chain_length: int = 10, num_chains: int = 10, persistent_db_path: Path | None = None, num_processes: int = 1) → DuckDBPyConnection¶

Run metamorphic testing on the SUT using previously registered relations.

Parameters:

sut (Callable[[T], R]) – The ‘system under test’ that is currenting being tested.
input_data (list[T]) – The input data to be transformed and used as input into the SUT. Each input object in the input data should be serializable by pickling.
search_strategy (SearchStrategy, optional) – The search strategy to use when generating metamorphic relation chains. The serach strategy defaults to SearchStrategy.RANDOM.
chain_length (int, optional) – The number of relations in each generated metamorphic relation chain. The chain length defaults to 10.
num_chains (int, optional) – The number of metamorphic chains to generate. The number of chains defaults to 10.
persistent_db_path (Path, optional) – The location to save the database file for the resultant duckdb connection. If no value is specified, the duckdb connection is made in-memory and thus the data in the connection is lost when the processes exits.
num_processes (int, optional) – The number of processes to use when performing metamorphic testing.

Invariants¶

chrysalis.invariants.equals(curr: T, prev: T) → bool¶: Check if current value equals previous value.

chrysalis.invariants.greater_than(curr: float, prev: float) → bool¶: Check if current value is greater than previous value.

chrysalis.invariants.greater_than_equal(curr: float, prev: float) → bool¶: Check if current value is greater than or equal to previous value.

chrysalis.invariants.is_same_sign(curr: float, prev: float) → bool¶: Check if current and previous values have the same sign.

chrysalis.invariants.less_than(curr: float, prev: float) → bool¶: Check if current value is less than previous value.

chrysalis.invariants.less_than_equal(curr: float, prev: float) → bool¶: Check if current value is less than or equal to previous value.

chrysalis.invariants.not_equals(curr: T, prev: T) → bool¶: Check if current value does not equal previous value.

chrysalis.invariants.not_same_sign(curr: float, prev: float) → bool¶: Check if current and previous values have different signs.

Tables¶

class chrysalis.tables.KnowledgeBase¶

Bases: Generic

A collection of metamorphic relations.

Relations are registered in (transformation, invariant) pairs. Under the hood, a relation stores single transformation and a list of invariants.

Is it important to note that lambda functions cannot be used for transformations or invariants.

property invariant_ids: dict[str, str]¶: Return mappings from invariant name to invariant id.

classmethod load_previous(current: Self, conn: DuckDBPyConnection) → Self¶

Load a previous knowledge base.

When replaying metamorphic testing, the knowledge base used to perform the original testing must be available. The previous knowledge base can be loaded from the current knowledge base if the current knowledge base is a superset (includes transformations, invariants, and relations).

Parameters:

current (KnowledgeBase) – The currently knowledge base loaded in the session. This knowledge base must be a superset of the knowledge base that is to be loaded.
conn (duckdb.DuckDBPyConnection) – A DuckDB connection that contains the results of a previous metamorphic testing run.

register(transformation: Callable[[T], T], invariant: Callable[[R, R], bool])¶

While there are other ways to add transformations, invariants, and relations, this should be the default method.

property relations: dict[str, Relation]¶: Return a mapping of all registered relations.

property transformation_ids: dict[str, str]¶: Return mappings from transformation name to transformation id.

class chrysalis.tables.TemporarySqlite3RelationConnection(knowledge_base: KnowledgeBase)¶

Bases: TemporaryDirectory

A temporary sqlite3 database designed to be used for transactional inserts.

During metamorphic testing, we want to record the order of which transformations are applied and the result of each invariant tested in a relation chain. This data is available after each step that is executed in the engine. By definition, this pattern follows the transactional processing pattern and thus justifies using sqlite3.

This database maintains two tables that are required to store the results of execution of a relation chain. The first table is the transformation table which records the order of which tranformations are applied to input data. The second table failed_invariant represents the result of a individual invariant tested for a given transformation that failed. It is important to note, it is possible that multiple invariants apply to the same transformation and thus a single transformation can have multiple invariants that fail.

Given this database design, it is not required to store the result of transformations on the input data. Instead, the transformations can be reapplied to the input data to acheive indiviudal instances of transformed data that is requested. This dramatically reduces the amount of data that is required to be stored.

Since this is a temporary sqlite3 connection, it is expected that all data will be extracted before exiting the context manager.

__enter__(*args, **kwargs) → tuple[Connection, Path]¶

Create a database in the temporary directory created during initialization.

In addition to creating the database, configure the sqlite3 database rules and create the relevant tables.

chrysalis.tables.sqlite_to_duckdb(sqlite_db: Path, output_db_path: Path | None = None) → DuckDBPyConnection¶

Convert a chrysalis sqlite3 database into a duckdb database.

Converting a sqilte3 database to a duckdb database is non-trivial due to the conversion needing to be done one table at a time. Additionally, some tables, such as the applied_transformation table require additional care as self-referential foreign key constraints can become problematic.

Execution¶

class chrysalis.execution.Engine(sut: Callable[[T], R], input_data: list[T], search_space: SearchSpace, sqlite_conn: Connection, sqlite_db: Path, num_processes: int = 8, writer: Writer | None = None)¶

Bases: Generic

A class responsible for execution in metamorphic testing.

The engine performs the actual execution of tests specificed by provided relation chains. To start, an engine is initialized with a static SUT (system under test) and input dataset. Then, the SUT can be tested using provided relation chains (likely generated by the search space class). Each relation chain is completely independent from all other relation chains, although multiple chains can be passed to execute at once for performance reasons. Additionally, all input data objects are tested for each specified relation chain.

Performing metamorphic testing on a single input data object and relation chain combination is an iterative process. First, the results of execution of the SUT on the input data is determined. Then, the input data is transformed by the transformation specified by the first relation in the relation chain. Lastly, the result of execution of the SUT on the transformed data is compared to the previous result so it can be determined if each invariant for the given transformation held. This process is repeated for each transformation in the relation chain.

Under the hood, a sqlite database is maintained which denotes the results of executing each relation chain on all of the input data. This data is kept so it can be used later for debugging if an invariant failed.

execute(chain_length: int, num_chains: int) → None¶

Execute a list of provided relation chains and store the results.

Execution of a relation chain involves iteratively applying transformations to each item in the input data and ensuring all invariants hold. At the moment, we do not anticipate transformations causing errors regardless of the input. It is important to note that multiple relations can reference the same transformation and thus multiple invariant can be checked during a single step in a relation chain.

results_to_duckdb(db_path: Path | None = None) → DuckDBPyConnection¶

Convert the sqlite3 database kept during execution into a duckdb database.

Sqlite databases are designed for transactional processing, which fits our use case when inserting transaction and invariant records, especially if the program is using multiple cores. Once all the results have been accumulated, we can convert the database into a duckdb database due to duckdb’s better performance on analytical queries and better data compression.