Skip to content

PipelineNodeExpectation

laktory.models.PipelineNodeExpectation ¤

Bases: BaseModel

Definition of an expectation for a given node output DataFrame. It consists of an SQL expression of the expectation and of an action to be taken if the expectation is not met. Similar to DLT Table Expectation. Currently only supported with DLT pipeline orchestrator.

ATTRIBUTE DESCRIPTION
name

Name of the expectation

TYPE: str

expression

SQL expression definition the constraint

TYPE: str

action

Action to take when expectation is not met. ALLOW: Write invalid records to the output DataFrame, but log exception. DROP: Drop Invalid records to the output DataFrame and log exception. FAIL: Raise exception when invalid records are found.

TYPE: Literal['ALLOW', 'DROP', 'FAIL']

Examples:

from laktory import models

e = models.PipelineNodeExpectation(
    name="valid timestamp",
    expression="col(“timestamp”) > '2012-01-01'",
    action="DROP",
)
print(e)
'''
variables={} name='valid timestamp' expression="col(“timestamp”) > '2012-01-01'" action='DROP'
'''
References