PipelineNodeDataSource

laktory.models.datasources.PipelineNodeDataSource

Bases: BaseDataSource

Data Source using an upstream pipeline node. Using a pipeline node data source defines the interdependencies between each node in a pipeline. Depending on the selected pipeline orchestrator and the context, a pipeline node data source might read the data from:

• memory
• upstream node sink
• DLT table

ATTRIBUTE	DESCRIPTION
node_name	Name of the upstream pipeline node TYPE: Union[str, None]

Examples:

from laktory import models

brz = models.PipelineNode(
    name="brz_stock_prices",
    source={"path": "/Volumes/sources/landing/events/yahoo-finance/stock_price"},
    sink={"path": "/Volumes/sources/landing/tables/brz_stock_prices"},
)

slv = models.PipelineNode(
    name="slv_stock_prices",
    source={"node_name": "brz_stock_prices"},
    sink={"path": "/Volumes/sources/landing/tables/slv_stock_prices"},
)

pl = models.Pipeline(name="pl-stock-prices", nodes=[brz, slv])

# pl.execute(spark=spark)