DLT Pipeline

laktory.models.resources.databricks.DLTPipeline

Bases: BaseModel, PulumiResource, TerraformResource

Databricks Delta Live Tables (DLT) Pipeline

ATTRIBUTE	DESCRIPTION
access_controls	Pipeline access controls TYPE: list[AccessControl]
allow_duplicate_names	If False, deployment will fail if name conflicts with that of another pipeline. TYPE: bool
catalog	Name of the unity catalog storing the pipeline tables TYPE: Union[str, None]
channel	Name of the release channel for Spark version used by DLT pipeline. TYPE: Literal['CURRENT', 'PREVIEW']
clusters	Clusters to run the pipeline. If none is specified, pipelines will automatically select a default cluster configuration for the pipeline. TYPE: list[PipelineCluster]
configuration	List of values to apply to the entire pipeline. Elements must be formatted as key:value pairs TYPE: dict[str, str]
continuous	If True, the pipeline is run continuously. TYPE: bool
development	If True the pipeline is run in development mode TYPE: Union[bool, str]
edition	Name of the product edition TYPE: Literal['CORE', 'PRO', 'ADVANCED']
libraries	Specifies pipeline code (notebooks) and required artifacts. TYPE: list[PipelineLibrary]
name	Pipeline name TYPE: str
name_prefix	Prefix added to the DLT pipeline name TYPE: str
name_suffix	Suffix added to the DLT pipeline name TYPE: str
notifications	Notifications specifications TYPE: list[PipelineNotifications]
photon	If True, Photon engine enabled. TYPE: bool
serverless	If True, serverless is enabled TYPE: bool
storage	A location on DBFS or cloud storage where output data and metadata required for pipeline execution are stored. By default, tables are stored in a subdirectory of this location. Change of this parameter forces recreation of the pipeline. (Conflicts with catalog). TYPE: str
target	The name of a database (in either the Hive metastore or in a UC catalog) for persisting pipeline output data. Configuring the target setting allows you to view and query the pipeline output data from the Databricks UI. TYPE: str

Examples:

Assuming the configuration yaml file

import io
from laktory import models

# Define pipeline
pipeline_yaml = '''
name: pl-stock-prices

catalog: dev
target: finance

clusters:
  - name : default
    node_type_id: Standard_DS3_v2
    autoscale:
      min_workers: 1
      max_workers: 2

libraries:
  - notebook:
      path: /pipelines/dlt_brz_template.py
  - notebook:
      path: /pipelines/dlt_slv_template.py
  - notebook:
      path: /pipelines/dlt_gld_stock_performances.py

access_controls:
  - group_name: account users
    permission_level: CAN_VIEW
  - group_name: role-engineers
    permission_level: CAN_RUN

'''
pipeline = models.resources.databricks.DLTPipeline.model_validate_yaml(
    io.StringIO(pipeline_yaml)
)

References

• Databricks Pipeline
• Pulumi Databricks Pipeline

ATTRIBUTE	DESCRIPTION
resource_type_id	dlt TYPE: str
additional_core_resources	permissions TYPE: list[PulumiResource]

Attributes

resource_type_id property

resource_type_id

dlt

additional_core_resources property

additional_core_resources

• permissions

---

laktory.models.resources.databricks.dltpipeline.PipelineLibraryFile

Bases: BaseModel

Pipeline Library File specifications

ATTRIBUTE	DESCRIPTION
path	Workspace filepath TYPE: str

---

laktory.models.resources.databricks.dltpipeline.PipelineLibraryNotebook

Bases: BaseModel

Pipeline Library Notebook specifications

ATTRIBUTE	DESCRIPTION
path	Workspace notebook filepath TYPE: str

---

laktory.models.resources.databricks.dltpipeline.PipelineLibrary

Bases: BaseModel

Pipeline Library specifications

ATTRIBUTE	DESCRIPTION
file	File specifications TYPE: str
notebook	Notebook specifications TYPE: PipelineLibraryNotebook

---

laktory.models.resources.databricks.dltpipeline.PipelineNotifications

Bases: BaseModel

Pipeline Notifications specifications

ATTRIBUTE	DESCRIPTION
alerts	Alert types TYPE: list[Literal['on-update-success', 'on-update-failure', 'on-update-fatal-failure', 'on-flow-failure']]
recipients	List of user/group/service principal names TYPE: list[str]

---

laktory.models.resources.databricks.dltpipeline.PipelineCluster

Bases: Cluster

Pipeline Cluster. Same attributes as laktory.models.Cluster, except for

• autotermination_minutes
• cluster_id
• data_security_mode
• enable_elastic_disk
• idempotency_token
• is_pinned
• libraries
• no_wait
• node_type_id
• runtime_engine
• single_user_name
• spark_version

that are not allowed.