Installation

For typical usage, a simple pip install will do the trick.

pip install laktory

In the spirit of having a package that is as lightweight as possible, only a few core dependencies will be installed:

• networkx: Creation manipulation of networks for creating pipeline DAG.
• pydantic: All laktory models derived from Pydantic BaseModel.
• settus: Cloud-based settings management system.
• typer: Library for building CLI applications.

For configuration-specific or more advanced features like testing transformations with Spark, deploying using Pulumi or running a job on Databricks, you will have to install some of the optional dependencies.

If you've got Python 3.8+ and pip installed, you're good to go. It is generally recommended to use a virtual environment for the installation.

The pip installation not only install the python package, but also Laktory CLI that can be invoked with

laktory --help

IaC Backends

Laktory supports multiple IaC backends. You will need to install the CLI of your desired backend manually:

• terraform
• pulumi

Optional Dependencies

Dataframes

If you want to run your pipeline locally or test some of the transformations, you will have to install the dataframe library used by your transformations. Available options are spark and polars.

• Apache Spark

  pip install laktory[spark]
  

  For running spark locally, you also need to follow instructions provided here. If you use homebrew to install java, your JAVA_HOME and SPARK_HOME environment variables should look something like:
  • JAVA_HOME=/opt/homebrew/opt/java
  • SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.0/libexec

• Polars

  pip install laktory[polars]
  

Infrastructure as Code

If you are using Pulumi as the IaC backend, you will want to run

pip install laktory[pulumi]

Orchestrator

If you want to run your pipeline remotely using one of the supported orchestrator you will have to install their respective packages.

• Databricks

  pip install laktory[databricks]
  

Cloud Provider

If you plan on deployed cloud-specific resources or want to retreive values from a secrets manager through settus, you will have to install cloud-specific packages.

• Microsoft Azure:

  pip install laktory[azure]
  

• Amazon Web Services (AWS)

  pip install laktory[aws]
  

• Google Cloud Platform (GCP)

  pip install laktory[gcp]
  

Git-based installation

If you need or prefer installing Laktory from git, you can use:

pip install git+https://github.com/okube-ai/laktory.git@main