Open repository in Console
Updated over 2 years ago
Indexed over 2 years ago


Example of a dbt transform on Splitgraph Cloud with Github Actions

If you're reading this on Splitgraph: this is built from a GitHub repository and a dbt project at https://github.com/splitgraph/dbt-transform-example. It uses data from trase/supply-chains.

If you're reading this on GitHub: This repository contains a dbt project that builds a model from the data in https://splitgraph.com/trase/supply-chains and outputs it to https://splitgraph.com/splitgraph/dbt-transform-example.

How it works

  • The dbt project defines a model that depends on a source named trase_supply_chains.
  • The splitgraph.yml file maps this data source to the dataset at https://splitgraph.com/trase/supply-chains.
  • The GitHub Action workflow generates a temporary clone link to this repository (using $GITHUB_TOKEN) and injects it into splitgraph.credentials.yml
  • It then runs sgr cloud sync with the --wait parameter to submit the dbt model to Splitgraph Cloud and wait for it to complete.
  • Splitgraph Cloud alters the model to point to trase/supply-chains and runs it against that dataset, snapshotting the output into a versioned data image.
  • The result is at https://splitgraph.com/splitgraph/dbt-transform-example.

Using this in your own projects

  • Fork this repository
  • Get a pair of Splitgraph API credentials on the settings page and set them up as GitHub Secrets:
  • Change splitgraph/dbt-transform-example in splitgraph.yml and the GitHub Action workflow file to use your own username (e.g. someuser/dbt-transform-example)
  • Run the workflow manually

Local development with the Splitgraph engine

$ cat ~/.dbt/profiles.yml 
      dbname: splitgraph
      pass: password
      port: 6432
      schema: dbt_transform_example
      threads: 32
      type: postgres
      user: sgr
  target: prod
  • Set up the dataset: sgr clone trase/supply-chains:latest && sgr checkout --layered trase/supply-chains:latest
  • Run dbt: dbt build
  • The resultant model will be in dbt_transform_example.top_exporter_metrics. You can connect to the engine with a normal PostgreSQL client by running psql $(sgr config -n).
  • top_exporter_metrics
    1 Object
     | 323 kB | 
    7.25k Rows
Upstream Metadata