sgr command line client reference

These are all the commands available in the commandline Splitgraph client.

sgr

Splitgraph command line client: manage and build Postgres schema images.

sgr [OPTIONS] COMMAND [ARGS]...

build

Build Splitgraph images.

This executes a Splitfile, building a new image or checking it out from cache if the same image had already been built.

Examples:

sgr build my.splitfile

Executes my.splitfile and writes its output into a new repository with a random name unless the name is specified in the Splitfile.

sgr build my.splitfile -o mynew/repo

Executes my.splitfile and writes its output into mynew/repo.

sgr build my_other.splitfile -o mynew/otherrepo --args PARAM1 VAL1 --args PARAM2 VAL2

Executes my_other.splitfile with parameters PARAM1 and PARAM2 set to VAL1 and VAL2, respectively.
sgr build [OPTIONS] SPLITFILE

Options

-a, --args <args>

Parameters to be substituted into the Splitfile. All parameters mentioned in the file must be specified in order for the Splitfile to be executed.

-o, --output-repository <output_repository>

Repository to store the result in.

Arguments

SPLITFILE

Required argument

checkout

Check out a Splitgraph image into a Postgres schema.

This downloads the required physical objects and materializes all tables.

Image spec must be of the format [NAMESPACE/]REPOSITORY[:HASH_OR_TAG]. Note that currently, the schema that the image is checked out into has to have the same name as the repository. If no image hash or tag is passed, “HEAD” is assumed.

If -u or --uncheckout is passed, this instead deletes the checked out schema (assuming there are no pending changes) and removes the HEAD pointer.

If --force isn’t passed and the schema has pending changes, this will fail.

sgr checkout [OPTIONS] IMAGE_SPEC

Options

-f, --force

Discard all pending changes to the schema

-u, --uncheckout

Delete the checked out copy instead

Arguments

IMAGE_SPEC

Required argument

cleanup

Prune unneeded objects from the driver.

This deletes all objects from the cache that aren’t required by any local repository.

sgr cleanup [OPTIONS]

clone

Clone a remote Splitgraph repository into a local one.

The lookup path for the repository is governed by the SG_REPO_LOOKUP and SG_REPO_LOOKUP_OVERRIDE config parameters and can be overriden by the command line --remote option.

sgr clone [OPTIONS] REMOTE_REPOSITORY [LOCAL_REPOSITORY]

Options

-r, --remote <remote>

Alias or full connection string for the remote driver

-d, --download-all

Download all objects immediately instead on checkout.

Arguments

REMOTE_REPOSITORY

Required argument

LOCAL_REPOSITORY

Optional argument

commit

Commit changes to a checked-out Splitgraph repository.

This produces a new image with all changes packaged up into a new image. Where a table hasn’t been created or had its schema changed, this will delta compress the changes. For all other tables (or if -s has been passed), this will store them as full table snapshots.

sgr commit [OPTIONS] REPOSITORY

Options

-s, --include-snap

Include the full table snapshots. This consumes more space, but makes checkouts faster.

-m, --message <message>

Optional commit message

Arguments

REPOSITORY

Required argument

config

Print the current Splitgraph configuration.

This takes into account the local config file, the default values and all overrides specified via environment variables.

This command can be used to dump the current Splitgraph configuration into a file:

sgr config --no-shielding --config-format > .sgconfig

…or save a config file overriding an entry:

SG_REPO_LOOKUP=driver1,driver2 sgr config -sc > .sgconfig
sgr config [OPTIONS]

Options

-s, --no-shielding

If set, doesnt replace sensitive values (like passwords) with asterisks

-c, --config-format

Output configuration in the Splitgraph config file format

diff

Show differences between two Splitgraph images.

The two images must be in the same repository. The actual targets of this command depend on the number of arguments passed:

sgr diff REPOSITORY

Return the differences between the current HEAD image and the checked out schema.

sgr diff REPOSITORY TAG_OR_HASH

Return the differences between the image and its parent.

sgr diff REPOSITORY TAG_OR_HASH_1 TAG_OR_HASH_2

Return the differences from the first (earlier) image to the second image.
sgr diff [OPTIONS] REPOSITORY [TAG_OR_HASH_1] [TAG_OR_HASH_2]

Options

-v, --verbose

Include the actual differences rather than just the total number of updated rows.

-t, --table-name <table_name>

Show the differences for a single table.

Arguments

REPOSITORY

Required argument

TAG_OR_HASH_1

Optional argument

TAG_OR_HASH_2

Optional argument

import

Import tables into a Splitgraph repository.

Imports a table or a result of a query from a local Splitgraph repository or a Postgres schema into another Splitgraph repository.

Examples:

sgr import noaa/climate:my_tag climate_data my/repository

Create a new image in my/repository with the climate_data table included. This links the new image to the physical object, meaning that the history of the climate_data table is preserved. If no tag is specified, the ‘latest’ (not the HEAD image or current state of the checked out image) image is used.

sgr import noaa/climate:my_tag "SELECT * FROM climate_data" my/repository climate_data

Create a new image in my/repository with the result of the query stored in the climate_data table. This creates a new physical object without any linkage to the original data, so the history of the climate_data table isn’t preserved. The SQL query can interact with multiple tables in the source image.

sgr import other_schema other_table my/repository

Since other_schema isn’t a Splitgraph repository, this will copy other_schema.other_table into a new Splitgraph object and add the other_table table to a new image in my/repository.

Note that importing doesn’t discard or commit pending changes in the target Splitgraph repository: a new image is created with the new table added, the new table is materialized in the repository and the HEAD pointer is moved.

sgr import [OPTIONS] IMAGE_SPEC TABLE_OR_QUERY TARGET_REPOSITORY
           [TARGET_TABLE]

Arguments

IMAGE_SPEC

Required argument

TABLE_OR_QUERY

Required argument

TARGET_REPOSITORY

Required argument

TARGET_TABLE

Optional argument

init

Initialize a new repository/driver.

Examples:

sgr init

Initializes the current local Splitgraph driver by writing some bookkeeping information. This is required for the rest of sgr to work.

sgr init new/repo

Creates a single image with the hash 00000... in new/repo
sgr init [OPTIONS] [REPOSITORY]

Arguments

REPOSITORY

Optional argument

log

Show the history of a Splitgraph repository.

By default, this shows the history of the current branch, starting from the HEAD pointer and following its parent chain.

If -t or --tree is passed, this instead renders the full image tree. The repository doesn’t need to have been checked out in this case.

sgr log [OPTIONS] REPOSITORY

Options

-t, --tree

Arguments

REPOSITORY

Required argument

mount

Mount foreign databases as Postgres schemas.

Uses the Postgres FDW interface to create a local Postgres schema with foreign tables that map to tables in other databases.

See a given mount handler’s documentation for handler-specific parameters.

sgr mount [OPTIONS] COMMAND [ARGS]...

provenance

Reconstruct the provenance of an image.

This crawls the history of a Splitgraph image to produce a list of images that were used by the Splitfile that created it, or a Splitfile with the same effect.

IMAGE_SPEC must be of the form [NAMESPACE/]REPOSITORY[:HASH_OR_TAG]. If no tag is specified, latest is used.

Examples:

Assume my/repo is produced by the following Splitfile:

FROM MOUNT [...] IMPORT external_table
FROM noaa/climate IMPORT {SELECT * FROM rainfall_data WHERE state = 'AZ'} AS rainfall_data

my/repo will have 2 images: one having hash_1 (with the external_table imported from a mounted database) and one having hash_2 (with both external_table and the rainfall_data containing the result of the query run against the then-latest image in the noaa/climate repository).

In this case:

sgr provenance my/repo

Returns a list of repositories and images that were imported by the Splitfile that constructed this image:

my/repo:[hash_2] depends on:
noaa/climate:[hash_3]

Where hash_3 is the hash of the latest image in the noaa/climate repository at the time the original Splitfile was run. However:

sgr provenance -f my/repo

Will try to reconstruct the Splitfile that can be used to build this image. Since the FROM MOUNT command isn’t reproducible (requires access to the original external database, which is a moving target), this will fail.

If -e is passed, this will base the image on the first image that can’t be reproduced:

sgr provenance -ef my/repo

# Splitfile commands used to reconstruct my/repo:[hash of the second layer]
FROM my/repo:[hash_1]
FROM noaa/climate:[hash_3] IMPORT {SELECT * FROM rainfall_data WHERE state = 'AZ'}
sgr provenance [OPTIONS] IMAGE_SPEC

Options

-f, --full

Recreate the Splitfile used to create this image

-e, --error-on-end

If False, bases the recreated Splitfile on the last image where the provenance chain breaks

Arguments

IMAGE_SPEC

Required argument

prune

Cleanup dangling images from a repository.

This includes images not pointed to by any tags (or checked out) and those that aren’t required by any of such images.

Will ask for confirmation of the deletion, unless -y is passed. If -r (--remote) is passed, this will perform deletion on a remote Splitgraph driver (registered in the config) instead, assuming the user has write access to the remote repository.

This does not delete any physical objects that the deleted repository/images depend on: use sgr cleanup to do that.

sgr prune [OPTIONS] REPOSITORY

Options

-r, --remote <remote>

Perform the deletion on a remote instead, specified by its alias

-y, --yes

Agree to deletion without confirmation

Arguments

REPOSITORY

Required argument

publish

Publish tagged Splitgraph images to the catalog.

Only images with a tag can be published. The image must have been pushed to the registry beforehand with sgr push.

sgr publish [OPTIONS] REPOSITORY TAG

Options

-r, --readme <readme>
--skip-provenance

Don’t include provenance in the published information.

--skip-previews

Don’t include table previews in the published information.

Arguments

REPOSITORY

Required argument

TAG

Required argument

pull

Pull changes from an upstream repository.

sgr pull [OPTIONS] REPOSITORY

Options

-d, --download-all <download_all>

Download all objects immediately instead on checkout.

Arguments

REPOSITORY

Required argument

push

Push changes from a local repository to the upstream.

The actual destination is decided as follows:

  • Remote driver: remote argument (either driver alias as specified in the config or a connection string, then the upstream configured for the repository.
  • Remote repository: remote_repository argument, then the upstream configured for the repository, then the same name as the repository.

-h and -o allow to upload the objects to somewhere else other than the external drivers. Currently, uploading to an S3-compatible host via Minio is supported: see splitgraph.hooks.s3 for information on handler options and how to register a new upload handler.

sgr push [OPTIONS] REPOSITORY [REMOTE_REPOSITORY]

Options

-r, --remote <remote>

Alias or full connection string for the remote driver

-h, --upload-handler <upload_handler>

Upload handler

-o, --upload-handler-options <upload_handler_options>

Upload handler parameters

Arguments

REPOSITORY

Required argument

REMOTE_REPOSITORY

Optional argument

rebuild

Rebuild images against different dependencies.

Examines the provenance of a Splitgraph image created by a Splitfile and reruns it against different images than the ones that were imported by the original run.

Examples:

sgr rebuild my/repo --against noaa/climate:old_data

Reconstructs the Splitfile used to create my/repo:latest, replaces all imports from noaa/climate with imports from noaa/climate:old_data and reruns the Splitfile.

sgr rebuild my/repo:other_tag -u

Rebuilds my_repo:other_tag against the latest versions of all of its dependencies.

Image caching still works in this case: if the result of the rebuild already exists, the image will be checked out.

sgr rebuild [OPTIONS] IMAGE_SPEC

Options

-u, --update

Rederive the image against the latest version of all dependencies.

-a, --against <against>

Images to substitute into the reconstructed Splitfile, of the form [NAMESPACE/]REPOSITORY[:HASH_OR_TAG]. Default tag is ‘latest’.

Arguments

IMAGE_SPEC

Required argument

rm

Delete schemas, repositories or images.

If the target of this command is a Postgres schema, this performs DROP SCHEMA CASCADE.

If the target of this command is a Splitgraph repository, this deletes the repository and all of its history.

If the target of this command is an image, this deletes the image and all of its children.

In any case, this command will ask for confirmation of the deletion, unless -y is passed. If -r (--remote) is passed, this will perform deletion on a remote Splitgraph driver (registered in the config) instead, assuming the user has write access to the remote repository.

This does not delete any physical objects that the deleted repository/images depend on: use sgr cleanup to do that.

Examples:

sgr rm temporary_schema

Deletes temporary_schema from the local driver.

sgr rm --driver splitgraph.com username/repo

Deletes username/repo from the Splitgraph registry.

sgr rm -y username/repo:old_branch

Deletes the image pointed to by old_branch as well as all of its children (images created by a commit based on this image), as well as all of the tags that point to now deleted images, without asking for confirmation. Note this will not delete images that import tables from the deleted images via Splitfiles or indeed the physical objects containing the actual tables.
sgr rm [OPTIONS] IMAGE_SPEC

Options

-r, --remote <remote>

Perform the deletion on a remote instead, specified by its alias

-y, --yes

Agree to deletion without confirmation

Arguments

IMAGE_SPEC

Required argument

show

Show information about a Splitgraph image. This includes its parent, comment and creation time.

Image spec must be of the format [NAMESPACE/]REPOSITORY[:HASH_OR_TAG]. If no tag is specified, HEAD is used.

sgr show [OPTIONS] IMAGE_SPEC

Options

-v, --verbose

Also show all tables in this image and the objects they map to.

Arguments

IMAGE_SPEC

Required argument

sql

Run an SQL statement against the Splitgraph driver.

There are no restrictions on the contents of the statement: this is the same as running it from any other PostgreSQL client.

If --schema is specified, the statement is run with the search_path set to that schema. This means that these statements are equivalent:

sgr sql "SELECT * FROM "noaa/climate".table"
sgr sql -s noaa/climate "SELECT * FROM table"
sgr sql [OPTIONS] SQL

Options

-s, --schema <schema>

Run SQL against this schema.

-a, --show-all

Returns all results of the query.

Arguments

SQL

Required argument

status

Show the status of the Splitgraph driver. If a repository is passed, show information about the repository. If not, show information about all repositories local to the driver.

sgr status [OPTIONS] [REPOSITORY]

Arguments

REPOSITORY

Optional argument

tag

Manage tags on images.

Depending on the exact invocation, this command can tag a Splitgraph image, list all tags in a repository or delete a tag.

Examples:

sgr tag noaa/climate

List all tagged images in the noaa/climate repository and their tags.

sgr tag noaa/climate:abcdef1234567890

List all tags assigned to the image noaa/climate:abcdef1234567890...

sgr tag noaa/climate:abcdef1234567890 my_new_tag

Tag the image noaa/climate:abcdef1234567890... with my_new_tag. If the tag already exists, this will raise an error, unless -f is passed, which will overwrite the tag.

sgr tag noaa/climate my_new_tag

Tag the current HEAD of noaa/climate with my_new_tag.

sgr tag --remove noaa/climate:my_new_tag

Remove the tag my_new_tag from noaa/climate.
sgr tag [OPTIONS] IMAGE_SPEC [TAG]

Options

-f, --force

Overwrite the tag if it already exists.

-r, --remove

Remove the tag instead.

Arguments

IMAGE_SPEC

Required argument

TAG

Optional argument

upstream

Get or set the upstream for a repository.

This shows the default repository used for pushes and pulls as well as allows to change it to a different remote driver and repository.

The remote driver alias must exist in the config file.

Examples:

sgr upstream my/repo --set splitgraph.com username/repo

Sets the upstream for my/repo to username/repo existing on the splitgraph.com driver

sgr upstream my/repo --reset

Removes the upstream for my/repo.

sgr upstream my/repo

Shows the current upstream for my/repo.
sgr upstream [OPTIONS] REPOSITORY

Options

-s, --set <set_to>

Set the upstream to a driver alias + repository

-r, --reset

Delete the upstream

Arguments

REPOSITORY

Required argument