splitgraph.engine.postgres package


splitgraph.engine.postgres.engine module

Default Splitgraph engine: uses PostgreSQL to store metadata and actual objects and an audit stored procedure to track changes, as well as the Postgres FDW interface to upload/download objects to/from other Postgres engines.

class splitgraph.engine.postgres.engine.AuditTriggerChangeEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.PsycopgEngine, splitgraph.engine.ChangeEngine

Change tracking based on an audit trigger stored procedure

discard_pending_changes(schema, table=None)

Discard recorded pending changes for a tracked schema / table


Get list of tables that have changed content

get_pending_changes(schema, table, aggregate=False)

Return pending changes for a given tracked table

  • schema – Schema the table belongs to

  • table – Table to return changes for

  • aggregate – Whether to aggregate changes or return them completely


If aggregate is True: tuple with numbers of (added_rows, removed_rows, updated_rows). If aggregate is False: List of (primary_key, change_type, change_data)


Return a list of tables that the audit trigger is working on.


Return True if the tracked schema has pending changes and False if it doesn’t.


Install the audit trigger on the required tables


Remove triggers from tables and delete their pending changes

class splitgraph.engine.postgres.engine.PostgresEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.AuditTriggerChangeEngine, splitgraph.engine.ObjectEngine

An implementation of the Postgres engine for Splitgraph

apply_fragments(objects, target_schema, target_table, extra_quals=None, extra_qual_args=None)

Apply multiple fragments to a target table as a single-query batch operation.

  • objects – List of tuples (object_schema, object_table) that the objects are stored in.

  • target_schema – Schema to apply the fragment to

  • target_table – Table to apply the fragment to

  • extra_quals – Optional, extra SQL (Composable) clauses to filter new rows in the fragment on (e.g. SQL(“a = %s”))

  • extra_qual_args – Optional, a tuple of arguments to use with extra_quals

download_objects(objects, remote_engine)

Download objects from the remote engine to the local cache

  • objects – List of object IDs to download

  • remote_engine – A remote ObjectEngine to download the objects from.

:return List of object IDs that were downloaded.

dump_object(schema, table, stream)

Dump a table to a stream using an engine-specific binary format.

  • schema – Schema the table lives in

  • table – Table to dump

  • stream – A file-like stream to dump the object into

load_object(schema, table, stream)

Load a table from a stream using an engine-specific binary format.

  • schema – Schema to create the table in. Must already exist.

  • table – Table to create. Must not exist.

  • stream – A file-like stream to load the object from

store_fragment(inserted, deleted, schema, table, source_schema, source_table)

Store a fragment of a changed table in another table

  • inserted – List of PKs that have been updated/inserted

  • deleted – List of PKs that have been deleted

  • schema – Schema to store the change in

  • table – Table to store the change in

  • source_schema – Schema the source table is located in

  • source_table – Name of the source table

upload_objects(objects, remote_engine)

Upload objects from the local cache to the remote engine

  • objects – List of object IDs to upload

  • remote_engine – A remote ObjectEngine to upload the objects to.

class splitgraph.engine.postgres.engine.PsycopgEngine(conn_params, name)

Bases: splitgraph.engine.SQLEngine

Postgres SQL engine backed by a Psycopg connection.


Commit and close the engine’s backing connection


Commit the engine’s backing connection


Engine-internal Psycopg connection.


Helper function to drop a database using the admin connection


database – Database name to drop

dump_table_sql(schema, table_name, stream, columns='*', where='', where_args=None)

Dump the table contents in the SQL format :param schema: Schema the table is located in :param table_name: Name of the table :param stream: A file-like object to write the result into. :param columns: SQL column spec. Default ‘*’. :param where: Optional, an SQL WHERE clause :param where_args: Arguments for the optional WHERE clause.

get_primary_keys(schema, table)

Inspects the Postgres information_schema to get the primary keys for a given table.

get_table_size(schema, table)

Return the table disk usage, in bytes.


Create the Splitgraph Postgres database and install the audit trigger

lock_table(schema, table)

Acquire an exclusive lock on a given table, released when the transaction commits / rolls back.


Rollback the engine’s backing connection

run_sql(statement, arguments=None, return_shape=<ResultShape.MANY_MANY: 4>)

Run an arbitrary SQL statement with some arguments, return an iterator of results. If the statement doesn’t return any results, return None.

run_sql_batch(statement, arguments, schema=None)

Run a parameterized SQL statement against multiple sets of arguments. Other engines can override if they support a more efficient batching mechanism.

splitgraph.engine.postgres.engine.make_conn(server, port, username, password, dbname)

Initializes a connection a Splitgraph Postgres engine.


Psycopg connection object

Module contents