splitgraph.engine.postgres package


splitgraph.engine.postgres.engine module

Default Splitgraph engine: uses PostgreSQL to store metadata and actual objects and an audit stored procedure to track changes, as well as the Postgres FDW interface to upload/download objects to/from other Postgres engines.

class splitgraph.engine.postgres.engine.AuditTriggerChangeEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.PsycopgEngine, splitgraph.engine.ChangeEngine

Change tracking based on an audit trigger stored procedure

discard_pending_changes(schema, table=None)

Discard recorded pending changes for a tracked schema / table


Get list of tables that have changed content

get_pending_changes(schema, table, aggregate=False)

Return pending changes for a given tracked table

  • schema – Schema the table belongs to
  • table – Table to return changes for
  • aggregate – Whether to aggregate changes or return them completely

If aggregate is True: tuple with numbers of (added_rows, removed_rows, updated_rows). If aggregate is False: List of (primary_key, change_type, change_data)


Return a list of tables that the audit trigger is working on.


Return True if the tracked schema has pending changes and False if it doesn’t.


Install the audit trigger on the required tables


Remove triggers from tables and delete their pending changes

class splitgraph.engine.postgres.engine.PostgresEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.AuditTriggerChangeEngine, splitgraph.engine.ObjectEngine

An implementation of the Postgres engine for Splitgraph

apply_diff_object(source_schema, source_table, target_schema, target_table)

Apply a changeset stored in a table to a certain target table.

  • source_schema – Schema the changeset is stored in
  • source_table – Table the changeset is stored in
  • target_schema – Schema to apply the changeset to
  • target_table – Table to apply the changeset to
download_objects(objects, remote_engine)

Download objects from the remote engine to the local cache

  • objects – List of object IDs to download
  • remote_engine – A remote ObjectEngine to download the objects from.
dump_object(schema, table, stream)

Dump a table to a stream using an engine-specific binary format.

  • schema – Schema the table lives in
  • table – Table to dump
  • stream – A file-like stream to dump the object into
load_object(schema, table, stream)

Load a table from a stream using an engine-specific binary format.

  • schema – Schema to create the table in. Must already exist.
  • table – Table to create. Must not exist.
  • stream – A file-like stream to load the object from
store_diff_object(changeset, schema, table, change_key)

Store a changeset in a table

  • changeset – List of (pk, action (0 for Insert, 1 for Delete, 2 for Update), action_data) where action_data is None for Delete and {‘c’: [column_names], ‘v’: [column_values]} that have been inserted/updated otherwise.
  • schema – Schema to store the changeset in
  • table – Table to store the changeset in
  • change_key – Change key as returned by get_change_key
upload_objects(objects, remote_engine)

Upload objects from the local cache to the remote engine

  • objects – List of object IDs to upload
  • remote_engine – A remote ObjectEngine to upload the objects to.
class splitgraph.engine.postgres.engine.PsycopgEngine(conn_params, name)

Bases: splitgraph.engine.SQLEngine

Postgres SQL engine backed by a Psycopg connection.


Commit and close the engine’s backing connection


Commit the engine’s backing connection


Engine-internal Psycopg connection. Will (re)open if closed/doesn’t exist.


Helper function to drop a database using the admin connection

Parameters:database – Database name to drop
get_primary_keys(schema, table)

Inspects the Postgres information_schema to get the primary keys for a given table.


Create the Splitgraph Postgres database and install the audit trigger


Rollback the engine’s backing connection

run_sql(statement, arguments=None, return_shape=<ResultShape.MANY_MANY: 4>)

Run an arbitrary SQL statement with some arguments, return an iterator of results. If the statement doesn’t return any results, return None.

run_sql_batch(statement, arguments, schema=None)

Run a parameterized SQL statement against multiple sets of arguments. Other engines can override if they support a more efficient batching mechanism.

splitgraph.engine.postgres.engine.make_conn(server, port, username, password, dbname)

Initializes a connection a Splitgraph Postgres engine.

Returns:Psycopg connection object

Module contents