splitgraph.engine.postgres package

Submodules

splitgraph.engine.postgres.engine module

Default Splitgraph engine: uses PostgreSQL to store metadata and actual objects and an audit stored procedure to track changes, as well as the Postgres FDW interface to upload/download objects to/from other Postgres engines.

class splitgraph.engine.postgres.engine.AuditTriggerChangeEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.PsycopgEngine, splitgraph.engine.ChangeEngine

Change tracking based on an audit trigger stored procedure

discard_pending_changes(schema, table=None)

Discard recorded pending changes for a tracked schema / table

get_changed_tables(schema)

Get list of tables that have changed content

get_pending_changes(schema, table, aggregate=False)

Return pending changes for a given tracked table

Parameters:
  • schema – Schema the table belongs to
  • table – Table to return changes for
  • aggregate – Whether to aggregate changes or return them completely
Returns:

If aggregate is True: tuple with numbers of (added_rows, removed_rows, updated_rows). If aggregate is False: List of (primary_key, change_type, change_data)

get_tracked_tables()

Return a list of tables that the audit trigger is working on.

has_pending_changes(schema)

Return True if the tracked schema has pending changes and False if it doesn’t.

track_tables(tables)

Install the audit trigger on the required tables

untrack_tables(tables)

Remove triggers from tables and delete their pending changes

class splitgraph.engine.postgres.engine.PostgresEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.AuditTriggerChangeEngine, splitgraph.engine.ObjectEngine

An implementation of the Postgres engine for Splitgraph

apply_diff_object(source_schema, source_table, target_schema, target_table)

Apply a changeset stored in a table to a certain target table.

Parameters:
  • source_schema – Schema the changeset is stored in
  • source_table – Table the changeset is stored in
  • target_schema – Schema to apply the changeset to
  • target_table – Table to apply the changeset to
download_objects(objects, remote_engine)

Download objects from the remote engine to the local cache

Parameters:
  • objects – List of object IDs to download
  • remote_engine – A remote ObjectEngine to download the objects from.
dump_object(schema, table, stream)

Dump a table to a stream using an engine-specific binary format.

Parameters:
  • schema – Schema the table lives in
  • table – Table to dump
  • stream – A file-like stream to dump the object into
load_object(schema, table, stream)

Load a table from a stream using an engine-specific binary format.

Parameters:
  • schema – Schema to create the table in. Must already exist.
  • table – Table to create. Must not exist.
  • stream – A file-like stream to load the object from
store_diff_object(changeset, schema, table, change_key)

Store a changeset in a table

Parameters:
  • changeset – List of (pk, action (0 for Insert, 1 for Delete, 2 for Update), action_data) where action_data is None for Delete and {‘c’: [column_names], ‘v’: [column_values]} that have been inserted/updated otherwise.
  • schema – Schema to store the changeset in
  • table – Table to store the changeset in
  • change_key – Change key as returned by get_change_key
upload_objects(objects, remote_engine)

Upload objects from the local cache to the remote engine

Parameters:
  • objects – List of object IDs to upload
  • remote_engine – A remote ObjectEngine to upload the objects to.
class splitgraph.engine.postgres.engine.PsycopgEngine(conn_params, name)

Bases: splitgraph.engine.SQLEngine

Postgres SQL engine backed by a Psycopg connection.

close()

Commit and close the engine’s backing connection

commit()

Commit the engine’s backing connection

connection

Engine-internal Psycopg connection. Will (re)open if closed/doesn’t exist.

delete_database(database)

Helper function to drop a database using the admin connection

Parameters:database – Database name to drop
get_primary_keys(schema, table)

Inspects the Postgres information_schema to get the primary keys for a given table.

initialize()

Create the Splitgraph Postgres database and install the audit trigger

rollback()

Rollback the engine’s backing connection

run_sql(statement, arguments=None, return_shape=<ResultShape.MANY_MANY: 4>)

Run an arbitrary SQL statement with some arguments, return an iterator of results. If the statement doesn’t return any results, return None.

run_sql_batch(statement, arguments, schema=None)

Run a parameterized SQL statement against multiple sets of arguments. Other engines can override if they support a more efficient batching mechanism.

splitgraph.engine.postgres.engine.make_conn(server, port, username, password, dbname)

Initializes a connection a Splitgraph Postgres engine.

Returns:Psycopg connection object

Module contents