splitgraph.engine.postgres package

Submodules

splitgraph.engine.postgres.engine module

Default Splitgraph engine: uses PostgreSQL to store metadata and actual objects and an audit stored procedure to track changes, as well as the Postgres FDW interface to upload/download objects to/from other Postgres engines.

class splitgraph.engine.postgres.engine.AuditTriggerChangeEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.PsycopgEngine, splitgraph.engine.ChangeEngine

Change tracking based on an audit trigger stored procedure

discard_pending_changes(schema, table=None)

Discard recorded pending changes for a tracked schema / table

get_changed_tables(schema)

Get list of tables that have changed content

get_pending_changes(schema, table, aggregate=False)

Return pending changes for a given tracked table

Parameters
  • schema – Schema the table belongs to

  • table – Table to return changes for

  • aggregate – Whether to aggregate changes or return them completely

Returns

If aggregate is True: tuple with numbers of (added_rows, removed_rows, updated_rows). If aggregate is False: List of (primary_key, change_type, change_data)

get_tracked_tables()

Return a list of tables that the audit trigger is working on.

has_pending_changes(schema)

Return True if the tracked schema has pending changes and False if it doesn’t.

track_tables(tables)

Install the audit trigger on the required tables

untrack_tables(tables)

Remove triggers from tables and delete their pending changes

class splitgraph.engine.postgres.engine.PostgresEngine(conn_params, name)

Bases: splitgraph.engine.postgres.engine.AuditTriggerChangeEngine, splitgraph.engine.ObjectEngine

An implementation of the Postgres engine for Splitgraph

apply_fragments(objects, target_schema, target_table, extra_quals=None, extra_qual_args=None)

Apply multiple fragments to a target table as a single-query batch operation.

Parameters
  • objects – List of tuples (object_schema, object_table) that the objects are stored in.

  • target_schema – Schema to apply the fragment to

  • target_table – Table to apply the fragment to

  • extra_quals – Optional, extra SQL (Composable) clauses to filter new rows in the fragment on (e.g. SQL(“a = %s”))

  • extra_qual_args – Optional, a tuple of arguments to use with extra_quals

download_objects(objects, remote_engine)

Download objects from the remote engine to the local cache

Parameters
  • objects – List of object IDs to download

  • remote_engine – A remote ObjectEngine to download the objects from.

:return List of object IDs that were downloaded.

dump_object(schema, table, stream)

Dump a table to a stream using an engine-specific binary format.

Parameters
  • schema – Schema the table lives in

  • table – Table to dump

  • stream – A file-like stream to dump the object into

load_object(schema, table, stream)

Load a table from a stream using an engine-specific binary format.

Parameters
  • schema – Schema to create the table in. Must already exist.

  • table – Table to create. Must not exist.

  • stream – A file-like stream to load the object from

store_fragment(inserted, deleted, schema, table, source_schema, source_table)

Store a fragment of a changed table in another table

Parameters
  • inserted – List of PKs that have been updated/inserted

  • deleted – List of PKs that have been deleted

  • schema – Schema to store the change in

  • table – Table to store the change in

  • source_schema – Schema the source table is located in

  • source_table – Name of the source table

upload_objects(objects, remote_engine)

Upload objects from the local cache to the remote engine

Parameters
  • objects – List of object IDs to upload

  • remote_engine – A remote ObjectEngine to upload the objects to.

class splitgraph.engine.postgres.engine.PsycopgEngine(conn_params, name)

Bases: splitgraph.engine.SQLEngine

Postgres SQL engine backed by a Psycopg connection.

close()

Commit and close the engine’s backing connection

commit()

Commit the engine’s backing connection

connection

Engine-internal Psycopg connection.

delete_database(database)

Helper function to drop a database using the admin connection

Parameters

database – Database name to drop

dump_table_sql(schema, table_name, stream, columns='*', where='', where_args=None)

Dump the table contents in the SQL format :param schema: Schema the table is located in :param table_name: Name of the table :param stream: A file-like object to write the result into. :param columns: SQL column spec. Default ‘*’. :param where: Optional, an SQL WHERE clause :param where_args: Arguments for the optional WHERE clause.

get_primary_keys(schema, table)

Inspects the Postgres information_schema to get the primary keys for a given table.

get_table_size(schema, table)

Return the table disk usage, in bytes.

initialize()

Create the Splitgraph Postgres database and install the audit trigger

lock_table(schema, table)

Acquire an exclusive lock on a given table, released when the transaction commits / rolls back.

rollback()

Rollback the engine’s backing connection

run_sql(statement, arguments=None, return_shape=<ResultShape.MANY_MANY: 4>)

Run an arbitrary SQL statement with some arguments, return an iterator of results. If the statement doesn’t return any results, return None.

run_sql_batch(statement, arguments, schema=None)

Run a parameterized SQL statement against multiple sets of arguments. Other engines can override if they support a more efficient batching mechanism.

splitgraph.engine.postgres.engine.make_conn(server, port, username, password, dbname)

Initializes a connection a Splitgraph Postgres engine.

Returns

Psycopg connection object

Module contents