Objects

Splitgraph objects, or fragments, are building blocks of Splitgraph tables. Each object is immutable and content-addressable (objects with same contents have the same object ID). Multiple tables can link to a single object.

sgr stores object metadata in the splitgraph_meta.objects table on the engine. The metadata contains the LTHash of rows added/deleted by the object and the object index that is a very short (a few KB) summary of the contents of the object.

With these features, sgr can satisfy queries to large datasets by only downloading a few objects instead of the whole image.

sgr stores object data as tables in the splitgraph_meta schema on the engine. Physically, objects are stored in cstore_fdw files, a columnar store for PostgreSQL that allows for superior read performance and low IO load. This allows sgr to rapidly swap CStore fragments as flat files without the extra overhead of loading them into PostgreSQL.

Each object has an extra Boolean column, sg_ud_flag, that determines whether the corresponding row in this object overwrites (upserts) or deletes a row in case of a primary key conflict. This allows Splitgraph objects to support delta compression.

More information on Splitgraph object internals is available.