Objects
Splitgraph objects, or fragments, are building blocks of Splitgraph tables. Each object is immutable and content-addressable (objects with same contents have the same object ID). Multiple tables can link to a single object.
Splitgraph stores object metadata in the splitgraph_meta.objects
table on the engine. The
metadata contains the LTHash of rows
added/deleted by the object and the object index
that is a very short (a few KB) summary of the contents of the object.
With these features, Splitgraph can satisfy queries to large datasets by only downloading a few objects instead of the whole image.
Splitgraph stores object data as tables in the splitgraph_meta
schema on the engine. Physically, objects are stored in cstore_fdw files, a columnar store for PostgreSQL that allows for
superior read performance and low IO load.
This allows Splitgraph to rapidly swap CStore fragments as flat files without the extra overhead of loading them into PostgreSQL.
Each object has an extra Boolean column, sg_ud_flag
, that determines whether the corresponding
row in this object overwrites (upserts) or deletes a row in case of a primary key conflict.
This allows Splitgraph objects to support delta compression.
More information on Splitgraph object internals is available.