Splitgraph objects, or fragments, are building blocks of Splitgraph tables. Each object is immutable and content-addressable (objects with same contents have the same object ID). Multiple tables can link to a single object.
sgr stores object metadata in the
splitgraph_meta.objects table on the
engine. The metadata contains the
LTHash of rows
added/deleted by the object and the object index
that is a very short (a few KB) summary of the contents of the object.
With these features,
sgr can satisfy queries to large datasets by only
downloading a few objects instead of the whole image.
sgr stores object data as tables in the
splitgraph_meta schema on the
engine. Physically, objects are stored in
cstore_fdw files, a columnar store
for PostgreSQL that allows for
superior read performance and low IO load.
sgr to rapidly swap CStore fragments as flat files without the
extra overhead of loading them into PostgreSQL.
Each object has an extra Boolean column,
sg_ud_flag, that determines whether
the corresponding row in this object overwrites (upserts) or deletes a row in
case of a primary key conflict. This allows Splitgraph objects to support delta
More information on Splitgraph object internals is available.