Splitgraph has been acquired by EDB! Read the blog post.

Tracking dataset provenance

There's one other advantage to using Splitfiles instead of building images manually: provenance tracking. When you create an image with a Splitfile, the metadata of the image includes the specifics of the actual command that created it.

This means that we can output a list of images and repositories that were used in an image's creation.

Example

$ sgr provenance example/output:latest

example/output:4fd86d48715d2a3e250810fbdaa43d21ed0a894cd00de9bc118d9191e6294741 depends on:
example/repo_2:97900a8279627ec22b355d1211dd824f78f5840b6a0e30e1d285f4e5c6120016
example/repo_1:103cb2da2da000f3dce9512e3cc67434d7a3c977c0df259411c73a2687244706

Since the Splitfile in the previous section imported tables from the two generated repositories, sgr considers these repositories to be dependencies of example/output. (Note that you can still check out and use the example/output repository without those two repositories present).

We can also run the dependency query backwards, to find the downstream dependents of a given image:

$ sgr dependents example/repo_1:latest

example/repo_1:103cb2da2da000f3dce9512e3cc67434d7a3c977c0df259411c73a2687244706 is depended on by:
example/output:4fd86d48715d2a3e250810fbdaa43d21ed0a894cd00de9bc118d9191e6294741

We can also use this provenance data to reconstruct a Splitfile that will rebuild the newly created image, even without access to the original Splitfile itself:

$ sgr provenance example/output:latest --full

# Splitfile commands used to recreate example/output:4fd86d48715d2a3e250810fbdaa43d21ed0a894cd00de9bc118d9191e6294741
FROM example/repo_1:103cb2da2da000f3dce9512e3cc67434d7a3c977c0df259411c73a2687244706 IMPORT demo AS table_1
FROM example/repo_2:97900a8279627ec22b355d1211dd824f78f5840b6a0e30e1d285f4e5c6120016 IMPORT demo AS table_2
SQL {CREATE TABLE result
  AS SELECT table_1.key
          , table_1.value AS value_1
          , table_2.value AS value_2
     FROM table_1
          INNER JOIN table_2 ON table_1.key = table_2.key}

Note this is a more powerful feature than what Dockerfiles offer (in Docker, the final image is unaware of upstream build contexts). With sgr, the provenance information encodes everything necessary to rebuild an image. This means that we can rebuild the example/output:latest image against a different version of any of its dependencies by simply substituting a different image hash into this regenerated Splitfile:

$ sgr rebuild example/output:latest --against example/repo_2:new_data

Rerunning example/output:4fd86d48715d2a3e250810fbdaa43d21ed0a894cd00de9bc118d9191e6294741 against:
example/repo_2:new_data

Step 1/3 : FROM example/repo_1:103cb2da2da000f3dce9512e3cc67434d7a3c...
Resolving repository example/repo_1
 ---> Using cache
 ---> 27aeabd37f2c

Step 2/3 : FROM example/repo_2:new_data IMPORT demo AS table_2
Resolving repository example/repo_2
Importing 1 table from example/repo_2:48407a3cef8a into example/output
 ---> ca85929a90e7

Step 3/3 : SQL {CREATE TABLE result  AS SELECT table_1.key        ...
Executing SQL...
Committing example/output...
Processing table result
Computing table partitions
Indexing the partition key
Storing and indexing the table
100%|█████████████████| 1/1 [00:00<00:00,  6.33objs/s]
 ---> 46b6a361bebd

Since this example uses the same image from example/repo_1, sgr is able to cache the first step of the rebuild. But since we altered the example/repo_2 image (by specifying --against example/repo_2:new_data), sgr invalidates the subsequent layers and reruns their build steps. You can think of this invalidation in the same way as Dockerfile cache invalidation.

Let's examine the new result:

$ sgr sql --schema example/output "SELECT * FROM result ORDER BY key"

2       d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35        d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35_UPDATED
3       4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce        4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce_UPDATED
4       4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a        4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a
5       ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d        ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d
6       e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683        e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683
7       7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451        7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451
8       2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3        2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3
9       19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7        19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7

Since the first two rows are now missing from example/repo_2, the JOIN result doesn't have them either.

The Splitfile executor still caches images in this case, so if we do a rebuild against the original version of example/repo_1, sgr will check out the first version of example/output from the cache:

$ sgr rebuild example/output:latest --against example/repo_2:original_data

Rerunning example/output:46b6a361bebd3c16ea23030f91645f683487690b8c695f4585c1b01238ec1d1d against:
example/repo_2:original_data

Step 1/3 : FROM example/repo_1:103cb2da2da000f3dce9512e3cc67434d7a3c...
Resolving repository example/repo_1
 ---> Using cache
 ---> 27aeabd37f2c

Step 2/3 : FROM example/repo_2:original_data IMPORT demo AS table_2
Resolving repository example/repo_2
 ---> Using cache
 ---> ee4dbe8a8167

Step 3/3 : SQL {CREATE TABLE result  AS SELECT table_1.key        ...
 ---> Using cache
 ---> 4fd86d48715d