Custom Splitfile commands
It is possible to define custom Splitfile commands that follow the normal Splitfile execution semantics (cache invalidation and idempotency), allowing users to write their own transformation or data import stages.
A custom command has to extend
and implement at least one method,
execute(), that runs the command and
returns the command context hash which is combined with the hash of the previous
image to create the hash of the new image.
Additionally, the custom command can also implement
calc_hash() if it's able
to calculate the command context hash without actually running the command.
The Python class can then be registered in the
.sgconfig file as follows:
For example, for a simple data importer that loads a file from HTTP,
calc_hash() could return the hash of the file's last modified timestamp and
execute() would perform the actual computation-heavy data import.
Custom commands are currently not supported by provenance tracking.
See the Python API reference for more information on building custom Splitfile commands.