splitgraph.config package


splitgraph.config.argument_config module


Get the raw (argN, argN+1) tuples from sys.argv

We could also use click to parse the flags ahead of time, and then pass the parsed flag object into SystemConfigGetters. But this way, we avoid having to pass down variables from click just to get config values.

splitgraph.config.argument_config.get_argument_config_value(key, default_return=None)

Get get the value of an argument, where value is the argument immediately following the argument matching a key in ARG_KEYS, e.g.:

SYS_ARGS = [“–namespace”, “foo”] –> return “foo”

Otherwise, return default_return

splitgraph.config.config module


Create and return a dict of all known config values

splitgraph.config.config.lazy_get_config_value(key, default_return=None)

Get the config value for a key in the following precedence Otherwise return default_return


Given an existing config_dict, update after reading sys.argv and overwriting any keys.

Return updated copy of config_dict.


Given an existing config_dict, update after reading os.environ and overwriting any keys.

Return updated copy of config_dict.

splitgraph.config.config.update_config_dict_from_file(config_dict, sg_config_file)

Given an existing config_dict, update after reading sg_config_file and overwriting any keys according to the rules in config_file_config

Return updated copy of config_dict.

splitgraph.config.config_file_config module


Transform a config_dict to “accumulate” objects “nested” via key name

Because ConfigParser does not support nesting, we implement our own syntax via the key names of sections. The ‘:’ character can be used in section names to specify the left and right key. Example:

.ini config             new config_dict

[remote: remote1]   |   {
SG_ENV_VAR=foo      |       **rest_of_config_dict,
                    |       "remotes": {
[origin: origin1]   |           "remote1": {
SG_ENV_VAR=bar      |               "SG_ENV_VAR": "foo"
                    |           }
[origin: origin2]   |       },
SG_ENV_VAR=bax      |       "origins": {
                    |           "origin1": {
                    |               "SG_ENV_VAR": "bar"
                    |           },
                    |           "origin2": {
                    |               "SG_ENV_VAR": "bax"
                    |           }
                    |       }
                    |   }

:return a new, updated copy of config_dict

splitgraph.config.config_file_config.get_config_dict_from_config_file(sg_file, **kwargs)

Create a dict from ConfigParser, apply transformations to it.

Return parsed and transformed config_dict.

splitgraph.config.config_file_config.get_config_dict_from_file(sg_file, **kwargs)
splitgraph.config.config_file_config.hoist_section(config_dict, section='defaults')

If a section exists called <hoist_section>, hoist it to the top level This is useful for overriding default configs from within a config file

Transform config_dict to “hoist” any config values from a section into the top level (thus, overriding environment variables), when the name of the section matches hoist_section.

Return a new, updated copy of config_dict.

splitgraph.config.config_file_config.transform_config_dict(config_dict, **kwargs)

Apply transformations to the raw ConfigParser.config object

  1. hoist_section

  2. accumulate_lists

Return the a new, updated copy of config_dict.

splitgraph.config.default_config module

splitgraph.config.default_config.get_default_config_value(key, default_return=None)

Get the hard-coded default value of a config key. Otherwise return default_return.

splitgraph.config.environment_config module

splitgraph.config.environment_config.get_environment_config_value(key, default_return=None)

Get the environment variable value of the environment variable matching key. Otherwise return default_return.

splitgraph.config.keys module

splitgraph.config.keys.SENSITIVE_KEYS = ['SG_ENGINE_PWD', 'SG_ENGINE_ADMIN_PWD', 'SG_S3_PWD']

Warning: Every key in DEFAULTS must have a key in ARGUMENT_KEY_MAP If you add/remove keys from DEFAULTS, make sure to do so here too.

splitgraph.config.system_config module

splitgraph.config.system_config.file_exists(_dir, filename)

Get the location of an existing SG_CONFIG_FILE on the system with a valid name. Do not attempt to parse the config file, just return its location.

Otherwise, return default_return


Get any explicitly defined config file directories, which are directories where we should search for files from VALID_CONFIG_FILE_NAMES.

This list is defined similar to $PATH, as a colon (:) delimited string, either in:

  • argument flag –config-dirs

  • or environment key SG_CONFIG_DIRS

Or a single directory defined in either

  • argument flag –config-dir

  • environment key SG_CONFIG_DIR

If both plural and single are defined, join them together.

Return a list of valid paths that are set, or an empty list.

Print a warning if any paths to not exist.


Get the explicitly defined location of config file, if defined.

The location will either be defined in:

  • argument flag –config-file

  • or environment key SG_CONFIG_FILE

In keeping with assumptions about priority, argument flag has higher priority than environment value.

If the location is set, and points to an existing file, return location.

Otherwise return None

Print a warning if location is set but points to non-existing file.

splitgraph.config.system_config.get_system_config_value(key, default_return=None)

Module contents

The CONFIG object is created and exported once __at import time__ Calling CONFIG[“KEY”] directly should be sufficient in most cases, except when a config value has changed since importing CONFIG. In that case, create_config_dict() can provide an updated config dict

Priority (highest to lowest):

  1. Command line argument values

  2. Environment variable values

  3. Config file values in [defaults] section

  4. DEFAULTS (see keys.py)