Query the Data Delivery Network
Query the DDNThe easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432 and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.
For example, you can query the feed_the_future_mozambique_baseline_population table in this repository, by referencing it like:
"datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latest"."feed_the_future_mozambique_baseline_population"or in a full query, like:
SELECT
":id", -- Socrata column ID
"c05_n", -- g3.04n. Decide to give away Transportation
"g03_b",
"zia_district_cd",
"c11_a", -- g3.08a. Decide to borrow from NGO
"f06", -- ZIF_Whl_day_no_eat_freq
"e07_g",
"c02_h", -- g3.02h. Owns most of Nonfarm Equip
"d01", -- ZID_Roof_tp
"b01_2", -- g2.01b Participate in Cash crop farming
"g404_e",
"d07", -- ZID_Electr_yn
"c05_f", -- g3.04f. Decide to give away Farm Equip - non mech
"c10_b", -- g3.07b. Taken any loans from Informal lender
"g04_d",
"c02_c", -- g3.02c. Owns most of Small Livestock
"c01a_l", -- g3.1a.l Anyone Owns Cell Phone
"c02_k", -- g3.02k. Owns most of Small Durable
"b03_6", -- g2.03f Input for spend money from Fishing or fishpond culture
"f04b", -- g6.02 How satisfied, scale of 1-10
"modd_missing",
"feelinputdecagr", -- Inadequacy in input in productive decisions
"b02_6", -- g2.02f Input for Fishing or fishpond culture
"b03_1", -- g2.03a Input for spend money from Food crop farming
"f03", -- ZIF_Sleep_hungry_yn
"e07_a",
"c11_d", -- g3.08d. Decide to borrow from Friends
"c06_b", -- g3.05b. Decide to rent out Large Livestock
"c02_b", -- g3.02b. Owns most of Large Livestock
"c12_d", -- g3.09d. Decide how to use money from Friends
"ea_20p",
"c01a_k", -- g3.1a.k Anyone Owns Small Durable
"c12_e", -- g3.09e. Decide how to use money from Group
"c09_d", -- g3.06d. Decide to buy new Chicken
"c06_m", -- g3.05m. Decide to rent out Other Land
"c04_a", -- g3.03a. Decide to sell Agriculture
"g02_g",
"c10_e", -- g3.07e. Taken any loans from Group
"c06_i", -- g3.05i. Decide to rent out House
"b03_2", -- g2.03b Input for spend money from Cash crop farming
"c05_b", -- g3.04b. Decide to give away Large Livestock
"c05_k", -- g3.04k. Decide to give away Small Durable
"f04", -- ZIF_Sleep_hungry_freq
"g01_i",
"f01", -- ZIF_No_food_yn
"c01a_m", -- g3.1a.m Anyone Owns Other Land
"b03_4", -- g2.03d Input for spend money from Non-farm
"c05_j", -- g3.04j. Decide to give away Large Durable
"g02_i",
"c04_n", -- g3.03n. Decide to sell Transportation
"c12_a", -- g3.09a. Decide how to use money from NGO
"c02_i", -- g3.02i. Owns most of House
"e02_a", -- g4.01 Speak in public to help decide
"g03_d",
"c11_b", -- g3.08b. Decide to borrow from Informal lender
"b01_3", -- g2.01c Participate in Livestock raising
"c02_f", -- g3.02f. Owns most of Farm Equip - non mech
"d04", -- ZID_Rooms_num
"e07_k",
"leisuretime", -- Inadequacy in leisure
"c04_l", -- g3.03l. Decide to sell Cell Phone
"zia_cluster_cd",
"c01b_e", -- g3.1b.e Household owns Fish Pond
"zia_int_outcome",
"g04_e",
"c06_n", -- g3.05n. Decide to rent out Transportation
"c06_l", -- g3.05l. Decide to rent out Cell Phone
"g02_h",
"c02_g", -- g3.02g. Owns most of Farm Equip - mech
"b03_3", -- g2.03c Input for spend money from Livestock raising
"c01a_f", -- g3.1a.f Anyone Owns Farm Equip - non mech
"hp",
"c09_a", -- g3.06a. Decide to buy new Agriculture
"g404_f",
"c11_c", -- g3.08c. Decide to borrow from Formal lender
"c06_e", -- g3.05e. Decide to rent out Fish Pond
"d11", -- ZID_Fuel_src_2
"c09_k", -- g3.06k. Decide to buy new Small Durable
"c01a_i", -- g3.1a.i Anyone Owns House
"id_code", -- ZIG1_Respondent_id
"g404_c",
"c06_f", -- g3.05f. Decide to rent out Farm Equip - non mech
"zia_location_cd",
"e07_d",
"c02_e", -- g3.02e. Owns most of Fish Pond
"incdec_count", -- Inadequacy in control over use of income
"c04_m", -- g3.03m. Decide to sell Other Land
"c09_m", -- g3.06m. Decide to buy new Other Land
"g404_a",
"f05", -- ZIF_Whl_day_no_eat_yn
"credjanydec_any", -- Inadequacy in access to and decisions on credit
"d05", -- ZID_Toilet_tp
"c04_i", -- g3.03i. Decide to sell House
"zia_visit2_dt",
"modf_missing",
"b01_4", -- g2.01d Participate in Non-farm
"a12", -- ZIG1_Int_alone
"a05", -- ZIG1_Sex_g1
"c05_m", -- g3.04m. Decide to give away Other Land
"c01a_j", -- g3.1a.j Anyone Owns Large Durable
"g02_b",
"c01b_g", -- g3.1b.g Household owns Farm Equip - mech
"c09_j", -- g3.06j. Decide to buy new Large Durable
"c09_h", -- g3.06h. Decide to buy new Nonfarm Equip
"c10_c", -- g3.07c. Taken any loans from Formal lender
"g02_c",
"aa",
"e07_c",
"g01_d",
"b01_6", -- g2.01f Participate in Fishing or fishpond culture
"c09_e", -- g3.06e. Decide to buy new Fish Pond
"c01a_n", -- g3.1a.n Anyone Owns Transportation
"c05_h", -- g3.04h. Decide to give away Nonfarm Equip
"g03_c",
"c09_i", -- g3.06i. Decide to buy new House
"c09_f", -- g3.06f. Decide to buy new Farm Equip - non mech
"c01b_l", -- g3.1b.l Household owns Cell Phone
"c01b_a", -- g3.1b.a Household owns Agriculture
"c01a_a", -- g3.1a.a Anyone Owns Agriculture
"pbs_id",
"g01_b",
"b02_1", -- g2.02a Input for Food crop farming
"ci", -- Individual disempowerment score
"c02_n", -- g3.02n. Owns most of Transportation
"ci_above_20p", -- Individual indicator that woman lacks gender parity
"c06_a", -- g3.05a. Decide to rent out Agriculture
"e07_j",
"c06_g", -- g3.05g. Decide to rent out Farm Equip - mech
"c05_i", -- g3.04i. Decide to give away House
"c01b_c", -- g3.1b.c Household owns Small Livestock
"c01a_b", -- g3.1a.b Anyone Owns Large Livestock
"he",
"d06", -- ZID_Drink_water_src
"c04_j", -- g3.03j. Decide to sell Large Durable
"c12_b", -- g3.09b. Decide how to use money from Informal lender
"raiprod_any", -- Inadequacy in autonomy in production
"d02", -- ZID_Floor_tp
"c01b_i", -- g3.1b.i Household owns House
"e07_h",
"g01_c",
"g05_e",
"c05_l", -- g3.04l. Decide to give away Cell Phone
"c05_e", -- g3.04e. Decide to give away Fish Pond
"c01a_g", -- g3.1a.g Anyone Owns Farm Equip - mech
"b02_4", -- g2.02d Input for Non-farm
"c02_j", -- g3.02j. Owns most of Large Durable
"g404_i",
"a04", -- Primary Secondary
"d10", -- ZID_Electr_yn_2
"c06_h", -- g3.05h. Decide to rent out Nonfarm Equip
"c01a_c", -- g3.1a.c Anyone Owns Small Livestock
"rp",
"ci_gap_20p", -- Individual empowerment gap for women without parity
"npoor_z105", -- Inadequacy in workload
"c01b_h", -- g3.1b.h Household owns Nonfarm Equip
"c05_c", -- g3.04c. Decide to give away Small Livestock
"speakpublic_any", -- Inadequacy in speaking in public
"c10_d", -- g3.07d. Taken any loans from Friends
"weaiwght", -- WEAI gender specific weight
"c02_d", -- g3.02d. Owns most of Chicken
"f02", -- ZIF_No_food_freq
"g05_c",
"c04_e", -- g3.03e. Decide to sell Fish Pond
"d09", -- ZID_Drink_water_src_2
"urbrur", -- Urban Rural - 1:urban; 2:rural
"c01b_f", -- g3.1b.f Household owns Farm Equip - non mech
"zi_start_dt",
"wt_weai_2", -- WEAI weight for Females
"c11_e", -- g3.08e. Decide to borrow from Group
"provname",
"d08", -- ZID_Fuel_src
"c01b_m", -- g3.1b.m Household owns Other Land
"wt_weai_1", -- WEAI weight for Males
"zia_region_cd",
"c10_a", -- g3.07a. Taken any loans from NGO
"b03_5", -- g2.03e Input for spend money from Wage and salary employment
"e07_e",
"e02_b", -- g4.02 Speak in public to ensure proper wages
"g04_b",
"g04_c",
"g02_d",
"g01_g",
"g01_e",
"e07_f",
"g404_j",
"g05_d",
"gpi",
"g404_b",
"c01b_j", -- g3.1b.j Household owns Large Durable
"c05_g", -- g3.04g. Decide to give away Farm Equip - mech
"g404_g",
"e02_c", -- g4.03 Speak in public to protest
"c12_c", -- g3.09c. Decide how to use money from Formal lender
"c09_c", -- g3.06c. Decide to buy new Small Livestock
"c04_d", -- g3.03d. Decide to sell Chicken
"c09_l", -- g3.06l. Decide to buy new Cell Phone
"c01a_d", -- g3.1a.d Anyone Owns Chicken
"d03", -- ZID_Ext_wall_tp
"g05_b",
"c09_b", -- g3.06b. Decide to buy new Large Livestock
"c06_j", -- g3.05j. Decide to rent out Large Durable
"e07_i",
"c06_d", -- g3.05d. Decide to rent out Chicken
"c05_d", -- g3.04d. Decide to give away Chicken
"c01b_k", -- g3.1b.k Household owns Small Durable
"c01a_e", -- g3.1a.e Anyone Owns Fish Pond
"c01b_d", -- g3.1b.d Household owns Chicken
"ch_20p", -- Condition of disempowerment k
"g404_h",
"c05_a", -- g3.04a. Decide to give away Agriculture
"c04_k", -- g3.03k. Decide to sell Small Durable
"c04_h", -- g3.03h. Decide to sell Nonfarm Equip
"c04_g", -- g3.03g. Decide to sell Farm Equip - mech
"c04_f", -- g3.03f. Decide to sell Farm Equip - non mech
"c09_n", -- g3.06n. Decide to buy new Transportation
"c02_m", -- g3.02m. Owns most of Other Land
"c04_c", -- g3.03c. Decide to sell Small Livestock
"c01b_b", -- g3.1b.b Household owns Large Livestock
"c02_a", -- g3.02a. Owns most of Agriculture
"c02_l", -- g3.02l. Owns most of Cell Phone
"b02_5", -- g2.02e Input for Wage and salary employment
"b02_3", -- g2.02c Input for Livestock raising
"c06_c", -- g3.05c. Decide to rent out Small Livestock
"b02_2", -- g2.02b Input for Cash crop farming
"c01b_n", -- g3.1b.n Household owns Transportation
"c04_b", -- g3.03b. Decide to sell Large Livestock
"zia_supervisor_cd",
"zia_visit2_yn",
"jown_count", -- Inadequacy in ownership of assets
"g03_e",
"g02_e",
"jrightanyagr", -- Inadequacy in purchase,sale, or transfer of assets
"districname",
"g01_h",
"e07_b",
"b01_1", -- g2.01a Participate in Food crop farming
"g404_k",
"b01_5", -- g2.01e Participate in Wage and salary employment
"g404_d",
"a06", -- ZIA_Household Type
"c09_g", -- g3.06g. Decide to buy new Farm Equip - mech
"c06_k", -- g3.05k. Decide to rent out Small Durable
"c01a_h", -- g3.1a.h Anyone Owns Nonfarm Equip
"country",
"groupmember_any" -- Inadequacy in group member
FROM
"datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latest"."feed_the_future_mozambique_baseline_population"
LIMIT 100;Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy with SQL in under 60 seconds.
Query Your Local Engine
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"Read the installation docs.
Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.
It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgrcan manage the image, container and volume for you.
There are a few ways to ingest data into the local engine.
For external repositories, the Splitgraph Engine can "mount" upstream data sources by using sgr mount. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.
For hosted datasets (like this repository), where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr cloneand sgr checkout.
Cloning Data
Because datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latest is a Splitgraph Image, you can clone the data from Spltgraph Cloud to your local engine, where you can query it like any other Postgres database, using any of your existing tools.
First, install Splitgraph if you haven't already.
Clone the metadata with sgr clone
This will be quick, and does not download the actual data.
sgr clone datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hyCheckout the data
Once you've cloned the data, you need to "checkout" the tag that you want. For example, to checkout the latest tag:
sgr checkout datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latestThis will download all the objects for the latest tag of datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy and load them into the Splitgraph Engine. Depending on your connection speed and the size of the data, you will need to wait for the checkout to complete. Once it's complete, you will be able to query the data like you would any other Postgres database.
Alternatively, use "layered checkout" to avoid downloading all the data
The data in datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latest is 0 bytes. If this is too big to download all at once, or perhaps you only need to query a subset of it, you can use a layered checkout.:
sgr checkout --layered datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latestThis will not download all the data, but it will create a schema comprised of foreign tables, that you can query as you would any other data. Splitgraph will lazily download the required objects as you query the data. In some cases, this might be faster or more efficient than a regular checkout.
Read the layered querying documentation to learn about when and why you might want to use layered queries.
Query the data with your existing tools
Once you've loaded the data into your local Splitgraph Engine, you can query it with any of your existing tools. As far as they're concerned, datahub-usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy is just another Postgres schema.