Query the Data Delivery Network
Query the DDNThe easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432 and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.
For example, you can query the feed_the_future_mozambique_baseline_population table in this repository, by referencing it like:
"usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latest"."feed_the_future_mozambique_baseline_population"or in a full query, like:
SELECT
":id", -- Socrata column ID
"c06_c", -- g3.05c. Decide to rent out Small Livestock
"c01b_n", -- g3.1b.n Household owns Transportation
"c04_b", -- g3.03b. Decide to sell Large Livestock
"jown_count", -- Inadequacy in ownership of assets
"g02_e",
"c01a_h", -- g3.1a.h Anyone Owns Nonfarm Equip
"country",
"groupmember_any", -- Inadequacy in group member
"c09_i", -- g3.06i. Decide to buy new House
"c01b_l", -- g3.1b.l Household owns Cell Phone
"c01b_a", -- g3.1b.a Household owns Agriculture
"c01a_a", -- g3.1a.a Anyone Owns Agriculture
"b02_1", -- g2.02a Input for Food crop farming
"ci_above_20p", -- Individual indicator that woman lacks gender parity
"c06_a", -- g3.05a. Decide to rent out Agriculture
"e07_j",
"c05_i", -- g3.04i. Decide to give away House
"he",
"c04_j", -- g3.03j. Decide to sell Large Durable
"raiprod_any", -- Inadequacy in autonomy in production
"c01b_i", -- g3.1b.i Household owns House
"g01_c",
"a04", -- Primary Secondary
"c01a_c", -- g3.1a.c Anyone Owns Small Livestock
"c10_d", -- g3.07d. Taken any loans from Friends
"c02_d", -- g3.02d. Owns most of Chicken
"g05_c",
"c01b_f", -- g3.1b.f Household owns Farm Equip - non mech
"c11_e", -- g3.08e. Decide to borrow from Group
"provname",
"d08", -- ZID_Fuel_src
"c01b_m", -- g3.1b.m Household owns Other Land
"c10_a", -- g3.07a. Taken any loans from NGO
"e07_i",
"d07", -- ZID_Electr_yn
"c10_b", -- g3.07b. Taken any loans from Informal lender
"g04_d",
"modd_missing",
"feelinputdecagr", -- Inadequacy in input in productive decisions
"f03", -- ZIF_Sleep_hungry_yn
"c11_d", -- g3.08d. Decide to borrow from Friends
"c12_d", -- g3.09d. Decide how to use money from Friends
"ea_20p",
"c12_e", -- g3.09e. Decide how to use money from Group
"c09_d", -- g3.06d. Decide to buy new Chicken
"c06_m", -- g3.05m. Decide to rent out Other Land
"g02_g",
"c05_b", -- g3.04b. Decide to give away Large Livestock
"c05_k", -- g3.04k. Decide to give away Small Durable
"c01a_m", -- g3.1a.m Anyone Owns Other Land
"b03_4", -- g2.03d Input for spend money from Non-farm
"c05_j", -- g3.04j. Decide to give away Large Durable
"c04_n", -- g3.03n. Decide to sell Transportation
"c02_i", -- g3.02i. Owns most of House
"e02_a", -- g4.01 Speak in public to help decide
"d04", -- ZID_Rooms_num
"e07_k",
"leisuretime", -- Inadequacy in leisure
"c04_l", -- g3.03l. Decide to sell Cell Phone
"zia_int_outcome",
"c06_n", -- g3.05n. Decide to rent out Transportation
"c06_l", -- g3.05l. Decide to rent out Cell Phone
"g02_h",
"b03_3", -- g2.03c Input for spend money from Livestock raising
"c02_h", -- g3.02h. Owns most of Nonfarm Equip
"f06", -- ZIF_Whl_day_no_eat_freq
"c11_a", -- g3.08a. Decide to borrow from NGO
"c05_n", -- g3.04n. Decide to give away Transportation
"g404_i",
"g03_b",
"zia_district_cd",
"e07_g",
"b01_2", -- g2.01b Participate in Cash crop farming
"g404_e",
"c05_f", -- g3.04f. Decide to give away Farm Equip - non mech
"c02_c", -- g3.02c. Owns most of Small Livestock
"c01a_l", -- g3.1a.l Anyone Owns Cell Phone
"c02_k", -- g3.02k. Owns most of Small Durable
"b03_6", -- g2.03f Input for spend money from Fishing or fishpond culture
"f04b", -- g6.02 How satisfied, scale of 1-10
"b02_6", -- g2.02f Input for Fishing or fishpond culture
"b03_1", -- g2.03a Input for spend money from Food crop farming
"e07_a",
"c06_b", -- g3.05b. Decide to rent out Large Livestock
"c02_b", -- g3.02b. Owns most of Large Livestock
"c01a_k", -- g3.1a.k Anyone Owns Small Durable
"c04_a", -- g3.03a. Decide to sell Agriculture
"c10_e", -- g3.07e. Taken any loans from Group
"c06_i", -- g3.05i. Decide to rent out House
"b03_2", -- g2.03b Input for spend money from Cash crop farming
"f04", -- ZIF_Sleep_hungry_freq
"g01_i",
"f01", -- ZIF_No_food_yn
"g02_i",
"c12_a", -- g3.09a. Decide how to use money from NGO
"g03_d",
"c11_b", -- g3.08b. Decide to borrow from Informal lender
"b01_3", -- g2.01c Participate in Livestock raising
"c02_f", -- g3.02f. Owns most of Farm Equip - non mech
"zia_cluster_cd",
"c01b_e", -- g3.1b.e Household owns Fish Pond
"g04_e",
"c02_g", -- g3.02g. Owns most of Farm Equip - mech
"hp",
"c09_a", -- g3.06a. Decide to buy new Agriculture
"g404_f",
"c06_e", -- g3.05e. Decide to rent out Fish Pond
"c01a_i", -- g3.1a.i Anyone Owns House
"g404_c",
"c06_f", -- g3.05f. Decide to rent out Farm Equip - non mech
"e07_d",
"c04_m", -- g3.03m. Decide to sell Other Land
"g404_a",
"f05", -- ZIF_Whl_day_no_eat_yn
"zia_visit2_dt",
"modf_missing",
"b01_4", -- g2.01d Participate in Non-farm
"a12", -- ZIG1_Int_alone
"g02_b",
"c01b_g", -- g3.1b.g Household owns Farm Equip - mech
"g02_c",
"e07_c",
"g01_d",
"b01_6", -- g2.01f Participate in Fishing or fishpond culture
"c01a_n", -- g3.1a.n Anyone Owns Transportation
"g03_c",
"c09_f", -- g3.06f. Decide to buy new Farm Equip - non mech
"pbs_id",
"g01_b",
"ci", -- Individual disempowerment score
"c02_n", -- g3.02n. Owns most of Transportation
"c06_g", -- g3.05g. Decide to rent out Farm Equip - mech
"c01b_c", -- g3.1b.c Household owns Small Livestock
"c01a_b", -- g3.1a.b Anyone Owns Large Livestock
"d06", -- ZID_Drink_water_src
"c12_b", -- g3.09b. Decide how to use money from Informal lender
"d02", -- ZID_Floor_tp
"c01a_f", -- g3.1a.f Anyone Owns Farm Equip - non mech
"c11_c", -- g3.08c. Decide to borrow from Formal lender
"d11", -- ZID_Fuel_src_2
"c09_k", -- g3.06k. Decide to buy new Small Durable
"id_code", -- ZIG1_Respondent_id
"zia_location_cd",
"c02_e", -- g3.02e. Owns most of Fish Pond
"incdec_count", -- Inadequacy in control over use of income
"c09_m", -- g3.06m. Decide to buy new Other Land
"credjanydec_any", -- Inadequacy in access to and decisions on credit
"d05", -- ZID_Toilet_tp
"c04_i", -- g3.03i. Decide to sell House
"a05", -- ZIG1_Sex_g1
"c05_m", -- g3.04m. Decide to give away Other Land
"c01a_j", -- g3.1a.j Anyone Owns Large Durable
"c09_j", -- g3.06j. Decide to buy new Large Durable
"c09_h", -- g3.06h. Decide to buy new Nonfarm Equip
"c10_c", -- g3.07c. Taken any loans from Formal lender
"aa",
"c09_e", -- g3.06e. Decide to buy new Fish Pond
"c05_h", -- g3.04h. Decide to give away Nonfarm Equip
"e07_h",
"g05_e",
"c05_l", -- g3.04l. Decide to give away Cell Phone
"c05_e", -- g3.04e. Decide to give away Fish Pond
"d10", -- ZID_Electr_yn_2
"c06_h", -- g3.05h. Decide to rent out Nonfarm Equip
"c01b_h", -- g3.1b.h Household owns Nonfarm Equip
"c05_c", -- g3.04c. Decide to give away Small Livestock
"c04_e", -- g3.03e. Decide to sell Fish Pond
"e07_e",
"e02_b", -- g4.02 Speak in public to ensure proper wages
"g04_b",
"gpi",
"c01b_j", -- g3.1b.j Household owns Large Durable
"c04_d", -- g3.03d. Decide to sell Chicken
"d03", -- ZID_Ext_wall_tp
"g05_b",
"c01b_k", -- g3.1b.k Household owns Small Durable
"ch_20p", -- Condition of disempowerment k
"g404_h",
"c09_n", -- g3.06n. Decide to buy new Transportation
"c01b_b", -- g3.1b.b Household owns Large Livestock
"b03_5", -- g2.03e Input for spend money from Wage and salary employment
"g05_d",
"c05_g", -- g3.04g. Decide to give away Farm Equip - mech
"c09_l", -- g3.06l. Decide to buy new Cell Phone
"c01a_d", -- g3.1a.d Anyone Owns Chicken
"d01", -- ZID_Roof_tp
"c01a_e", -- g3.1a.e Anyone Owns Fish Pond
"c01b_d", -- g3.1b.d Household owns Chicken
"c02_m", -- g3.02m. Owns most of Other Land
"jrightanyagr", -- Inadequacy in purchase,sale, or transfer of assets
"districname",
"b01_1", -- g2.01a Participate in Food crop farming
"b01_5", -- g2.01e Participate in Wage and salary employment
"b02_4", -- g2.02d Input for Non-farm
"c01a_g", -- g3.1a.g Anyone Owns Farm Equip - mech
"c02_a", -- g3.02a. Owns most of Agriculture
"c02_j", -- g3.02j. Owns most of Large Durable
"c06_k", -- g3.05k. Decide to rent out Small Durable
"c09_g", -- g3.06g. Decide to buy new Farm Equip - mech
"a06", -- ZIA_Household Type
"g404_d",
"g404_k",
"e07_b",
"g01_h",
"g03_e",
"zia_visit2_yn",
"zia_supervisor_cd",
"b02_2", -- g2.02b Input for Cash crop farming
"b02_3", -- g2.02c Input for Livestock raising
"b02_5", -- g2.02e Input for Wage and salary employment
"c02_l", -- g3.02l. Owns most of Cell Phone
"c04_c", -- g3.03c. Decide to sell Small Livestock
"c04_f", -- g3.03f. Decide to sell Farm Equip - non mech
"c04_g", -- g3.03g. Decide to sell Farm Equip - mech
"c04_h", -- g3.03h. Decide to sell Nonfarm Equip
"c04_k", -- g3.03k. Decide to sell Small Durable
"c05_a", -- g3.04a. Decide to give away Agriculture
"c05_d", -- g3.04d. Decide to give away Chicken
"c06_d", -- g3.05d. Decide to rent out Chicken
"c06_j", -- g3.05j. Decide to rent out Large Durable
"c09_b", -- g3.06b. Decide to buy new Large Livestock
"c09_c", -- g3.06c. Decide to buy new Small Livestock
"c12_c", -- g3.09c. Decide how to use money from Formal lender
"e02_c", -- g4.03 Speak in public to protest
"g404_g",
"g404_b",
"g404_j",
"e07_f",
"g01_e",
"g01_g",
"g02_d",
"g04_c",
"zia_region_cd",
"wt_weai_1", -- WEAI weight for Males
"wt_weai_2", -- WEAI weight for Females
"zi_start_dt",
"urbrur", -- Urban Rural - 1:urban; 2:rural
"d09", -- ZID_Drink_water_src_2
"f02", -- ZIF_No_food_freq
"weaiwght", -- WEAI gender specific weight
"speakpublic_any", -- Inadequacy in speaking in public
"npoor_z105", -- Inadequacy in workload
"ci_gap_20p", -- Individual empowerment gap for women without parity
"rp"
FROM
"usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latest"."feed_the_future_mozambique_baseline_population"
LIMIT 100;Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy with SQL in under 60 seconds.
This repository is an "external" repository. That means it's hosted elsewhere, in this case at data.usaid.gov. When you queryusaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy:latest on the DDN, we "mount" the repository using the socrata mount handler. The mount handler proxies your SQL query to the upstream data source, translating it from SQL to the relevant language (in this case SoQL).
We also cache query responses on the DDN, but we run the DDN on multiple nodes so a CACHE_HIT is only guaranteed for subsequent queries that land on the same node.
Query Your Local Engine
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"Read the installation docs.
Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.
It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgrcan manage the image, container and volume for you.
There are a few ways to ingest data into the local engine.
For external repositories (like this repository), the Splitgraph Engine can "mount" upstream data sources by using sgr mount. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.
For hosted datasets, where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr cloneand sgr checkout.
Mounting Data
This repository is an external repository. It's not hosted by Splitgraph. It is hosted by data.usaid.gov, and Splitgraph indexes it. This means it is not an actual Splitgraph image, so you cannot use sgr clone to get the data. Instead, you can use the socrata adapter with the sgr mount command. Then, if you want, you can import the data and turn it into a Splitgraph image that others can clone.
First, install Splitgraph if you haven't already.
Mount the table with sgr mount
sgr mount socrata \
"usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy" \
--handler-options '{
"domain": "data.usaid.gov",
"tables": {
"feed_the_future_mozambique_baseline_population": "7dr5-x8hy"
}
}'That's it! Now you can query the data in the mounted table like any other Postgres table.
Query the data with your existing tools
Once you've loaded the data into your local Splitgraph engine, you can query it with any of your existing tools. As far as they're concerned, usaid-gov/feed-the-future-mozambique-baseline-population-7dr5-x8hy is just another Postgres schema.