Query the Data Delivery Network
Query the DDNThe easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432
and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.
For example, you can query the sidewalk_snow_removal_nov_2019_edmonton_insight
table in this repository, by referencing it like:
"edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq:latest"."sidewalk_snow_removal_nov_2019_edmonton_insight"
or in a full query, like:
SELECT
":id", -- Socrata column ID
"browserexperience_user_agent",
"plugintype_no_detectable_plugin",
"browserversion",
"browsertype",
"completiontime",
"invitationdate",
"starteddate",
"completeddate",
"invite_to_join",
"d6", -- Do you own or rent your primary residence in the City of Edmonton? Own Rent I prefer not to answer
"d4", -- Including yourself, how many people live in your household?
"d3", -- Are you? Under 18 years old 18-24 years old 25-34 years old 35-44 years old 45-54 years old 55-64 years old 65 to 74 years old 75+ years old I prefer not to answer
"d2", -- [IF LIVE IN EDMONTON] Which neighbourhood do you live in? Dropdown list including don’t know/prefer not to answer
"d1", -- DEMOGRAPHIC QUESTIONS FOR OPEN LINK SURVEY ONLY Do you currently? Live in Edmonton Live in surrounding areas I prefer not to answer
"snow_q10_other_please_specify", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): Other please specify
"snow_q10_volunteered_help_to_shovel_the_sidewalk", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): Volunteered help to shovel the sidewalk
"snow_q10_no_warnings_issued_straight_to_ticket", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): No warnings issued (straight to ticket)
"snow_q8_other_please_specify", -- What prevents you from making a complaint to 311? (Please check all that apply): Other please specify
"snow_q8_like_to_keep_the_peace_in_the_neighbourhood", -- What prevents you from making a complaint to 311? (Please check all that apply): Like to ‘keep the peace’ in the neighbourhood
"snow_q8_i_don_t_have_faith_it_will_be_addressed", -- What prevents you from making a complaint to 311? (Please check all that apply): I don’t have faith it will be addressed
"snow_q8_i_don_t_walk_around_the_neighbourhood_so_have_no_need_t", -- What prevents you from making a complaint to 311? (Please check all that apply): I don’t walk around the neighbourhood so have no need to complain
"snow_q7_no", -- Have you submitted complaints to 311 of sidewalks that are not cleared in your neighbourhood in the last 3 years: No (GO TO Q8)
"snow_q6_i_am_very_satisfied_with_how_my_neighbours_are_keeping_", -- Thinking of sidewalk snow removal in your neighbourhood, which of the following statements apply to you? (Please check all that apply.): I am very satisfied with how my neighbours are keeping up with removing snow and ice from the sidewalks adjacent to their private property
"snow_q5_other_please_specify", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Other please specify
"snow_q5_nobody_lives_at_my_property", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Nobody lives at my property
"snow_q5_snow_gets_too_packed_down_before_i_can_get_to_it", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Snow gets too packed down before I can get to it
"snow_q5_do_not_have_access_to_or_own_a_shovel", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Do not have access to or own a shovel
"snow_q5_safety_concerns_being_out_alone_shovelling_sidewalk", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Safety concerns being out alone shovelling sidewalk
"snow_q5_work_schedule_prevents_me_from_getting_to_it_during_day", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Work schedule prevents me from getting to it during daylight hours
"snow_q5_too_busy_to_get_to_it", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Too busy to get to it
"snow_q5_permanent_mobility_issues_eg_disability_health_issue_in", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Permanent mobility issues (eg. disability, health issue, injury)
"snow_q4_i_always_shovel_the_snow_or_remove_ice_from_the_sidewal", -- (ASK ONLY WHEN 3 = YES) Thinking of sidewalk snow removal adjacent to your property, which of the following statements apply to you? (Please check all that apply.): I always shovel the snow or remove ice from the sidewalk
"snow_q4_receiving_a_ticket_warning_prompted_me_to_keep_the_side", -- (ASK ONLY WHEN 3 = YES) Thinking of sidewalk snow removal adjacent to your property, which of the following statements apply to you? (Please check all that apply.): Receiving a ticket/warning prompted me to keep the sidewalk more clear of snow/ice
"snow_q3", -- Are you or someone in your household responsible for sidewalk snow removal adjacent to your property? Yes No Other please specify
"snow_q2", -- Do you think the fine for not shovelling the snow at $100 is Too high Too low Just right Other please specify
"snow_q1_before_today_i_knew_that_i_will_be_billed_charged_for_a", -- Thinking of sidewalk snow removal adjacent to private properties in the City of Edmonton, which of the following statements apply to you: Before today, I knew that I will be billed/charged for any snow removal from sidewalks adjacent to my property
"snow_q1_before_today_i_knew_the_fine_for_not_shoveling_the_snow", -- Thinking of sidewalk snow removal adjacent to private properties in the City of Edmonton, which of the following statements apply to you: Before today, I knew the fine for not shovelling the snow is $100
"responsedate",
"index",
"d7", -- What is your current employment status? Employed full-time (30+ hours a week) Employed part-time (0-30 hours a week) Homemaker Post-secondary student High School Student Unemployed Permanently unable to Work Retired Other I prefer not to answer
"snow_q11", -- Would anyone in your household be willing to be a volunteer to shovel a residential sidewalk for someone in need (eg. someone with mobility issues) in your neighbourhood? Yes No Maybe
"snow_q10_recognition_from_the_city_of_edmonton", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): Recognition from the City of Edmonton
"snow_q10_financial_assistance_to_hire_a_contractor", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): Financial assistance to hire a contractor
"snow_q10_increased_enforcement_presence", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): Increased enforcement presence
"snow_q9_willing_to_risk_getting_a_fine", -- Thinking of those people in your neighbourhood who do not shovel the snow or remove ice from the sidewalk after every snowfall or freeze/thaw cycle, what might be the main reasons? (Please check all that apply): Willing to risk getting a fine
"snow_q9_do_not_have_access_to_or_own_a_shovel_or_an_ice_scraper", -- Thinking of those people in your neighbourhood who do not shovel the snow or remove ice from the sidewalk after every snowfall or freeze/thaw cycle, what might be the main reasons? (Please check all that apply): Do not have access to or own a shovel or an ice scraper
"snow_q9_safety_concerns_i_e_being_out_alone_injuries_etc", -- Thinking of those people in your neighbourhood who do not shovel the snow or remove ice from the sidewalk after every snowfall or freeze/thaw cycle, what might be the main reasons? (Please check all that apply): Safety concerns, i.e. being out alone, injuries, etc.
"snow_q9_too_busy_to_get_to_it", -- Thinking of those people in your neighbourhood who do not shovel the snow or remove ice from the sidewalk after every snowfall or freeze/thaw cycle, what might be the main reasons? (Please check all that apply): Too busy to get to it
"snow_q9_mobility_issues_eg_disability_health_issue_injury", -- Thinking of those people in your neighbourhood who do not shovel the snow or remove ice from the sidewalk after every snowfall or freeze/thaw cycle, what might be the main reasons? (Please check all that apply): Mobility issues (eg. disability, health issue, injury)
"snow_q8_311_app_was_not_working", -- What prevents you from making a complaint to 311? (Please check all that apply): 311 app was not working
"snow_q8_i_don_t_pay_attention_to_other_sidewalks_in_the_neighbo", -- What prevents you from making a complaint to 311? (Please check all that apply): I don’t pay attention to other sidewalks in the neighbourhood
"snow_q7_not_sure_can_t_remember", -- Have you submitted complaints to 311 of sidewalks that are not cleared in your neighbourhood in the last 3 years: Not sure/Can’t remember (GO TO Q9)
"snow_q6_none_of_the_above", -- Thinking of sidewalk snow removal in your neighbourhood, which of the following statements apply to you? (Please check all that apply.): None of the above [exclusive]
"snow_q4_i_have_received_a_ticket_or_warning_for_not_shovelling_", -- (ASK ONLY WHEN 3 = YES) Thinking of sidewalk snow removal adjacent to your property, which of the following statements apply to you? (Please check all that apply.): I have received a ticket or warning for not shovelling the snow in the past 3 years
"snow_q1_i_know_where_the_free_sand_is_located_in_my_neighbourho", -- Thinking of sidewalk snow removal adjacent to private properties in the City of Edmonton, which of the following statements apply to you: I know where the free sand is located in my neighbourhood
"snow_q1_before_today_i_knew_the_city_provided_free_sand_to_comm", -- Thinking of sidewalk snow removal adjacent to private properties in the City of Edmonton, which of the following statements apply to you: Before today, I knew the City provided free sand to communities
"snow_q9_nobody_has_complained_yet", -- Thinking of those people in your neighbourhood who do not shovel the snow or remove ice from the sidewalk after every snowfall or freeze/thaw cycle, what might be the main reasons? (Please check all that apply): Nobody has complained yet
"snow_q6_i_have_fallen_on_a_residential_sidewalk_that_is_adjacen", -- Thinking of sidewalk snow removal in your neighbourhood, which of the following statements apply to you? (Please check all that apply.): I have fallen on a residential sidewalk that is adjacent to private property because it wasn’t adequately cleared
"snow_q7_yes", -- Have you submitted complaints to 311 of sidewalks that are not cleared in your neighbourhood in the last 3 years: Yes (GO TO Q9)
"snow_q5_short_term_but_recoverable_mobility_issues_eg_disabilit", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Short term, but recoverable, mobility issues (eg. disability, health issue, injury)
"snow_q5_don_t_want_to_risk_injury_while_shovelling", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Don’t want to risk injury while shovelling
"snow_q5_do_not_have_access_to_or_own_an_ice_scraper", -- (ASK ONLY WHEN 4 [I always shovel] = NOT Chosen) What were the main reasons why you do not always shovel the snow or remove ice from the sidewalk? (Please check all that apply): Do not have access to or own an ice scraper
"d5", -- Do you live in Apartment/condo building Single family house Townhouse/duplex I prefer not to answer
"snow_q8_no_need_sidewalks_are_cleared_in_my_neighbourhood", -- What prevents you from making a complaint to 311? (Please check all that apply): No need, sidewalks are cleared in my neighbourhood
"snow_q4_receiving_a_snow_removal_bill_prompted_me_to_keep_the_s", -- (ASK ONLY WHEN 3 = YES) Thinking of sidewalk snow removal adjacent to your property, which of the following statements apply to you? (Please check all that apply.): Receiving a snow removal bill prompted me to keep the sidewalk more clear of snow/ice
"snow_q1_before_today_i_knew_there_is_a_bylaw_requiring_owners_t", -- Thinking of sidewalk snow removal adjacent to private properties in the City of Edmonton, which of the following statements apply to you: Before today, I knew there is a bylaw requiring owners to shovel the snow
"snow_q10_higher_fines", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): Higher fines
"devicetype",
"plugintype_flash",
"snow_q8_311_would_not_take_the_complaint", -- What prevents you from making a complaint to 311? (Please check all that apply): 311 would not take the complaint
"snow_q10_none_of_the_above", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): None of the above
"snow_q1_none_of_the_above", -- Thinking of sidewalk snow removal adjacent to private properties in the City of Edmonton, which of the following statements apply to you: None of the above [exclusive]
"snow_q8_fear_of_retaliation", -- What prevents you from making a complaint to 311? (Please check all that apply): Fear of retaliation
"browserexperience_javascript_enabled",
"ostype",
"respondentsource_icm_power_surveys",
"pointsearned",
"version",
"responsestatus",
"respondentlocale",
"snow_q10_neighbours_start_complaining", -- In your opinion, what would motivate people to remove the snow/ice from the sidewalks more regularly? (Please check all that apply): Neighbours start complaining
"snow_q9_other_please_specify" -- Thinking of those people in your neighbourhood who do not shovel the snow or remove ice from the sidewalk after every snowfall or freeze/thaw cycle, what might be the main reasons? (Please check all that apply): Other (please specify)
FROM
"edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq:latest"."sidewalk_snow_removal_nov_2019_edmonton_insight"
LIMIT 100;
Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq
with SQL in under 60 seconds.
Query Your Local Engine
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
Read the installation docs.
Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.
It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr
, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgr
can manage the image, container and volume for you.
There are a few ways to ingest data into the local engine.
For external repositories, the Splitgraph Engine can "mount" upstream data sources by using sgr mount
. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.
For hosted datasets (like this repository), where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr clone
and sgr checkout
.
Cloning Data
Because edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq:latest
is a Splitgraph Image, you can clone the data from Spltgraph Cloud to your local engine, where you can query it like any other Postgres database, using any of your existing tools.
First, install Splitgraph if you haven't already.
Clone the metadata with sgr clone
This will be quick, and does not download the actual data.
sgr clone edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq
Checkout the data
Once you've cloned the data, you need to "checkout" the tag that you want. For example, to checkout the latest
tag:
sgr checkout edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq:latest
This will download all the objects for the latest
tag of edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq
and load them into the Splitgraph Engine. Depending on your connection speed and the size of the data, you will need to wait for the checkout to complete. Once it's complete, you will be able to query the data like you would any other Postgres database.
Alternatively, use "layered checkout" to avoid downloading all the data
The data in edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq:latest
is 0 bytes. If this is too big to download all at once, or perhaps you only need to query a subset of it, you can use a layered checkout.:
sgr checkout --layered edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq:latest
This will not download all the data, but it will create a schema comprised of foreign tables, that you can query as you would any other data. Splitgraph will lazily download the required objects as you query the data. In some cases, this might be faster or more efficient than a regular checkout.
Read the layered querying documentation to learn about when and why you might want to use layered queries.
Query the data with your existing tools
Once you've loaded the data into your local Splitgraph Engine, you can query it with any of your existing tools. As far as they're concerned, edmonton-ca/sidewalk-snow-removal-nov-2019-edmonton-insight-y795-rveq
is just another Postgres schema.