Query the Data Delivery Network
Query the DDNThe easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432
and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.
For example, you can query the dec_land_acquisition_annual_reports_beginning_1990
table in this repository, by referencing it like:
"ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest"."dec_land_acquisition_annual_reports_beginning_1990"
or in a full query, like:
SELECT
":id", -- Socrata column ID
"land_acquisition_category", -- Indicates type of acquisition as it relates to the state’s Open Space Conservation Plan and the purpose for which the land was purchased. Open Space Plan Category Meaning <blank> Data is Missing AC Access AFE Adirondack Forest Easement AFP Adirondack Forest Preserve AFP/AFE Adirondack Forest Preserve/ Adirondack Forest Easement ARA Aquifer Recharge Area BLA Boundary Line Agreement CFE Catksill Forest Easement CFP Catskill Forest Preserve E-ESB Exceptional Scenic Beauty – 1986 Bond Act EFC Exceptional Forest Character EFP Exceptional Forest or Plan Community E-FWL Freshwater Wetlands – 1986 Bond Act E-IWA Inland Waterway Access – 1986 Bond Act E-OS Open Space - 1986 Bond Act ESB Exceptional Scenic Beauty ESB-CE Exceptional Scenic Beauty – Conservation Easement E-TR Trailways - 1986 Bond Act E-UC Unique Area - 1986 Bond Act EX Exchange FP Floodplain Preservation FWL Freshwater Wetlands GW Greenways HA Heritage Area IWA Inland Waterway Access NYC-MOA NYC Watershed Agreement (Memorandum of Agreement) OS Open Space PB Pine Barrens PFR Public Fishing Rights PL Parklands PLA Enhancement of Public Lands (access) PLB Enhancement of Public Lands (buffer) PLC Enhancement of Public Lands (consolidation) Q-AC Access - 1972 Bond Act Q-FWL Freshwater Wetlands – 1972 Bond Act Q-SR Stream Rights - 1972 Bond Act Q-TWL Tidal Wetlands - 1972 Bond Act SCR Scenic Resources SLP Shoreline Protection SP Small Project SRA State Reforestation Area TOJ Transfer of Jurisdiction TR Trailways TWA Tidal Waterway Access TWL Tidal Wetlands UA Unique Area UC Unique Character WA Waterway Access WH Wildlife Habitat WL Working Landscape WSP Watershed Protection
"project_name_alias", -- Name given to project during acquisition process. The property name may change after acquisition because the project name often includes the seller’s family name. The project name does not generally match the name eventually given to the land unit after the acquisition is complete.
"acres_conservation_easement", -- Acres of land purchase through a conservation easement. Conservation easement - a power invested in a qualified private land conservation organization (often called a "land trust") or government (municipal, county, state or federal) to constrain, as to a specified land area, the exercise of rights otherwise held by a landowner so as to achieve certain conservation purposes. It is an interest in real property established by agreement between a landowner and land trust or unit of government. The conservation easement "runs with the land," meaning it is applicable to both present and future owners of the land. As with other real property interests, the grant of conservation easement is recorded in the local land records; the grant becomes a part of the chain of title for the property. In New York State, conservation easements are governed by Title 3 of Article 49 of the state’s Environmental Conservation Law, which defines a conservation easement as an easement, covenant, restriction or other interest in real property, created under and subject to the provisions of Title 3 of article 49 of the Environmental Conservation Law which limits or restricts development, management or use of such real property for the purpose of preserving or maintaining the scenic, open, historic, archaeological, architectural, or natural condition, character, significance or amenities of the real property in a manner consistent with the public policy and purpose set forth in section 49-0301 of the Environmental Conservation Law, provided that no such easement shall be acquired or held by the state which is subject to the provisions of Article XIV of the State Constitution.
"dec_region", -- Administrative region for the New York State Department of Environmental Conservation http://www.dec.ny.gov/about/50230.html
"county", -- County in which majority of project acreage is located.
"project_number", -- The DEC assigned number associated with the project.
"acres_fee", -- Acres of land purchased by DEC Fee simple - an absolute interest in land over which the holder has complete freedom of disposition during his/her life. Note: negative values in Acres (Fee) represent transfers to other state agencies, conveyances to New York City Department of Environmental Protection for drinking water protection (conservation easement retained by DEC), or an exchange of land (a net gain in acreage for DEC).
"notes", -- Additional information
"deed_vesting_date", -- Date the deed was recorded by the county clerk.
"equivalent_stream_miles" -- The unit of measurement for Public Fishing Rights (PFR) easements, which allow the public to walk along the stream banks for the sole purpose of fishing. Easements along stream banks are normally 33 feet in width, but can vary depending on the purchased rights. One equivalent stream mile is one mile of stream bank on both sides of the stream. If a PFR easement covers one mile on one bank it is considered one half mile of equivalent stream miles. For more information on PFR: http://www.dec.ny.gov/outdoor/7746.html. Equivalent Stream Miles is not used as a unit of measurement for fee acquisition nor other forms of easements (i.e. conservation easement).
FROM
"ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest"."dec_land_acquisition_annual_reports_beginning_1990"
LIMIT 100;
Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz
with SQL in under 60 seconds.
Query Your Local Engine
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
Read the installation docs.
Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.
It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr
, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgr
can manage the image, container and volume for you.
There are a few ways to ingest data into the local engine.
For external repositories, the Splitgraph Engine can "mount" upstream data sources by using sgr mount
. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.
For hosted datasets (like this repository), where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr clone
and sgr checkout
.
Cloning Data
Because ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest
is a Splitgraph Image, you can clone the data from Spltgraph Cloud to your local engine, where you can query it like any other Postgres database, using any of your existing tools.
First, install Splitgraph if you haven't already.
Clone the metadata with sgr clone
This will be quick, and does not download the actual data.
sgr clone ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz
Checkout the data
Once you've cloned the data, you need to "checkout" the tag that you want. For example, to checkout the latest
tag:
sgr checkout ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest
This will download all the objects for the latest
tag of ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz
and load them into the Splitgraph Engine. Depending on your connection speed and the size of the data, you will need to wait for the checkout to complete. Once it's complete, you will be able to query the data like you would any other Postgres database.
Alternatively, use "layered checkout" to avoid downloading all the data
The data in ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest
is 0 bytes. If this is too big to download all at once, or perhaps you only need to query a subset of it, you can use a layered checkout.:
sgr checkout --layered ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest
This will not download all the data, but it will create a schema comprised of foreign tables, that you can query as you would any other data. Splitgraph will lazily download the required objects as you query the data. In some cases, this might be faster or more efficient than a regular checkout.
Read the layered querying documentation to learn about when and why you might want to use layered queries.
Query the data with your existing tools
Once you've loaded the data into your local Splitgraph Engine, you can query it with any of your existing tools. As far as they're concerned, ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz
is just another Postgres schema.