Query the Data Delivery Network
Query the DDNThe easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432
and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.
For example, you can query the dec_land_acquisition_annual_reports_beginning_1990
table in this repository, by referencing it like:
"ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest"."dec_land_acquisition_annual_reports_beginning_1990"
or in a full query, like:
SELECT
":id", -- Socrata column ID
"land_acquisition_category", -- Indicates type of acquisition as it relates to the state’s Open Space Conservation Plan and the purpose for which the land was purchased. Open Space Plan Category Meaning <blank> Data is Missing AC Access AFE Adirondack Forest Easement AFP Adirondack Forest Preserve AFP/AFE Adirondack Forest Preserve/ Adirondack Forest Easement ARA Aquifer Recharge Area BLA Boundary Line Agreement CFE Catksill Forest Easement CFP Catskill Forest Preserve E-ESB Exceptional Scenic Beauty – 1986 Bond Act EFC Exceptional Forest Character EFP Exceptional Forest or Plan Community E-FWL Freshwater Wetlands – 1986 Bond Act E-IWA Inland Waterway Access – 1986 Bond Act E-OS Open Space - 1986 Bond Act ESB Exceptional Scenic Beauty ESB-CE Exceptional Scenic Beauty – Conservation Easement E-TR Trailways - 1986 Bond Act E-UC Unique Area - 1986 Bond Act EX Exchange FP Floodplain Preservation FWL Freshwater Wetlands GW Greenways HA Heritage Area IWA Inland Waterway Access NYC-MOA NYC Watershed Agreement (Memorandum of Agreement) OS Open Space PB Pine Barrens PFR Public Fishing Rights PL Parklands PLA Enhancement of Public Lands (access) PLB Enhancement of Public Lands (buffer) PLC Enhancement of Public Lands (consolidation) Q-AC Access - 1972 Bond Act Q-FWL Freshwater Wetlands – 1972 Bond Act Q-SR Stream Rights - 1972 Bond Act Q-TWL Tidal Wetlands - 1972 Bond Act SCR Scenic Resources SLP Shoreline Protection SP Small Project SRA State Reforestation Area TOJ Transfer of Jurisdiction TR Trailways TWA Tidal Waterway Access TWL Tidal Wetlands UA Unique Area UC Unique Character WA Waterway Access WH Wildlife Habitat WL Working Landscape WSP Watershed Protection
"dec_region", -- Administrative region for the New York State Department of Environmental Conservation http://www.dec.ny.gov/about/50230.html
"county", -- County in which majority of project acreage is located.
"project_number", -- The DEC assigned number associated with the project.
"project_name_alias", -- Name given to project during acquisition process. The property name may change after acquisition because the project name often includes the seller’s family name. The project name does not generally match the name eventually given to the land unit after the acquisition is complete.
"acres_fee", -- Acres of land purchased by DEC Fee simple - an absolute interest in land over which the holder has complete freedom of disposition during his/her life. Note: negative values in Acres (Fee) represent transfers to other state agencies, conveyances to New York City Department of Environmental Protection for drinking water protection (conservation easement retained by DEC), or an exchange of land (a net gain in acreage for DEC).
"acres_conservation_easement", -- Acres of land purchase through a conservation easement. Conservation easement - a power invested in a qualified private land conservation organization (often called a "land trust") or government (municipal, county, state or federal) to constrain, as to a specified land area, the exercise of rights otherwise held by a landowner so as to achieve certain conservation purposes. It is an interest in real property established by agreement between a landowner and land trust or unit of government. The conservation easement "runs with the land," meaning it is applicable to both present and future owners of the land. As with other real property interests, the grant of conservation easement is recorded in the local land records; the grant becomes a part of the chain of title for the property. In New York State, conservation easements are governed by Title 3 of Article 49 of the state’s Environmental Conservation Law, which defines a conservation easement as an easement, covenant, restriction or other interest in real property, created under and subject to the provisions of Title 3 of article 49 of the Environmental Conservation Law which limits or restricts development, management or use of such real property for the purpose of preserving or maintaining the scenic, open, historic, archaeological, architectural, or natural condition, character, significance or amenities of the real property in a manner consistent with the public policy and purpose set forth in section 49-0301 of the Environmental Conservation Law, provided that no such easement shall be acquired or held by the state which is subject to the provisions of Article XIV of the State Constitution.
"equivalent_stream_miles", -- The unit of measurement for Public Fishing Rights (PFR) easements, which allow the public to walk along the stream banks for the sole purpose of fishing. Easements along stream banks are normally 33 feet in width, but can vary depending on the purchased rights. One equivalent stream mile is one mile of stream bank on both sides of the stream. If a PFR easement covers one mile on one bank it is considered one half mile of equivalent stream miles. For more information on PFR: http://www.dec.ny.gov/outdoor/7746.html. Equivalent Stream Miles is not used as a unit of measurement for fee acquisition nor other forms of easements (i.e. conservation easement).
"deed_vesting_date", -- Date the deed was recorded by the county clerk.
"notes" -- Additional information
FROM
"ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest"."dec_land_acquisition_annual_reports_beginning_1990"
LIMIT 100;
Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz
with SQL in under 60 seconds.
This repository is an "external" repository. That means it's hosted elsewhere, in this case at data.ny.gov. When you queryny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz:latest
on the DDN, we "mount" the repository using the socrata
mount handler. The mount handler proxies your SQL query to the upstream data source, translating it from SQL to the relevant language (in this case SoQL).
We also cache query responses on the DDN, but we run the DDN on multiple nodes so a CACHE_HIT
is only guaranteed for subsequent queries that land on the same node.
Query Your Local Engine
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
Read the installation docs.
Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.
It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr
, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgr
can manage the image, container and volume for you.
There are a few ways to ingest data into the local engine.
For external repositories (like this repository), the Splitgraph Engine can "mount" upstream data sources by using sgr mount
. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.
For hosted datasets, where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr clone
and sgr checkout
.
Mounting Data
This repository is an external repository. It's not hosted by Splitgraph. It is hosted by data.ny.gov, and Splitgraph indexes it. This means it is not an actual Splitgraph image, so you cannot use sgr clone
to get the data. Instead, you can use the socrata
adapter with the sgr mount
command. Then, if you want, you can import the data and turn it into a Splitgraph image that others can clone.
First, install Splitgraph if you haven't already.
Mount the table with sgr mount
sgr mount socrata \
"ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz" \
--handler-options '{
"domain": "data.ny.gov",
"tables": {
"dec_land_acquisition_annual_reports_beginning_1990": "jwv3-emfz"
}
}'
That's it! Now you can query the data in the mounted table like any other Postgres table.
Query the data with your existing tools
Once you've loaded the data into your local Splitgraph engine, you can query it with any of your existing tools. As far as they're concerned, ny-gov/dec-land-acquisition-annual-reports-beginning-1990-jwv3-emfz
is just another Postgres schema.