ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2
Loading...

Query the Data Delivery Network

Query the DDN

The easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432 and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.

For example, you can query the dart_department_application_review_and_tracking_on table in this repository, by referencing it like:

"ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2:latest"."dart_department_application_review_and_tracking_on"

or in a full query, like:

SELECT
    ":id", -- Socrata column ID
    "short_description", -- This is a preliminary description entered when the application is first received.
    "date_received", -- Usually this is the date the permit application is received. There are two exceptions. For pre- applications this is the date the first material is received from the project sponsor. In the case of department-initiated actions, it is the date that the department mails the notice informing the permittee of an action.
    "enivronmental_justice", -- Environmental Justice and Permitting is a Commissioner issued policy of the Department that provides guidance for incorporating environmental justice concerns into the Department's permit review process. Its intent is to insure that all people regardless of race, color, or income have fair treatment and the opportunity for meaningful involvement with respect to the development, implementation, and enforcement of environmental laws, regulations, and policies. The policy became effective April 18, 2003.
    "stimulus_project", -- A Project that is for Federal Economic Funding under the American Recovery and Reinvestment Act of 2009.
    "facility", -- A facility is a property or location under the control of a single legally responsible party. Facilities may be composed of multiple contiguous parcels.
    "coastal_zone_status", -- This field indicates whether a project is located in a Coastal Management area and is subject to the Waterfront Revitalization and Coastal Resources Act.
    "shpa_status", -- SHPA stands for the New York State Historic Preservation Act of 1980. The Act declares historic preservation to be a public policy and in the public interest of the state and is found in Section 14.09, Subchapter A (Statutory Authority: Parks, Recreation and Historic Preservation Law, Section 3.09(8), Article 14). Among other requirements, this regulation requires state agencies to consult with the Office of Parks, Recreation and Historic Preservation if it appears that any projects being planned may or will cause any change, beneficial or adverse, in the quality of any historic, architectural, archeological or cultural property that is listed on the State or National Registers of Historic Places or that have been determined to be eligible for listing on the State Register.
    "permit_expration_date", -- This is the date the permit expires. Maximum permit terms are set in UPA. Not all permit types require an expiration date
    "final_disposition", -- The possible final dispositions are: Authorized by Consent Order, Denied, Discontinued Issued, and Insufficient Information. New Application needed, No Jurisdiction, Review Refused - work done prior to application, Withdrawn Waived (for Water Quality Certifications only) In the case of department- initiated modifications, suspensions, or revocations, the possible final dispositions are: Continue Existing Permit as Issued, Modify Permit, Revoke Permit, and Suspend Permit.
    "status", -- This is the current status of the application. Possible Application Status include: Completeness Determination Due, Incomplete, Complete, Written Comments Due, Suspended, Hearing Decision Due, Final Decision Due, Issued, Denied and Expired. Renewal applications for operational type permits that are timely and sufficient may also be SAPA extended, which allows the facility to continue to operate under the previous permit until a final decision is made on the renewal.
    "permit_effective_date", -- This is the date on which the permit is effective. Maximum permit terms are set in UPA.
    "other_known_ids", -- Others DEC ID Numbers associated with a specific project and site location.
    "application_id", -- The application ID is a 15-digit number that is parsed between the 1st and 2nd digits, the 5th and 6th digits, and the 10th and 11th digits. The first digit indicates the DEC region in which the regulated activity is located. A zero indicates it is in 2 or more regions. The first 10 digits represent the DEC ID.
    "dec_contact", -- Contact information for the DEC Permit Project Manager handling public comment.
    "location", -- Physical address of the facility or a description of its location.
    "enb_publication_date", -- ENB stands for Environmental Notice Bulletin. This is where Notices of Complete Application are published. If a particular application has been published in an ENB after 11/09/1999, then a link will be provided to the electronic Notice of Complete Application.
    "complete_status", -- This field can be blank, Incomplete, or Complete. If blank, completeness has not been determined or is not necessary. If Incomplete, the application is missing information needed to process it. If Complete, the permit application provides enough information for technical review to commence. In the case of federally- delegated permits (State Pollutant Discharge Elimination System permits, Air Title V, and Resource Conservation and Recovery Act permits), it means that a draft permit has been prepared. 
    "application_type", -- When the department sees a proposed activity for the first time it is NEW application. Once the permit is issued, following applications may be Renewals or Modifications. Application Type defines how the department is processing an application. In cases involving substantial modifications to a project, DEC may decide that a renewal, modification or Department Initiated Modification (DIM) will be treated as a new application.
    "permit_type", -- The permit type represents Regulatory Jurisdiction triggered by the type of activity being undertaken. A given action may have one or more permit types associated with it. Over time some permit types have become inactive as statutes or regulations have changed. Current Permit types are listed at http://www.dec.ny.gov/permits/6081.html Aquatic Pesticide Permits are non –UPA permits and are only tracked in DART when associated with a Tidal or Freshwater Wetland Permit type or when occurring in the Adirondack Park.
    "applicant", -- This is the entity that is legally responsible for making applications and if a permit is issued, for the ensuring that the conditions of permit are followed.
    "seqr_determination", -- SEQR stands for State Environmental Quality Review Act. There are four (4) possible determinations: Not Applicable, Negative Declaration, Conditioned Negative Declaration, and Positive Declaration. A Negative Declaration is one where the Lead Agency has determined that the project will not have a significant adverse impact on the environment. A Conditioned Negative Declaration is one where one or more significant adverse impacts may occur however mitigation measures have been identified and required by the Lead Agency to modify the proposed action. A Positive Declaration is one where the Lead Agency has determined that the project may have a significant adverse impact on the environment. In such as case an Environmental Impact Statement is required.
    "written_comments_due", -- This is the date by which written comments on an application must be received.
    "town_or_city", -- This is the municipality that the facility is located in.
    "upa_class", -- UPA stands for Uniform Procedures Act.   This statute and implementing regulations set procedures that the department is to follow in the processing of permit applications. There are two classes: Major and Minor. These are specifically defined in the implementing regulations, but generally minor projects are those which by their nature are not likely to have a significant effect on the environment. 
    "seqr_class", -- SEQR stands for State Environmental Quality Review Act. There are three classes: Type I, Type II, and Unlisted. Type II Actions are not subject to the procedural requirements of SEQR. There were two other classes of actions and these may show on older records: Exempt and Excluded. Like Type II Actions, these are also not subject to the procedural requirements of SEQR.
    "lead_agency" -- The lead agency is chosen from the involved agencies. These are agencies that are either undertaking an action, funding an action, or have approval authority for an action. The lead agency makes the SEQR determination and prepares the Negative or Positive Declaration. Where a Positive Declaration is prepared, it is also the job of the lead agency to manage the Environmental Impact Statement process.
FROM
    "ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2:latest"."dart_department_application_review_and_tracking_on"
LIMIT 100;

Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2 with SQL in under 60 seconds.

Query Your Local Engine

Install Splitgraph Locally
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
 

Read the installation docs.

Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.

It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgrcan manage the image, container and volume for you.

There are a few ways to ingest data into the local engine.

For external repositories, the Splitgraph Engine can "mount" upstream data sources by using sgr mount. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.

For hosted datasets (like this repository), where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr cloneand sgr checkout.

Cloning Data

Because ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2:latest is a Splitgraph Image, you can clone the data from Spltgraph Cloud to your local engine, where you can query it like any other Postgres database, using any of your existing tools.

First, install Splitgraph if you haven't already.

Clone the metadata with sgr clone

This will be quick, and does not download the actual data.

sgr clone ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2

Checkout the data

Once you've cloned the data, you need to "checkout" the tag that you want. For example, to checkout the latest tag:

sgr checkout ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2:latest

This will download all the objects for the latest tag of ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2 and load them into the Splitgraph Engine. Depending on your connection speed and the size of the data, you will need to wait for the checkout to complete. Once it's complete, you will be able to query the data like you would any other Postgres database.

Alternatively, use "layered checkout" to avoid downloading all the data

The data in ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2:latest is 0 bytes. If this is too big to download all at once, or perhaps you only need to query a subset of it, you can use a layered checkout.:

sgr checkout --layered ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2:latest

This will not download all the data, but it will create a schema comprised of foreign tables, that you can query as you would any other data. Splitgraph will lazily download the required objects as you query the data. In some cases, this might be faster or more efficient than a regular checkout.

Read the layered querying documentation to learn about when and why you might want to use layered queries.

Query the data with your existing tools

Once you've loaded the data into your local Splitgraph Engine, you can query it with any of your existing tools. As far as they're concerned, ny-gov/dart-department-application-review-and-tracking-on-mbk7-f2r2 is just another Postgres schema.

Related Documentation:

Loading...