Query the Data Delivery Network
Query the DDNThe easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432
and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.
For example, you can query the nyc_cv_pilot_ie_processed_data
table in this repository, by referencing it like:
"datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj:latest"."nyc_cv_pilot_ie_processed_data"
or in a full query, like:
SELECT
":id", -- Socrata column ID
"timerecordfollow", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"excessivecurvespeedtime", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"experimentgroup", -- translation of GrpID column created by NYC
"heard", -- Boolean value indicating whether the ASD detected that the alert was heard or detected.
"excessivezonespeed", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"sizelimit", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"version", -- The firmware running on the ASD at the time of the recorded event.
"x", -- the x value of the vehicle’s location relative to location of the event in meters
"excessivecurvespeed", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"heightlimit", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"eventid", -- Event ID
"locationsource", -- Provides details on which method the location of the host vehicle was determined. Note this parameter is only meaningful in ASD firmware versions 4.3 or higher. Values can include: "gpsrsu": location is determined through a combination of GSP GNSS sensors, vehicle speed and yaw rate, and triangulation to nearby RSUs configured to broadcast V2XLocate details to ASDs. "gps": location is determined through a combination of GSP GNSS sensors, vehicle speed and yaw rate. "directgps": location is determined by GPS GNSS sensors only. Note in this location mode, BSM transmissions are halted as location is not know precisely enough for safety applications, and no event warnings should be issued.
"active", -- Boolean value indicating whether the ASD delivered the warning to the driver. True if the audible alert was issued, False if the alert was not issued (if ASD is configured for pre-deployment conditions or if the vehicle is part of the control group which does not received warnings).
"sent", -- Boolean value indicating whether the CV application determined if conditions warrant that a warning should be issued.
"hostvehid", -- Temporary ID of the host vehicle at the time of the warning.
"targetid", -- Temporary ID of the target vehicle at the time of the warning. Valid only for V2V applications.
"seqnumhv", -- Sequence number of the host vehicle's BSM which triggered the warning that was issued. Refers to the MsgSeqNum in the list of BSM records contained in the EVENT record.
"seqnumtv", -- Sequence number of the target vehicle's BSM which triggered the warning that was issued. Refers to the MsgSeqNum in the list of BSM records contained in the EVENT record. Valid only for V2V applications.
"warningtype", -- The CV application which triggered the warning. The value can be: "bsw" for Blind Spot Warning (V2V), "eebl" for Emergency Electronic Brake Lights (V2V), "fcw" for Forward Crash Warning (V2V), "ima" for Intersection Movement Assist (V2V), "lcw" for Lane Change Warning (V2V), "vtrw" for Vehicle Turning Right Warning (V2V), "cspdomp" for Curve speed compliance (V2I, "evacinfo" for Emergency Communications and Evacuation Information (V2I), "ovcclearancelimit" for Oversize Vehicle Compliance (V2I), "pedinxwalk" for Pedestrian in Signalized Crosswalk Warning (V2I), "rlvw" for Red Light Violation Warning (V2I), "spdcomp" for Speed Compliance (V2I), "spdcompwz" for Speed Compliance in Work Zone(V2I). See https://cvp.nyc/cv-safety-apps for details.
"roi", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"timerecbefore", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"dataresolution", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"minspdthreshold", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"ttc", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"grpid", -- Integer representing the group to which the vehicle is assigned. ID <= 19: Pilot test vehicles. Vehicles used by the pilot team for ASD testing purposes throughout the pilot. ID = 20: Control group vehicle. Vehicles assigned to the control group operate in 'silent' mode throughout the deployment. ASDs should not deliver warnings to the drivers, but will still record EVENTS when applications determine a warning condition exists. ID >= 21: Treatment group vehicle. Vehicles assigned to the treatment group will all operate in 'silent' mode for the before deployment data collection period. At the start of the after or live deployment, all treatment vehicles will transition to 'active' mode, and start to deliver application warning messages to the drivers.
"eventstatus", -- Value is always “Obfuscated” for released EVENT records
"timebin", -- Time period of the day, extracted from the eventHeader_eventTimeBin field. The value can be "NT" for night (12-6am), "AM" for morning (6-10am), "MD" for midday (10am-3pm), "PM" for afternoon (3-8pm), "EV" for evening (8pm-12am), and "N/A" for unspecified time of day. This field is created for use within Socrata and is not present in the data sandbox.
"locationbin", -- String of the format “XX-YY-ZZZ”, to provide some details on where in NYC the event took place, where: XX indicates whether the roadway is near an road side unit (RSU), with "CV" if near and "NY" if not near; YY is a 2-character abbreviation of the NYC borough; ZZZ provides insights into the road type where the event occurred. Detailed description for YY and ZZZ can be found in the descriptions for columns bureauBin and roadTypeBin. Note when the event occurred outside of the NYC city limits, the code "nonNYC" is used as the entire entry for the eventLocationBin.
"weathercond", -- String describing of the weather conditions (from NWS)
"airtempurature", -- Air temperature, in degrees Fahrenheit
"precipitation1hr", -- String, inches of precipitation in last hour; "M" indicates data is not available
"windspeed", -- Integer, wind speed (knots)
"volpeid", -- internal event ID that made it easier to search through the datasets
"dummytime", -- needed to use datetime data type functionalities within python and SQL
"y", -- y value of the vehicle’s location relative to location of the event in meters
"mincurvespeed", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"excessivespeed", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"excessivespeedtime", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"excessivezonespeedtime", -- This is part of the set of parameters that contains configuration parameters which help define how the ASD applications operate and how data is recorded. It is noted that which parameters are included vary based on the CV application warning type that the EVENT record is documenting.
"warningstarttime", -- while NYC’s intention with the event data was to have the “0” time marker indicate when the event was triggered, this was not always the case in practice. There was sometimes up to a 10 second offset from 0 in the BSM data. Therefore, Volpe calculated this WarningStartTime value to indicate what time in the BSM data the event actually was triggered at.
"xdeg", -- Volpe’s previously developed functions for calculating vehicle relative kinematics and visualizing and animating events require latitude and longitude values instead of X and Y in meters. These are dummy values converted from cartesian coordinates to lat long values using “close to the equator” approximations
"ydeg" -- Volpe’s previously developed functions for calculating vehicle relative kinematics and visualizing and animating events require latitude and longitude values instead of X and Y in meters. These are dummy values converted from cartesian coordinates to lat long values using “close to the equator” approximations
FROM
"datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj:latest"."nyc_cv_pilot_ie_processed_data"
LIMIT 100;
Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj
with SQL in under 60 seconds.
Query Your Local Engine
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
Read the installation docs.
Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.
It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr
, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgr
can manage the image, container and volume for you.
There are a few ways to ingest data into the local engine.
For external repositories, the Splitgraph Engine can "mount" upstream data sources by using sgr mount
. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.
For hosted datasets (like this repository), where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr clone
and sgr checkout
.
Cloning Data
Because datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj:latest
is a Splitgraph Image, you can clone the data from Spltgraph Cloud to your local engine, where you can query it like any other Postgres database, using any of your existing tools.
First, install Splitgraph if you haven't already.
Clone the metadata with sgr clone
This will be quick, and does not download the actual data.
sgr clone datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj
Checkout the data
Once you've cloned the data, you need to "checkout" the tag that you want. For example, to checkout the latest
tag:
sgr checkout datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj:latest
This will download all the objects for the latest
tag of datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj
and load them into the Splitgraph Engine. Depending on your connection speed and the size of the data, you will need to wait for the checkout to complete. Once it's complete, you will be able to query the data like you would any other Postgres database.
Alternatively, use "layered checkout" to avoid downloading all the data
The data in datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj:latest
is 0 bytes. If this is too big to download all at once, or perhaps you only need to query a subset of it, you can use a layered checkout.:
sgr checkout --layered datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj:latest
This will not download all the data, but it will create a schema comprised of foreign tables, that you can query as you would any other data. Splitgraph will lazily download the required objects as you query the data. In some cases, this might be faster or more efficient than a regular checkout.
Read the layered querying documentation to learn about when and why you might want to use layered queries.
Query the data with your existing tools
Once you've loaded the data into your local Splitgraph Engine, you can query it with any of your existing tools. As far as they're concerned, datahub-transportation-gov/nyc-cv-pilot-ie-processed-data-gu5j-r7xj
is just another Postgres schema.