Query the Data Delivery Network
Query the DDNThe easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432
and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.
For example, you can query the lobbyist_bimonthly_reports_beginning_2019
table in this repository, by referencing it like:
"ny-gov/lobbyist-bimonthly-reports-beginning-2019-t9kf-dqbc:latest"."lobbyist_bimonthly_reports_beginning_2019"
or in a full query, like:
SELECT
":id", -- Socrata column ID
"expense_reimbursed_by_client", -- Yes or No. Indicates if the itemized expense was reimbursed by the Client to a Principal Lobbyist.
"monitoring_only", -- Yes or No. If a Filer did not actually Lobby a specific Party, they may select ‘Monitoring Only’. In such cases, the Filer is not required to disclose the names of any ‘Party(ies) Lobbied’ for that specific Focus.
"contractual_client_name", -- Organization or individual name. The name of the individual or organization that retains the services of a Principal Lobbyist for the benefit of itself or another.
"principal_lobbyist_name", -- Organization or individual name. An individual or organization that is retained to lobby for a client, or that is lobbying on its own behalf. The Principal Lobbyist is registered in LA for a biennial (two-year) period and is required to submit Bi-Monthly reports detailing the lobbying activities that occurred during each Bi-Monthly Reporting period.
"form_submission_id", -- Form identification number to help users distinguish how many rows of data are attributable to one distinct filing.
"reporting_period", -- Each Bi-Monthly report contains information for a two-month period of lobbying activity. Bi-Monthly reporting periods are: Jan/Feb; Mar/Apr; May/June; July/Aug; Sep/Oct; and Nov/Dec.
"co_lobbyist", -- Organization or individual name. All Lobbyists who are retained by a Client on the same single retainer agreement or contract are Co-Lobbyists and must file individual Lobbying reports with the Commission.
"coalition_name", -- Identifies the name of the Coalition. A Coalition is defined as a group of otherwise-unaffiliated entities or members who pool funds or resources for the primary purpose of engaging in Lobbying Activities on behalf of the members of the Coalition.
"department_lobbied", -- Certain ‘Party Names’, when selected from the LA Database, require filers to include additional information. Unit, Division, Department Lobbied – OR – Social Media Platform associated to the specified Party Lobbied.
"type_of_lobbying_communication", -- Output options are: Direct Lobbying, Grassroots Lobbying, Both (Direct and Grassroots) or Monitoring Only. Distinguishes how attempts to influence the specific Lobbying Focus was conducted. If ‘Monitoring Only’ was selected, no Parties Lobbied will be affiliated to that specific Lobbying Focus (Lobbying Activity).
"focus_identifying_number", -- May include either an identifying number or description of the Lobbying Focus.
"coalition_contribution_expense", -- Indicates the Expense Purpose of the Contribution selected from a pre-defined list of expense purposes.
"if_pooled_funds_were_used", -- Dollar amount. Any Expenses incurred by the Member on behalf of the Unstructured Coalition, which can be from the Member’s own direct Contribution and/or from the Unstructured Coalition’s pool of funds.
"total_amount_of_the_expense", -- Any Expenses incurred by the Member on behalf of the Unstructured Coalition, which can be from the Member’s own direct Contributions and/or from the Unstructured Coalition’s pool of funds.
"contribution_amount", -- Dollar amount. Dollar amount of the Coalition contribution. Contributions are the funds or resources provided to the Coalition and are disclosed in the applicable Bi-Monthly Reporting period in which the contribution was given to the Coalition.
"expense_details_amount", -- Dollar amount. The dollar amount attributable to the itemized expense. Only applicable in the case where an Itemized Expense is paid to an organization and an expense detail is required.
"expense_details_employer", -- Name of the organization the Individual associated to the expense detail is employed by. Only applicable in the case where an Itemized Expense is paid to an organization and an expense detail is required.
"expense_details_title", -- Title of individual. Only applicable in the case where an Itemized Expense is paid to an organization and an expense detail is required.
"filing_type", -- Output options are: ‘Original’; or ‘Amendment’. Distinguishes between an original filing or an amended filing.
"expense_paid_to", -- Organization or individual name. The name of the Individual or Organization the itemized expense was paid to.
"itemized_expenses", -- Dollar amount. Any expense over $75 must be reported as an Itemized Expense, reporting the payee, nature of expense, and value.
"reimbursed_expenses", -- Dollar amount. The aggregate value of all Expenses (regardless of value of the individual Expenses) that were reimbursed by the Client.
"sub_lobbyist", -- Organization or individual name. A Sub-Lobbyist is a Lobbyist who is engaged to perform services by another Lobbyist, as part of an agreement between the main Lobbyist and a Contractual Client. There are separate retainer agreements or contracts: one between the main Lobbyist and Client, and one between the main Lobbyist and Sub Lobbyist to do a portion of the lobbying that the main Lobbyist has agreed to perform on behalf of the Client.
"beneficial_client_name", -- Organization or individual name. The specific individual or organization on whose behalf and at whose request or behest Lobbying Activity is conducted. Beneficial Clients are generally referred to as the ‘true client’ as they are directly benefiting from the lobbying activity. A Beneficial Client and Contractual Client can be, and often are, the same.
"was_an_expense_incurred_on", -- Yes or No.
"individual_lobbyist_name", -- The first and last name of the individual(s) who personally engages in Direct or Grassroots Lobbying. Generally, these individuals either work for or are designated by the Principal Lobbyist and can include individuals lobbying on their own behalf.
"expense_type", -- Individual or Organization. Indicates whether an Itemized Expense is paid to an Individual or an Organization. If an Itemized Expense is paid to an Organization on behalf of an Individual (or Individuals), additional information entitled an “Expense Detail” is required and includes: • Name of individual(s) (Multiple individuals can be added) Title and Employer; • Amount of Detailed Expense that is attributed to the individual listed.
"person_lobbied", -- Certain ‘Party Names’, when selected from the LA Database, require filers to include additional information. First and last name – OR – title of the person lobbied associated to the specified Party Lobbied.
"party_name", -- The ‘targets’ of lobbying activity. The people and/or entities Filers are attempting to influence. Parties Lobbied are selected from a drop-down list . The LA Database includes a list of ‘Party Names’ that, if selected, do not require any additional information from the Filer. This includes, but is not limited to, the names of every State Legislator and Legislative Committee (and ‘staff member’ of every such Legislator and Committee), the Governor, Lieutenant Governor, and Executive Chamber, and New York City Council Members.
"government_body", -- Every ‘Party’ listed in the LA Database contains two components: the ‘Government Body’ and the main ‘Party’ name. Every ‘Party’ name is categorized into specific Government Bodies. Below is a list of all government bodies represented in LA available for the Bi-monthly. • A communication sent to entire Senate; • A communication sent to entire Assembly; • A communication sent to entire Legislature (includes all members of the Senate and Assembly); • A communication sent to all NYC Council Members; • A communication sent to an entire specific county Legislature (Example: Albany County Legislature); • Senate Committee; • Assembly Committee; • State Agency; • NYS Assembly; • NYS Senate; • Executive Chamber; • NYS School Districts; • State and Local Public Authorities and Local Development Corporations; • NYC Council Members and County Legislators; • Industrial Development Agency; • Village; • Town; • City; • County; • Improvement/Special Districts: o County Special District; o Town Special District; o Consolidated Health District; o Fire District; o Independent Special District.
"lobbying_focus_type", -- Lobbying Activities include the Focus Type, an identifying number or description, and the targets of the lobbying for each Lobbying Activity reported. Lobbying Focuses are divided into State or Municipal Activities. State Lobbying Focuses: • State Bill; • State Executive Order; • State Funding; • State Land Use; • State Permits/Licensing; • State Procurement; • State Regulation/Rate-making/Rule; • State Resolution; • State Tribal Compact Agreement – NYS Indian Nations. Municipal Lobbying Focuses: • Municipal Bill; • Municipal Executive Order; • Municipal Funding; • Municipal Land Use; • Municipal Ordinance; • Municipal Permits/Licensing; • Municipal Procurement; • Municipal Rule/Regulation/Ratemaking/Rule; • Municipal Resolution.
"lobbying_subjects", -- Filer selected information via a drop-down menu. Description of the general subject matter areas/issues lobbied on for the specified reporting period. Examples: Education – General; Energy & Natural Resources – Waste Management; Health – Medicine/ Medicaid.
"total_contribution_amount", -- Report the Total Contribution amount which includes the Member’s Contributions and the Member’s portion of incurred Expenses related to an Unstructured Coalition
"expense_details_name", -- Name of individual. Only applicable in the case where an Itemized Expense is paid to an organization and an expense detail is required.
"expense_date", -- Date formatted mm/dd/yyyy. Date the expense occurred.
"expense_purpose", -- Pre-defined descriptions of the purpose of an itemized expense. 1. Advertising – Billboards Expenses related to purchasing and maintaining a billboard advertisement. 2. Advertising – Consulting Expenses related to hiring a consultant for the purpose of using their expertise in evaluating advertising, public relations, communications and/or marketing goals. 3. Advertising – Design Services for Media Buy Expenses related to hiring and utilizing the services of a graphic designer. 4. Advertising – Flyers/Posters Expenses related to staff time spent creating, designing, purchasing, and printing flyers and posters. 5. Advertising – Media Relations and Strategy Expenses related to the analysis of specific advertising goals for the purpose of developing a targeted media relations strategy related to Lobbying or advocacy efforts. 6. Advertising – Multi-Platform Media Buy (Print, Online, TV, Web) Expenses related to purchasing advertisements that include multiple platforms such as print, television, and online advertising. (i.e. purchasing an advertisement in the Times Union print newspaper and online web edition). 7. Advertising – Newspaper Inserts Expenses related to the design, development (including staff time), and market research related to the purchase and distribution of newspaper inserts. 8. Advertising – Print Media/Publishing fees over $500 Expenses related to printing and publishing materials for advertising that is more than $500. 9. Advertising – Social Media Platforms (Facebook, Twitter, Instagram, etc.) Expenses related to the design and execution of advertisements posted to Social Media accounts including but not limited to Facebook, Twitter, Instagram, etc. 10. Advertising – Television Expenses related to airing advertisements on television stations/channels. 11. Advocacy – Civic Engagement/Community Organizing Expenses related to staff time preparing a community event where lobbying activity will occur. 12. Advocacy – Email marketing/Email blasts Expenses related to the design, purchase of software applications/licenses, development and target audience research related to email marketing and mass emails. 13. Advocacy – Expenses Related to Hiring an Outside Firm to Handle Administrative Tasks Expenses related to the hiring of an outside entity to complete administrative tasks related to lobbying such as scheduling meetings. 14. Advocacy – Legislative Bill Tracking Expenses related to subscriptions fees or hiring law firms to track legislative bills on search engines such as: Legislative Retrieval System (LRS), and State Watch. 15. Advocacy – Legislative Research Expenses related to subscriptions to legal search engines such as West Group or Lexis, POLITICO or hiring law firms to conduct Legislative Research. 16. Advocacy – Mass Mailings/Bulk Mailing Expenses related to the purchase of supplies and staff time spent developing and preparing mass/bulk mailings. 17. Advocacy – Phone Advocacy Expenses related to organizing, paying staff/volunteers, phone equipment and service fees related to phone advocacy efforts. 18. Advocacy – Postage and/or Printing over $500 Expenses related to postage, printing and mailing items valued more than $500. 19. Advocacy – Technology (cellphones, iPad, hotspots) Expenses related to purchasing devices, data plans, and internet services/personal hotspots to be used specifically for Lobbying Activities. 20. Social Event – Booth Rentals/Meeting Space Fees Expenses related to renting a conference space and/or booth table for a Lobby Day or other social event. 21. Social Event – Buses/transportation for Lobby Day Expenses related to transportation (bus, train, rental cars) of volunteers and staff members but NOT LOBBYISTS to a Lobby Day or other social event. 22. Social Event – Catering/Meals for Lobby Day Expenses related to supplying food for attendees of a Lobby Day. 23. Social Event – Civic Engagement/Community Organizing Expenses related to staff time planning and preparing for a Lobby Day or other social event. 24. Social Event – Consulting Expenses related to hiring an event planner or organizer for Lobby Day or other social event. 25. Social Event – Equipment and A/V Rentals Expenses related to the rental and usage of technology and audio-visual equipment for a Lobby Day or other social event. 26. Social Event – Event Sponsorship Expenses related to the act of supporting an individual, organization or event financially for the purpose of creating exposure to a brand or set of issues, access to attendee information, and/or speaking opportunities. 27. Social Event – Lodging Expenses related to hotel reservations and accommodations for staff and volunteers who attend a Lobby Day or other social event. 28. Social Event – Parking fees Expenses related to fees for parking for a Lobby Day or other social event. 29. Social Event – Print fees over $500 Expenses related to printing of name badges, and informational materials/pamphlets distributed at a Lobby Day or other social event. 30. Social Event – Promotional Materials Expenses related to the creation and purchase of materials used to promote an organizations message during a Lobby Day or other social event, such as shirts, hats, mugs, pencils and stress balls. 31. Social Event – Rallies Expenses related to staff time for setting up and organizing a Rally. 32. Social Event – Reception/Banquet Expenses related to staff time organizing a reception, renting out the banquet space, and catering/food expenses. 33. Social Event – Rental Fees Expenses related to renting equipment such as microphones and tables for events such as a Lobby Day or other social event. 34. Social Event – Travel Reimbursement (Train, Airfare, Car, Hotel) Expenses related to reimbursing staff and or volunteers for travel expenses they paid for themselves. 35. Social Event – Volunteer Training Expenses related to organizing and holding training events for volunteers prior to an event, such as a Lobby Day or other social event. 36. Social Media – Media Relations and Strategy Expenses related to hiring an individual or company to establish and/or oversee an organization’s social media presence. 37. Social Media – Online Advocacy (sponsored posts) Expenses related to search engine optimization and sponsoring, such as expenses incurred to create, or promote or place content on social media. This would also include the cost of hiring an individual to place content related to the Organization on the individual’s own Social Media accounts based on the individual’s popularity on social media (i.e., an “influencer”).
"lobbying_expenses_for_non", -- Dollar amount. Non-lobbying staff salaries include Compensation paid to those professional and clerical employees who do not engage in Direct or Grassroots Lobbying Activity. Reported in the aggregate.
"expenses_less_than_75", -- Dollar amount. The aggregate value of all lobbying expenses less than $75 regardless of value of the individual expenses. Example: Ten expenses valued at $60 each would be reported as $600.
"level_of_government", -- Output options are: State, Municipal, or Both (State and Municipal). Level of Government lobbied.
"compensation", -- Dollar amount. All direct or indirect payments of salaries or other things of value provided to a Lobbyist in exchange for Lobbying or services that are otherwise in furtherance of lobbying activity during the applicable Bi-Monthly Reporting Period.
"reporting_year", -- A Principal Lobbyist submits a Lobbyist Statement of Registration for a biennial (two-year) period. Example: 2021-2022; 2023-2024. The reporting year distinguishes which year of the biennial period the specific filing is submitted for.
"was_the_expense_paid_for", -- Yes or No.
"unique_id" -- This information is used for internal and technical data identification purposes only. It is not part of the filing information reported by the filer.
FROM
"ny-gov/lobbyist-bimonthly-reports-beginning-2019-t9kf-dqbc:latest"."lobbyist_bimonthly_reports_beginning_2019"
LIMIT 100;
Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query ny-gov/lobbyist-bimonthly-reports-beginning-2019-t9kf-dqbc
with SQL in under 60 seconds.
This repository is an "external" repository. That means it's hosted elsewhere, in this case at data.ny.gov. When you queryny-gov/lobbyist-bimonthly-reports-beginning-2019-t9kf-dqbc:latest
on the DDN, we "mount" the repository using the socrata
mount handler. The mount handler proxies your SQL query to the upstream data source, translating it from SQL to the relevant language (in this case SoQL).
We also cache query responses on the DDN, but we run the DDN on multiple nodes so a CACHE_HIT
is only guaranteed for subsequent queries that land on the same node.
Query Your Local Engine
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
Read the installation docs.
Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.
It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr
, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgr
can manage the image, container and volume for you.
There are a few ways to ingest data into the local engine.
For external repositories (like this repository), the Splitgraph Engine can "mount" upstream data sources by using sgr mount
. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.
For hosted datasets, where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr clone
and sgr checkout
.
Mounting Data
This repository is an external repository. It's not hosted by Splitgraph. It is hosted by data.ny.gov, and Splitgraph indexes it. This means it is not an actual Splitgraph image, so you cannot use sgr clone
to get the data. Instead, you can use the socrata
adapter with the sgr mount
command. Then, if you want, you can import the data and turn it into a Splitgraph image that others can clone.
First, install Splitgraph if you haven't already.
Mount the table with sgr mount
sgr mount socrata \
"ny-gov/lobbyist-bimonthly-reports-beginning-2019-t9kf-dqbc" \
--handler-options '{
"domain": "data.ny.gov",
"tables": {
"lobbyist_bimonthly_reports_beginning_2019": "t9kf-dqbc"
}
}'
That's it! Now you can query the data in the mounted table like any other Postgres table.
Query the data with your existing tools
Once you've loaded the data into your local Splitgraph engine, you can query it with any of your existing tools. As far as they're concerned, ny-gov/lobbyist-bimonthly-reports-beginning-2019-t9kf-dqbc
is just another Postgres schema.