cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk
Loading...

Query the Data Delivery Network

Query the DDN

The easiest way to query any data on Splitgraph is via the "Data Delivery Network" (DDN). The DDN is a single endpoint that speaks the PostgreSQL wire protocol. Any Splitgraph user can connect to it at data.splitgraph.com:5432 and query any version of over 40,000 datasets that are hosted or proxied by Splitgraph.

For example, you can query the nonbudgeted_special_revenue_and_trusts table in this repository, by referencing it like:

"cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk:latest"."nonbudgeted_special_revenue_and_trusts"

or in a full query, like:

SELECT
    ":id", -- Socrata column ID
    "total_available_for_spending", -- The amount available in an account for expenditure in the fiscal year. Equal to Total Enacted Budget Appropriation + Retained Revenue Collected + Other Spending Authorization minus Planned Savings (9C Spending Cuts).
    "fiscal_year", -- The period during which a fiscal year's budget may be expended.  This period for any fiscal year is from July 1st of the preceding calendar year through June 30th of the current calendar year. It is the period within which all goods and/or services must be received to be payable with those fiscal year funds.  For example, fiscal year 2018 began on July 1, 2017 and ended on June 30, 2018.
    "balance_forward", -- Balance Forward/Prior Appropriation Continued (PAC). If authorized by the Legislature, appropriation accounts may carry forward unexpended balances into the following fiscal year. This “Prior Appropriation Continued” amount is reflected in the state accounting system as a “Balance Forward” in the current fiscal year and an equal “Beginning Balance” in the following fiscal year.
    "transfer_in", -- A department may contract with other state departments to administer spending authorized for a particular account, or else transfer reserve funds to other accounts (for example, when central reserves are established to fund collective bargaining agreements).  When such transactions occur, funds are shown as a “transfer in” to the receiving department, and a “transfer out” from the sending department.
    "revenue_collected",
    "beginning_balance", -- If authorized by the Legislature, spending accounts may carry forward unexpended balances from the prior fiscal year. The balances are reflected in the state accounting system as a “Beginning Balance” or a “Prior Appropriation Continued”. These amounts are identical to the “Balance Forward” in prior fiscal year for the same account.
    "appropriation_account_name", -- The name of an account corresponding to an eight-digit appropriation/account number.
    "cabinet_secretariat", -- A Secretariat (also known as Cabinet) is defined as an executive office that reports to the Governor. Note:  In CTHRU, Cabinet/Secretariat also includes offices that do not report to the Governor, such as constitutional offices (e.g., Office of the Treasurer), and independent agencies (e.g., Office of the Comptroller).  Executive Departments usually report to Secretariats.  For example, the Executive Office of Health and Human Services (EHS) is a Secretariat and the Department of Public Health is a Department within that Secretariat.
    "fund_name",
    "unexpended",
    "total_expenses", -- The total amount expended in the fiscal year; for the current active fiscal year (but not for closed fiscal years), this includes the amount of expenses accrued but not yet paid out as cash to vendors. This gap may occur because there is often a grace period (of up to 45 days) between when an expense is incurred (e.g., when it is billed) and when the payment is made to the vendor.
    "transfer_out", -- See definition for “Transfer In”.
    "appropriation_account_number", -- A unique eight-digit number, as identified in the General Appropriation Act (GAA) or subsequent budget bills, or established administratively, with the legislative authorization for a department to spend funds to fulfill the department’s legislative mission, including specifically identified purposes.
    "department_code", -- A Department is an entity of state government with a specific mission, established by the legislature, which may either report to cabinet-level units of government, known as executive offices or secretariats, or be independent divisions or departments.  Each department has an assigned three-letter code in the accounting system; for example, the Office of the Comptroller is CTR.
    "fund_number", -- Funds are fiscal and accounting entities, almost always established in general law, used to deposit revenues that are dedicated to specific purposes and finance expenditures for those purposes. For instance, income tax revenue is deposited in the Commonwealth’s General Fund to finance general government expenditures, while motor fuel tax collections and vehicle registration fees are deposited in the Commonwealth Transportation Fund, and are used to fund transportation-related spending, including the Massachusetts Department of Transportation (MassDOT) and Commonwealth debt service on bonds used to finance road and bridge construction. Each fund is designated in the state accounting system with a four-digit code. For example, the General Fund is designated with the digits 0010 and the Commonwealth Transportation Fund is designated with the digits 0103. The funds included in this section of CTHRU are only those subject to the Commonwealth’s annual appropriations process (“Budgeted Funds”). 
    "department_name" -- A Department is an entity of state government with a specific mission, established by the legislature, which may either report to cabinet-level units of government, known as executive offices or secretariats, or be independent divisions or departments.  Each department has an assigned three-letter code in the accounting system; for example, the Office of the Comptroller is CTR.
FROM
    "cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk:latest"."nonbudgeted_special_revenue_and_trusts"
LIMIT 100;

Connecting to the DDN is easy. All you need is an existing SQL client that can connect to Postgres. As long as you have a SQL client ready, you'll be able to query cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk with SQL in under 60 seconds.

Query Your Local Engine

Install Splitgraph Locally
bash -c "$(curl -sL https://github.com/splitgraph/splitgraph/releases/latest/download/install.sh)"
 

Read the installation docs.

Splitgraph Cloud is built around Splitgraph Core (GitHub), which includes a local Splitgraph Engine packaged as a Docker image. Splitgraph Cloud is basically a scaled-up version of that local Engine. When you query the Data Delivery Network or the REST API, we mount the relevant datasets in an Engine on our servers and execute your query on it.

It's possible to run this engine locally. You'll need a Mac, Windows or Linux system to install sgr, and a Docker installation to run the engine. You don't need to know how to actually use Docker; sgrcan manage the image, container and volume for you.

There are a few ways to ingest data into the local engine.

For external repositories, the Splitgraph Engine can "mount" upstream data sources by using sgr mount. This feature is built around Postgres Foreign Data Wrappers (FDW). You can write custom "mount handlers" for any upstream data source. For an example, we blogged about making a custom mount handler for HackerNews stories.

For hosted datasets (like this repository), where the author has pushed Splitgraph Images to the repository, you can "clone" and/or "checkout" the data using sgr cloneand sgr checkout.

Cloning Data

Because cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk:latest is a Splitgraph Image, you can clone the data from Spltgraph Cloud to your local engine, where you can query it like any other Postgres database, using any of your existing tools.

First, install Splitgraph if you haven't already.

Clone the metadata with sgr clone

This will be quick, and does not download the actual data.

sgr clone cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk

Checkout the data

Once you've cloned the data, you need to "checkout" the tag that you want. For example, to checkout the latest tag:

sgr checkout cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk:latest

This will download all the objects for the latest tag of cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk and load them into the Splitgraph Engine. Depending on your connection speed and the size of the data, you will need to wait for the checkout to complete. Once it's complete, you will be able to query the data like you would any other Postgres database.

Alternatively, use "layered checkout" to avoid downloading all the data

The data in cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk:latest is 0 bytes. If this is too big to download all at once, or perhaps you only need to query a subset of it, you can use a layered checkout.:

sgr checkout --layered cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk:latest

This will not download all the data, but it will create a schema comprised of foreign tables, that you can query as you would any other data. Splitgraph will lazily download the required objects as you query the data. In some cases, this might be faster or more efficient than a regular checkout.

Read the layered querying documentation to learn about when and why you might want to use layered queries.

Query the data with your existing tools

Once you've loaded the data into your local Splitgraph Engine, you can query it with any of your existing tools. As far as they're concerned, cthru-data-socrata/nonbudgeted-special-revenue-and-trusts-uaqe-2tpk is just another Postgres schema.

Related Documentation:

Loading...