
Like almost everyone these days, Splitgraph has a pretty elaborate CI/CD system.
It handles everything from building
our [Docker images](https://www.splitgraph.com/blog/makefile)
to [running tests](https://www.splitgraph.com/blog/integration-tests)
to [staging/production deployments](https://www.splitgraph.com/blog/docker-compose-production)
. To achieve this, we use the extensive GitLab CI/CD tool. Through a combination
of a base `.gitlab-ci.yml` configuration file coupled with arbitrary scripts, it
lets us perform a variety of potential DevOps actions.

One such example that recently attracted our attention was something GitLab
calls [review apps](https://docs.gitlab.com/ee/ci/review_apps/). The concept is
pretty straight-forward: a CI/CD job that deploys code changes from a branch,
integrated with GitLab's UI. This deployment could be a work-in-progress or a
proof-of-concept. GitLab review apps let developers, designers or product owners
preview and iterate on their work faster.

We decided to make use of this functionality in our own workflow. We call it
**preview environments**, as the name is a better fit for our use case (more on
that below).

## Splitgraph infrastructure

Our proprietary product, Splitgraph Cloud, can be deployed to private as well as
public clouds. In the latter case, we use a set of multi-cloud Terraform
templates that cover the public cloud triumvirate of Azure, AWS and GCP.

Terraform allows us to write maintainable and re-usable infrastructure-as-code
in a cloud-agnostic way. Thus our preview environments not only have to deal
with application-level deployments, but also with provisioning the cloud
infrastructure during the initial setup (as well as cleaning up on tear-down).
This makes them veritable environments, in parallel to staging and production.

Terraform also allows us to persist the state of all our existing preview
environments between CI/CD job runs. For this, it uses the remote backend
configuration (which in our case is set to a GCS bucket).

## Setup

Here's an example of the preview environment job spec in the `.gitlab-ci.yml`
configuration file:

```yaml
deploy_preview:
  image: registry.gitlab.com/splitgraph/splitgraph-cloud/cd-environment:development
  stage: preview
  rules:
  - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
    when: manual
    allow_failure: true
  environment:
    name: preview/$CI_COMMIT_REF_SLUG
    url: https://www.$CI_COMMIT_REF_SLUG.splitgraph.io
    deployment_tier: development
    auto_stop_in: 3 days
    on_stop: terminate_preview
  script:
  - source "$CI_PROJECT_DIR"/.ci/envs/preview.env
  - $CI_PROJECT_DIR/.ci/preview_env.sh deploy
```

A couple of noteworthy points:

- We use a custom "Docker-in-Docker" image with Terraform to manage the
  infrastructure. We also use `yq` and our open
  source [sgr client](https://github.com/splitgraph/sgr) to seed all our
  preview environments with
  a [whole lot of data](https://www.splitgraph.com/blog/40k-sql-datasets) upon
  the initial deploy.
- The job can only run if the previous build and test stage succeeded.
- The job can be run with or without an open Merge Request, and it needs to be
  manually triggered. It cannot run on our main branch.
- The name and domain for the preview environment are generated based on the
  URL-friendly `CI_COMMIT_REF_SLUG` [predefined variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)
  .

The actual provisioning and deployment is handled through the `preview_env.sh`
script. Before that, source a `.env` file to populate some environment
variables. These include Terraform variables, as well as some variables
controlling app-level settings. One more thing we had to resort to here is trim
the name of the preview environment for branches with long names. This is to
avoid the length constraints put on us by the cloud providers or Let's Encrypt,
since we use this name for various resources too.

## Deployment

When provisioning/deploying, in the `preview_env.sh` script we first check
whether a Terraform workspace for the given branch already exists:

```bash
workspace_exists=0
terraform workspace select "$CI_COMMIT_REF_SLUG" && workspace_exists=1
```

**If the workspace doesn't exist**, it means that the environment has not been
provisioned, so the job then executes:

```bash
terraform workspace new "$CI_COMMIT_REF_SLUG"
terraform apply -auto-approve

source ./first_deploy.sh
```

Terraform then creates the required resources. This includes the VPC, firewalls,
subnetwork, virtual machines, disks, as well as Cloudflare domain records and
Gitlab project variables (for per-environment secrets, e.g. SSH keys).

In case of preview environments we always deploy a 2-node cluster. The main VM
hosts our DBs and the backend services, while the lighter, auxiliary VM, hosts
analytics and monitoring services.

The script `first_deploy.sh` subsequently takes care of a number of other
important tasks:

- generating a deploy artifact
- uploading and executing it
- spinning up all our containerized services in the cluster reliably (with the
  help of Nomad)
- registering admin/test users
- renewing certificates using LetsEncrypt

**If the workspace does exist**, then the job has been triggered on a
pre-existing preview environment. This means we only need to upgrade our app to
pick up the latest changes for the given branch:

```bash
"$CI_PROJECT_DIR"/.ci/deploy/deploy.sh
```

The complete initial setup takes about 15 minutes. A redeploy (if all services
are updated) takes about 10.

## Output

Finally, in either case we use chatops to distribute/persist some
environment-specific user details (e.g. URL, user credentials, etc.) in a
Markdown-formatted message. If a merge request is open, the job posts a note on
it with the details. If not, the job pushes them directly to a corresponding
Mattermost channel through a hook.

## Termination

There is one more job, called `terminate_preview`, which is used for
de-provisioning the environment. Besides being executable manually, note that it
is specified as `environment:on_stop` reference for the `deploy_preview` job.
This means that once the `environment:auto_stop_in` period since the last deploy
job ran expires, the job gets triggered automatically by GitLab.

It sources the same `preview.env` and then
executes `$CI_PROJECT_DIR/.ci/preview_env.sh terminate`:

```bash
terraform destroy -auto-approve
terraform workspace select default
terraform workspace delete "$CI_COMMIT_REF_SLUG"
```

thereby ensuring that all resources are cleaned up, including the Terraform
workspace.

## Conclusion

In summary, we utilize preview environments for smoke testing and rapid
iteration, in an on-demand, fully fledged production-grade Splitgraph instance.
The infrastructure provisioning and application deployment done in this case
exercise the same code that we run for our customer environments and production
releases. For this reason we view preview environments as an extension of
integration tests and, in fact, are planning to start using them for automated
end-to-end tests.


We talk about how we use GitLab's review apps functionality to preview and test Splitgraph Cloud deployments.

Preview Environments: Spinning up temporary Splitgraph instances from any commit


At the time of writing, the [Splitgraph registry](/explore) runs on a single 30 EUR/month [Scaleway](https://scaleway.com/) instance. Despite that, we index and let you query [over 40,000 public datasets](/blog/40k-sql-datasets) with simple SQL.

This series of articles discusses our [infrastructure](/blog/topic/infrastructure) and how we stay lean while still providing a lot of value to data scientists and engineers.

To run Splitgraph in production, we use Docker Compose. We found it to be a surprisingly effective solution. This article is a collection of tips and tricks for using Docker Compose that we discovered. We'll talk about:

  * structuring a Compose dev-prod configuration using [override mixins](https://docs.docker.com/compose/extends)
  * an automated deploy workflow
  * configuration generation, injection and live reloading
  * how we monitor our services in production

## Why Compose?

We are eventually interested in moving to Kubernetes or Nomad (we're even writing a script that automatically converts Compose files to Nomad jobs!). But, these solutions felt like overkill for our use case.

Besides, we relied on Docker Compose a lot when building Splitgraph. Currently, we run [integration tests in CI](/blog/integration-tests) using it. We wanted to match the dev, CI and production workflow as closely as possible.

## Compose configs and mixins

Docker Compose allows you to combine multiple Compose configurations into one at runtime. This feature is called ["overrides"](https://docs.docker.com/compose/extends).

Here's a sample Compose definition for a "prod" service:

```yaml
version: "3.4"
services:
  registry-db:
    image: ${DOCKER_REPO}/registry-db:${DOCKER_TAG}
    expose:
      - 5431
    env_file:
      - ${CONFIG_HOME}/config/registry-db.env
    restart: on-failure
    user: "postgres:pgsockets"
    volumes:
      - registry_dbdata:/var/lib/postgresql/data/pgdata
      - pg_pgb_sockets:/var/pgsockets
      - ${CONFIG_HOME}/config/registry-db.sgadmin.password:/registry-db.sgadmin.password:cached
...
```

Our production configuration normally includes:

  * The actual Docker image
  * A [policy](https://docs.docker.com/compose/compose-file/#restart) to automatically restart containers that exited with an error
  * Bind mounts of configuration files
  * [Environment files](https://docs.docker.com/compose/compose-file/#env_file)
  * Mounts of data volumes
  * Docker healthcheck definitions

The development Compose file is a mixin on top of the production configuration. For this service, it's:

```yaml
version: "3.4"
services:
  registry-db:
    ports:
      - "0.0.0.0:5430:5431"
    volumes:
      - ./src/py/splitgraph/splitgraph:/splitgraph/splitgraph
      - ./src/luamods/main:/home/postgres/luamods
      - ./src/py/sgr_admin/sgr_admin/embedded/dbs:/home/postgres/pymods
      - ./components/registry/registry-db/etc/postgresql:/etc/postgresql
    environment:
      - SG_LOGLEVEL=DEBUG
```

This can include:

  * Bind mounts for source code for scripted languages
  * Flags for verbose logging
  * Using a more heavyweight Docker image with dev tooling or hot reloads
  * Publishing ports on the host to aid debugging

This lets us avoid repetition in our configuration. We also use this idea to split up different aspects of the stack into separate Compose files. We call these "service groups".

We have runscripts that wrap Compose overrides to give the developer a single script. They can use it to run Compose commands on a service group. The script can inject the "dev" configuration if needed:

```
#!/usr/bin/env bash -e

if [[ "$1" == "--dev" ]] ; then
  DEV=1
  shift
fi

export CONFIG_HOME=${CONFIG_HOME-"$PWD"}

exec docker-compose -f docker-compose.prod.yml \\
	$( [[ -n $DEV ]] && echo "-f docker-compose.dev.yml" ) \\
	-f docker-compose.dbs.build.yml \\
	-f docker-compose.dbs.prod.yml \\
	$( [[ -n $DEV ]] && echo "-f docker-compose.dbs.dev.yml" ) $@
```

## Deployment workflow

We deploy Splitgraph using GitLab CI. The deploy process runs in several stages.

### Building the production bundle

First, we use our Makefile (we wrote more about how we use Makefiles in [a previous blog post](/blog/makefile)) to build a "deploy bundle". This bundle is self-contained and includes only the files needed to run Splitgraph in production:

  * Production Docker Compose configs
  * Runscripts that combine those configs into "service groups"
  * Other useful scripts for production: installation, backups, configuration management, helper script to load into the admin shell
  * Configuration templates

We store the actual tag (commit hash) that we're deploying in a `.env` file in the bundle that every script sources.

### Deploy home directory

On the server, we have a "deploy home" directory that contains service configuration files. It also contains a `config.json` file that includes:

  * configuration flags for the current deploy
  * secrets
  * third-party API keys

The CI script copies the bundle into this directory and extracts it. We keep copies of previous deploy bundles too, each one in a timestamped directory. This means we can roll back to an earlier version in case of a failed deploy by running that bundle's installation script.

### Pulling and tagging the containers

The installation script first pulls all required containers from the GitLab CI registry. At this point, it uses the deploy's commit hash as the tag.

The script then retags these containers with a `:production` tag. This is to avoid Compose reloading all services on every deploy. If an image's tag (not only the SHA hash) has changed, Compose will treat the service's configuration as changed. Keeping the tag fixed prevents that.

### Configuration generation

The script then runs the "configurator". It's a container with a Python application that uses Jinja. It uses the bundle's config templates and the `config.json` file to regenerate configuration for all services that we run.

We wrote more about the [configurator](/blog/integration-tests) in a previous blog post.

We bind mount configuration into containers. We prefer this to baking it into the image. This lets us make emergency hotfixes without having to go through the CI pipeline.

### Database migrations

Next, the installation script runs schema migrations. We check all our migrations into version control and run integration tests on them in CI.

These migrations form part of the "admin" container. We also use this container to run other Splitgraph instance management tasks.

Finally, we implement a lot of Splitgraph functionality as PostgreSQL functions in languages like `PL/Python` or `PL/Lua`. We package those up as PostgreSQL extensions which we also install at migration time.

### Reloading services

We have several service groups that we deploy one after another. We use `./service-runscript.sh up -d` to upgrade most of them. This means that Docker Compose only recreates containers for services with changed specifications. You can read more about this behavior in the [Compose documentation](https://docs.docker.com/compose/reference/up/).

This is not a zero downtime deploy, but we try to minimize excess container restarts by pinning third-party container hashes. For example, instead of `haproxy:latest`, we use `haproxy@sha256:e6f9faf0c2a0cf2d2d5a53307351fa896d...`.

In some cases, we run many copies of the same service group for high availability. This is the case for our [REST API](/docs/query/postgrest). There, the script waits until the replica is healthy before upgrading the next one:

```bash
#!/bin/bash -e

# Wait for a container to become healthy.

CONTAINER=$1


timeout=120
while true; do
  status=$(docker inspect -f '{{.State.Health.Status}}' "$CONTAINER")
  if [ "$status" != "starting" ]; then
    break
  fi
  if (( timeout < 0 )); then
    echo "Timed out waiting for $CONTAINER to become healthy"
    exit 1
  fi
  sleep 2
  timeout=$(( timeout - 2))
  echo "Waiting for $CONTAINER to become healthy, $timeout s.."
done

if [ "$status" == "unhealthy" ]; then
  echo "$CONTAINER is unhealthy!"
  exit 1
fi

echo "$CONTAINER is healthy."
```

### Reloading configuration

Most containers don't get recreated or restarted on deploy, which helps us minimize downtime. But what do we do if we need to reload a service's configuration without restarting it?

The answer is simple. Lots of services like HAProxy, Prometheus, PgBouncer or NGINX support configuration reloads via a `SIGHUP` signal. You can send a UNIX signal to a container with Docker:

```shell-session
$ docker kill --signal SIGHUP container_name
```

All we need to do to apply new configuration is send `SIGHUP` to relevant containers after a deploy. If the configuration has errors, these services will silently switch back to the old one.

There's a caveat here with bind mounts. You have to make sure you're bind mounting **the directory** with the configuration file rather than the actual file. This is because of how Docker bind mounts interact with Linux inodes. More on [Docker GitHub](https://github.com/moby/moby/issues/15793#issuecomment-135411504).

### Healthchecks and monitoring

As the final step, the deploy runs `docker-compose ps` to make sure all containers that have healthchecks pass them. If there are unhealthy containers, the deploy script raises an alert in our team chat.

We also use [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) to scrape various container statistics, plot them and alert on them. In particular, we have alerting rules on:

  * HAProxy backend availability
  * High HTTP response latencies
  * High node resource usage
  * High rate of non-2xx responses
  * Failed Airflow jobs

We check our Prometheus configs into version control. We do the same with Grafana dashboards. We load those from configuration files using Grafana's [provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) feature.

## Conclusion

In this article, we discussed how we use Docker Compose to run Splitgraph in production, as well as how to use it for automated deploys.

This is the final article in the series that talks about our [build, test and deploy infrastructure](/blog/topic/infrastructure). If you're interested in learning more about Splitgraph, you can check our [frequently asked questions](/docs/getting-started/frequently-asked-questions) section, follow our [quick start guide](/docs/sgr-advanced/getting-started/five-minute-demo) or visit our [website](https://www.splitgraph.com).


We finish our overview of Splitgraph's infrastructure by talking about why and how we use Docker Compose to run the Splitgraph registry in production.

Splitgraph infrastructure, part 3: Using Docker Compose in production


Serverless and edge computing have allowed application developers to bring their applications closer to the end user.

Instead of maintaining a group of servers in a single location, developers can let companies like Cloudflare, Fastly or Akamai handle their content delivery.

With [Function as a service](https://en.wikipedia.org/wiki/Function_as_a_service), companies pay for what they use. They can avoid having to provision a server that stays idle most of the time.

In this article, we want to talk about these trends and how we can apply them to databases. We'll also talk about our decision to make the API for our Splitgraph registry work over a public SQL connection. We'll use this experience to propose the idea of a **data delivery network**.

## How content delivery networks work

Content delivery networks provide a straightforward way to scale a read-only HTTP layer. They use existing HTTP cache semantics like the Cache-Control header. The developer only needs to point their DNS records to use the CDN's nameservers. The CDN handles everything else for them. It has points of presence around the world and peering agreements with other ISPs. It can selectively cache data, handle DDoS protection and offer extra services on top.

The value proposition behind edge computing is simple. For a lot of companies, scaling compute is not their core competency. They can spend time and money provisioning servers and configuring something like Varnish. Or, they can use services that will handle scaling and caching for them.

However, applications still need to run SQL queries. A CDN doesn't completely help an application that performs client-side rendering. The database becomes the next performance bottleneck in scaling a service.

There are many ways to scale a database, for example, replication or sharding. But again, this requires specialist knowledge about a database that is easy to get wrong.

## Why do you need a backend, anyway?

Let's change gears and consider a classic Web application. It consists of the frontend, the backend and the database.

There are several purposes that a backend serves:

  * **Business logic**. The backend converts higher level API calls into low-level SQL queries. It prepares data for presentation and writes it back when needed.
  * **Authorization**. The backend acts as a security barrier, validating API calls. This is necessary because the frontend is running on the user's machine: the client is not trusted.
  * **Multiplexing**. A database connection has a larger overhead than an HTTP connection. A backend can shunt hundreds of simultaneous clients over to a few database connections.

### Alternatives to CRUD services

One big issue with writing RESTful backends is that there's a lot of boilerplate. The programmer has to write very similar code to handle every action. They have to care of validation, typechecking and handling edge cases.

Libraries like [PostgREST](https://postgrest.org/en/latest/) and [PostGraphile](https://www.graphile.org/postgraphile/) have helped developers decrease iteration times. They introspect database schemas and generate REST and GraphQL APIs for them.

PostgREST and PostGraphile perform their authorization using database methods like [row level security](https://postgrest.org/en/v7.0.0/auth.html). In essence, they decrease the size of the ["trusted computing base"](https://en.wikipedia.org/wiki/Trusted_computing_base).

Often, services that use these kinds of tools don't even have a separate backend. Client side code can call the automatically generated GraphQL/REST API directly.

### Splitgraph's architecture

The database can perform a lot of work that the backend does more quickly and more efficiently.

We use this idea in the API for the Splitgraph registry that allows you to push and pull [data images](/docs/concepts/images). A [Splitgraph client](/docs/sgr-advanced/architecture/sgr-client) can access it over a normal PostgreSQL connection to `postgresql://data.splitgraph.com:5432/sgregistry`.

Our API implements all **business logic** as PostgreSQL functions. This has a few immediate advantages:

  * Lets PostgreSQL precompile them
  * Avoids an extra hop from the backend, decreasing latency
  * Makes basic validation and type checking trivial. It's not possible to call a function with a wrong number of arguments or different types.

For more complex logic, we wrote it in higher-level languages like [PL/Python](https://www.postgresql.org/docs/current/plpython.html) or PL/Lua. PostgreSQL even supports languages like C or JavaScript.

We solved the problem of **multiplexing and authorization** by adding [PgBouncer](https://www.pgbouncer.org/), a connection pooler, in front of our database. Our fork of PgBouncer injects a signed cookie into every transaction as a local variable. Downstream procedures validate this cookie for authentication and authorization. This lets us decouple PostgreSQL users from application users. Multiple inbound sessions can use the same connection.

Our fork of PgBouncer even inspects queries on the fly and filters them. This makes sure that the client can only call Splitgraph SQL API functions.

For the web frontend at [www.splitgraph.com](https://www.splitgraph.com/), we use PostGraphile. Besides not having to write an extra API server, it lets us generate TypeScript client code.

## Data delivery network

We can apply these ideas and concepts to the problem of building a **"data delivery network"**. Such a network would completely abstract away all the issues around making sure that data is available at the edge. It can also provide plenty of other useful services.

Here's a quick sketch of what a DDN's administration interface would look like:

![](https://raw.githubusercontent.com/splitgraph/splitgraph.com/master/content/blog/images/20200713-data-delivery-network/admin-panel.png)

To use a DDN, a developer would create a read-only account on their database and give the DDN the credentials. It will then make a few services available:

The DDN will create an **SQL endpoint**. Any existing SQL client or application will be able to connect to it and run queries.

Besides SQL, the DDN will also be able to introspect the origin database and provide **REST and GraphQL API endpoints**. A client, running in the user's web browser, can use these endpoints instead of a backend server.

The DDN will be able to **cache** read-only SQL transactions with configurable policies. It will only forward the query to the origin database if there's a cache miss or expiry.

The client code doesn't need to be trusted. The DDN can intercept and **firewall** queries or **rate limit** them. To simplify migrations, the DDN can **rewrite** queries on the fly before forwarding them.

The DDN's work doesn't need to stop at handling queries. It can also manage **data imports and exports**. For example, it can make data from other services available to clients. Or, it can export data to Google Sheets or a data warehouse.

In the case of Splitgraph, we envision you being able to even run a `JOIN` across a public Splitgraph image and your private data.

## Conclusion

The database is the next frontier of serverless and edge computing. One of Splitgraph's goals is building a data delivery network to handle these problems.

If you're interested in learning more about Splitgraph, you can check our [frequently asked questions](/docs/getting-started/frequently-asked-questions) section, follow our [quick start guide](/docs/sgr-advanced/getting-started/five-minute-demo) or visit our [website](https://www.splitgraph.com).


We discuss the trends of serverless and edge computing, talk about why our SQL server is open to the public and propose the idea of a data delivery network.

Throwing away the backend: Towards a data delivery network


The [Splitgraph registry](https://www.splitgraph.com/explore) is a complex piece of software that consists of about 60 containers that use a dozen programming languages.

Besides application code, our business logic also lives in multiple configuration files (Varnish, NGINX, HAProxy...) and SQL functions.

We would like to have confidence that we exercise our code while still maintaining the freedom to make big architectural changes. This is why we have taken a liking to loose end-to-end integration tests.

In this series of articles, we're discussing Splitgraph's internal build, test and deploy infrastructure. Today, we'll be talking about tests.

## GitLab CI

We use GitLab CI for our builds and tests. We talked about builds in the [previous article](/blog/makefile).

[GitLab CI](https://docs.gitlab.com/ee/ci/) allows us to provide a Docker image to run CI in. However, we want to run our integration tests _in_ Docker as well. So, we use a Docker-in-Docker image as the base for our CI image. There aren't many dependencies on top of that:

```dockerfile
FROM docker:stable-dind

ENV GLIBC_VERSION="2.27-r0"

# use docker-compose that supports Buildkit builds
ENV DOCKER_COMPOSE_VERSION="1.25.0"

RUN apk add --no-cache \\
    # bash for buildscripts, gettext for envsubst, make to build the thing,
    # openssh-client to deploy it
    curl bash gettext make openssh-client git jq \\
    # libc -- required for docker-compose
    && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub \\
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk \\
    && apk add glibc-${GLIBC_VERSION}.apk \\
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-bin-${GLIBC_VERSION}.apk \\
    && apk add glibc-bin-${GLIBC_VERSION}.apk \\
    # docker-compose itself
    && wget https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` -O docker-compose \\
    && chmod +x docker-compose \\
    && mv docker-compose /usr/local/bin/docker-compose

RUN wget -q -O mkcert https://github.com/FiloSottile/mkcert/releases/download/v1.4.0/mkcert-v1.4.0-linux-amd64 \\
    && chmod +x mkcert \\
    && mv mkcert /usr/bin/mkcert

RUN apk add libcap

ENV DOCKER_TLS_CERTDIR=

COPY builder-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/builder-entrypoint.sh"]
CMD []
```

The only dependencies for running the full test suite are:

  * Docker and Docker Compose
  * Make
  * mkcert to generate self-signed SSL certificates

Everything else in this image (gettext, openssh-client, jq) is used for deployment.

## Overview of tests

With these dependencies, anyone run the full test suite with a single command, `make test`. We also have Make test targets for specific parts of the stack. Each of these targets is self contained:

  * builds the required images
  * starts them with Docker Compose
  * runs the test suite in a separate container

Most of our tests are written in Python and use the [Pytest](https://docs.pytest.org/en/latest/) framework. We have some unit tests for our Lua code too where we use [Busted](https://olivinelabs.com/busted/).

### End-to-end tests

A lot of our business logic is spread around multiple configuration files, third-party components or SQL code. We even consider the way we connect components together via Docker Compose an important part of our infrastructure that we want to automatically test.

As an example, we have a [REST API](https://www.splitgraph.com/docs/query/postgrest) that we make available for every dataset pushed to Splitgraph. It consists of multiple components:

  * An instance of [Openresty](https://openresty.org/en/) that intercepts HTTP requests and sends a request to a...
  * ...component that mounts the dataset into a temporary schema using [layered querying](https://www.splitgraph.com/docs/sgr-advanced/large-datasets/layered-querying)...
  * Openresty rewrites the HTTP request to be in terms of this temporary schema and forwards it to...
  * ...custom version of [PostgREST](https://postgrest.org/en/latest/) that can perform introspection and schema discovery on the fly.

The only way to be sure that this kind of setup works correctly is end-to-end tests.

We also have a suite of "acceptance" tests that run the public Splitgraph client against the registry. They exercise the whole Splitgraph Cloud stack like a normal user would:

  * registering on the registry
  * pulling a data image
  * making changes to it
  * pushing it back
  * querying it via our REST API

### Pytest

We use Pytest to exercise even the components that aren't written in Python. This is because of its versatility and support for [fixtures](https://docs.pytest.org/en/stable/fixture.html).

For example, we have fixtures that set up some users and some repositories. These fixtures run against the dev registry. They're reusable and make it easy for anyone to exercise their new feature.

While we sometimes use mocks, we're not big fans of them. To us, the best way to test something that uses a database connection is to actually spin up a database and run code against it. This is especially true because a lot of our code is inside SQL functions.

### Dev-prod parity

Our integration tests run against a stack that is _very_ similar to what we run in production. We even use the same Docker Compose files and run the same configuration generation.

This eliminates a whole class of problems where production containers have slightly different configuration or are mis-wired.

We try to keep development and production containers the same. In particular, we deliberately build production Python containers to use editable installs for the Python package. In development, we bind mount Python code into the container. This lets us iterate faster while still maintaining dev-prod parity.

## Configuration generation

The kinds of tests we want to run can touch many containers, including HAProxy, our authentication gateway or our registry. Running these tests requires a fundamentally different approach to configuration generation.

### Jinja

We use [Jinja templates](https://jinja.palletsprojects.com/en/2.11.x/) to build configuration files for all services. This includes third-party ones like HAProxy or Varnish.

We made a small change to Jinja's templating to allow for secret generation. Here's an example template for an `.sgconfig` file that we use for Splitgraph instances powering our [REST API](https://www.splitgraph.com/docs/query/postgrest):

```ini
[defaults]
...
SG_ENGINE_DB_NAME=splitgraph
SG_ENGINE_USER=sgr
SG_ENGINE_PWD={{ password.engine.sgr }}
SG_ENGINE_ADMIN_USER=sgr
SG_ENGINE_ADMIN_PWD={{ password.engine.sgr }}
SG_ENGINE_POSTGRES_DB_NAME=postgres
SG_META_SCHEMA=splitgraph_meta

[remote: registry]
SG_ENGINE_HOST=haproxy
SG_ENGINE_PORT=5431
SG_IS_REGISTRY=true
SG_ENGINE_USER=registry_api_user
SG_ENGINE_PWD={{ password.registry.api_user }}
```

When Jinja sees a variable like `{{ password.engine.sgr }}`, it first checks a special `config.json` file to see if it exists in there. If not, it generates a random password and writes it to that file. All other configuration files that reference this variable get the same password.

We use this `config.json` file to store other configuration relevant to the deployment. For example, a developer running the registry on their own machine can disable the monitoring stack with a single flag. This removes all monitoring backends from the HAProxy config.

The disadvantage of this approach is that all secrets are in a single file. However, having this kind of setup in place means that it's easy for us to add an extra database role or an extra service. We can rotate credentials easily when needed. We're also free to secure interactions even between internal services.

### CI and production

We use this workflow in CI as well. The configurator generates all configuration files from these templates. We use random passwords and throw them away at the end of the CI run. Because we run end-to-end tests, this can quickly identify issues with misconfigured roles or missing credentials.

In production, we bind mount these configuration files into the containers. This allows us to avoid rebuilds for hotfixes to configuration. For example, if we need to change some HAProxy configuration on the fly, we can do that, then send a `SIGHUP` signal to the container. This will reload the configuration file without downtime.

## Conclusion

In this article, we talked about how we approach testing at Splitgraph. Instead of relying on mocks, we try to test the whole stack as it would run in production, using powerful tools like Docker Compose.

In the next article in this series, we'll talk about how we run the Splitgraph registry in production and how we deploy and monitor it.

If you're interested in learning more about Splitgraph, you can check our [frequently asked questions](/docs/getting-started/frequently-asked-questions) section, follow our [quick start guide](/docs/sgr-advanced/getting-started/five-minute-demo) or visit our [website](https://www.splitgraph.com).


We continue our overview of Splitgraph's internal build infrastructure by talking about how we run end-to-end integration tests. We also discuss using Jinja to generate configuration and inject secrets into our components.

Splitgraph infrastructure, part 2: Integration testing with Docker Compose


## Introduction

At the time of writing, the [Splitgraph registry](/explore) runs using about 60 Docker containers. About half of them we build ourselves in CI. Our stack has code written in C, Haskell, Lua, Python, SQL and TypeScript.

Despite this complexity, any developer is able to build and spin up the whole stack in a matter of minutes. Our continuous integration system can build, test and deploy a change to any part of the codebase in about half an hour.

In this series of blog posts, we'll discuss Splitgraph's internal continuous integration and test infrastructure. We'll start by talking about how we use Make, Docker Compose and BuildKit to efficiently build Docker images.

## Docker

Every Splitgraph service runs in Docker. We use [GitLab CI](https://docs.gitlab.com/ee/ci/introduction/) to build images and push them to the GitLab registry.

We tag images with `$DOCKER_REPO/$image:$DOCKER_TAG`. In CI, the tag is the current commit hash, for example, `registry.gitlab.com/some_repo/some_image:abcdef12`. This lets us easily deploy any commit to production and roll back to previous versions.

However, there is a lot of nuance around using Docker for this.

### BuildKit

[BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/) is an advanced Docker image build engine. We use BuildKit globally for all builds. Its advantages over normal Docker builds are:

  * Running multi-stage image builds concurrently
  * More caching methods. For example, referencing a common `apt` cache without having to delete it from the image in the same build step.
  * Build cache import/export

To use BuildKit, you need to export a couple of environment variables:

```bash
# Get docker build to use BuildKit
export DOCKER_BUILDKIT=1

# Get Compose to use CLI to build images (so that it uses BuildKit too)
export COMPOSE_DOCKER_CLI_BUILD=1
```

### Custom build contexts

A common issue with Docker builds is the need to include files from the outside of the directory of the Dockerfile.

For example, we keep our Lua codebase in one part of the source tree. Multiple services depend on it. We do not want to run Docker from the root of the codebase, since then every build will send the whole source tree to the Docker daemon.

One potential solution is using a `.dockerignore` file to specify which directories should be excluded from the build context. However, this is not going to work for services that share only some dependencies. One service's excluded directory might be required by another service.

We solve this by having custom build scripts for some images. These scripts use `tar` to package up a build context that includes all files that the Dockerfile needs. The build script then pipes this context to `docker build`:

```
#!/bin/bash -ex

SRC_DIR_REL="../../../src"

THIS_DIR="$(cd -P -- "$(dirname -- "$0")" && pwd -P)"

cd "$THIS_DIR"

# Create a custom build context: besides the Dockerfile and
# configs from this directory, we also want the Lua modules
# from src.

tar \
  -cf - . "$SRC_DIR_REL"/luamods \\
  | docker build -t "$DOCKER_REPO"/openresty:"$DOCKER_TAG" \\
    $( [[ -n $CACHE_FROM ]] && echo "--cache-from=$CACHE_FROM" ) \\
    --build-arg BUILDKIT_INLINE_CACHE=1 \\
    -f Dockerfile -
```

This lets the Dockerfile reference any file in the source tree.

### Caching

Public GitLab CI instances run every build in a fresh environment. This means that the Docker daemon state doesn't persist between builds. Luckily, BuildKit allows us to use some advanced caching patterns.

The script above passes `--cache-from=some_repo/image:some_tag` to `docker build`. This tells BuildKit which image it can use as Docker cache. At the beginning of a CI run, we run [`docker manifest inspect`](https://docs.docker.com/engine/reference/commandline/manifest_inspect/) to find a suitable tag from a previous build.

Even better, if the cache image isn't on the local Docker daemon, BuildKit will selectively pull parts of it from the registry to perform the build.

On our developer machines, we have a script that sets `$DOCKER_CACHE_REPO` and `$DOCKER_CACHE_TAG` to the last successful GitLab build. This means that any developer can bring their environment up to date quickly.

Note that this requires you to pass `--build-arg BUILDKIT_INLINE_CACHE=1` to the invocation. This gets BuildKit to put cache metadata inside the image. See the [Docker documentation](https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources) for more information.

### Docker Compose

Splitgraph's codebase contains a few dozen [Docker Compose](https://docs.docker.com/compose/) files. They define various aspects of building and testing the Splitgraph registry. We even use Docker Compose to run Splitgraph in production.

We do this by relying on Compose's less well-known feature: the ability to [combine multiple Compose files](https://docs.docker.com/compose/extends/). For example:

```
$ docker-compose -f docker-compose.prod.yml \\
                 -f docker-compose.service_group_1.yml \\
                 -f docker-compose.service_group_2.yml \\
                 up -d
```

We reuse a lot of the same Compose configuration in development, CI and production. This gives us confidence in our deploys. We will discuss this in a future article.

We also use Compose to build services where we do not need a custom build context. Here is an example Compose snippet that builds our base Python toolchain:

```yaml
  pybase:
    image: ${DOCKER_REPO}/pybase:${DOCKER_TAG}
    build:
      context: ./src/py/base
      dockerfile: Dockerfile
      cache_from: ["${DOCKER_CACHE_REPO}/pybase:${DOCKER_CACHE_TAG}"]
      args:
        BUILDKIT_INLINE_CACHE: 1
```

## Bringing it all together: Make

Until now, we only discussed building one image. How do we manage building and keeping up to date thirty of them, especially if some images depend on others?

The answer is Make. Make is a famous UNIX tool which we can easily adapt to being a generic task execution engine.

Make works using file modification timestamps. Every Make rule describes how to build a file:

```makefile
output.txt: input_1.txt input_2.txt
	cat input_1.txt input_2.txt > output.txt
```

If the output is older than any of the inputs, Make rebuilds it.

But Make targets don't have to be actual executables. In our build system, Make targets build Docker images. Every target produces an empty sentinel file in the `.build` directory. Downstream targets can depend on these files. This lets us define dependency trees between images and factor some common functionality out into base images.

For example, we have a `pybase` image that contains common Python tooling. Python services inherit from it and have two stages:

  * Install required Python packages into a virtual environment
  * Copy the virtual environment into a lightweight `python:slim` image

Our Make tasks also depend on the files that go into the build context of the image. So, if the build context doesn't change, Make doesn't even need to run Docker.

This synergizes well with Git. Git checkouts only bump timestamps of files that have changed. After a checkout, Make will not rebuild images that didn't change between Git commits.

Finally, we use Make for other development tasks which are out of scope for this article:

  * Running integration tests
  * Starting up the whole stack, including the registry, the website and our [query REST API](/docs/query/postgrest), in development
  * Running code formatters and other utilities

## Sample Makefile

This is a snippet from the actual (now almost 600 lines long) Makefile that we use to build and deploy the Splitgraph registry. It brings all these ideas together:

```makefile
# Tag images as $DOCKER_REPO/$image:$DOCKER_TAG
# We override this in CI to push to the GitLab registry directly.
export DOCKER_REPO ?= splitgraph
export DOCKER_TAG ?= development

# Cache from the same image in dev.
# In CI, we detect a suitable image to use as cache.
export DOCKER_CACHE_REPO ?= $(DOCKER_REPO)
export DOCKER_CACHE_TAG ?= $(DOCKER_TAG)

# Force to use buildkit for all images and for docker-compose to invoke
# Docker via CLI (otherwise buildkit isn't used for those images)
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

# BuildKit uses fancy output by default, making CI logs for multiple jobs
# unreadable.
export BUILDKIT_PROGRESS=plain

SHELL=/bin/bash

INTERACTIVE:=$(shell [ -t 0 ] && echo 1)
ifdef INTERACTIVE
	# For interactive invocations, interleave everything, since otherwise
	# shell targets (e.g. local_registry.shell) will hang forever not echoing
	# output.
	MAKEFLAGS := -j 4 --no-builtin-rules
else
	MAKEFLAGS := -j 4 --no-builtin-rules --output-sync=line
endif

# find invocation that ignores various files we do not want to include
# in the build context
FEXCL := -name __pycache__ -prune -o -not -name __pycache__ \\
		-a -not -path "*.egg-info*" \\
		-a -not -path "*htmlcov*" \\
		-a -not -path "*.mypy_cache*" \\
		-a -not -path "*.pytest_cache*" \\
		-a -not -path ".coverage" \\
		-a -not -path "*/node_modules/*" \\
		-a -not -path "*/.next/*" \\
		-a -not -path "*/libmdx/*" \\
		-print

# --- SAMPLE SERVICE ---

.PHONY: service_group.build

# Sample service that uses Docker Compose to build the image
.build/service_1: $(shell find src/py/service_1 $(FEXCL))
	docker-compose -f docker-compose.base.build.yml build service_1
	mkdir -p .build && touch $@

# Sample service that uses a build script
.build/service_2: $(shell find src/pgexts $(FEXCL))
	CACHE_FROM=${DOCKER_CACHE_REPO}/service_2:${DOCKER_CACHE_TAG} components/service_2/build.sh
	mkdir -p .build && touch $@

# PHONY Makefile task that builds both services.
service_group.build: .build/service_1 .build/service_2
```

## Conclusion

In this article, we talked about using Makefiles to manage builds of multiple Docker images. We think this approach strikes a good balance between abstracting away complexity and still being extensible enough to handle a diverse codebase.

In the future articles in this series, we'll discuss other aspects of how we test the Splitgraph registry and run it in production.

If you're interested in learning more about Splitgraph, you can check our [frequently asked questions](/docs/getting-started/frequently-asked-questions) section, follow our [quick start guide](/docs/sgr-advanced/getting-started/five-minute-demo) or visit our [website](https://www.splitgraph.com).


We begin our overview of Splitgraph's internal build infrastructure by discussing how we build Docker images in development and CI using Make and Docker BuildKit.

Splitgraph has been acquired by EDB! Read the blog post.

infrastructure

Preview Environments: Spinning up temporary Splitgraph instances from any commit

Splitgraph infrastructure, part 3: Using Docker Compose in production

Throwing away the backend: Towards a data delivery network

Splitgraph infrastructure, part 2: Integration testing with Docker Compose

Splitgraph infrastructure, part 1: Using Make to build multiple Docker images efficiently

Product

Support

Company

Splitgraph

Splitgraph has been acquired by EDB! Read the blog post.

infrastructure

Preview Environments: Spinning up temporary Splitgraph instances from any commit

Splitgraph infrastructure, part 3: Using Docker Compose in production

Throwing away the backend: Towards a data delivery network

Splitgraph infrastructure, part 2: Integration testing with Docker Compose

Splitgraph infrastructure, part 1: Using Make to build multiple Docker images efficiently

Product

Support

Company

Community

Splitgraph