## Introduction From the outset one of the key tenets of [Seafowl](/seafowl/docs/getting-started/introduction)'s architecture is separation of compute and storage, true to its aim of providing fast analytical queries at the edge. On one hand, this enables running Seafowl on thin, distributed and [scalable nodes](/seafowl/docs/guides/scaling-multiple-nodes) or serverless platforms with scale-to-zero capabilities. On the other hand, the data itself is kept in [Parquet](https://csvbase.com/blog/3) files in dedicated object stores, which can provide for cheap and voluminous storage capacities on the cloud. As of recently, the actual physical layout of table data accommodates the [Delta Lake](https://delta.io/) semantics, marking a transition from a previous custom storage layer that we used.[^1] In this blog post we'll be taking a look at what this means in practice, and at the same time expose some basic facts and features of the Delta protocol. ## A brief recap of table formats and data lakes In general, table format refers to the underlying storage arrangement that represents a given table in a given database. In the case of Seafowl, it made most sense to go with flat storage of Parquet files for a variety of performance and storage benefits, a setup commonly referred to as a data lake. Initially, we implemented a [custom](/blog/seafowl-time-travel) Seafowl data lake table format, that (among other things) involved keeping track of which Parquet file corresponds to which table (version) via a metadata catalog (i.e. a Postgres or SQLite DB). This also meant a lot of custom logic for scanning and partition pruning of tables. However, we soon realized that this niche is being commoditized quickly, and that it would be beneficial to migrate to one of the new standards. In particular, there are three open and competing table formats that bring an ACID layer to data lakes, one of which will likely become the de-facto industry standard eventually. Besides Delta Lake, there are also [Apache Iceberg](https://iceberg.apache.org/) and [Apache Hudi](https://hudi.apache.org/), and while they all offer a variety of advanced features and open-source implementations, we've found that the most mature Rust implementation is that of Delta via the [`delta-rs`](https://github.com/delta-io/delta-rs) project. It also has an active and welcoming community of maintainers, so we finally decided to replace our custom storage layer with delta-rs instead. Lastly, note that in conjunction with other components, namely the internal metadata catalog, and a SQL-native query engine powered by DataFusion, the combined capabilities of Seafowl make up for what is dubbed a Lakehouse.[^2] This is a term popularized by Databricks, but gaining industry-wide acceptance, which denotes a hybrid of a data lake and a data warehouse. ## A quick tour To demonstrate the new storage layer in action, let's try out a toy example, where we use Seafowl to perform analytical queries on a table that is periodically synced with an external source. First, we'll spin up a Seafowl process: ```bash $ SEAFOWL__FRONTEND__HTTP__WRITE_ACCESS=any ./target/debug/seafowl 2023-04-11T06:58:04.654Z INFO seafowl > Starting Seafowl 0.3.2 2023-04-11T06:58:04.659Z INFO seafowl > Loading the configuration from seafowl.toml 2023-04-11T06:58:04.680Z INFO seafowl > Starting the PostgreSQL frontend on 127.0.0.1:6432 2023-04-11T06:58:04.680Z WARN seafowl > The PostgreSQL frontend doesn't have authentication or encryption and should only be used in development! 2023-04-11T06:58:04.680Z INFO seafowl > Starting the HTTP frontend on 0.0.0.0:8080 2023-04-11T06:58:04.680Z INFO seafowl > HTTP access settings: read any, write any ``` ... and run some queries: ```bash $ curl -H "Content-Type: application/json" http://localhost:8080/q -d@-<

Splitgraph has been acquired by EDB! Read the blog post.

storage

A Lakehouse by the sea: Migrating Seafowl storage layer to delta-rs

Product

Support

Company

Community

Splitgraph