Show HN: Turbolite – S3에서 250ms 미만의 Cold JOIN 쿼리를 제공하는 SQLite VFS

turbolite is a SQLite VFS in Rust that serves point lookups and joins directly from S3 with sub-250ms cold latency. It also offers page-level compression (zstd) and encryption (AES-256) for efficiency and security at rest, which can be used separately from S3. turbolite is experimental. It is new and contains bugs. It may corrupt your data. Please be careful. Object storage is getting fast. S3 Express One Zone delivers single-digit millisecond GETs and Tigris is also extremely fast. The gap between local disk and cloud storage is shrinking, and turbolite exploits that. The design and name are inspired by turbopuffer's approach of ruthlessly architecting around cloud storage constraints. The project's initial goal was to beat Neon's 500ms+ cold starts. Goal achieved. If you have one database per server, use a volume. turbolite explores how to have hundreds or thousands of databases (one per tenant, one per workspace, one per device), don't want a volume for each one, and you're okay with a single write source. turbolite ships as a Rust library, a SQLite loadable extension (.so /.dylib ), and language packages for Python and Node.js, plus Github deps for Go. Any S3-compatible storage works (AWS S3, Tigris, R2, MinIO, etc.). It's a standard SQLite VFS operating at the page level, so most SQLite features should work: FTS, R-tree, JSON, WAL mode, etc. If you want to contribute to turbolite or find bugs, please create a pull request or open an issue. | Query | Type | Cold (S3 Express) | Cold (Tigris) | |---|---|---|---| | Post + user | point lookup + join | 77ms | 192ms | | Profile | multi-table join (5 JOINs) | 190ms | 524ms | | Who-liked | index search + join | 129ms | 340ms | | Mutual friends | multi-search join | 82ms | 183ms | | Indexed filter | covered index scan | 74ms | 173ms | | Full scan + filter | full table scan | 586ms | 984ms | 1M rows, 1.5GB at with nothing cached, every byte from S3. EC2 c5.2xlarge + S3 Express One Zone (same AZ, ~4ms GET latency). Fly performance-8x + Tigris (~25ms GET latency). Both: 8 dedicated vCPU, 16GB RAM, 8 prefetch threads. See Benchmarking and Storage backend matters. Benchmarks are organized by cache level (what's already on local disk when the query runs): | Cache level | What's cached | What's fetched from S3 | When this happens | |---|---|---|---| | none | nothing | everything | Fresh start, empty cache | | interior | interior B-tree pages | index + data pages | First query after connection open | | index | interior + index pages | data pages only | Normal turbolite operation | | data | everything | nothing | Equivalent to local SQLite | interior is the most realistic cold benchmark: interior pages load eagerly on connection open, so by the time you run your first query, they're cached. Index pages aggressively prefetch on first access in the background and may not be ready yet. pip install turbolite import turbolite # tiered database — serve cold queries from S3-compatible storage (Tigris) conn = turbolite.connect("my.db", mode="s3", bucket="my-bucket", endpoint="https://t3.storage.dev") conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)") conn.execute("INSERT INTO users VALUES (1, 'alice', '[email protected]')") conn.commit() alice = conn.cursor().execute("SELECT * FROM users").fetchone() print(alice[1]) >>> "alice" See Installation for Node, Go, Rust, local-only mode, and using the .so loadable extension directly turbolite is designed for S3's constraints over filesystem constraints. Every decision flows from this model: | S3 Constraint | Implication | |---|---| | Round trips are slow | Minimize request count. Batch writes, prefetch reads aggressively. | | Bandwidth is a bottleneck | Maximize bandwidth utilization. | | PUTs and GETs charge per-operation | A 64KB GET costs the same as a 16MB GET. Optimize request count, not byte efficiency. | | Objects are immutable | Never update in place. Write new versions, swap a pointer. No partial-write corruption. | | Storage is cheap | Don't optimize for space. Over-provision, keep old versions, let GC clean up later. | turbolite adds introspection and indirection layers between SQLite and S3 that efficiently groups, compresses, tracks, and fetches pages. SQLite uses a B-tree index and requests for one page at a time. It knows page N is at byte offset N * page_size . And those pages are distributed randomly throughout the pagemap for efficient random access. But on S3, fetching one page per request would mean thousands of potentially random GETs per query. But pages are not created equally. SQLite has different types of pages. turbolite separates page groups by type: interior B-tree, index leaf, and data leaf pages. Interior pages are touched on every query to route lookups to leaf pages. turbolite detects them, stores them in compressed bundles in S3, and loads them eagerly on VFS open. After that, every B-tree traversal is a cache hit. Index leaf pages get the same treatment: separate bundles