Why Arrays as a Universal Data Model

The data industry keeps shipping a new system for every type and every job: warehouses for tables, feature stores for ML, file managers for blobs, specialist engines for genomics or LiDAR. Each one lays bytes on disk and fetches them for a query. The question that should come before the next purchase is simpler: is there one data model that can store, govern, and process all of it — tables, images, video, point clouds, variants, metadata, and whatever arrives next — at the performance of a purpose-built store?

StarNET’s answer, when we design a Data Platform, is that multi-dimensional arrays are that model. Not arrays as a Python convenience, and not dense scientific cubes alone. Dense and sparse arrays, with a storage engine that owns layout, IO, and APIs so SQL, notebooks, and domain tools share one foundation.

That is the difference between a lakehouse that only unifies tables, and a platform that can also govern the pixels, sequences, and point clouds sitting next to those tables.

Why another system per data type fails

You can treat everything as tabular and force the rest into object storage with a catalog note. That is how most estates look today. It is also why teams pay twice for authentication, access control, and logging, then spend the rest of the week joining systems that were never designed to be joined.

Three costs show up immediately:

• Data diversity. Finance tables are not the only asset. Imaging, audio, genomics, LiDAR, and flat files carry decisions that a warehouse never sees. Analytics, data science, and ML all want those collections — with different access patterns.
• Vendor sprawl. A warehouse plus an ML platform plus a metadata store plus a file manager overlaps on the expensive parts (identity, governance, HA) and underlaps on the hard part: one query that mixes a table with an image stack or a variant set.
• Split governance. Each product has its own ACLs and audit trail. Central policy becomes a spreadsheet of exceptions. If you need one control plane, you build it in-house — again.

A universal database is useless if it is slow at any of those types. Skepticism is rational: general systems usually lose to specialists. Arrays are interesting because they can be as fast as specialists when you choose dimensions, tiling, and density correctly.

What has to be universal

Every database, lake, and warehouse shares the same skeleton: persist data on a medium, index it, authorize access, plan a query, execute it, return a subset. If the model can represent every payload, those subsystems can be built once.

The model has to do two things at once:

1. Capture the shape of the data (dense grids and sparse events).
2. Expose a selection primitive that maps to real workloads: range conditions on coordinates — slicing a subarray — with optional filters on payload fields.

If you only have tables, you already live in a special case of that idea (rows as points in a key space). If you only have dense tensors, you miss LiDAR, variants, and most fact tables. Universality starts when both are first-class.

The array data model

Treat an array as a multi-dimensional space of cells. Each cell is identified by a unique set of dimension coordinates and can hold one or more attributes (integers, floats, strings, bytes). Optional array metadata is just key-value context on the whole object.

Dense arrays have an integer domain on every dimension. Every cell exists and stores a value. Images, video frames, and regularly sampled rasters fit here. Coordinates are not stored; they are inferred from the schema.

Sparse arrays allow empty cells. Dimensions may be heterogeneous — floats, strings, “infinite” domains — and duplicate coordinates can be allowed. LiDAR, genomic variants, and most tables fit here. Coordinates must be stored, because emptiness is the common case.

Two modeling choices dominate performance later:

• Dimension vs. attribute. Put a field on a dimension if workloads slice ranges on it. Put it on an attribute if you filter or project it after the slice.
• Dense vs. sparse. If nearly every cell has a value, stay dense. If almost every cell is empty, go sparse. If the data is dense in space but coordinates are floats or gappy integers, use a dense array with dimension labels — a lookup vector that maps real-world values to integer coordinates — then slice as usual.

Slicing is the primitive. In numpy terms, A[0:2, 1:3] is the cells whose first dimension is 0–1 and second is 1–2. In SQL: SELECT attr FROM A WHERE d1 >= 0 AND d1 <= 1 AND d2 >= 1 AND d2 <= 2. Multi-range slices (several intervals per dimension) are the same idea.

Arrays are optimized for those range conditions on dimensions. Attribute filters still work; they are not where the engine spends its layout budget.

Use cases one model can hold

Two myths keep arrays in a scientific niche: that they are only for HPC, and that they are only dense. Drop both and the map looks like a data platform, not a lab notebook.

Tabular data is not a counterexample. It is a configuration. Slice-by-row-id looks like 1D arrays. Slice-by-customer-and-date looks like a sparse array on those dimensions. The same engine can serve BI and a LiDAR viewport if both are arrays underneath.

That is the Data Platform claim: stop buying a new store because the payload is not a table. Change the schema of the array.

Why this can be as fast as a specialist

IO dominates most analytical and scientific queries. The on-disk format decides whether a slice is a few sequential reads or a scatter of random ones. Arrays give you knobs that tables-as-files do not: global order (how a multi-dimensional space is linearized into a 1D file) and tiles (the atomic unit of IO, compression, encryption, checksums).

A practical layout:

• One file (or object) per dimension or attribute — columnar grouping for compression, vectorization, and projecting a subset of fields.
• Dense arrays omit coordinate files; sparse arrays materialize them.
• Variable-length values get a companion offset file.
• Values are not compressed as one blob. They are chunked into tiles so a small slice does not inflate a terabyte.

Dense tiling is set by space-tile extent per dimension, cell order inside the tile, and tile order across the space. Tile shape is geometric.

Sparse tiling cannot use space tiles alone — some tiles would be empty, others huge. You still pick an order (the same three parameters, or a Hilbert curve), then cut tiles by capacity (non-empty cells per tile) so compressibility and IO stay balanced.

Global order is the one shot you get at locality. Disk is 1D; queries are ND. Cells that your typical slice needs should sit near each other. If you know the slice shape (time windows, spatial bounding boxes, genomic loci), you can pick an order that makes those slices cheap. If you do not, you pick an order that is acceptable for most patterns. The point is not a magic default. It is that the model lets you encode the access pattern into the layout — for every data type, not only facts in a warehouse.

Indexing and access

Dense slicing can be arithmetic. Schema plus global order plus tiling plus “no empty cells” tells you which cell slabs (contiguous runs on disk) satisfy the query. No extra index is required. A parallel reader fetches tiles, decompresses, and copies slabs into result buffers. Done well, this is extremely fast.

Sparse slicing cannot infer empty cells. You store non-empty coordinates and index them with something small enough to load on query start — typically an R-tree of tile minimum bounding rectangles. Walk the tree for overlapping tiles, fetch those tiles in parallel, then test coordinates that only partially overlap the slice. Multi-threading and vectorization make this competitive with purpose-built spatial and genomic stores.

That split is why “just put the cube in Parquet” fails. A 2×2 dense array [[1, 2], [3, 4]] serializes as 1, 2, 3, 4 in row-major order. An array engine knows cell (1, 0) is 3 from the schema. A table format has no dimensionality: those four values are a column. Locating (1, 0) requires extra coordinate columns and a scan. Multiply that by tiling, several dimensions, and mixed cell/tile orders, and you see why relational engines never owned scientific workloads — and why a lakehouse table format is not a universal model by itself.

Five decisions that decide performance

1. Dimensions vs. attributes. Slice keys become dimensions. Everything else is an attribute.
2. Number of dimensions. Returns diminish. Use a handful (often four or five) with real pruning power. More dimensions is not more platform.
3. Global order. Locality versus your common slice shapes. Wrong order means extra IO you cannot buy back with more compute.
4. Tile size. Tiles much larger than the slice waste IO and decompression. Tiles much smaller than the slice add overhead. Match them to the query, not to a default.
5. Dense vs. sparse vs. labeled dense. Full grids stay dense. Mostly empty spaces stay sparse. Dense values on non-integer or gappy axes use labels.

Arrays can model tables efficiently. Tables cannot model arrays efficiently. That asymmetry is the argument for putting arrays under a Data Platform rather than stretching a warehouse until images and variants show up.

Build a storage engine, not a format spec

A model and a file layout are not a product. Several array systems proved the pitfalls:

• Dense only — fine for rasters, blind to LiDAR, genomics, and tables.
• One language — a Python-only implementation walls off JVM, R, and C++ estates.
• One backend — a single-file format that shone on Lustre fails on immutable object stores (S3, GCS, Azure Blob) unless IO is redesigned.
• Spec without an engine — every SQL engine and Spark job reimplements parsing, and any format change breaks the ecosystem.

The engine should own density and sparsity; ship a fast core (typically C++) with APIs in the languages your teams already use; abstract POSIX, HDFS, S3, GCS, Azure, and even RAM; and let Spark, Trino, and ML libraries call the API instead of the bytes. Then you can evolve the format without a committee rewriting every consumer.

On top of that baseline, a database-grade platform still needs versioning / time travel and compute push-down (attribute predicates, aggregations) so you do not copy data into every runtime. Open engines in this design family exist; the consulting job is to put them in an architecture with identity, catalog, and products — not to drop a new file format into the lake and call it universal.

What this means for a StarNET data platform

We do not ask every client to throw away Iceberg or Delta. Table formats remain the right default for BI marts and many lakehouse gold products. We do ask whether the rest of the estate — imaging, telemetry, geospatial, genomics, documents — will stay orphaned in buckets with a different security model.

A practical sequence:

1. Inventory types and slices, not only tables. What is dense, what is sparse, what is queried by range?
2. Keep open table formats where the workload is tabular and the engines are already paid for.
3. Introduce an array engine where specialists have proliferated or where Parquet cannot express the access pattern.
4. Unify governance — identity, audit, classification — across table and array objects so “universal” is not only a storage slogan.
5. Expose one self-serve path so domain teams do not open a ticket to read a raster and another to read a fact table.

If that sounds like lakehouse plus mesh, it is: same operating model, wider payload. The array model is how the platform stays honest when the data is not a rectangle of columns.

Getting started

Avoid a greenfield “replace the warehouse with arrays” program. Sequence for the slices that already hurt.

1. Pick one non-tabular collection the business already uses (plant imagery, well logs, claims attachments, store video, genomic panels).
2. Write down the slice predicates (time, region, sample, asset id) — those become dimensions.
3. Prototype dense vs. sparse vs. labeled dense against those predicates and measure IO, not just query syntax.
4. Put the same identity and catalog policy on that array that you use for gold tables.
5. Only then decide whether a second domain should share the engine.

The anti-pattern is a new format with no owners, no tests, and no access model. Arrays will not save a lake that was already ungoverned.

Conclusion

Multi-dimensional arrays — dense and sparse — are a credible universal model for a data platform that has outgrown “everything is a table.” They capture the payloads organizations actually hold, and they give you layout control so slicing can match specialist systems. The model and the on-disk format are not enough. You need a storage engine that hides the spec, speaks many APIs, and runs on the cloud backends you already operate.

"A universal platform is not a single vendor logo. It is one model, one control plane, and performance you would still choose if a specialist were on the table." — StarNET Team

If your estate is still a warehouse, a lake, and a pile of domain files with three audit stories, start with an inventory of types and access patterns. The architecture will only be as universal as the data you are willing to put on the same foundation.