On-prem S3-compatible object storage has gone from niche to mainstream, driven by backup targets, data lakes, application object stores and the simple economics of owning capacity rather than renting it. But the software, whether MinIO, Ceph or another platform, runs on physical nodes, and the node hardware decides whether your object store is cheap and resilient or slow and lopsided. This is the physical-node design guide behind on-prem S3: how to spec the drives, metadata flash, host bus adapters and networking for an object storage node, and how the node profile changes with the workload.
What object storage asks of hardware
Object storage has a different hardware profile from block or file storage. It is built for scale and resilience over raw low latency, storing data as objects across many nodes and tolerating the loss of drives and whole nodes by design. That means the node hardware is optimised for capacity, throughput and density rather than the microsecond latency a database demands, and the cluster as a whole, not any single node, provides the performance and durability.
Because the platform expects to spread data and rebuild around failures, the individual node can be built from commodity-style components: dense drives, sensible memory and fast networking, without the exotic features a latency-critical system needs. The skill is in balancing those components so each node pulls its weight and the cluster scales cleanly, rather than over-building a single node that the software was never designed to lean on.
Drives: capacity-led, presented raw
The drives are the heart of an object node, and for most object workloads they are capacity-led: large hard drives that maximise the petabytes per node at the lowest cost per terabyte. Object storage is forgiving of individual drive latency because the durability comes from spreading data across many drives and nodes, so the priority is capacity and aggregate throughput rather than the per-drive speed that matters in a database.
As with any software-defined storage, present the drives to the object platform through a plain host bus adapter in pass-through mode, never a hardware RAID controller, because the software wants to own each disk and manage redundancy through replication or erasure coding itself. Size the HBA and any SAS expanders so they can carry the throughput of all the drives at once, chosen from our host bus adapters range, so the controller never becomes the bottleneck.
- •Capacity-led large drives for the lowest cost per usable terabyte
- •Durability comes from the cluster, so per-drive latency matters less
- •Present disks via HBA pass-through; let the software own redundancy
- •Size the HBA and expander path to the aggregate drive throughput
Metadata, small objects and a flash tier
Object stores keep metadata, the index of what objects exist and where, and that metadata is accessed far more often than any single object, so putting it on fast flash markedly improves the responsiveness of the whole store. A node therefore typically pairs its bulk capacity drives with a smaller amount of NVMe dedicated to metadata and, where the workload has many small objects, to the small-object data itself.
Small objects are the workload that most changes the hardware profile. A store full of large objects, big backup files or media assets, is throughput-bound and happy on capacity drives, whereas a store with billions of small objects becomes metadata- and operations-bound and benefits from far more flash and memory. Knowing which kind of object workload you are building for is the first thing that shapes the node, because the two profiles look quite different.
Memory and the node profile
Memory on an object node serves the storage software, caches metadata and absorbs the work of recovery when a drive or node fails. A capacity-led node storing large objects needs sensible, ECC-protected memory but not extravagant amounts; a metadata-heavy, small-object node needs considerably more, because the index and caching demands rise with the object count rather than the raw capacity.
This is why there is no single object node spec, only profiles. A capacity profile maximises drives and keeps memory and flash modest; a balanced profile adds memory and a metadata flash tier for mixed workloads; a performance profile leans heavily on flash and memory for small-object or latency-sensitive stores. Choosing the profile up front, against the real workload, is how you avoid both an under-resourced node and an over-built one.
Networking and node count
Networking is central to object storage because the cluster constantly moves data between nodes for replication, recovery and rebalancing, on top of the client traffic. A node wants fast, redundant links, with 25GbE a sensible baseline and 100GbE where the node is dense or the throughput high, and on busy clusters it is common to separate the internal cluster traffic from the client-facing traffic so they do not contend.
Object storage is also inherently a multi-node architecture: durability and availability come from spreading data with replication or erasure coding across enough nodes to survive failures, so the smallest sensible deployment is several balanced nodes rather than one. Build below that floor and you have the object API without the resilience that is the whole point. We design the node profile and the cluster size together, around dense platforms such as the HPE Apollo.
Putting it together
Start from the object workload: large objects mean a capacity profile of dense drives, modest memory and a little metadata flash; billions of small objects mean a performance profile heavy on flash and memory. Present drives through an HBA sized to the throughput, give the node fast redundant networking with cluster and client traffic separated, and deploy enough nodes for the durability scheme. We design S3-compatible nodes around the HPE Apollo and the right host bus adapters and drive tiers.