I want to research a good option for (almost) infinitely scalable file storage system that is fast to use by many concurrent nodes. Nodes are running Ubuntu 20.04. File system needs to be highly fault tolerant and it should handle automatic backups by configuration.
System needs to support basic POSIX file permissions.
There are two options:
1. Use managed (remote) service such as Azure Files (SMB). Even within Azure network the latencies are too high for high performance applications. This option would need a local RAM cache, which I believe is supported already by cachefilesd.
2. Manage the cluster by yourself using open-souce software such as:
- MooseFS (free version has a single point of failure)
- POSIX file permissions
- High availability
- Fault tolerant
- Built-in backups via fault tolerance or a tool to use
- High amount of concurrent reads, almost none concurrent writes
What I need to research the best option based on which is easier to setup/manager, performance and especially high availability and fault tolerance. Cluster should be able to handle random node crashes.
I want benchmarks for each good option from you. I also want benchmarks against standard HDD and SDD to get a baseline.