Research distributed file storage systems on linux on Azure

I want to research a good option for (almost) infinitely scalable file storage system that is fast to use by many concurrent nodes. Nodes are running Ubuntu 20.04. File system needs to be highly fault tolerant and it should handle automatic backups by configuration.

System needs to support basic POSIX file permissions.

There are two options:

1. Use managed (remote) service such as Azure Files (SMB). Even within Azure network the latencies are too high for high performance applications. This option would need a local RAM cache, which I believe is supported already by cachefilesd.

2. Manage the cluster by yourself using open-souce software such as:

- MooseFS (free version has a single point of failure)

- BeeGFS

- LizardFS

- Ceph

- Quobyte

Requirements are:

- POSIX file permissions

- High availability

- Fault tolerant

- Built-in backups via fault tolerance or a tool to use

- High amount of concurrent reads, almost none concurrent writes

What I need to research the best option based on which is easier to setup/manager, performance and especially high availability and fault tolerance. Cluster should be able to handle random node crashes.

I want benchmarks for each good option from you. I also want benchmarks against standard HDD and SDD to get a baseline.

Taidot: Linux, Ubuntu, Verkonhallinta, Pilvitietotekniikka , Järjestelmänvalvoja

Näytä lisää: file storage script download, custom file storage, php file upload bar linux, custom user file storage, website multi user file management systems, online file storage project requirements management manager, secure file storage linux, java distributed file storage, distributed file storage system, azure file storage, azure file storage api, azure file storage api c#, create directory in azure file storage c#, upload file to azure file storage c#, azure file storage class kubernetes, create file in azure file storage c#, azure file storage rest api example, download file from azure file storage, azure file storage pricing

Projektin tunnus: #28310720