Filesystem for 1 million+ files

Question

I currently have a VPS with 1&1, running a number of Spring Boot, Node and PHP services, and have had zero problems with the service at all - until I hit the 500,000 inodes limit last night. I am hosting a large number of map tile images, and associated meta data files, and I'm looking in the region of around 1.2 million files initially.

I have contacted them with regards to expanding the 500k limit, which shouldn't be impossible since the actual size of the data on file so far is barely 4Gb, but in the event of things not working out with this particular provider, what are my options?

Is 500,000 a typical inode limit? Are there any providers that have much higher limits? Any other suggestions for hosting such a service?

Answer 1

To answer the question "Are there any providers that have much higher limits?":

You are using a Virtual Private Server. The entire system is in your hands, only the hardware is in your provider hands. Hence your issue is only related to the software configuration you set-up (or the default one you left as is), not related to the VPS provider.

Inode limitation

In most filesystems, inode count limitation comes from the formating process and can not be upgraded later (except by reformating the partition).

When you use mkfs.ext4 to format a partition, you can explicit inodes ratio (-i bytes-per-inode) or directly inode count (-N xxx). The defaults comes from the configuration file /etc/mke2fs.conf (inode_ratio = 16384 except if you specify profiles big or huge with mkfs argument -T).

The default ratio of 16384 bytes-per-inode leads to 524288 inodes max on a 8GB partition.

This limitation is difficult to upgrade

From mkfs.ext4 :

Be warned that it is not possible to change this ratio on a filesystem after it is created, so be careful deciding the correct value for this parameter. Note that resizing a filesystem changes the numer of inodes to maintain this ratio.

I see three possibilities:

1. Backup you data (and have a cup of tee..), format your partition with more inodes and copy back your data.

2. As suggested by @berserck you can create a loopback partition in a file. This is less efficient and increase disk segmentation but it works. Don't forget to have it auto-mounted on boot.

3. If you have free disk space after you partition, or if you can have some by extending your storage somehow, then you can resize your partition which will lineary increase the inode count.