Recently one of our EC2 instance was down due to insufficient disk space. To make it up immediately we had increased its disk space dynamically using EC2 console.
After debugging we found that there was an unexpected spike in rails logs ( some custom logs also ) and logrotate failed to rotate them due to insufficient memory.
After increasing memory, we still, getting ‘Errno::ENOSPC: No space left on device’ in some Sidekiq jobs. Every though more than 50% disk space was available.
After googling, got that might be running out of inodes, but it seems that also not an issue.
But in the above output found a strength file system for /tmp called 'overflow'. So googled out on that term and got the exact reason for an issue.
Here is that reason,
When the system boots and the hard drive is full, nothing can write to /tmp. So during init, a tmpfs is created and mounted. This way your system can safely boot because it can write to /tmp.
So started looking on, how to reset back tmp, found many solutions but those seem to be risky, but out of that found nice solution provided by Jarrod’s blog.
He just asked to unmount an overflow partition so that /tmp will go back to the normal pointing.
But after running this command was getting an error,
umount: /tmp: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
Again his blog helped. Found comment in his blog for exactly the above issue.
And finally /tmp got mounted back to normal.
After debugging we found that there was an unexpected spike in rails logs ( some custom logs also ) and logrotate failed to rotate them due to insufficient memory.
After increasing memory, we still, getting ‘Errno::ENOSPC: No space left on device’ in some Sidekiq jobs. Every though more than 50% disk space was available.
After googling, got that might be running out of inodes, but it seems that also not an issue.
But in the above output found a strength file system for /tmp called 'overflow'. So googled out on that term and got the exact reason for an issue.
Here is that reason,
When the system boots and the hard drive is full, nothing can write to /tmp. So during init, a tmpfs is created and mounted. This way your system can safely boot because it can write to /tmp.
So started looking on, how to reset back tmp, found many solutions but those seem to be risky, but out of that found nice solution provided by Jarrod’s blog.
He just asked to unmount an overflow partition so that /tmp will go back to the normal pointing.
sudo umount overflow
But after running this command was getting an error,
umount: /tmp: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
Again his blog helped. Found comment in his blog for exactly the above issue.
sudo umount -l /tmp
And finally /tmp got mounted back to normal.
