Vagrant up not working, it requires a manual fsck #10208. Cawecoy opened this issue Sep 14, 2018 5 comments Labels.
Two days ago I corrupted my file system during a failed resume from standby on Fedora 19. This feature has never quite worked correctly and randomly makes the kernel panic. Usually, I hard reboot my laptop and everything is fine but that time, something went wrong and when it came back up:
Git-fsck tests SHA-1 and general object sanity, and it does full tracking of the resulting reachability and everything else. It prints out any corruption it finds (missing or bad objects), and if you use the -unreachable flag it will also print out objects that exist but that aren’t reachable from any of the specified head nodes (or the default set, as mentioned above). How do I run fsck manually?Helpful? Please support me on Patreon: thanks & praise to God, and with thanks to the m. Re: RUN fsck MANUALLY. Boot the system from a disc or USB drive. Then open a terminal and run the command. Code: sudo fsck /dev/sda1. If you ask for help, do not abandon your request. Please have the courtesy to check for responses and thank the people who helped you. Blog Linode System Administration Guides Android Apps for Ubuntu Users. The fsck command is a wrapper for the various Linux filesystem checkers (fsck.) and accepts different options depending on the file system’s type. Check the manual pages for more information about a specific checker. For example, to view the options available for fsck.ext4, type: man fsck.ext4 Repair Corrupted File System #.
In this case /dev/sda2 is my root partition and since it was mounted even in maintenance mode, attempting to run fsck on it would output:
Which makes sense as common knowledge tells us that running fsck on a mounted file system will most likely do more damage to it.
The best option
Your best option is simply to boot into another Linux, be it on a different partition, a USB drive or a CD and run fsck manually on the faulty partition, which can easily be unmounted if necessary because no OS is using it. Easy. Normally yes, but my stupid Macbook Pro 2008 cannot boot though USB into anything else other than Mac OS X, my cd drive has been dead for years and recently, I got rid of my OS X partition. To make things more complicated, I’m in Thailand at the moment and obviously not able to take apart my computer to grab the hard drive and stick into a working system.
The other option (if you cannot boot into another Linux)
Manual Fsck Linux
In order to assess the damage, I ran fsck in dry-run mode and piped the output to more to make reading more practical:
From there, I could ensure that no critical files had been damaged and while keeping in mind that it’s always a gamble to use a corrupted file system, I proceeded to boot into the system to make some backups. That out of the way, I did some research on the web on how to fix a root file system that I had to boot into and sadly, not many things turned up for its not an ideal solution. Forcing the system to do it a boot time by creating a file named forcefsck and writing y in it (echo y > /forcefsck) at root no longer works and adding fsck.mode=force on the kernel command line did not fix the problem as fsck will not fix errors on its own without authorization, ie: someone to enter yes on the keyboard. Tried a few other tricks but none worked. I had no choice but keep my fingers crossed and use the system as is.
A few days later, I decided to get back to the issue and while researching alternative solutions, I read that it was possible to fix errors on a read-only file system, which it turns out can also be used to boot into. And it worked, so for posterity here is the technique:
- Put your root partition into read-only mode by modifying the faulty partition’s line on /etc/fstab (but remember your old settings):
Note: If you’re already in maintenance mode at this point, you may be able to remount your file system in read only mode by running “mount -o remount,ro /” and skipping the reboot (thanks Jay).
- Reboot
- Switch to runlevel 1 just to minimize the amount of interfering processes (skip this step if you are running the session over SSH [thanks Josh]):
- Fix your file system (replace /dev/sda2 with your partition’s device), which should now work because the root partition is in read only:
- Reboot
- Make your root file system readable/writable:
- Restore your /etc/fstab to its original state.
- Reboot
Voilà, your system is safe to use again. Hopefully this will have gotten you out of a sticky situation like it did for me. If errors keep coming up, it’s probably a sign that your hard-drive is failing and before you loose it completely, you should mirror your data to a new one.
Performing fsck on non-root filesystem is fairly straight forward. But, for root filesytem, you cannot perform fsck when it is mounted.
This quick tutorial explains how to force filesytem check for a root filesystem.
Root Filesystem
In this example, /dev/sda1 partition is the root filesystem that is mounted as /
If you run a fsck to check your root filesystem, you’ll get the following error message, as /dev/sda1 is mounted, and your cannot perform fsck on a mounted filesystem.
If you are performing a fsck on a non-root file system, you can just unmount that partition and perform fsck.
But in this case, we like to perform fsck on a root filesystem. So, what is the solution?
Also, if you are new to fsck, refer to this: 10 Linux Fsck Command Examples to Check and Repair Filesystem
Tune2fs Output Before Reboot
Before we reboot the system, let us check when was the last time fsck performed a check on the root filesystem.
For this, use tune2fs command, and grep for “check” as shown below.
As we see from the above output:
- Last checked: indicates the last time the root filesystem check happened. This happened on Nov.
- Check internal: indicates how often the root filesystem will be checked. In this example, it will wait for another 6 months before fsck will be executed on this filesystem during reboot.
- Next check after: indicates the date and time after which when you perform a reboot, the filesystem will be checked. This will happen only after May.
But, in our case, we don’t want to wait until May. We want to perform root filesystem fsck check now.
Create /forcefsck File to Force Check Root Filesystem
What Is A Manual Fsck
So, to force a root filesystem check, one of the easy way is to reboot the system, and force fsck to perform the filesystem when the system is starting up before the root filesystem is mounted by the kernel.
To force the fsck during reboot, first create an empty file called forcefsck under / as shown below.
Now reboot the system:
Tune2fs Output After Reboot
If you have console access, you can see that fsck will be performing the check on the / filesystem during the reboot.
After the check, fsck will automatically delete the /forcefsck file that we created earlier. After the reboot, you won’t see this file anymore.
Now, if you execute tune2fs, you’ll see the “Last checked” field was updated with the current timestamp. This confirms that fsck performed the root filesystem check during the reboot.
You’ll also see the value of “Next check after” changed accordingly. i.e 6 months from the time we rebooted the system.