Let’s first not talk about why this can happen, but deleting /lib, /usr/lib, or some other essential runtime files happens quite a lot (as you can see: here, here, here, and here). In this post, I will only discuss what happens when you delete /lib on Linux and how to recover from that.
The easy solution for everything is to replace the missing files, but this can be difficult if /lib is deleted because we won’t have ld-linux, which is needed to run any dynamic executable. When you deleted /lib, all non-static executable (such as ls, cat, etc, will output):
No such file or directory
You will also be unable to open any new connection using ssh, or open a new tmux window/pane if you are using tmux. So you can only rely on your current shell built in, and some static executables that you have on the system.
If you have a static busybox installed, then it can be your rescue. You can use wget from busybox to download libraries from a clean system. For your information: Debian has busybox installed by default, but the default is not the static version.
If you are worried that this kind of problem might happen to you in the future: Install the static version of the busybox binary, and confirm that it is the correct version.
Bash to the rescue
I assume right now that you don’t have a static busybox, and you don’t even have any static executables (which is the situation in many cases, like in the default install of minimal Debian). My solution for this is to download a static busybox from another machine.
I also assume that you have bash installed (which is the default for most systems). Bash has a lot of default built-ins that we can use. There is a solution from here that can be used to download a file using only built-in bash functions. Other solutions on this thread rely on external command (such as cat). Please note that you need to set the environment variable LANG to C; Otherwise, this script will incorrectly handle Unicode bytes.
Of course, we can’t chmod the destination file to be executable, so we need to overwrite an existing executable. If you have busybox installed (even if it is the non-static version), you can overwrite this file. At this point, you can start the rescue mission: for example, use wget to download fresh /lib from another system.
Please note that busybox can’t function with a name that is not a busybox applet name. So if you overwrite for example, the fmt binary with busybox, then it won’t work (it will say: applet not found). If you don’t have busybox, I suggest overwriting cp, then you can use cp to create a copy of cp as busybox (which will be executable).
No bash? printf can help
If you have a more advanced shell (e.g: zsh), it has TCP modules already built in. You can easily use nc from another machine to send a file to the target machine. Now, let’s assume that you have a very basic shell, for example: dash. Most shell (including dash), has printf as built-in, and we can use this to construct binary files.
Most (all?) shell’s built-in printf implementation supports \ooo where ooo is 3 digit octal. First approach is to just convert busybox, but this file is quite big (2 megabyte). Copy-pasting large printf commands is tedious and is error-prone. We need a small static binary that can help us.
This printf trick will also work for other OS, if you can create a small binary for that OS.
Creating a small ELF for Linux
You can create a very tiny executable if you use assembly directly, but let’s try to do this using C, so it can be portable across different architectures. The smallest useful program that I can think of is just to copy from stdin to stdout, so we can prepare netcat on a machine:
cat busybox | nc -v -l -p 10000
and then we can do this from the borked machine:
fdio < /dev/tcp/192.168.1.168/10000 > busybox
The source code can be like this:
#include "unistd.h"
int main()
{
char x;
while (1) {
int c = read(0, &x, 1);
if (c!=0) break;
c = write(1, &x, 1);
if (c!=0) break;
}
return 0;
}
If we try to compile this with standard C library (on AMD64 machine), the result is 776KB.
$ gcc -Os -static fd.c $ du -hs a.out 768K a.out
The Linux kernel source code contains a nolibc implementation that we can use. Using this compilation option:
gcc -Os -Wl,--build-id=none -fno-asynchronous-unwind-tables -fno-ident -s -nostdlib -nodefaultlibs -static -include nolibc.h fd.c -lgcc -o fd
We get a 4536 bytes binary. Quite good. If we add -z max-page-size=0x04, we can even get a smaller size.
gcc -Os -Wl,--build-id=none -z max-page-size=0x04 -fno-asynchronous-unwind-tables -fno-ident -s -nostdlib -nodefaultlibs -static -include nolibc.h fd.c -lgcc -o fd
It is now 672 bytes. Small enough to transfer. We can convert this using Python.
import sys
with open(sys.argv[1], "rb") as f:
data = f.read()
start = 0
width = 20
targetname = sys.argv[2]
while True:
part = data[start:start+width]
if part=='':
break
a = ''.join(['\\'+(oct(ord(i)).zfill(3))[-3:] for i in part])
dest = '>'
if start>0:
dest += '>'
dest += ' ' + targetname
print("printf '{}' {} ".format(a, dest))
start += width
We can then copy paste this to our ssh session, then do the /dev/tcp redirection trick.
Of course, we can also write a complete program that makes the TCP connection instead of relying on bash redirection.
I hope you will never need this knowledge
This problem occurred to me a few days ago when I updated my Solar Powered Pi Zero, and somehow /lib got deleted (not sure what caused it). This is not a very important machine, and I could have just reimaged the MicroSD card and be done with it, but I was curious if I could recover from the error.
I hope you will never have this error on your production/important machine, but if you have this problem in the future, I hope this post will help you recover from the situation.