About

Overview

  1. Enable USB boot mode on the Pi (enables booting via network)
  2. Setup dnsmasq on the server (provides /boot)
  3. Setup NFS on the server (privides /)

Enable USB boot mode

To start, we need to enable USB boot mode in your Pi 3’s firmware.  I’m assuming you’re working in a terminal on your Pi running some version of Raspbian.  You should be working as root or using sudo to perform these tasks.

Issue the following command to append program_usb_boot_mode=1 to your /boot/config.txt file:

echo program_usb_boot_mode=1 | sudo tee -a /boot/config.txt

Can check /boot/config.txt to make sure it actually got appended with your favorite text editor, cat, less, whatever.  Once you’ve verified that line is present in /boot/config.txt, reboot the Pi.

shutdown -r now

Once it comes back up, login, and issue:

vcgencmd otp_dump | grep 17:

Assuming what we did worked, you should see:

17:3020000a

If you get this result, you have successfully enabled USB boot mode on your Pi 3.  If you don’t, double check that what we wanted to append to /boot/config.txt actually got appended, and reboot again.

This doesn’t break SD card booting functionality.  It just means that you can also boot from USB and ethernet now.

More information can be found here.

You wont be needing that SD card in the Pi anymore, but we’ll be using the contents of /boot and / on the SD card in the setup of dnsmasq and NFS.  Attach the SD card you were using in the Pi to your server.  On the server, let’s make a couple of directories to mount SD card partitions on:

mkdir /mnt/raspbian-boot
mkdir /mnt/raspbian-root

Depending on how you attached the SD card to your server, the exact path to the SD card partitions will vary.  Mount the first one (the smaller one) on /mnt/raspbian-boot and the second one (the larger one) on /mnt/raspbian-root.  NOTE:  You’ll need to change these paths to suit how your machine sees the SD card when you attach it.

mount /path/to/sdcard/boot-partition /mnt/raspbian-boot
mount /path/to/sdcard/root-partition /mnt/raspbian-root

We’ll be copying the contents of /mnt/raspbian-boot and /mnt/raspbian-root later on.

Setup dnsmasq

dnsmasq can do a number for things.  The man pages describe dnsmasq as “A lightweight DHCP and caching DNS server.”  As it turns out, we aren’t going to be using dnsmasq for either of these.  Instead, dnsmasq is going to serve your Pi the /boot part of the filesystem.  When your Pi starts up, having since enabled USB boot mode, it will broadcast bootp requests over the network its attached to.  Once our dnsmasq service on our server is up and listening, dnsmasq will get these requests, and respond by sending files needed by the Pi to bootstrap.

Start by installing the required packages not on the Pi, but on the server/computer/toaster the Pi will be booting off of:

apt install dnsmasq tcpdump

We should test to make sure the server is hearing the bootp requests the Pi is sending out before going any further.  You can check this by issuing:

tcpdump -i <insert network interface here> port bootpc

Powercycle the Pi, wait a few seconds, and you should start seeing things pop up after that command. If you do see something, great! We have connectivity!  If not, double check that both the Pi and your server are really on the same network.  Exit with Ctrl-c and we can continue.

Make a directory for your Pi’s /boot files to live in.  Let’s call it /nfs/client1/boot:

mkdir -p /nfs/client1/boot

Copy the contents of /mnt/raspbian-boot to /nfs/client1/boot:

rsync -xa --progress /mnt/raspbian-boot/* /nfs/client1/boot

Just for testing, open up the permissions of /nfs/client1/boot to make sure dnsmasq can DEFINITELY access those files and send them out:

chmod -R 777 /nfs/client1/boot

Move your current /etc/dnsmasq.conf out of the way:

mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig

This is my entire /etc/dnsmasq.conf on the machine (KDE Neon, but its Ubuntu, whatever) where dnsmasq is running:

#port=0
dhcp-range=192.168.1.255,proxy
log-dhcp
enable-tftp
tftp-root=/nfs/client1/boot
pxe-service=0,"Raspberry Pi Boot "

Make a new /etc/dnsmasq.conf in the editor of your choice (as root) and copy/paste my configuration into it.

You may need to change the 192.168.1.255 part to suit your particular LAN.  The IP of my server is 192.168.1.106 and has a subnet of /24 (I usually check mine with ip a).  If your subnet is /24, which it usually is in residential LAN environments, the IP listed there will be first.three.octets.255.  The 192.168.1 part may change depending on how things are in your home/apartment/office.  I’m assuming that if you’ve made it this far you can figure out what your IP and subnet are.  Just in case.

Regarding the #port=0 part: This part (basically) determines whether or not dnsmasq is doing DNS things.  I should have been allowed to disable this (port=0), but in my case, if I want to have DNS resolution working on the server, I did need to leave it commented out.  I leave this part here for you to test.  I got everything working, Pi 3 booting, great.  Then I went to do something on the server that involved internet and couldn’t because DNS wasn’t working.  Commented that part out, restarted dnsmasq, and voila, DNS works again.

Once your /etc/dnsmasq.conf is ready to go, restart dnsmasq.  On Ubuntu I use:

systemctl enable dnsmasq
systemctl restart dnsmasq

Now we can reboot the Pi 3 and see if stuff actually gets sent back to it.  It won’t boot, because it still needs the rest of the filesystem to do that, but dnsmasq should still pass at least SOMETHING to it.  Monitor with

tail -f /var/log/syslog | grep dnsmasq

You should see a whole bunch of stuff fly by indicating files being sent and not found and a whole bunch of other stuff.

Ctrl-c to exit and we can continue.

On my particular network, even if my server gets bootp requests from the Pi, the Pi doesnt always get stuff from dnsmasq.  I’m not sure why, but sometimes, the Pi will turn on, send out its requests, the server will see them, but stuff won’t get sent back.  Unplug the power, plug back in, and then it’ll work.  I suspect this is an issue with network equipment more than with dnsmasq configuration.  See here.  The errors I was getting are attributed to out of date code burned into the Pi 3.  Usage of the newer bootcode.bin that comes with current versions of Raspbian resolve this issue.  More information about this fix can be found at the end of this guide.

dnsmasq takes care of /boot, now we need to setup NFS, which takes care of everything else.

Setup NFS

On the server, install packages:

apt-get install nfs-kernel-server

Add an entry to /etc/exports:

echo "/nfs/client1 *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports

Copy the contents of the second SD card partition into our NFS export directory /nfs/client1:

rsync -xa --progress /mnt/raspbian-root/* /nfs/client1/

Enable and (re)start services so our changes take effect:

systemctl enable rpcbind
systemctl enable nfs-kernel-server
systemctl start rpcbind
systemctl start nfs-kernel-server

Make sure NFS is doing its thing using showmount:

showmount -e localhost

You should get something back like:

$ showmount -e localhost
Export list for localhost:
/nfs/client1 *

We need to adjust /nfs/client1/boot/cmdline.txt to use this NFS export. This is what my /nfs/client1/boot/cmdline.txt looks like:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.1.106:/nfs/client1,vers=3 rw ip=dhcp rootwait elevator=deadline fsck.repair=yes rootwait

You can see that everything is the same except for the root= part.  Adjust 192.168.1.106 to the IP of your server.

Adjust /nfs/client1/etc/fstab to only include the proc line (delete or comment out the others):

proc            /proc           proc    defaults          0       0

You’re ready to test!

Consideration: /boot on SD card might still be a good idea bootcode.bin in the FAT32 partition is all you need, put /boot in the NFS share

I have two reasons for continuing to use an SD card: inconsistent boots over the network, and updates touching files in /boot.

If I knew that 100% of the time I could boot via (my) network, I might still be cardless.  This is a minor annoyance as I don’t reboot very often, but the degree of inconsistency I experienced would be a thorn in my side.  Updates to how the Pi establishes a tftp session have been made in bootcode.bin.  What I’ve done in my particular setup, is leave the SD card in the Pi, with only that file in the FAT32 partition (the /boot partition).  The Pi searches that partition for that file on boot regardless of any other configuration file.  At that point of the boot process, the Pi isn’t even considering mountpoints.  It looks for a partition of that type (FAT32) and tries to find that file.  If it does, it uses that file to do things instead of whats burned into the chip when it left the factory.  100% boot success over tftp using only that file in that partition.

Updates touching files in /boot is more of a concern.  I don’t want to have to think about whether a package install or an update is going to mess with something in that partition before I run apt.   Figuring out a way to give the Pi access to the contents of /tftpboot is definitely on my list of to-do’s for this guide.  So, while I still have dnsmasq running, I’m not really taking advantage of it.  Pi boots (reliably) off of /boot on the SD card, and then proceeds to pull everything over NFS.  Thanks to the recommendations of some fine people on Reddit and #raspberrypi on Freenode, the separate /tftpboot and /nfs/client1 directories have been combined.  dnsmasq gets its things from /nfs/client1/boot, NFS remains unchanged.  This allows updates and installations to update the contents of /boot on the server just as it would if it were still on the SD card.

Credits

HOW TO BOOT FROM A USB MASS STORAGE DEVICE ON A RASPBERRY PI 3

Network boot a bunch of Raspberry Pi 3

NETWORK BOOT YOUR RASPBERRY PI