Introduction
If you have a Raspberry Pi in a remote location, far away from where you are, then it’s important to be able to reach the Pi from the distance (from outside networks) all the time. Unfortunately things can go wrong. For example it might happen that the modem or the router to which the remote Pi is attached temporarily looses the Internet connection and sometimes the Pi’s network interface is not able to automatically recover from this problem and reconnect. Or there could be some trouble on the Pi itself which could cause the connection to be lost and not recovered. No matter how well things are set up, it could always happen that the Raspberry Pi goes offline and thus becomes unavailable from the distance. This is a problem, because the only way to get the machine back online is to go to the remote location and reboot it. That means wasted time and possibly also unnecessary fuel consumption (for traveling to the Pi’s location). If the Pi is used in headless mode (no screen, keyboard or mouse attached), then the problem is even more serious because you cannot communicate with the Pi, so the only way to reboot it is to pull the power plug, which is not a healthy thing to do. You should always shut down the OS from software before removing power:
sudo shutdown -h now
Failure to do this may result in damage to the SD card (it happens rarely, but it does happen sometimes).
So if we need our remote Raspberry Pi (or maybe even a local Pi) to recover the lost connection, it must be able to reboot itself whenever the problem of connection loss occurs. In some cases it might be enough to just restart the network interface service instead of rebooting:
sudo service network restart
However, in other cases something else might be stuck, so the safest solution (from the point of view of recovering online availability) is the brute method of the Raspberry Pi rebooting itself (unless of course some software runs on it which should not be restarted if it can be avoided).
The most important questions that need to be answered before we can implement the solution are:
- How long is the Pi allowed to be offline before it decides to reboot itself?
- How can the Pi detect that it’s unreachable from an outside network?
- How can the Pi reboot itself?
Executing tasks periodically and automatically
To execute a task periodically and automatically on the Raspberry Pi (and on Linux in general), you have to add a cron job. cron is the Linux system scheduler, which executes the tasks specified in its list periodically. To edit the list of scheduled cron tasks, open it with the following command:
sudo crontab -e
If you need to add a new scheduled task, add a line to the bottom of the file, in the following form:
Minute Hour DayOfMonth Month DayOfWeek Task
Every line contains 6 words. The first 5 words form a pattern which defines when to run the task and the last word is the path to the task (script, program) to be executed.
The time pattern formed by the first 5 words is quite flexible. You can use wildcards (*) instead of each word, lists of values, intervals of values or single values. A few examples:
- 0 17 10 3 1 example.sh – executes the example script at 17:00 on the 10th of March if it falls on Monday
- 0 17 * * 1 example.sh – executes the example script at 17:00 every Monday
- 0 17 * * * example.sh – executes the example script every day at 17:00
- 15 * * * * example.sh – executes the example script every hour, when the minute equals 15
- 0,15,30,45 example.sh – executes the example script every hour, when the minute equals 0,15,30 or 45
- 0 17-20 * * * example sh – executes the example script at 17:00, 18:00, 19:00 and 20:00, every day
- */15 * * * * example.sh – executes the example script every 15 minutes
- * * * * * example.sh – executes the example script every minute
As you can see from these examples, the smallest scheduling interval allowed by cron is 1 minute. When you open the cron tab with sudo (sudo crontab -e), you are actually opening the root user‘s cron tab, so the tasks added to the list will be executed as root. For more details about cron, see the related Wikipedia page.
Attempt to guarantee online availability of the Raspberry Pi
The simplest approach to getting around the loss of online availability is scheduling a periodic reboot. This can be done by adding a cron job to the root’s cron tab:
sudo crontab -e
Add the following line to the bottom of the cron file:
0 0 * * * /sbin/shutdown -r
Finally save and close the cron file (in case of nano CTRL+o, CTRL+x)
The above line schedules a system reboot once a day, every day, at midnight (minutes=0, hours=0).
The main problem with this approach is that it just executes a blind reboot periodically, without checking if it’s really needed. In most cases it won’t be needed, the reboot will be unnecessary. It’s not the best of ideas to do this. It might not be a tragedy to execute an unnecessary reboot once a day, but what if the Internet connection is lost soon after the reboot? Can we afford to not have online availability for the next almost 24 hours? Probably not. But that means that we would need to schedule a more frequent reboot, perhaps every 4 hours. The Pi would keep rebooting itself (mostly unnecessarily) quite frequently. So this solution is just not good enough.
A better approach would be for the Pi to check if it can access the outside world and only reboot itself if it cannot. For example it could periodically try to download the main Google search page (http://www.google.com), or some other page that is almost for sure available online all the time. In case it could not download the given page, it would reboot itself. Unfortunately this solution isn’t good enough either because what we’re interested in is being able to access the Pi from outside, not the Pi accessing the outside world (the other direction of communication).
A real solution
And so we arrive to the final solution, which is for the Raspberry Pi to periodically check if it has been successfully accessed from outside and reboot itself if it hasn’t been accessed from outside for more than a given period of time. Unfortunately this means that somebody has to access the Pi from time to time, otherwise it will think that it’s unreachable and reboot itself. If the Pi only reboots itself when it detects that it has not been accessed for a fairly long period of time (perhaps 24 hours), then a human user could log onto it every day to prevent the rebooting when it’s accessible. This can work, but it’s a bit inconvenient. It’s a lot better if a second Raspberry Pi, located in an outside network, periodically logs into the first Pi and executes a command which leaves some trace behind to let the first Pi know that it has been accessed from the outside and that it should not reboot itself. Unfortunately for this solution a second Raspberry Pi is needed, but the good thing is that the solution can be implemented both ways, meaning that both Pis can keep each other in check. If the possibility of communication is lost in ether direction, the Pi that becomes inaccessible will see that the other one did not log into it for more than the allowed time frame and it will reboot itself.
The actual implementation
The first thing that we need is a shell script that resides on the Pi which must be accessible from outside. This script will check its own last modification time and reboot the Pi if more time has elapsed since the last modifiation than allowed. Using the modification time of the script to decide whether a reboot is needed or not is convenient because no other file is needed. Below is a shell script to achieve this (also available for download here):
# How many minutes to allow to pass before rebooting the machine?
MAX_MINUTES_SINCE_LAST_TOUCH=240 #4 hours
# How many minutes of warning (broadcast to logged in users) before actual reboot?
MINUTES_OF_WARNING_BEFORE_REBOOT=2
# Obtain the time when this script was last touched
t_last_touch=`stat -c %Y $0`
# Calculate the difference in seconds between
# the current time and the time of the last touch
t_diff_sec=$(( $(date +”%s”) – $t_last_touch ))
# Calculate the difference in minutes
t_diff_min=$(( $t_diff_sec / 60 ))
# Display the elpased time
echo “Time elapsed since the last modification of ${0##*/}: $t_diff_min min ($t_diff_sec sec).”
# If more than the given number of minutes have elapsed
# since the last login, then reboot the machine
# Give logged in users 5 minutes to save their stuff
# (5 minutes warning before the actual reboot)
# Touch this script file before reboot to reset the counter
if (( $t_diff_min >= $MAX_MINUTES_SINCE_LAST_TOUCH ))
then
touch $0
/sbin/shutdown -r +$MINUTES_OF_WARNING_BEFORE_REBOOT
fi
Let’s assume that this script is saved as /soft/scripts/rebooter.sh
Make sure that the script’s access rights are set properly, so that the users who need to run it can do so. For example you may allow it’s owner and the users who are in the owner’s group read/write/execute the script and everybody else just read it:
sudo chmod 774 /soft/scripts/rebooter.sh
The second thing that we need is a cron job that executes this script periodically. For example the below line executes the script every 15 minutes:
0,15,30,45 * * * * /soft/scripts/rebooter.sh
Note that it’s safer to execute it when the system time minute value equals 0, 15, 30 or 45 than to execute it simply every 15 minutes (*/15 for the minute pattern is less safe than 0,15,30,45) because if something goes wrong and the reboot timer is not reset to 0 after a reboot, then you will still have a maximum of 15 minutes to log in and fix the problem. If the cron entry’s minute pattern would be */15 instead, then the script would run again after every reboot, and if the timer were not reset, it would keep rebooting the machine in an infinite loop.
The third thing needed on the Pi for which we are trying to guarantee online availability is resetting the reboot counter to 0 after each reboot, so that after a reboot the checking script does not think that it needs to reboot the Pi again. To achieve this, open the /etc/rc.local script file (which is executed by the operating system automatically after every reboot) and add a line to it which modifies the rebooter script’s last modification time:
open the file: sudo nano /etc/rc.local
add the line: touch /soft/scripts/rebooter.sh
save the file: CTRL+o
and exit the editor: CTRL+x
Finally, one last thing is needed: on the second Raspberry Pi (which keeps the first one in check) a periodic task must be scheduled to log into the first Pi and update the modification time of the rebooter script. In other words, we must add a cron job to the second Pi to execute a remote command over SSH on the first Pi, to update the modification time of the script. If the first Pi becomes inaccessible, the second Pi will fail executing this remote command and the modification time of the rebooter script residing on the first Pi will not be updated, which will cause the periodically executed rebooter script to reboot the fist Pi after a while. Log into the second Pi and add the needed line to the crontab:
sudo crontab -e
add the line: */30 * * * * ssh [email protected] ‘touch /soft/scripts/rebooter.sh’
save it and close the editor (in case of nano CTRL+o, CTRL+x)
This line will cause the second Pi to log in every 30 minutes as user1 into the first Pi (reachable at the address pi1.com) and update the modification time of the rebooter script.
For this to work, the user from the second Pi must be able to log in without a password, as a trusted user, onto the first Pi as user user1, via SSH. The cron job is executed as root, so both the root user and user1 must be granted passwordless SSH access to the first Pi, when logging in automatically as user1 onto the first Pi. To understand why this is needed, how it works and for a guide to help you set up the passwordless, automatic SSH login, read this article.
Notes
In the form presented above the script reboots the Pi on which it exists if more then 4 hours have passed since it was last modified, but it detects this only when it is run by cron. In our example we scheduled the script to run every 15 minutes. This means that the maximum time for which the Pi can be offline (in the worst case) is a little less than 4 hours and 15 minutes. The second Pi tries to update the rebooter script’s modification time every 30 minutes, which gives it about 8 attempts to successfully update it before a reboot. SSH connections may sometime fail, but 8 opportunities should be more than plenty. The good thing about this setup is that if the connection between the two machines fails temporarily (let’s say for 2 hours), but then it recovers, the Pi won’t be rebooted unnecessarily because the second Pi will still have time to update the modification time of the rebooter script. Of course, these values (4 hours, 15 minutes, 30 minutes) can be tailored to you own needs, but keep in mind that if the time allowed by the rebooter script to pass before it reboots the machine is very short, then there might not be enough buffer for temporary SSH connection failures. Also, it’s not very healthy for the Pi to reboot itself very frequently. The second Pi should probably not touch the rebooter script’s modification time very frequently because the Raspberry Pi uses and SD card as long term storage medium and SD cards allow a limited amount of writes along their lifetime, so the number of writes should be minimized when possible. Don’t take this too seriously, though. The number of writes supported by modern SD cards is quite huge.
Testing
To test that your setup works, you can manually run the rebooter script from the terminal on the first Pi (which hosts the rebooter script). When not enough time has elapsed for a reboot, it displays the number of seconds/minutes elapsed since the lats modification. If everything works well, you should see this counter reset after each 30 minutes (when the other Pi logs in and updates the script’s modification time). You should also see the counter reset after a reboot.
