Saturday, July 11, 2015

Raspberry Pi as backup sever of remote files and remote mySQL databases

Project Goals

To use a raspberry pi with an external USB hard disk as a backup media for my data (files and mySQL databases) stored on a remote server - not on my LAN.

Why?

Because I had a raspberry pi around and an available and unused external USD drive that was begging for some action!

Before you start

0. You have an OS on your raspberry pi

1. You have ssh access to it.

2. The pi can access the internet.

3. You have ssh access from your pi to the remote server.

4. You would like to schedule the backup to run daily.

5. You would like to always perform full data backups.

ok, we have our list of requirements, lets jump into the implementation of the project.

Procedure

 

SSH connections

 

1. From a computer on your network, ssh to your Pi (password is raspberry)

ssh pi@

2. From your pi, ssh to your remote server (mine was not on default port 22, so I had to change it to 2222)

sudo ssh -p 2222 username@remote_server
So you have confirmed your ssh connection to your pi and from your pi to the remote server that contains the files you would like to backup.

 

External HDD

 

1. Find out what is your drive device name:

sudo blkid

You will get a list that looks like this:

/dev/mmcblk0p6: LABEL="root" UUID="2b177520-9ff1-4ad5-95f5-6ec7cf61ccaf" TYPE="ext4"
/dev/sda1: UUID="cb980263-9238-469d-87ea-5b1ce2f5f421" TYPE="ext2"


You can see my drive is at /dev/sda1 and is formatted as ext2.

2. Mount the device:

sudo mount /dev/sda1 /mnt

3. Create your backup folder on the drive:

sudo mkdir /mnt/backup

 

Prepare for database backups

 

1. Run the following commands to install some tools you will need to reach and dump (save) your remote databases:

sudo apt-get update
sudo apt-get install mysql-server

2. Do a manual test to see if you can backup mysql databases from your remote host.

sudo mkdir /test
sudo mkdir /test/databases

sudo mysqldump --host -u   -p > /mnt/backup/test/databases/.sql

On my setup, I had to specifically allow remote mysql connections from my ISP on cpanel, otherwise I cannot reach the database from outside.

 

Prepare ssh connection from pi to remote host

 

We will automate the backup procedure and we want it to be fully automatic, so we do not wish to have to enter the ssh connection password all the time we make a connection request. Proceed as follows to implement a no-password ssh connection:

on localhost (where you want to backup to, i.e. your pi)

ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub "@remote-host -p 2222"

You may or may not have to specify the connection port depending on your remote host ssh implementation. As you can see above I had to force the user of another port.

Now try to connect from your pi. You should not be asked for a password at this point:

ssh -port 2222 username@remote_host

 

The bash file

 

Use any variation of the following:

#!/bin/bash

#Create folder under /backup with a name of the format 20151020_23:59:00
DATE=`date +%m.%d.%Y_%H.%M.%S`
HOST=""

#Backup files
sudo mkdir -p /mnt/backup/$DATE
sudo chown pi /mnt/backup/$DATE
mkdir -p /mnt/backup/$DATE/log
mkdir -p /mnt/backup/$DATE/databases
touch /mnt/backup/$DATE/log/backup.log

date >> /mnt/backup/$DATE/log/backup.log
echo "******************************Starting..." >> /mnt/backup/$DATE/log/backup.log

#############################################START BACKUP PROCEDURES####################################
#Backups on Saturday:
if [[ $(date +%u) -eq 6 ]]
then
        #DECLARE ALL FOLDERS TO BACKUP ON SATURDAYS WITHOUT TRAILING SLASH:
        declare -a arr=("" "")
        for i in "${arr[@}}"
        do
                echo "BACKUP OF" >> /mnt/backup/$DATE/log/backup.log
                echo $i >> /mnt/backup/$DATE/log/backup.log
                IFS=/ read -a path <<< "$i"
                rsync -arv -e 'ssh -p 2222' @:/$i/ /mnt/backup/$DATE/${path[0]}/ >> /mnt/backup/$DATE/log/backup.log
        done
else
        echo "Not Saturday. Weekly backup not performed." >> /mnt/backup/$DATE/log/backup.log
fi

#Backups daily
#DECLARE ALL FOLDERS TO BACKUP DAILY WITHOUT TRAILING SLASH:
declare -a arr=("" "")
for i in "${arr[@]}"
do
        echo "BACK UP OF " >> /mnt/backup/$DATE/log/backup.log
        echo $i >> /mnt/backup/$DATE/log/backup.log
        IFS=/ read -a path <<< "$i"
        rsync -arv -e "ssh -p 2222" @:/$i/ /mnt/backup/$DATE/${path[0]} >> /mnt/backup/$DATE/log/backup.log
done

#DB BACKUP ON SATURDAY:
if [[ $(date +%u) -eq 6 ]]
then
        sudo mysqldump --host $HOST -u -p > /mnt/backup/$DATE/databases/
else
        echo "Not Saturday. Weekly database backup not performed"
fi

sudo mysqldump --host $HOST -u -p > /mnt/backup/$DATE/databases/

#Delete folder of name backup_XXXXX older than XXXX months
#coming later

#Send report email
#coming later

echo "******************************Finished..." >> /mnt/backup/$DATE/log/backup.log
date >> /mnt/backup/$DATE/log/backup.log

This file will do a full backup of certain files on Saturday and certains files daily. The same for the databases. It will create a folder with the date and time at which it runs and will create a log file to store debug information.

Ensure that the file runs manually (for example, if it's called backup.sh):

./backup.sh

 

Make this an automatically scheduled cron job

 

Run the following command to edit your crontab:

crontab -e

And enter the following command (this will run the file everyday at 2:00 am. Of course you can set it to match your requirements):

0 2 * * * /home/pi/backup.sh

 

Setup FTP access to your pi

 

I need to be able to access the files from computers on my LAN. One way that works for me is to setup an FTP server on the pi with the connection folder being /mnt/backup. Everytime I FTP to the pi, I'm sent to the backup folder where I can see all the folders per day and get what I'm interested in.

To install the FTP server:

sudo apt-get install vsftpd


Configure the FTP server to not allow anonymous logins and to redirect connecting users to the /mnt/backup directory:

nano /etc/vsftpd.conf

Edit the following configuration file to match your case:
anonymous_enable=NO
#
# Uncomment this to allow local users to log in.
local_enable=YES
#
# Uncomment this to enable any form of FTP write command.
write_enable=YES
#
# Default umask for local users is 077. You may wish to change this to 022,
# if your users expect that (022 is used by most other ftpd's)
local_umask=022
# Activate directory messages - messages given to remote users when they
# go into a certain directory.
dirmessage_enable=YES
#
# Activate logging of uploads/downloads.
xferlog_enable=YES
#
# Make sure PORT transfer connections originate from port 20 (ftp-data).
connect_from_port_20=YES
#
# If you want, you can have your log file in standard ftpd xferlog format
xferlog_std_format=YES
#
local_root= /mnt/backup/
chroot_local_user=YES
pam_service_name=vsftpd
userlist_enable=YES
#enable for standalone mode
listen=YES
tcp_wrappers=YES
use_localtime=YES


I ran into a few issues with FTP the server. Namely I had to create the file vsftpd.user_list file:

sudo touch vsftpd.user_list

Then I had to make my backup folder accessible to the FTP server:

sudo chmod a-w /mnt/backup

 

Things to do

 

I'm happily retrieving my data everyday but there's a couple of things I need to improve. These may be the subject of another post if there's any interest:

1. Receive report email

I have not taken too much time reading it, but this page seems to show the way to go to have email sent from the pi: http://www.sbprojects.com/projects/raspberrypi/exim4.php

2. Spin-down my external drive

My drive seems to be always operating even when it's not doing anything (which is most of the time actually) so I would like to save power and working hours by having it go into sleep mode or something like that. Not sure if that can be achieved with an external USB drive but the following pages are good places to start:


Good Luck! 

Saturday, September 6, 2014

Weather station github page

I'm actively updating all the code and libraries for the new weather station project. I'm collecting all the old information for historic purposes, as follows:

1. github project pages are here: https://github.com/RuiAlves/weather

2. Original SHT1x arduino library: https://www.dropbox.com/s/12cpvenrntuworb/SHT1x-master_orig.zip?dl=0

***NOTE extract the library above, rename the extracted folder from "SHT1x-master" to "SHT1" and copy it into the libraries forlder of your arduino IDE installation.

3. Original Timer 1 arduino library: https://www.dropbox.com/s/plrpx35agdubf74/TimerOne-r11.zip?dl=0

 ***NOTE extract the library above,create a folder called "Timer1" and copy it to the libraries folder of your arduino IDE installation.

You will need to shutdown and restart the IDE for the libraries to be ready for use by the system.

Once you do this with the original IDE 0017 code, the code verification will fail on the creation of an instance of Webserver. No doubt this is now done another way. I have to investigate how much the http library has changed in the last few years. To work!

More information will be posted as it becomes available.

Sunday, August 31, 2014

Personal Weather Station V2 - Revamping an old project

After more than 4 years working reliably, I decided to retire my weather station... No, I did not, I decided it was time for an upgrade!!!

Inside the weather station controller enclosure - 4 years after. You have served me well!

The weather station project was one of the most interesting projects I ever did and I miss all the excitement that went along with it at the time. Even now it's one of the most visited projects on my site (right after the wifi speakers project).

I have recently come across a good use for the data that the weather station collects: I now live near the sea and being the outdoors man that I am, I like to be out as much as I can and the current is very important to know before going out. Yeah, I can just look out the window but what's the fun in that? Actually there's another think the weather data is important for and that is all my friends that do not live near but that still come over frequently to enjoy the beauties of the sea. It would be great to provide an updated current weather view along with a camera feed on the current sea or beach conditions.

As such, the main project goals of the this project are:

1. Use as much of the existing weather station hardware as possible (might be a challenge as a lot of the equipment is already 4 years old and it shows!)
The weather station sensors as they stand after 4 years of reliable data gathering (note the wind speed sensor with a broken segment...). A little more rust then when I got it but everything still works well.
2. Eliminate the need for a dedicated network (I currently have a separate network for the weather station. I would like to install a wireless bridge to connect the weather station to a single network.
A refurbished and quite old Buffalo wifi router will be our wireless bridge.

3. Move data storage to my own cloud server instead of on my computer or thinkspeak.
4. Provide a connection page to all users with the current weather data, historical weather data and a camera feed.
5. Collect, store and display the following data:
  • Temperature
  • Relative humidity
  • Wind direction (will not be perfect because of installation restrictions)
  • Wind speed (will not be perfect because of installation restrictions)
  • Luminosity
  • Dust/ Particles sensor
  • Provide camera feed of current conditions
6. Technically, I will use the following equipment, software and tools:
The Sharp GP2Y10 particle sensor.
  • [hardware] Old buffalo router as network bridge, model WZR2-G300N, where I intend to replace the stock firmware by 
  • [software] DD-WRT to allow for more advanced features. Apparently this router is not supported by the open-source firmware, but there's some files that were made available that have seen users report successful firmware changes. I will make the files available soon as I had to so some digging around to get them. The instructions for what needs to be done are here - look for the post dated Fri Jul 30, 2010 11:14 by REPASSAC.
I will use this blog as data collection tool for me and for other users that may require the information that I collect as I go (like software I will use and all code that I develop or equipment datasheets).

I will usually work on this on Sundays, so don't expect posts more frequent 

Wednesday, March 5, 2014

Hack into 'Jurassic Park' like you're Samuel L. Jackson

via theverge.com

For a movie full of giant dinosaurs, it's strange that one of Jurassic Park's most memorable scenes involves portly programmer Dennis Nedry — played by Seinfeld regular Wayne Knight — wagging his finger and repeating the words "Ah-ah-ah! You didn't say the magic word!" Now you can try to wrest control of the dinosaur-packed park back from the nefarious Nedry, using the recreation of the movie's UNIX network found at JurassicSystems.com.

Monday, January 13, 2014

change unity launcher buttons size

run the following script as root

to run from the command line: sudo script_name.py XX, where XX is the icon size (popular size is 32)

logout and login again and you should see smaller icons in the laucher.

---copy and paste from this line to the end to a file called "script.py"
#!/usr/bin/python

import shutil
import sys
import os

def getparent (content, line_nr):
    for i in range (line_nr - 1, -1, -1):
        if "{" in content[i].lstrip():
            return content[i].lstrip().split(" ")[0]

def change_line (content, key, value, parent):
    line_nr = 0
    for line in content:
        if line.lstrip().startswith(key + ":"):
            lparent = getparent(content, line_nr)
            if parent == lparent:
                content[line_nr] = key + ": " + value + "\n"
                return content
        line_nr = line_nr + 1       

def load_file (filename):
    file = open (filename, "r")
    content = file.readlines()
    file.close()
    return content
   
def write_file (filename, content):
    shutil.copyfile (filename, filename + ".bak")
    file = open (filename, "w")
    for line in content:
        file.write (line)
    file.close()
    return
           
shell_qml = "/usr/share/unity-2d/shell/Shell.qml"
icontile_qml = "/usr/share/unity-2d/shell/common/IconTile.qml"
launcherlist_qml = "/usr/share/unity-2d/shell/launcher/LauncherList.qml"
launcheritem_qml = "/usr/share/unity-2d/shell/launcher/LauncherItem.qml"

if (len(sys.argv) > 1):
    icon_size = int(sys.argv[1])
else:
    sys.exit("Please enter your desired icon size as the first argument.")
   
if not os.geteuid() == 0:
    sys.exit("Script must be run as root.")

content = load_file (shell_qml)
content = change_line (content, "width", str (icon_size + 16), "LauncherLoader")
write_file (shell_qml, content)
content = load_file (icontile_qml)
content = change_line (content, "sourceSize.width", str (icon_size), "Image")
content = change_line (content, "sourceSize.height", str (icon_size), "Image")
write_file (icontile_qml, content)
content = load_file (launcherlist_qml)
content = change_line (content, "property int tileSize", str (icon_size + 6), "AutoScrollingListView")
content = change_line (content, "property int selectionOutlineSize", str (icon_size + 16), "AutoScrollingListView")
write_file (launcherlist_qml, content)

Tuesday, May 14, 2013

Resettting your mySQL root password

When you need to reset your root password for your mySQL server, proceed this way:

# sudo /etc/init.d/mysql stop
# sudo mysqld_safe --skip-grant-tables &
# sudo mysql -u root

on the mySQL console:

mysql> use mysql;
mysql> update user set password=PASSWORD("NEW-ROOT-PASSWORD") where User='root';
mysql> flush privileges;
mysql> quit


# sudo /etc/init.d/mysql stop
# sudo /etc/init.d/mysql start






To confirm you indeed have access to mySQL with the new password:


# mysql -u root -p