[rsnapshot-discuss] Weekly backup bombing out not finding daily.6

[Phill Edwards]

I have rsnapshot set up to do 7 daily backups and a weekly backup. The
target for the backups are on a remote NAS which gets mounted/unmounted by
a script run as cmd_preexec and cmd_postexec.

This is all working fine for the daily backups. But when it comes to do the
weekly backup I get this error:

/mnt/pme-nas2/Backups/daily.6 not present (yet), nothing to copy

If I mount the NAS manually before running rsnapshot the weekly backup
works OK.

As the remote NAS gets mounted properly for the daily backups, I'm
wondering why it doesn't for the weekly. Does the weekly copy section
happen before the cmd_preexec is run, perhaps? This is how it's looking to
me because I have echo statements in the cmd_preexec that don't show any
output before I ge the "not present yet" error. Any suggestions for how I
can resolve this?

[Phill Edwards]

I have made similar mistake in the past: I started the weekly before the
daily has completed.
45 23 * * 0-6 /usr/bin/rsnapshot daily # daily backup ran at 11:45pm
10 2 * * 7 /usr/bin/rsnapshot weekly # Weekly backup at 3:10am Sunday

Thanks Anthony, that's interesting. But I'm not yet convinced it's the root
cause of my problem. I set the weekly up to run BEFORE the daily based on
some very clear advice I read on a QNAP forum. I also read it on the
rsnapshot HOW itself at
http://rsnapshot.org/rsnapshot/docs/docbook/rest.html. It says "It is
usually a good idea to schedule the larger intervals to run a bit before
the lower ones. For example, in the crontab above, notice that daily runs
30 minutes before hourly. This helps prevent race conditions where the
daily would try to run before the hourly job had finished. This same
strategy should be extended so that a weekly entry would run before the
daily and so on".

So now I'm confused. Is my problem really because I'm running weekly before
daily? I don't actually see how it can be because the daily.0/1/2/3/4/5/6
are always there so the weekly should just work - so I have scheduled it to
run BEFORE daily. In what order do other folks reading this set up daily vs
weekly?

I am very suspicious that the real problem is that when running any
interval which isn't the most frequent, the rsnapshot script doesn't run
cmd_preexec (which in my case means the target for the backup isn't
mounted).

Can anyone confirm?

[Phill Edwards]

Post by Phill Edwards
So now I'm confused. Is my problem really because I'm running weekly
before daily? I don't actually see how it can be because the
daily.0/1/2/3/4/5/6 are always there so the weekly should just work - so I
have scheduled it to run BEFORE daily. In what order do other folks reading
this set up daily vs weekly?
I am very suspicious that the real problem is that when running any
interval which isn't the most frequent, the rsnapshot script doesn't run
cmd_preexec (which in my case means the target for the backup isn't
mounted).
Can anyone confirm?

I just saw this which confirms my theory. For anyone in future...DO NOT use
cmd_preexec to mount the target for the backups. While it seems like a
logical place to do it, cmd_preexec only runs for the most frequent
interval backups. So if, for example, you had daily, weekly and monthly,
cmd_preexec will only be run for the daily backups.

[Phill Edwards]

Post by Phill Edwards
So now I'm confused. Is my problem really because I'm running weekly

Post by Phill Edwards
before daily? I don't actually see how it can be because the
daily.0/1/2/3/4/5/6 are always there so the weekly should just work - so I
have scheduled it to run BEFORE daily. In what order do other folks reading
this set up daily vs weekly?
I am very suspicious that the real problem is that when running any
interval which isn't the most frequent, the rsnapshot script doesn't run
cmd_preexec (which in my case means the target for the backup isn't
mounted).
Can anyone confirm?

I just saw this which confirms my theory. For anyone in future...DO NOT
use cmd_preexec to mount the target for the backups. While it seems like
a logical place to do it, cmd_preexec only runs for the most frequent
interval backups. So if, for example, you had daily, weekly and monthly,
cmd_preexec will only be run for the daily backups.

Sorry forgot to paste the link I saw -
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/

[Nico Kadel-Garcia]

Post by Phill Edwards

Post by Phill Edwards

Post by Phill Edwards
So now I'm confused. Is my problem really because I'm running weekly
before daily? I don't actually see how it can be because the
daily.0/1/2/3/4/5/6 are always there so the weekly should just work - so I
have scheduled it to run BEFORE daily. In what order do other folks reading
this set up daily vs weekly?
I am very suspicious that the real problem is that when running any
interval which isn't the most frequent, the rsnapshot script doesn't run
cmd_preexec (which in my case means the target for the backup isn't
mounted).
Can anyone confirm?

I just saw this which confirms my theory. For anyone in future...DO NOT
use cmd_preexec to mount the target for the backups. While it seems like a
logical place to do it, cmd_preexec only runs for the most frequent interval
backups. So if, for example, you had daily, weekly and monthly, cmd_preexec
will only be run for the daily backups.

Sorry forgot to paste the link I saw -
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/

Ouch. Also see my advice at the bottom of that thread. If I have to
play games like remounting an attached drive or NFS mount in "rw" mode
to run rsnapshot, I've tended to do it in a wrapper that can execute
"trap" functions if something fails horribly.

[Phill Edwards]

Post by Phill Edwards
I just saw this which confirms my theory. For anyone in future...DO NOT

Post by Phill Edwards
use cmd_preexec to mount the target for the backups. While it seems
like a logical place to do it, cmd_preexec only runs for the most
frequent interval backups. So if, for example, you had daily, weekly and
monthly, cmd_preexec will only be run for the daily backups.

Sorry forgot to paste the link I saw -
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/

For the benefit of the archive, I thought I would post the rsnapshot
wrapper shell script that I wrote in case anyone finds it useful in future.
And if anyone has any feedback on how it could be improved, please shout
out! The script essentially mounts the remote backup target (assumes it's a
NFS share) if it's not already mounted, then runs rsnapshot if that all
works OK, and finally unmounts it. If you wanted to mount a non-NFS share
it would be a simple change to the script.


#!/bin/sh
#
-----------------------------------------------------------------------------
#
# Script: pme_rsnapshot_wrapper.sh
# Created by PME to act as a wrapper script for rsnapshot. It's called from
# cron and mounts the target NFS backup share, checks it's mounted and then
# runs rsnapshot. When rsnapshot's finished it unmounts the target share.
#
# This is instead of using cmd_prexec and cmd_postexec configs in rsnapshot
# because they're only run fior the most frequent interval backup (ie daily)
# so don't get run for weekly and monthly backups.
#
# Parameters:
# $1 = rsnapshot backup interval: daily, weekly or monthly
# $2 = "DUMMYRUN" if don't actually want the rsnapshot backup to run.
#
# Usage:
# pme_rsnapshot_wrapper.sh daily|weekly|monthly [DUMMYRUN]
#
# Example command to mount NFS share on NAS2 is:
# /bin/mount -t nfs remote.server.name:/mnt/sda2 /mnt/nas2/
#
#
-----------------------------------------------------------------------------

#
-----------------------------------------------------------------------------
# Set config variables
#
-----------------------------------------------------------------------------

# Location for target of backups.
BKUP_TARGET="remote.server.name:/mnt/sda2"

# Mount point - don't put a / on end cos it will fail grep of /proc/mounts
MNT_POINT="/mnt/nas2"

# rsnapshot executable and path (this is on a QNAP NAS)
RSNAP_EXE="/share/MD0_DATA/.qpkg/Entware-ng/bin/rsnapshot"

#
-----------------------------------------------------------------------------
# END OF CONFIG. Shouldn't need to change anything below here (unless you
want
# to add a new backup interval).
#
-----------------------------------------------------------------------------


#
-----------------------------------------------------------------------------
# Function to mount remote backup target on local dir.
#
-----------------------------------------------------------------------------
mountBackupTarget () {
if grep -qs "$MNT_POINT" /proc/mounts; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: $MNT_POINT is already
mounted"
return 0
else
echo "`date` pme_rsnapshot_wrapper.sh: INF: mounting $BKUP_TARGET on
$MNT_POINT"
/bin/mount -t nfs $BKUP_TARGET $MNT_POINT
return $?
fi
}

#
-----------------------------------------------------------------------------
# Function to UNmount backup target after backup finished.
#
-----------------------------------------------------------------------------
unmountBackupTarget () {
echo "`date` pme_rsnapshot_wrapper.sh: INF: unmounting $BKUP_TARGET"
/bin/umount $BKUP_TARGET
}

#
-----------------------------------------------------------------------------
# MAIN
#
-----------------------------------------------------------------------------
# Check num of parameters passed is correct (should be at least 1)
if [ $# -lt 1 ]; then
echo "`date` pme_rsnapshot_wrapper.sh: ERR: Incorrect number of params:
\$1=$1"
echo " usage: pme_rsnapshot_wrapper.sh
daily|weekly|monthly"
exit 1

# Execute mount command
elif [ "$1" == "daily" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot \"$1\" requested"

# Execute UNmount command
elif [ "$1" == "weekly" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot \"$1\" requested"

# Execute UNmount command
elif [ "$1" == "monthly" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot \"$1\" requested"

# Command line parameter was some other rubbish.
else
echo "`date` pme_rsnapshot_wrapper.sh: ERR: Incorrect param \"$1\""
echo " usage: pme_rsnapshot_wrapper.sh
daily|weekly|monthly"
exit 1
fi


#
-----------------------------------------------------------------------------
# Mount remote backup target and run rsnapshot if successful.
#
-----------------------------------------------------------------------------
mountBackupTarget
RETVAL=$? # Gets return value of the mount command.

if [ "$RETVAL" -eq 0 ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: $BKUP_TARGET mounted OK"

if [ "$2" == "DUMMYRUN" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: $2 so rsnapshot not
executed"
else
echo "`date` pme_rsnapshot_wrapper.sh: INF: running $RSNAP_EXE $1"
$RSNAP_EXE $1 # Run rsnapshot backup
fi

unmountBackupTarget # UNmount backup directory when rsnapshot completed.

else
echo "`date` pme_rsnapshot_wrapper.sh: ERR: $BKUP_TARGET mount failed
with code $RETVAL"
exit 1
fi

[Patrick O'Callaghan]

I haven't really been following this thread but I have one question: are
you running rsnapshot on the NAS or on the client(s)? The fact that you
mount the NAS via NFS leads me to suspect the former. It's worth noting
that this is by far the worst (i.e. least efficient) way to do it. I used
to do it this way a few years ago because it seemed so natural, but in fact
rsnapshot works much better if you run it on the NAS and connect to the
client(s) via the rsync protocol. This is not very clear from the
documentation - I guess the author just assumed it was obvious - but when I
changed methods I saw a speedup by an order of magnitude.

The reason is that treating the NAS as if it were local means that the
receiving rsync (which is still running on the client machine!) is reading
huge quantities of data in order to check for modified files. Doing it the
other way means the receiver runs on the NAS and the network is only used
to send summaries and changed files, with the resulting boost in
efficiency. It also means you don't need NFS for backups (though it's still
useful for restores).

Sorry of I've misunderstood what you are doing, but I feel this point is
worth emphasizing and should be included in the docs.

poc

Post by Phill Edwards
I just saw this which confirms my theory. For anyone in future...DO NOT

Post by Phill Edwards

Post by Phill Edwards
use cmd_preexec to mount the target for the backups. While it seems
like a logical place to do it, cmd_preexec only runs for the most
frequent interval backups. So if, for example, you had daily, weekly and
monthly, cmd_preexec will only be run for the daily backups.

Sorry forgot to paste the link I saw -
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/

For the benefit of the archive, I thought I would post the rsnapshot
wrapper shell script that I wrote in case anyone finds it useful in future.
And if anyone has any feedback on how it could be improved, please shout
out! The script essentially mounts the remote backup target (assumes it's a
NFS share) if it's not already mounted, then runs rsnapshot if that all
works OK, and finally unmounts it. If you wanted to mount a non-NFS share
it would be a simple change to the script.
#!/bin/sh
#
-----------------------------------------------------------------------------
#
# Script: pme_rsnapshot_wrapper.sh
# Created by PME to act as a wrapper script for rsnapshot. It's called from
# cron and mounts the target NFS backup share, checks it's mounted and then
# runs rsnapshot. When rsnapshot's finished it unmounts the target share.
#
# This is instead of using cmd_prexec and cmd_postexec configs in rsnapshot
# because they're only run fior the most frequent interval backup (ie daily)
# so don't get run for weekly and monthly backups.
#
# $1 = rsnapshot backup interval: daily, weekly or monthly
# $2 = "DUMMYRUN" if don't actually want the rsnapshot backup to run.
#
# pme_rsnapshot_wrapper.sh daily|weekly|monthly [DUMMYRUN]
#
# /bin/mount -t nfs remote.server.name:/mnt/sda2 /mnt/nas2/
#
#
-----------------------------------------------------------------------------
#
-----------------------------------------------------------------------------
# Set config variables
#
-----------------------------------------------------------------------------
# Location for target of backups.
BKUP_TARGET="remote.server.name:/mnt/sda2"
# Mount point - don't put a / on end cos it will fail grep of /proc/mounts
MNT_POINT="/mnt/nas2"
# rsnapshot executable and path (this is on a QNAP NAS)
RSNAP_EXE="/share/MD0_DATA/.qpkg/Entware-ng/bin/rsnapshot"
#
-----------------------------------------------------------------------------
# END OF CONFIG. Shouldn't need to change anything below here (unless you
want
# to add a new backup interval).
#
-----------------------------------------------------------------------------
#
-----------------------------------------------------------------------------
# Function to mount remote backup target on local dir.
#
-----------------------------------------------------------------------------
mountBackupTarget () {
if grep -qs "$MNT_POINT" /proc/mounts; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: $MNT_POINT is already
mounted"
return 0
else
echo "`date` pme_rsnapshot_wrapper.sh: INF: mounting $BKUP_TARGET on
$MNT_POINT"
/bin/mount -t nfs $BKUP_TARGET $MNT_POINT
return $?
fi
}
#
-----------------------------------------------------------------------------
# Function to UNmount backup target after backup finished.
#
-----------------------------------------------------------------------------
unmountBackupTarget () {
echo "`date` pme_rsnapshot_wrapper.sh: INF: unmounting $BKUP_TARGET"
/bin/umount $BKUP_TARGET
}
#
-----------------------------------------------------------------------------
# MAIN
#
-----------------------------------------------------------------------------
# Check num of parameters passed is correct (should be at least 1)
if [ $# -lt 1 ]; then
\$1=$1"
echo " usage: pme_rsnapshot_wrapper.sh
daily|weekly|monthly"
exit 1
# Execute mount command
elif [ "$1" == "daily" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot \"$1\" requested"
# Execute UNmount command
elif [ "$1" == "weekly" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot \"$1\" requested"
# Execute UNmount command
elif [ "$1" == "monthly" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot \"$1\" requested"
# Command line parameter was some other rubbish.
else
echo "`date` pme_rsnapshot_wrapper.sh: ERR: Incorrect param \"$1\""
echo " usage: pme_rsnapshot_wrapper.sh
daily|weekly|monthly"
exit 1
fi
#
-----------------------------------------------------------------------------
# Mount remote backup target and run rsnapshot if successful.
#
-----------------------------------------------------------------------------
mountBackupTarget
RETVAL=$? # Gets return value of the mount command.
if [ "$RETVAL" -eq 0 ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: $BKUP_TARGET mounted OK"
if [ "$2" == "DUMMYRUN" ]; then
echo "`date` pme_rsnapshot_wrapper.sh: INF: $2 so rsnapshot not
executed"
else
echo "`date` pme_rsnapshot_wrapper.sh: INF: running $RSNAP_EXE $1"
$RSNAP_EXE $1 # Run rsnapshot backup
fi
unmountBackupTarget # UNmount backup directory when rsnapshot completed.
else
echo "`date` pme_rsnapshot_wrapper.sh: ERR: $BKUP_TARGET mount failed
with code $RETVAL"
exit 1
fi
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data
untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Phill Edwards]

Post by Patrick O'Callaghan
I haven't really been following this thread but I have one question: are
you running rsnapshot on the NAS or on the client(s)? The fact that you
mount the NAS via NFS leads me to suspect the former. It's worth noting
that this is by far the worst (i.e. least efficient) way to do it. I used
to do it this way a few years ago because it seemed so natural, but in fact
rsnapshot works much better if you run it on the NAS and connect to the
client(s) via the rsync protocol. This is not very clear from the
documentation - I guess the author just assumed it was obvious - but when I
changed methods I saw a speedup by an order of magnitude.
The reason is that treating the NAS as if it were local means that the
receiving rsync (which is still running on the client machine!) is reading
huge quantities of data in order to check for modified files. Doing it the
other way means the receiver runs on the NAS and the network is only used
to send summaries and changed files, with the resulting boost in
efficiency. It also means you don't need NFS for backups (though it's still
useful for restores).

Hi Patrick, I'm not sure what you mean by "client" in this scenario. My
setup is as follows. I have a QNAP NAS where all our important files are
stored. In addition I have a D-Link NAS which I'm using as the target for
backups of the QNAP NAS, so if the QNAP blows up we have a backup copy of
everything on the D-Link.

I'm running rsnapshot on the QNAP NAS which backs up the data onto the
D-LInk NAS over NFS. I'm not sure which of these would be the client in
your terminology. But I have noticed that the backups from the QNAP to the
D-Link are quite slow so any suggestions on how to speed them up are
welcome.

[Patrick O'Callaghan]

Post by Phill Edwards

Post by Patrick O'Callaghan
I haven't really been following this thread but I have one question: are

you running rsnapshot on the NAS or on the client(s)? The fact that you
mount the NAS via NFS leads me to suspect the former. It's worth noting
that this is by far the worst (i.e. least efficient) way to do it. I used
to do it this way a few years ago because it seemed so natural, but in fact
rsnapshot works much better if you run it on the NAS and connect to the
client(s) via the rsync protocol. This is not very clear from the
documentation - I guess the author just assumed it was obvious - but when I
changed methods I saw a speedup by an order of magnitude.

Post by Phill Edwards

Post by Patrick O'Callaghan
The reason is that treating the NAS as if it were local means that the

receiving rsync (which is still running on the client machine!) is reading
huge quantities of data in order to check for modified files. Doing it the
other way means the receiver runs on the NAS and the network is only used
to send summaries and changed files, with the resulting boost in
efficiency. It also means you don't need NFS for backups (though it's still
useful for restores).

Post by Phill Edwards
Hi Patrick, I'm not sure what you mean by "client" in this scenario. My

setup is as follows. I have a QNAP NAS where all our important files are
stored. In addition I have a D-Link NAS which I'm using as the target for
backups of the QNAP NAS, so if the QNAP blows up we have a backup copy of
everything on the D-Link.

Post by Phill Edwards
I'm running rsnapshot on the QNAP NAS which backs up the data onto the

D-LInk NAS over NFS. I'm not sure which of these would be the client in
your terminology. But I have noticed that the backups from the QNAP to the
D-Link are quite slow so any suggestions on how to speed them up are
welcome.


In this context client means "machine whose files are being backed up" and
server is "machine where the backups are stored". So in your case the QNAP
is the client and the D-Link is the server.

Post by Phill Edwards
From what you say, running rsnapshot on the QNAP is what you don't want to

do. You should be running it on the D-Link. This takes a little more work
to set up but is very much worth it in terms of efficiency. You need to
configure the QNAP to allow rsync access. If you're concerned about
security, best do this as rsync over ssh, which means first of all setting
up ssh login from the D-Link to the QNAP. As you want to do this as a batch
process, the login should use token authentication, not passwords. Make
sure that any intervening firewalls (e.g. on the QNAP itself) will let ssh
through.

After that, it's a question of configuring your rsnapshot.conf file (on the
*server* side) to use rsync. As an example, I have lines like these (where
Bree is my client machine):

backup Bree::backup-home ./home
backup Bree::backup-etc ./etc
backup Bree::backup-local ./local
backup Bree::backup-opt ./opt
...
(note that this is slightly different from the default, as I use a separate
rsnapshot root for each possible client).

And on Bree itself, I have these lines in /etc/rsyncd.conf:

[backup-home]
path=/home

[backup-etc]
path=/etc

[backup-local]
path=/usr/local

[backup-opt]
path=/opt
...

The main headache getting this all to work was actually with ssh. For some
reason it kept losing the authentication token so rsnapshot would fail. I
finally figured out that the problem was with my DNS setup. The server
wasn't resolving Bree's name correctly :-)

Anyway, best of luck.

poc

[Phill Edwards]

Post by Patrick O'Callaghan
In this context client means "machine whose files are being backed up" and
server is "machine where the backups are stored". So in your case the QNAP
is the client and the D-Link is the server.
From what you say, running rsnapshot on the QNAP is what you don't want to
do. You should be running it on the D-Link. This takes a little more work
to set up but is very much worth it in terms of efficiency. You need to
configure the QNAP to allow rsync access. If you're concerned about
security, best do this as rsync over ssh, which means first of all setting
up ssh login from the D-Link to the QNAP. As you want to do this as a batch
process, the login should use token authentication, not passwords. Make
sure that any intervening firewalls (e.g. on the QNAP itself) will let ssh
through.
After that, it's a question of configuring your rsnapshot.conf file (on
the *server* side) to use rsync. As an example, I have lines like these
The main headache getting this all to work was actually with ssh. For some
reason it kept losing the authentication token so rsnapshot would fail. I
finally figured out that the problem was with my DNS setup. The server
wasn't resolving Bree's name correctly :-)

Thanks very much for this advice, Patrick. I'll give it a try. Hopefully
the limited linux O/S on my QNAP NAS won't prevent it from working.

Out of interest, any idea why "pulling" it from the client to the server
should be so much faster than "pushing" it from the client?

[Patrick O'Callaghan]

Post by Phill Edwards
Thanks very much for this advice, Patrick. I'll give it a try. Hopefully
the limited linux O/S on my QNAP NAS won't prevent it from working.
Out of interest, any idea why "pulling" it from the client to the server
should be so much faster than "pushing" it from the client?

As I said in an earlier message, it's because of how rsync works. There are
two rsync processes, one at each end of the connection. Call them the
source and the target processes, or S and T for short. On startup S sends a
list of candidate files to T. T checks which of them it doesn't already
have and asks S to send them. It also checks which of the files it already
has may have changed, firstly by looking at sizes and then at checksums.
The exact behaviour is controlled by rsync options but we'll take the
default case. When a file exists at both ends but is different, there is a
rolling checksum that tries to minimize the number of sections of file to
re-send in order to bring the T version up to date. (This is a very crude
summary but it's enough for now).

OK, what happens when we're doing this on NFS? Both S and T are running *on
the same machine* (the client). S is looking at local files and T at
NFS-mounted remote files, so for T to do any checksumming it has to read
the file contents over NFS. This is a big performance hit, which you have
to pay *even when the files are not actually different*. In other words, in
the NFS scenario the cost of rsnapshot is dominated not by the number of
changed files, but by the total size of the files you want to check.

On the other hand, using the rsync "pull" method, T and S are on different
machines (so potentially faster for that reason) and only files that
actually might be different will be copied. Big win.

Cheers

poc

[Phill Edwards]

Post by Phill Edwards
From what you say, running rsnapshot on the QNAP is what you don't want to

Post by Patrick O'Callaghan
do. You should be running it on the D-Link. This takes a little more work
to set up but is very much worth it in terms of efficiency. You need to
configure the QNAP to allow rsync access. If you're concerned about
security, best do this as rsync over ssh, which means first of all setting
up ssh login from the D-Link to the QNAP. As you want to do this as a batch
process, the login should use token authentication, not passwords. Make
sure that any intervening firewalls (e.g. on the QNAP itself) will let ssh
through.
After that, it's a question of configuring your rsnapshot.conf file (on
the *server* side) to use rsync.

I don't know if I set something up incorrectly here, by my experience of
this was totally different to your's. I changed my rsnapshot backup so that
the D-Link NAS2 runs rsnapshot and "pulls" the files from the QNAP NAS over
a CIFS connection. Unfortunately this was massively slower than my original
setup - more than 3 times slower. I took some timings:

New method - D-Link NAS pulls from QNAP over CIFS - 12.3GB home
directory tree for one user - 145 minutes.
Original method - QNAP NAS pushes to D-Link NAS over NFS - 12.3GB home
directory tree for one user - 44 minutes.

As you can see from these timings, mounting the remote D-Link NAS over NFS
and pushing files to it from the QNAP was a LOT quicker.

As I say, I don't know whether I set something up incorrectly, but I don't
think so. Interestingly the QNAP TS-410 has a slower CPU (ARM Marvell
800MHz) than the D-Link DNS-320L (ARM Marvell 1000MHz) so you would have
thought it would run rsnapshot/rsync slower. They both have 256MB RAM.

[Patrick O'Callaghan]

I changed my rsnapshot backup so that the D-Link NAS2 runs rsnapshot and
"pulls" the files from the QNAP NAS over a CIFS connection

I suspect you didn't entirely understand the reasoning behind this. Who
said anything about CIFS? That's just mounting the filesystems in reverse.
I'm not surprised it didn't work well. The whole point is to back up using
the rsync protocol *and not a remote filesystem*.

poc

[Phill Edwards]

Post by Patrick O'Callaghan

I changed my rsnapshot backup so that the D-Link NAS2 runs rsnapshot and

"pulls" the files from the QNAP NAS over a CIFS connection

Post by Patrick O'Callaghan
I suspect you didn't entirely understand the reasoning behind this. Who

said anything about CIFS? That's just mounting the filesystems in reverse.
I'm not surprised it didn't work well. The whole point is to back up using
the rsync protocol *and not a remote filesystem*.
Sorry, I muddled up SSH and CIFS. The remote D-Link NAS connected to the
QNAP by SSH. There was no CIFS involved. So that means rsync was running on
the QNAP, and maybe it ran very slowly?

[Patrick O'Callaghan]

Post by Patrick O'Callaghan

I changed my rsnapshot backup so that the D-Link NAS2 runs rsnapshot

and "pulls" the files from the QNAP NAS over a CIFS connection

Post by Patrick O'Callaghan
I suspect you didn't entirely understand the reasoning behind this. Who

said anything about CIFS? That's just mounting the filesystems in reverse.
I'm not surprised it didn't work well. The whole point is to back up using
the rsync protocol *and not a remote filesystem*.
Sorry, I muddled up SSH and CIFS. The remote D-Link NAS connected to the
QNAP by SSH. There was no CIFS involved. So that means rsync was running on
the QNAP, and maybe it ran very slowly?

In fact rsync is running on both machines. That's how it works. My advice
was to run rsnapshot on the backup server and configure it to run rsync
(optionally over ssh) to the client. It does this by running rsync at both
ends, the two processes cooperating to minimize transfers.

It's possible that the backup server (i.e. the D-Link in your case) is just
a slow machine and the ssh processing is taking too long (unlikely), or
perhaps it has slow disk drives. That could happen if its disks are on USB
2 ports for example, but of course the same would apply in your original
configuration. So I don't really understand what's happening.

In my case the backup server is an Iomega NAS, around 5 years old now, with
internal drives in a raid configuration. It's certainly not a fast machine
and as I said earlier I saw an order of magnitude speedup by changing
rsnapshot to pull with rsync rather than push over NFS.

poc

[Phill Edwards]

Post by Patrick O'Callaghan
In fact rsync is running on both machines. That's how it works. My advice
was to run rsnapshot on the backup server and configure it to run rsync
(optionally over ssh) to the client. It does this by running rsync at both
ends, the two processes cooperating to minimize transfers.
It's possible that the backup server (i.e. the D-Link in your case) is
just a slow machine and the ssh processing is taking too long (unlikely),
or perhaps it has slow disk drives. That could happen if its disks are on
USB 2 ports for example, but of course the same would apply in your
original configuration. So I don't really understand what's happening.
In my case the backup server is an Iomega NAS, around 5 years old now,
with internal drives in a raid configuration. It's certainly not a fast
machine and as I said earlier I saw an order of magnitude speedup by
changing rsnapshot to pull with rsync rather than push over NFS.

Thanks Patrick. I guess for anyone looking at this in future the
recommendation would be to try both options and see what works best for
their particular setup.

Appreciate all your help on this and it's been a good learning exercise for
me.

[Nico Kadel-Garcia]

Post by Phill Edwards

Post by Patrick O'Callaghan
In fact rsync is running on both machines. That's how it works. My advice
was to run rsnapshot on the backup server and configure it to run rsync
(optionally over ssh) to the client. It does this by running rsync at both
ends, the two processes cooperating to minimize transfers.
It's possible that the backup server (i.e. the D-Link in your case) is
just a slow machine and the ssh processing is taking too long (unlikely), or
perhaps it has slow disk drives. That could happen if its disks are on USB 2
ports for example, but of course the same would apply in your original
configuration. So I don't really understand what's happening.
In my case the backup server is an Iomega NAS, around 5 years old now,
with internal drives in a raid configuration. It's certainly not a fast
machine and as I said earlier I saw an order of magnitude speedup by
changing rsnapshot to pull with rsync rather than push over NFS.

Thanks Patrick. I guess for anyone looking at this in future the
recommendation would be to try both options and see what works best for
their particular setup.

Synology servers are rataher limited, proprietized environment. The
one I've looked at recently does not even have "perl" available to the
running operating system. It uses an ext4 backend filesystem, which is
very stable, but does have limits with excessive numbers of files in
single directories or filesystems that are too deep. It's LVM mounted,
but that allows only one LVM snapshot maximum, not multiple local
filesystem snapshots.

If one must do rsnapshot backups to a Synology server for reasons of
bulk storage and accessible backups, I'd strongly suggest using NFS
mounts, and limiting the number of "hourly" or most frequently
performed backups to control the burden of the hardlinks. NFS is much,
much less chatty than CIFS, and is likely to perform far better. It
almost always does for any production environment. The big advantage
of CIFS is the availability of the complex permission models only
available to NFSv4 for NFS environments, and those are not normally
that useful.

Post by Phill Edwards
Appreciate all your help on this and it's been a good learning exercise for
me.
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Patrick O'Callaghan]

Post by Patrick O'Callaghan
In fact rsync is running on both machines. That's how it works. My advice

Post by Patrick O'Callaghan
was to run rsnapshot on the backup server and configure it to run rsync
(optionally over ssh) to the client. It does this by running rsync at both
ends, the two processes cooperating to minimize transfers.
It's possible that the backup server (i.e. the D-Link in your case) is
just a slow machine and the ssh processing is taking too long (unlikely),
or perhaps it has slow disk drives. That could happen if its disks are on
USB 2 ports for example, but of course the same would apply in your
original configuration. So I don't really understand what's happening.
In my case the backup server is an Iomega NAS, around 5 years old now,
with internal drives in a raid configuration. It's certainly not a fast
machine and as I said earlier I saw an order of magnitude speedup by
changing rsnapshot to pull with rsync rather than push over NFS.

Thanks Patrick. I guess for anyone looking at this in future the
recommendation would be to try both options and see what works best for
their particular setup.
Appreciate all your help on this and it's been a good learning exercise
for me.

Happy to help, even though I didn't in the end :-)

Cheers

poc