[rsnapshot-discuss] mirror a rsnapshot backups

Discussion:

Charles Steaderman

2008-06-13 17:32:37 UTC

Permalink

--
Charlie Steaderman
***@poliac.com
VP Engineering
Poliac Research Corporation

Ahmed Kamal

2008-06-13 18:07:05 UTC

Permalink

AFAIK, using rsync-3 helps a lot with -H option

But better yet. Do not try to replicate the whole rsnapshot tree. Instead,
replicate say daily.1 offsite today, in a few weeks/months, the replicated
data will build up to be a replica of what's on your side. The benefit of
this approach is that rsnapshot is only needed to work on a small
subdirectory instead of scanning the whole huge tree. I have this setup on
my side and it's working great.

Regards

I am happily using rsnapshot 1.2.9 on a freebsd based system (FreeNAS). I
would like to mirror my rsnapshot my rsnapshot backups to another machine
for redundancy. Ideally, it would be on a remote machine across a secure
connection. I have about 189GB of backups spanning about 6 months. I have
been trying to us rsync to mirror the data across systems but I am have
great difficulty getting past rsync's use of large amounts of RAM when
copying data comprised of hard links. The problem is that the largest data
size that I can set for a process if 2GB and rsync wants more than that in
order to sync the data.
/usr/local/bin/rsync -aSH --stats -e 'ssh -ax' --delete-after
[server]:/mnt/data/.snapshots /mnt/remote/rsync
Any ideas short of moving to a 64bit system with 16GB of memory just to act
as a backup file server?
- Charlie
--
Charlie Steaderman
VP Engineering
Poliac Research Corporation
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

David Cantrell

2008-06-17 13:49:08 UTC

Permalink

Post by Charles Steaderman
I am happily using rsnapshot 1.2.9 on a freebsd based system (FreeNAS).
I would like to mirror my rsnapshot my rsnapshot backups to another
machine for redundancy...
The problem is that
the largest data size that I can set for a process if 2GB and rsync
wants more than that in order to sync the data.
/usr/local/bin/rsync -aSH ...

-H is a memory pig.

Post by Charles Steaderman
Any ideas short of moving to a 64bit system with 16GB of memory just to
act as a backup file server?

Use the rsnapshot-copy script that is in CVS to do the initial sync:
http://shorterlink.org/4144

Then run rsnapshot on the second machine and use it to backup the most
recent snapshot on the first machine, so that the two stay in sync.
Obviously you need to arrange matters so that the second machine always
backs up the first after the first has finished.

--
David Cantrell | Enforcer, South London Linguistic Massive

PERL: Politely Expressed Racoon Love

Charles Steaderman

2008-06-27 14:58:39 UTC

Permalink

David Cantrell

2008-06-27 16:03:25 UTC

Permalink

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
...

Please don't send HTML mail.

Thanks for your suggestion and a link to the rsnapshot-copy script. I
am trying to use it but I am having a problem. The script stops when it
rsync: read errors mapping ...: Input/Output error (5)
If I try to 'cat' the file on the rsnapshot server, it gets stuck,
indicating to me that the file is surely corrupted. I have fsck'd the
drive (freebsd UFS) and there are no errors.

fsck only checks the logical structure of the filesystem, it doesn't
check that all the files can be read. It sounds like you've got a bad
sector on the disk, which can't be read.

Unfortunately because rsnapshot uses hard links that also means that all
the other "copies" of that file in your backups will also be unreadable.
Hopefully you've caught this soon enough that you don't have any other
bad sectors on the disk and so can copy the rest of your data off it
before throwing it away!

It's for this reason that I periodically (once a month) run a little
script that reads every single file in my most recent backup, just to
make sure they *can* be read, as well as doing a test restore of a few
randomly selected files.

--
David Cantrell | A machine for turning tea into grumpiness

EIN KIRCHE! EIN KREDO! EIN PAPST!

Charles Marcus

2008-06-27 16:47:06 UTC

Permalink

Post by David Cantrell
It's for this reason that I periodically (once a month) run a little
script that reads every single file in my most recent backup, just to
make sure they *can* be read, as well as doing a test restore of a few
randomly selected files.

Interesting... care to share it? I'm a real klutz when it comes to
scripting, but I can usually adapt a working one to my needs...

Thanks!

--
Best regards,

Charles

David Cantrell

2008-06-27 20:35:21 UTC

Permalink

Post by Charles Marcus

Post by David Cantrell
It's for this reason that I periodically (once a month) run a little
script that reads every single file in my most recent backup, ...

Interesting... care to share it? I'm a real klutz when it comes to
scripting, but I can usually adapt a working one to my needs...

#!/usr/local/bin/perl

use File::Find::Rule;

# enable "slurp" mode
$/ = undef;

# find all the files under /mnt/backups/daily.0
foreach(File::Find::Rule->file()->in('/mnt/backups/daily.0')) {
# unset the "there was an error" variable
$! = '';
# try to open this file, and if it succeeds, read the whole thing,
# ignoring its contents
open(F, $_) && <F>;
# if anything went wrong, whinge
print "$_: $!\n" if($!);
close(F);
}

You'll need to also install the File::Find::Rule module from the CPAN:

$ sudo perl -MCPAN -e 'install "File::Find::Rule"'

Note that it should also whinge about files that you don't have
permission to read, so make sure to run it as root, and it doesn't
attempt to follow symlinks or read FIFOs, sockets or devices. That
generally doesn't matter for rsnapshot backups, but is something to bear
in mind if you decide to use it elsewhere.

I presume that something like this would work too:

find /mnt/backups/daily.0 -type f -print0|xargs -0 -n 1 cat>/dev/null

Assuming, of course, that cat reports errors to STDERR and that it
includes the filename in the report.

--
David Cantrell | Reality Engineer, Ministry of Information

Googling for "Swedish lizard overlord" gives
this sig as the first result.

WHAT DOES THAT TELL YOU, EH? EH?

Charles Marcus

2008-06-27 20:37:36 UTC

Permalink

Post by David Cantrell

Post by Charles Marcus

Post by David Cantrell
It's for this reason that I periodically (once a month) run a little
script that reads every single file in my most recent backup, ...

Interesting... care to share it? I'm a real klutz when it comes to
scripting, but I can usually adapt a working one to my needs...

#!/usr/local/bin/perl

Cool! Thanks a lot...

--
Best regards,

Charles

Clemens Feige

2008-06-30 15:16:41 UTC

Permalink

... but it appears that it
will never recover since rsync skips updating a new version of the file since
there exists a corrupted version that it cannot sync. This makes my very
suspicious of my backups in general in that a corrupted file (for whatever
reason) on the backup server will never be corrected and I don't now know how
many 'bad' files there may be. Are there options in rsnapshot/rsync that I could
use to force a copy of the source file if rsync has a problem with the
destination file?

Hi Charles.

Have you considered using the --checksum option for rsync?

This way rsync computes checksum for the original file and the backup
file and compares them.

Sure, this means more computing load to your system, but it should
immediately show up in the log file if there are errors with some files;
shouldn't it?

Clemens

Charles Steaderman

2008-11-05 18:55:24 UTC

Permalink

David,
I have acquired rsnapshot-copy and have successfully copied my snapshots
from one machine to another. I am now having a problem figuring out how
to setup rsnapshot to synchronize the most recent snapshot.

Setup
Primary machine:
o 7 Daily snapshots rolled over to 'n' Weekly snapshots
o Data stored at /mnt/data/.snapshots/

Secondary machine:
o rsnapshot-copy data stored at /mnt/external/.snapshots/

On the Secondary machine I setup an rsnapshot.conf file with

snapshot_root /mnt/external/.snapshots/

and tried to rsnapshot data from the Primary machine with

backup ***@primary:/mnt/data/.snapshots/daily.0 ./

but rsnapshot simply created
/mnt/external/.snapshots/mnt/data/.snapshots/daily.0 on the Secondary
machine. What am I missing? How can I get rsnapshot to ignore the path
of the backup source?

- Charlie

Post by David Cantrell

-H is a memory pig.

Post by Charles Steaderman
Any ideas short of moving to a 64bit system with 16GB of memory just to
act as a backup file server?

http://shorterlink.org/4144
Then run rsnapshot on the second machine and use it to backup the most
recent snapshot on the first machine, so that the two stay in sync.
Obviously you need to arrange matters so that the second machine always
backs up the first after the first has finished.

--
Charlie Steaderman
***@poliac.com
VP Engineering
Poliac Research Corporation
Phone: 952.707.6245
Cell: 612.242.6364

Matt McCutchen

2008-11-06 03:49:39 UTC

Permalink

Post by Charles Steaderman
On the Secondary machine I setup an rsnapshot.conf file with
snapshot_root /mnt/external/.snapshots/
and tried to rsnapshot data from the Primary machine with
but rsnapshot simply created
/mnt/external/.snapshots/mnt/data/.snapshots/daily.0 on the Secondary
machine. What am I missing? How can I get rsnapshot to ignore the path
of the backup source?

The default rsync_long_args parameter contains the --relative option,
which makes rsync include the source path. If you don't want that,
specify an rsync_long_args that does not contain --relative, e.g.:

rsync_long_args --delete --numeric-ids --delete-excluded

Matt

Charles Steaderman

2008-11-06 18:21:48 UTC

Permalink

Matt,
Thanks for the tip. Works perfectly!

- Charlie

Post by Matt McCutchen

The default rsync_long_args parameter contains the --relative option,
which makes rsync include the source path. If you don't want that,
rsync_long_args --delete --numeric-ids --delete-excluded
Matt
!DSPAM:4912695b164758362916074!

--
Charlie Steaderman
***@poliac.com
VP Engineering
Poliac Research Corporation
Phone: 952.707.6245
Cell: 612.242.6364