[rsnapshot-discuss] Question About Moving a Backup Directory

[Mark Phillips]

I have been using rsnapshot for awhile to backup two computers. The backup
drive is getting full. The backups for computer A are very large, and the
backup for computer B are much smaller. I have daily (0-6), weekly (0-4),
and monthly (0-4) backups. Under each of these directories, I have
directories A and B for the two computers. Eg the file structure is:

backup
daily.0
A
home
etc
B
home
etc
daily.1
A
B
......

I was wondering if I can do the following without messing up the backups I
have.

1. Copy all the backup files to a new larger hard drive (eg cp -a to
preserve links and file attributes).

2. Delete all the computer B directories from the new drive.

3. Delete all the computer A directories from the old drive.

4. Create a new rsnapshot.conf file that points to the new drive for the
computer A backups and just backs up computer A.

5. Remove the computer A backup information from the existing rsnapshot.
conf file.

6. Add cron new daily, weekly, and monthly entries for computer A using the
-c option for the new rsnapshot.conf file. Change the cron entries for
computer B to use the old rsnapshot.conf file with the -c option.

I would expect the the backups will continue without an issues - computer A
is backed up to the new larger drive, and computer B is backed up to the
old drive which now has a lot more free space.

Will this plan work? I assume there are not shared links between the
backups for the two computers. Do I need to verify this somehow?

Are there other cp options I need to use for this move?

Thanks,

Mark

[Mark Phillips]

Forgot to add that I am using rsnapshot version 1.4.1 from Debian testing.

Mark

Post by Mark Phillips
I have been using rsnapshot for awhile to backup two computers. The backup
drive is getting full. The backups for computer A are very large, and the
backup for computer B are much smaller. I have daily (0-6), weekly (0-4),
and monthly (0-4) backups. Under each of these directories, I have
backup
daily.0
A
home
etc
B
home
etc
daily.1
A
B
......
I was wondering if I can do the following without messing up the backups I
have.
1. Copy all the backup files to a new larger hard drive (eg cp -a to
preserve links and file attributes).
2. Delete all the computer B directories from the new drive.
3. Delete all the computer A directories from the old drive.
4. Create a new rsnapshot.conf file that points to the new drive for the
computer A backups and just backs up computer A.
5. Remove the computer A backup information from the existing rsnapshot.
conf file.
6. Add cron new daily, weekly, and monthly entries for computer A using
the -c option for the new rsnapshot.conf file. Change the cron entries for
computer B to use the old rsnapshot.conf file with the -c option.
I would expect the the backups will continue without an issues - computer
A is backed up to the new larger drive, and computer B is backed up to the
old drive which now has a lot more free space.
Will this plan work? I assume there are not shared links between the
backups for the two computers. Do I need to verify this somehow?
Are there other cp options I need to use for this move?
Thanks,
Mark

[Scott Hess]

I believe your plan would work.

As a general rule, I've generally found it easier to just get a drive big
enough for everything, copy everything over, then retire the old drive to a
drawer for a couple months. That reduces my need to stress out over
whether I'm doing something wrong and unrecoverable.

Along those lines, you can just leave the old backups in place until you've
convinced yourself they can be deleted. I would circle back and delete
them (carefully!), because otherwise you might later find yourself
wondering which data is currently valid, or making a mistake because you
think you're on a different system (can you tell I've been there?).

cp -a should work, but my rule of thumb is to use an rsync command-line
derived from rsnapshot.log. That means I don't have to worry about
screwing rsnapshot up with some minor mistake. Also, if/when I need to
interrupt the copy, I can do it safely without worrying about whether
restarting it will screw something up. [Also, if you're careful you can
start the backup after your weekly, excluding your shorter-term backups,
then once that's done do another pass covering everything (including
dailies) using the last weekly as --link-dest. But if you go this route,
test your planned rsync calls ahead of time on a subset, it's easy to get
this wrong.]

Note that you can use an include directive to share config between the
more-specific configuration files.

Good luck, and be careful,
scott

Post by Mark Phillips
I have been using rsnapshot for awhile to backup two computers. The backup
drive is getting full. The backups for computer A are very large, and the
backup for computer B are much smaller. I have daily (0-6), weekly (0-4),
and monthly (0-4) backups. Under each of these directories, I have
backup
daily.0
A
home
etc
B
home
etc
daily.1
A
B
......
I was wondering if I can do the following without messing up the backups I
have.
1. Copy all the backup files to a new larger hard drive (eg cp -a to
preserve links and file attributes).
2. Delete all the computer B directories from the new drive.
3. Delete all the computer A directories from the old drive.
4. Create a new rsnapshot.conf file that points to the new drive for the
computer A backups and just backs up computer A.
5. Remove the computer A backup information from the existing rsnapshot.
conf file.
6. Add cron new daily, weekly, and monthly entries for computer A using
the -c option for the new rsnapshot.conf file. Change the cron entries for
computer B to use the old rsnapshot.conf file with the -c option.
I would expect the the backups will continue without an issues - computer
A is backed up to the new larger drive, and computer B is backed up to the
old drive which now has a lot more free space.
Will this plan work? I assume there are not shared links between the
backups for the two computers. Do I need to verify this somehow?
Are there other cp options I need to use for this move?
Thanks,
Mark
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Tapani Tarvainen]

Post by Scott Hess
I believe your plan would work.

Yes, it should, it just takes a bit of care.

Post by Scott Hess
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months.

That's a good strategy as long as your backups fit on a single disk.

I've long ago started using LVM for more or less everything, it makes
it easy to add new disks and remove old ones, without even booting the
machine if you've got hotswap disk slots.

[Mark Phillips]

Scott,

Thanks for a great idea! I will get two new drives instead of one. That way
I can pull the old drive out and, as you said, "put it in a drawer". Then
change the rsnapshot config files to use one drive for computer A and one
for computer B. As the computer A drive fills up (~year), I can swap out
the drive in the drawer (after erasing it as I don't need the backups back
that far) and keep the older drive A backups in a drawer for another year.
I can keep swapping out drives as needed. No need to copy any files, worry
about deleting the wrong stuff, try out new rsync commands, etc. Brilliant!

Thanks again!

Mark

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a drive big
enough for everything, copy everything over, then retire the old drive to a
drawer for a couple months. That reduces my need to stress out over
whether I'm doing something wrong and unrecoverable.
Along those lines, you can just leave the old backups in place until
you've convinced yourself they can be deleted. I would circle back and
delete them (carefully!), because otherwise you might later find yourself
wondering which data is currently valid, or making a mistake because you
think you're on a different system (can you tell I've been there?).
cp -a should work, but my rule of thumb is to use an rsync command-line
derived from rsnapshot.log. That means I don't have to worry about
screwing rsnapshot up with some minor mistake. Also, if/when I need to
interrupt the copy, I can do it safely without worrying about whether
restarting it will screw something up. [Also, if you're careful you can
start the backup after your weekly, excluding your shorter-term backups,
then once that's done do another pass covering everything (including
dailies) using the last weekly as --link-dest. But if you go this route,
test your planned rsync calls ahead of time on a subset, it's easy to get
this wrong.]
Note that you can use an include directive to share config between the
more-specific configuration files.
Good luck, and be careful,
scott
On Fri, Oct 30, 2015 at 10:09 AM, Mark Phillips <

Post by Mark Phillips
I have been using rsnapshot for awhile to backup two computers. The
backup drive is getting full. The backups for computer A are very large,
and the backup for computer B are much smaller. I have daily (0-6), weekly
(0-4), and monthly (0-4) backups. Under each of these directories, I have
backup
daily.0
A
home
etc
B
home
etc
daily.1
A
B
......
I was wondering if I can do the following without messing up the backups
I have.
1. Copy all the backup files to a new larger hard drive (eg cp -a to
preserve links and file attributes).
2. Delete all the computer B directories from the new drive.
3. Delete all the computer A directories from the old drive.
4. Create a new rsnapshot.conf file that points to the new drive for the
computer A backups and just backs up computer A.
5. Remove the computer A backup information from the existing rsnapshot.
conf file.
6. Add cron new daily, weekly, and monthly entries for computer A using
the -c option for the new rsnapshot.conf file. Change the cron entries for
computer B to use the old rsnapshot.conf file with the -c option.
I would expect the the backups will continue without an issues - computer
A is backed up to the new larger drive, and computer B is backed up to the
old drive which now has a lot more free space.
Will this plan work? I assume there are not shared links between the
backups for the two computers. Do I need to verify this somehow?
Are there other cp options I need to use for this move?
Thanks,
Mark
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Helmut Hullen]

Hallo, Scott,

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a drive
big enough for everything, copy everything over, then retire the old
drive to a drawer for a couple months. That reduces my need to
stress out overwhether I'm doing something wrong and unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1 disk.
And since about 4 weekse this disk has errors ...

Viele Gruesse!
Helmut


------------------------------------------------------------------------------

[Scott Hess]

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a drive
big enough for everything, copy everything over, then retire the old
drive to a drawer for a couple months. That reduces my need to
stress out overwhether I'm doing something wrong and unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1 disk.
And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one of
the disks with have errors. With one disk, you have a higher chance that
the disk with errors contains all of your backups.

-scott

[Tapani Tarvainen]

Post by Helmut Hullen
Just for the record : I have put the backups of 3 machines onto 1 disk.
And since about 4 weekse this disk has errors ...

I've seen enough disks fail that I use some RAID setup for all
important data, including backups (nowadays RAID1, RAID6 or RAID10).
Disks are cheap, data is expensive.

[Helmut Hullen]

Hallo, Scott,

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months. That reduces my
need to stress out overwhether I'm doing something wrong and
unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one
of the disks with have errors. With one disk, you have a higher
chance that the disk with errors contains all of your backups.

Yes - I know ...

But i hoped it doesn't happen just here ...

Viele Gruesse!
Helmut


------------------------------------------------------------------------------

[Scott Hess]

Post by Helmut Hullen

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months. That reduces my
need to stress out overwhether I'm doing something wrong and
unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one
of the disks with have errors. With one disk, you have a higher
chance that the disk with errors contains all of your backups.

Yes - I know ...
But i hoped it doesn't happen just here ...

Where I ended up on this is that I keep a mirror of my backups on a
separate disk (actually two, one rotated offsite). Why I went with that
rather than a RAID-1 is because with my previous RAID-1 setup, I found that
the biggest danger was me doing something stupid, not the hardware going
south (once, I did something stupid because I thought the hardware was
going south).

For my needs, though, having a reliable backup from the relatively recent
past is much more important than minimizing downtime of the backup server.
I'm not happy if the server is out of service for a few days or a week, but
it wouldn't have actual concrete consequences. If I were doing this as
part of an IT group where other people depended on it, I'd likely make
different decisions!

-scott

[Mark Phillips]

I am sure everyone has seen this about hard drive failure rates -
https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/

If you haven't, it is very interesting.

Mark

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months. That reduces my
need to stress out overwhether I'm doing something wrong and
unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one
of the disks with have errors. With one disk, you have a higher
chance that the disk with errors contains all of your backups.

Yes - I know ...
But i hoped it doesn't happen just here ...

Where I ended up on this is that I keep a mirror of my backups on a
separate disk (actually two, one rotated offsite). Why I went with that
rather than a RAID-1 is because with my previous RAID-1 setup, I found that
the biggest danger was me doing something stupid, not the hardware going
south (once, I did something stupid because I thought the hardware was
going south).
For my needs, though, having a reliable backup from the relatively recent
past is much more important than minimizing downtime of the backup server.
I'm not happy if the server is out of service for a few days or a week, but
it wouldn't have actual concrete consequences. If I were doing this as
part of an IT group where other people depended on it, I'd likely make
different decisions!
-scott
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Rasmus Borup Hansen]

There's also a version for Q3:

https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/ <https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/>

Best,

Rasmus

Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.
-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
E: ***@intomics.com W: http://www.intomics.com/
P: [phone] P: [phone]

I am sure everyone has seen this about hard drive failure rates - https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/ <https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/>
If you haven't, it is very interesting.
Mark

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months. That reduces my
need to stress out overwhether I'm doing something wrong and
unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one
of the disks with have errors. With one disk, you have a higher
chance that the disk with errors contains all of your backups.

Yes - I know ...
But i hoped it doesn't happen just here ...
Where I ended up on this is that I keep a mirror of my backups on a separate disk (actually two, one rotated offsite). Why I went with that rather than a RAID-1 is because with my previous RAID-1 setup, I found that the biggest danger was me doing something stupid, not the hardware going south (once, I did something stupid because I thought the hardware was going south).
For my needs, though, having a reliable backup from the relatively recent past is much more important than minimizing downtime of the backup server. I'm not happy if the server is out of service for a few days or a week, but it wouldn't have actual concrete consequences. If I were doing this as part of an IT group where other people depended on it, I'd likely make different decisions!
-scott
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss <https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss>
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Mark Phillips]

A quick update to my original post.

I tried backing up my rsnapshot directory on my backup drive to a new drive
an it failed using rsync but succeeded using cp -a.

The details:
* backup drive - 2 TB ext4, 16GB free, 360 GB of old backups not related to
rsnapshot, and the rest in a rsnapshot directory.
* new drive - 3 TB ext4 empty.
* both are usb 3.0 drives
* I ran the copies with both drives plugged into separate USB 3.0 drives on
a Ubuntu 14.04 LTS machine
* the rsync command - sudo rsync -aP
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The rsync command was interrupted. Had to go out of town, so stopped
rsync, disconnected the drives, and took the laptop to Palo Alto. When I
came back, I restarted the same command and then many hours later it
failed.
* the cp cpmmand - sudo cp -a
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The cp command ran all night and never complained.

The rsync command failed with disk full error. And sure enough, the 3 TB
was completely full. I erased the drive and ran the cp command, and it
completed. df -h on the backup drive reveals:

Filesystem Size Used Avail Use% Mounted on
/dev/sdc1 2.7T 1.4T 1.3T 53%
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2

Which is what I would expect. I browsed the backup drive and it looks OK.

Any thoughts on why rsync failed? Something with my options or the fact I
had to interrupt the sync?

Thanks,

Mark

Post by Rasmus Borup Hansen
https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/
Best,
Rasmus
*Intomics is a contract research organization specialized in deriving core
biological insight from large scale data. We help our clients in the
pharmaceutical industry develop tomorrow's medicines better, faster, and
cheaper through optimized use of biomedical data.*
-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
P: [phone] P: [phone]
I am sure everyone has seen this about hard drive failure rates -
https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/
If you haven't, it is very interesting.
Mark

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months. That reduces my
need to stress out overwhether I'm doing something wrong and
unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one
of the disks with have errors. With one disk, you have a higher
chance that the disk with errors contains all of your backups.

Yes - I know ...
But i hoped it doesn't happen just here ...

Where I ended up on this is that I keep a mirror of my backups on a
separate disk (actually two, one rotated offsite). Why I went with that
rather than a RAID-1 is because with my previous RAID-1 setup, I found that
the biggest danger was me doing something stupid, not the hardware going
south (once, I did something stupid because I thought the hardware was
going south).
For my needs, though, having a reliable backup from the relatively recent
past is much more important than minimizing downtime of the backup server.
I'm not happy if the server is out of service for a few days or a week, but
it wouldn't have actual concrete consequences. If I were doing this as
part of an IT group where other people depended on it, I'd likely make
different decisions!
-scott
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Scott Hess]

You probably wanted -H on the rsync command:
-H, --hard-links preserve hard links
because:
-a, --archive archive mode; equals -rlptgoD (no
-H,-A,-X)

My rsync commands end up like:
rsync -axHSW --delete --numeric-ids

-scott

Post by Mark Phillips
A quick update to my original post.
I tried backing up my rsnapshot directory on my backup drive to a new
drive an it failed using rsync but succeeded using cp -a.
* backup drive - 2 TB ext4, 16GB free, 360 GB of old backups not related
to rsnapshot, and the rest in a rsnapshot directory.
* new drive - 3 TB ext4 empty.
* both are usb 3.0 drives
* I ran the copies with both drives plugged into separate USB 3.0 drives
on a Ubuntu 14.04 LTS machine
* the rsync command - sudo rsync -aP
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The rsync command was interrupted. Had to go out of town, so stopped
rsync, disconnected the drives, and took the laptop to Palo Alto. When I
came back, I restarted the same command and then many hours later it
failed.
* the cp cpmmand - sudo cp -a
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The cp command ran all night and never complained.
The rsync command failed with disk full error. And sure enough, the 3 TB
was completely full. I erased the drive and ran the cp command, and it
Filesystem Size Used Avail Use% Mounted on
/dev/sdc1 2.7T 1.4T 1.3T 53%
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2
Which is what I would expect. I browsed the backup drive and it looks OK.
Any thoughts on why rsync failed? Something with my options or the fact I
had to interrupt the sync?
Thanks,
Mark

Post by Rasmus Borup Hansen
https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/
Best,
Rasmus
*Intomics is a contract research organization specialized in deriving
core biological insight from large scale data. We help our clients in the
pharmaceutical industry develop tomorrow's medicines better, faster, and
cheaper through optimized use of biomedical data.*
-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
P: [phone] P: [phone]
I am sure everyone has seen this about hard drive failure rates -
https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/
If you haven't, it is very interesting.
Mark

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months. That reduces my
need to stress out overwhether I'm doing something wrong and
unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one
of the disks with have errors. With one disk, you have a higher
chance that the disk with errors contains all of your backups.

Yes - I know ...
But i hoped it doesn't happen just here ...

Where I ended up on this is that I keep a mirror of my backups on a
separate disk (actually two, one rotated offsite). Why I went with that
rather than a RAID-1 is because with my previous RAID-1 setup, I found that
the biggest danger was me doing something stupid, not the hardware going
south (once, I did something stupid because I thought the hardware was
going south).
For my needs, though, having a reliable backup from the relatively
recent past is much more important than minimizing downtime of the backup
server. I'm not happy if the server is out of service for a few days or a
week, but it wouldn't have actual concrete consequences. If I were doing
this as part of an IT group where other people depended on it, I'd likely
make different decisions!
-scott
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Mark Phillips]

Thanks!

Mark

Post by Scott Hess
-H, --hard-links preserve hard links
-a, --archive archive mode; equals -rlptgoD (no
-H,-A,-X)
rsync -axHSW --delete --numeric-ids
-scott

Post by Mark Phillips
A quick update to my original post.
I tried backing up my rsnapshot directory on my backup drive to a new
drive an it failed using rsync but succeeded using cp -a.
* backup drive - 2 TB ext4, 16GB free, 360 GB of old backups not related
to rsnapshot, and the rest in a rsnapshot directory.
* new drive - 3 TB ext4 empty.
* both are usb 3.0 drives
* I ran the copies with both drives plugged into separate USB 3.0 drives
on a Ubuntu 14.04 LTS machine
* the rsync command - sudo rsync -aP
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The rsync command was interrupted. Had to go out of town, so stopped
rsync, disconnected the drives, and took the laptop to Palo Alto. When I
came back, I restarted the same command and then many hours later it
failed.
* the cp cpmmand - sudo cp -a
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The cp command ran all night and never complained.
The rsync command failed with disk full error. And sure enough, the 3 TB
was completely full. I erased the drive and ran the cp command, and it
Filesystem Size Used Avail Use% Mounted on
/dev/sdc1 2.7T 1.4T 1.3T 53%
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2
Which is what I would expect. I browsed the backup drive and it looks OK.
Any thoughts on why rsync failed? Something with my options or the fact I
had to interrupt the sync?
Thanks,
Mark

Post by Rasmus Borup Hansen
https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/
Best,
Rasmus
*Intomics is a contract research organization specialized in deriving
core biological insight from large scale data. We help our clients in the
pharmaceutical industry develop tomorrow's medicines better, faster, and
cheaper through optimized use of biomedical data.*
-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
P: [phone] P: [phone]
I am sure everyone has seen this about hard drive failure rates -
https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/
If you haven't, it is very interesting.
Mark

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess
I believe your plan would work.
As a general rule, I've generally found it easier to just get a
drive big enough for everything, copy everything over, then retire
the old drive to a drawer for a couple months. That reduces my
need to stress out overwhether I'm doing something wrong and
unrecoverable.

Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that one
of the disks with have errors. With one disk, you have a higher
chance that the disk with errors contains all of your backups.

Yes - I know ...
But i hoped it doesn't happen just here ...

Where I ended up on this is that I keep a mirror of my backups on a
separate disk (actually two, one rotated offsite). Why I went with that
rather than a RAID-1 is because with my previous RAID-1 setup, I found that
the biggest danger was me doing something stupid, not the hardware going
south (once, I did something stupid because I thought the hardware was
going south).
For my needs, though, having a reliable backup from the relatively
recent past is much more important than minimizing downtime of the backup
server. I'm not happy if the server is out of service for a few days or a
week, but it wouldn't have actual concrete consequences. If I were doing
this as part of an IT group where other people depended on it, I'd likely
make different decisions!
-scott
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

------------------------------------------------------------------------------
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Nico Kadel-Garcia]

On Thu, Nov 5, 2015 at 10:05 AM, Mark Phillips

Post by Mark Phillips
A quick update to my original post.
I tried backing up my rsnapshot directory on my backup drive to a new drive
an it failed using rsync but succeeded using cp -a.
* backup drive - 2 TB ext4, 16GB free, 360 GB of old backups not related to
rsnapshot, and the rest in a rsnapshot directory.
* new drive - 3 TB ext4 empty.
* both are usb 3.0 drives
* I ran the copies with both drives plugged into separate USB 3.0 drives on
a Ubuntu 14.04 LTS machine
* the rsync command - sudo rsync -aP
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The rsync command was interrupted. Had to go out of town, so stopped
rsync, disconnected the drives, and took the laptop to Palo Alto. When I
came back, I restarted the same command and then many hours later it failed.
* the cp cpmmand - sudo cp -a
/media/mark/6997d1c2-5ed7-4f32-9d5e-564cadd7bc6c/rsnapshot/
/media/mark/6ea41ea3-230e-4357-bda0-7fcb7d009be2/
* The cp command ran all night and never complained.
The rsync command failed with disk full error. And sure enough, the 3 TB was
completely full. I erased the drive and ran the cp command, and it

Forgot to use "rsync -aPH --delete", didn't you?

The "-H" is critical.

------------------------------------------------------------------------------

[Helmut Hullen]

Hallo, Scott,

Post by Scott Hess

Post by Helmut Hullen

Post by Scott Hess

Post by Helmut Hullen
Just for the record : I have put the backups of 3 machines onto 1
disk. And since about 4 weekse this disk has errors ...

It's a tradeoff. With three disks, you have a higher chance that
one of the disks with have errors. With one disk, you have a
higher chance that the disk with errors contains all of your
backups.

Yes - I know ...
But i hoped it doesn't happen just here ...

Where I ended up on this is that I keep a mirror of my backups on a
separate disk (actually two, one rotated offsite). Why I went with
that rather than a RAID-1 is because with my previous RAID-1 setup, I
found that the biggest danger was me doing something stupid, not the
hardware going south (once, I did something stupid because I thought
the hardware was going south).

Nearly the same as my decision - I prefer two independent backups with
the disk(s) in two independent docking stations or so. Backing up the
"monthly" backups (or the "weekly" ones) should be enough.

With the above mentioned exception ... putting such a job onto the "to
do" list is not enough.

Viele Gruesse!
Helmut


------------------------------------------------------------------------------