[rsnapshot-discuss] Parallelism and deduplication

[Gandalf Corvotempesta]

Hi to all.
I'm trying to implement a new backup server based on rsnapshot
by adding parallelism (through "parallel") and deduplication.

Just need some confirms, this is what I have right now (and seems
to work perfectly):

/etc/rsnapshot.d/hosts/server1.{conf,passwd}
/etc/rsnapshot.d/hosts/server2.{conf,passwd}
/etc/rsnapshot.d/hosts/server3.{conf,passwd}
/etc/rsnapshot.d/hosts/serverN.{conf,passwd}
/etc/rsnapshot.d/rsnapshot.conf

/etc/rsnapshot.d/rsnapshot.conf is the base configuration,
included from each host's config:

$ cat /etc/rsnapshot.d/hosts/server1.conf
include_conf /etc/rsnapshot.d/rsnapshot.conf
snapshot_root /var/backups/rsnapshot/server1/
logfile /var/log/rsnapshot/server1.log
lockfile /var/run/rsnapshot/server1.pid
backup rsync://***@server1/everything/ ./
+rsync_long_args=--password-file=/etc/rsnapshot.d/hosts/server1.passwd


$ cat rsnapshot.conf
config_version 1.2
no_create_root 0
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
#cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
#cmd_preexec /path/to/preexec/script
#cmd_postexec /path/to/postexec/script
verbose 3
loglevel 5
lockfile /var/run/rsnapshot.pid
sync_first 1
rsync_short_args -a
rsync_long_args --delete --numeric-ids --relative --delete-excluded --stats
link_dest 1
rsync_numtries 3
retain daily 15
retain weekly 2
exclude /backups/
exclude /admin_backups/
exclude /reseller_backups/
exclude /user_backups/
exclude /tmp/
exclude /proc
exclude /sys
exclude /var/cache
exclude /var/log/lastlog
exclude /var/log/rsync*
exclude /var/lib/mlocate
#exclude /var/spool
exclude /media
exclude /mnt
exclude tmp/


Up to this, is pretty easy.
Now I addeded some parallelism, through parallel.
Just a single entry in crontab:

$ cat /etc/cron.d/rsnapshot
0 0 * * * root /usr/local/bin/parallel_rsnapshot 2>&1 > /dev/null


$ cat /usr/local/bin/parallel_rsnapshot
#!/bin/bash
RSNAPSHOT_SCHEDULER="/usr/local/bin/rsnapshot_scheduler"
HOSTS_PATH="/etc/rsnapshot.d/hosts"
PARALLEL_JOBS=5
# Run rsnapshot in parallel
parallel --jobs ${PARALLEL_JOBS} "${RSNAPSHOT_SCHEDULER} {}" :::
${HOSTS_PATH}/*.conf



And a custom rsnapshot scheduler that choose which backup level to run:
(this is a little bit semplified for posting on list)

$ cat /usr/local/bin/rsnapshot_scheduler
#!/bin/bash
RSNAPSHOT="/usr/bin/rsnapshot -v"
CONFIG=$1
HOST=$(/usr/bin/basename ${CONFIG} | sed 's/\.conf$//g')
LOG_PATH="/var/log/rsnapshot"
LOG_FILE="${LOG_PATH}/$(/usr/bin/basename ${CONFIG} | sed
's/\.conf$/\.log/g')"

function rsnap {
${RSNAPSHOT} -c $1 $2
}

if [ "$(date +%j)" -eq "001" ]; then
rsnap ${CONFIG} yearly
fi
if [ $(date +%d) -eq 1 ]; then
rsnap ${CONFIG} monthly
fi
if [ $(date +%w) -eq 0 ]; then
rsnap ${CONFIG} weekly
fi

rsnap ${CONFIG} sync

# Check if sync was OK. Run daily only if sync is OK.
SUCCESS=$(grep -ci "$(date +%d/%b/%Y).*sync: completed successfully"
${LOG_FILE})
if [ ${SUCCESS} -ne 1 ]; then
EMAIL_SUBJECT="Backup FAILED per ${HOST}"
else
EMAIL_SUBJECT="Backup OK per ${HOST}"
rsnap ${CONFIG} daily
fi

# Send full log report
grep -i "$(date +%d/%b/%Y)" ${LOG_FILE} | mailx -s "${EMAIL_SUBJECT}"
***@mydomain.tld





Now, some questions:

1) should I run weekly, monthly and yearly before or after the sync
process? If run before, i'll rotate some backups with no confirm that
the followind sync would be ok. This could lead to some missing backups
(the ones deleted by rotation). Probably, the first thing to do is a
sync. If all is OK, then rotate everything else. Right?

2) On remote server, I have a "pre-xfer" script that run some actions
and output some debug text (like "echo Running xy....")
Is possible to get the output logged by rsnapshot ?

3) I would like to add some deduplication feature. Actually, I can run
"hardlink" over the latest backup (for example, daily.0), but this will
"deduplicate" only files in the same backup pool. How can I deduplicate
common files acroll all pools ? Is safe to run "hardlinks" across the
whole rsnapshot directory? (hardlink /var/backups/rsnapshot/server1/*) ?
This would save much more space....

Thanks in advance.

[Gandalf Corvotempesta]

no one ?

Post by Gandalf Corvotempesta
Hi to all.
I'm trying to implement a new backup server based on rsnapshot
by adding parallelism (through "parallel") and deduplication.
Just need some confirms, this is what I have right now (and seems
/etc/rsnapshot.d/hosts/server1.{conf,passwd}
/etc/rsnapshot.d/hosts/server2.{conf,passwd}
/etc/rsnapshot.d/hosts/server3.{conf,passwd}
/etc/rsnapshot.d/hosts/serverN.{conf,passwd}
/etc/rsnapshot.d/rsnapshot.conf
/etc/rsnapshot.d/rsnapshot.conf is the base configuration,
$ cat /etc/rsnapshot.d/hosts/server1.conf
include_conf /etc/rsnapshot.d/rsnapshot.conf
snapshot_root /var/backups/rsnapshot/server1/
logfile /var/log/rsnapshot/server1.log
lockfile /var/run/rsnapshot/server1.pid
+rsync_long_args=--password-file=/etc/rsnapshot.d/hosts/server1.passwd
$ cat rsnapshot.conf
config_version 1.2
no_create_root 0
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
#cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
#cmd_preexec /path/to/preexec/script
#cmd_postexec /path/to/postexec/script
verbose 3
loglevel 5
lockfile /var/run/rsnapshot.pid
sync_first 1
rsync_short_args -a
rsync_long_args --delete --numeric-ids --relative
--delete-excluded --stats
link_dest 1
rsync_numtries 3
retain daily 15
retain weekly 2
exclude /backups/
exclude /admin_backups/
exclude /reseller_backups/
exclude /user_backups/
exclude /tmp/
exclude /proc
exclude /sys
exclude /var/cache
exclude /var/log/lastlog
exclude /var/log/rsync*
exclude /var/lib/mlocate
#exclude /var/spool
exclude /media
exclude /mnt
exclude tmp/
Up to this, is pretty easy.
Now I addeded some parallelism, through parallel.
$ cat /etc/cron.d/rsnapshot
0 0 * * * root /usr/local/bin/parallel_rsnapshot 2>&1 > /dev/null
$ cat /usr/local/bin/parallel_rsnapshot
#!/bin/bash
RSNAPSHOT_SCHEDULER="/usr/local/bin/rsnapshot_scheduler"
HOSTS_PATH="/etc/rsnapshot.d/hosts"
PARALLEL_JOBS=5
# Run rsnapshot in parallel
${HOSTS_PATH}/*.conf
(this is a little bit semplified for posting on list)
$ cat /usr/local/bin/rsnapshot_scheduler
#!/bin/bash
RSNAPSHOT="/usr/bin/rsnapshot -v"
CONFIG=$1
HOST=$(/usr/bin/basename ${CONFIG} | sed 's/\.conf$//g')
LOG_PATH="/var/log/rsnapshot"
LOG_FILE="${LOG_PATH}/$(/usr/bin/basename ${CONFIG} | sed
's/\.conf$/\.log/g')"
function rsnap {
${RSNAPSHOT} -c $1 $2
}
if [ "$(date +%j)" -eq "001" ]; then
rsnap ${CONFIG} yearly
fi
if [ $(date +%d) -eq 1 ]; then
rsnap ${CONFIG} monthly
fi
if [ $(date +%w) -eq 0 ]; then
rsnap ${CONFIG} weekly
fi
rsnap ${CONFIG} sync
# Check if sync was OK. Run daily only if sync is OK.
SUCCESS=$(grep -ci "$(date +%d/%b/%Y).*sync: completed successfully"
${LOG_FILE})
if [ ${SUCCESS} -ne 1 ]; then
EMAIL_SUBJECT="Backup FAILED per ${HOST}"
else
EMAIL_SUBJECT="Backup OK per ${HOST}"
rsnap ${CONFIG} daily
fi
# Send full log report
grep -i "$(date +%d/%b/%Y)" ${LOG_FILE} | mailx -s "${EMAIL_SUBJECT}"
1) should I run weekly, monthly and yearly before or after the sync
process? If run before, i'll rotate some backups with no confirm that
the followind sync would be ok. This could lead to some missing backups
(the ones deleted by rotation). Probably, the first thing to do is a
sync. If all is OK, then rotate everything else. Right?
2) On remote server, I have a "pre-xfer" script that run some actions
and output some debug text (like "echo Running xy....")
Is possible to get the output logged by rsnapshot ?
3) I would like to add some deduplication feature. Actually, I can run
"hardlink" over the latest backup (for example, daily.0), but this will
"deduplicate" only files in the same backup pool. How can I deduplicate
common files acroll all pools ? Is safe to run "hardlinks" across the
whole rsnapshot directory? (hardlink /var/backups/rsnapshot/server1/*) ?
This would save much more space....
Thanks in advance.

[Scott Hess]

On Mon, Apr 25, 2016 at 3:08 AM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta
I'm trying to implement a new backup server based on rsnapshot
by adding parallelism (through "parallel") and deduplication.

Meta-comment: Often I/O bandwidth or seeks are the limiting factor, and
running things in parallel will convert 5 serial 10-minute jobs into 5
parallel 60-minute jobs. Make sure you're testing for that in your case.
Especially the cp -al and rm -rf phases are unlikely to enjoy competing
with each other for resources, whereas rsync phases could plausibly
timeshare with each other.

Post by Gandalf Corvotempesta
1) should I run weekly, monthly and yearly before or after the sync
process? If run before, i'll rotate some backups with no confirm that
the followind sync would be ok. This could lead to some missing backups
(the ones deleted by rotation). Probably, the first thing to do is a
sync. If all is OK, then rotate everything else. Right?

Only the first level benefits from sync. I always run the longer-period
rsnapshot calls first, because when combined with use_lazy_deletes, those
runs can be relatively tightly constrained in runtime (though I leave a gap
to my next sync, to let all the lazy deletes clear out).

2) On remote server, I have a "pre-xfer" script that run some actions

Post by Gandalf Corvotempesta
and output some debug text (like "echo Running xy....")
Is possible to get the output logged by rsnapshot ?

How and where? Why not just have the script log somewhere directly?

3) I would like to add some deduplication feature. Actually, I can run

Post by Gandalf Corvotempesta
"hardlink" over the latest backup (for example, daily.0), but this will
"deduplicate" only files in the same backup pool. How can I deduplicate
common files acroll all pools ? Is safe to run "hardlinks" across the
whole rsnapshot directory? (hardlink /var/backups/rsnapshot/server1/*) ?
This would save much more space....

Running hardlink every time is probably going to take a long time and not
do much most of the time. If you are frequently generating duplicate data
across hosts (like from a script or something), you might either explore
whether you actually need to backup that data, or whether you can dedupe in
targeted directories. I will dedupe periodically either when I'm doing
some other maintenance or when I notice disk usage seems up.

I just dedupe across my entire volume. The premise of rsnapshot means that
hardlinks are fine, if they weren't the entire thing would fall over. Just
make sure you aren't deduping while rsnapshot is running. On most systems
you cannot hardlink across volumes.

-scott

[Gandalf Corvotempesta]

Post by Scott Hess
Meta-comment: Often I/O bandwidth or seeks are the limiting factor, and
running things in parallel will convert 5 serial 10-minute jobs into 5
parallel 60-minute jobs. Make sure you're testing for that in your case.
Especially the cp -al and rm -rf phases are unlikely to enjoy competing with
each other for resources, whereas rsync phases could plausibly timeshare
with each other.

This is something that I've not considered, you are right.
What if i'll use "link-dest" and not "cp -al" ? it should avoid the cp phase.

My biggest issue is to finish all rsync phase as fast as possible, to keep
load on backupped server low. I'm running rsync in the middle of the night,
running backup serially (I have to backup 98 servers) will result having
some rsync running during the day. This is not what I want.

"cp -al" is run on backup server, thus I don't care about load.

As improvement, I can run "rsnapshot sync" in parallel, and after all syncs,
I can run the daily rotation, even sequeantially.

But with "link-dest" the whole "cp" phase should be avoided, right ?

Post by Scott Hess
Only the first level benefits from sync. I always run the longer-period
rsnapshot calls first, because when combined with use_lazy_deletes, those
runs can be relatively tightly constrained in runtime (though I leave a gap
to my next sync, to let all the lazy deletes clear out).

Ok, but rotating before sync could lead to deleted backups (based on retention)
even if the following sync is not completed.

Post by Scott Hess
How and where? Why not just have the script log somewhere directly?

Yes, is the best solution.

Post by Scott Hess
Running hardlink every time is probably going to take a long time and not do
much most of the time. If you are frequently generating duplicate data
across hosts (like from a script or something), you might either explore
whether you actually need to backup that data, or whether you can dedupe in
targeted directories. I will dedupe periodically either when I'm doing some
other maintenance or when I notice disk usage seems up.

I don't want to dedup across hosts (at the moment).
I would like to dedup across multiple level for the same host, in example:

daily.3 has almost 90% of shared files with daily.2

I've tried to run "hardlink" across all backups for 1 host and saved 12GB

Post by Scott Hess
I just dedupe across my entire volume. The premise of rsnapshot means that
hardlinks are fine, if they weren't the entire thing would fall over. Just
make sure you aren't deduping while rsnapshot is running. On most systems
you cannot hardlink across volumes.

So, you are dedupe across daily.1 then daily.2 and so on or across all
backup leves at
the same time ? In example:

hardlink myserver/daily.1
hardlink myserver/daily.2

or

hardlink myserver/*

[Scott Hess]

On Sat, Apr 30, 2016 at 3:21 AM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Scott Hess
Meta-comment: Often I/O bandwidth or seeks are the limiting factor, and
running things in parallel will convert 5 serial 10-minute jobs into 5
parallel 60-minute jobs. Make sure you're testing for that in your case.
Especially the cp -al and rm -rf phases are unlikely to enjoy competing

with

Post by Scott Hess
each other for resources, whereas rsync phases could plausibly timeshare
with each other.

This is something that I've not considered, you are right.
What if i'll use "link-dest" and not "cp -al" ? it should avoid the cp phase.

I/O is I/O, you can shift it around, but this won't get rid of it.

Post by Gandalf Corvotempesta
My biggest issue is to finish all rsync phase as fast as possible, to keep
load on backupped server low. I'm running rsync in the middle of the night,
running backup serially (I have to backup 98 servers) will result having
some rsync running during the day. This is not what I want.

Do you want to run as fast as possible on the system being backed up?
Because in that case it will be faster to back up a single server at a
time, so that backups don't contend with each other on I/O.

Do you want to run with as little load as possible? Because in that case
running as fast as possible is likely to run at a higher load than
spreading things out.

Your last couple lines make it sound like you're having an issue where the
overall backup is taking too long. Rather than trying to parallelize 98
backups, I'd scan rsnapshot.log and look to see if there aren't five or six
backups which are causing the majority of the problems. Probably if you
take those five or six and put them in a separate rsnapshot flow, you can
end up with 2 parallel systems which modestly compete with each other,
rather than trying to have a single system with weird performance issues.
In that case you could even consider doing things like putting one set of
servers on a different volume so they aren't stepping on each other's toes
I/O wise.

"cp -al" is run on backup server, thus I don't care about load.
The "cp -al" pass runs before the rsync, so on the backup server in
isolation. Using link-dest would push the "cp -al" I/O into the rsync
itself, so the rsync will likely take longer.

As improvement, I can run "rsnapshot sync" in parallel, and after all syncs,

Post by Gandalf Corvotempesta
I can run the daily rotation, even sequeantially.

Running the sync in parallel should mostly mean running all of the rsync
calls in parallel, so that sounds like what you need.

Post by Gandalf Corvotempesta
But with "link-dest" the whole "cp" phase should be avoided, right ?

You avoid the point-in-time "cp -al" phase. You cannot avoid the I/O
overhead of creating the directory structure and populating it with
hardlinks.

Post by Gandalf Corvotempesta
Only the first level benefits from sync. I always run the longer-period

Post by Scott Hess
rsnapshot calls first, because when combined with use_lazy_deletes, those
runs can be relatively tightly constrained in runtime (though I leave a

gap

Post by Scott Hess
to my next sync, to let all the lazy deletes clear out).

Ok, but rotating before sync could lead to deleted backups (based on retention)
even if the following sync is not completed.

Don't know what you're getting at. If sync is failing, you aren't backing
up current data, but you won't lose material amounts of data until sync has
been failing for weeks, don't let yourself get into that situation.

Post by Gandalf Corvotempesta
Running hardlink every time is probably going to take a long time and not do

Post by Scott Hess
much most of the time. If you are frequently generating duplicate data
across hosts (like from a script or something), you might either explore
whether you actually need to backup that data, or whether you can dedupe

in

Post by Scott Hess
targeted directories. I will dedupe periodically either when I'm doing

some

Post by Scott Hess
other maintenance or when I notice disk usage seems up.

I don't want to dedup across hosts (at the moment).
daily.3 has almost 90% of shared files with daily.2
I've tried to run "hardlink" across all backups for 1 host and saved 12GB

If you have a lot of identical files between daily.3 and daily.2 and they
aren't being hardlinked, then there's something wrong with your system.
The entire point of rsnapshot is that if a file doesn't change between two
backups, then it should be hardlinked.

Post by Gandalf Corvotempesta
I just dedupe across my entire volume. The premise of rsnapshot means that

Post by Scott Hess
hardlinks are fine, if they weren't the entire thing would fall over.

Just

Post by Scott Hess
make sure you aren't deduping while rsnapshot is running. On most

systems

Post by Scott Hess
you cannot hardlink across volumes.

So, you are dedupe across daily.1 then daily.2 and so on or across all
backup leves at
hardlink myserver/daily.1
hardlink myserver/daily.2
or
hardlink myserver/*

I dedupe across the entire volume, so that if hosts share files (say they
are comparable Ubuntu installs or something), they don't duplicate.

-scott

[Nico Kadel-Garcia]

Post by Scott Hess
On Sat, Apr 30, 2016 at 3:21 AM, Gandalf Corvotempesta

Post by Gandalf Corvotempesta
"cp -al" is run on backup server, thus I don't care about load.

The "cp -al" pass runs before the rsync, so on the backup server in
isolation. Using link-dest would push the "cp -al" I/O into the rsync
itself, so the rsync will likely take longer.

"cp -al" is lightning fast: it's talking only to the local filesystem,
not to any seprate "rsync" process which may have to intelligently
parse and complain file systems. Don't discard this unless you have
to.

Post by Scott Hess

Post by Gandalf Corvotempesta
As improvement, I can run "rsnapshot sync" in parallel, and after all syncs,
I can run the daily rotation, even sequeantially.

Running the sync in parallel should mostly mean running all of the rsync
calls in parallel, so that sounds like what you need.

Post by Gandalf Corvotempesta
But with "link-dest" the whole "cp" phase should be avoided, right ?

You avoid the point-in-time "cp -al" phase. You cannot avoid the I/O
overhead of creating the directory structure and populating it with
hardlinks.

Which is typically signifacntly faster than doing sync funkiness later.

Post by Scott Hess

Post by Gandalf Corvotempesta

Post by Scott Hess
Only the first level benefits from sync. I always run the longer-period
rsnapshot calls first, because when combined with use_lazy_deletes, those
runs can be relatively tightly constrained in runtime (though I leave a gap
to my next sync, to let all the lazy deletes clear out).

Ok, but rotating before sync could lead to deleted backups (based on retention)
even if the following sync is not completed.

Do the fast ones first. If the slow ones run long, the short ones will
be missed or delayed, and if parallelized they will overlap in I/O,
bandwidth, and CPU resources. If you do the fast ones first, they're
out of the way and the long ones can use reasonably configured lock
files to say "I haven't finished, I'll do the next update!!"

Post by Scott Hess
Don't know what you're getting at. If sync is failing, you aren't backing
up current data, but you won't lose material amounts of data until sync has
been failing for weeks, don't let yourself get into that situation.

Post by Gandalf Corvotempesta

Post by Scott Hess
Running hardlink every time is probably going to take a long time and not do
much most of the time. If you are frequently generating duplicate data
across hosts (like from a script or something), you might either explore
whether you actually need to backup that data, or whether you can dedupe in
targeted directories. I will dedupe periodically either when I'm doing some
other maintenance or when I notice disk usage seems up.

I don't want to dedup across hosts (at the moment).
daily.3 has almost 90% of shared files with daily.2
I've tried to run "hardlink" across all backups for 1 host and saved 12GB

If you have a lot of identical files between daily.3 and daily.2 and they
aren't being hardlinked, then there's something wrong with your system. The
entire point of rsnapshot is that if a file doesn't change between two
backups, then it should be hardlinked.

Try hardlinking *one* backup of one host. You may find considerable
amounts of file duplication, such as duplicate copies of multiple
source trees for the same person working on the same host. I've had
this happen when working on kernels or other bulky source trees.

It can also be dangerous as futz to restore from this, because you can
hardcode links *inside* source trees which *should* be broken, and
where copying to one modifies both source trees. Been there, done
that, even seen it with identical files inside mysql databases where
they multiple databases had the same MyISAM tables copied into them
directly. Hilarity ensued.....

Post by Scott Hess

Post by Gandalf Corvotempesta

Post by Scott Hess
I just dedupe across my entire volume. The premise of rsnapshot means that
hardlinks are fine, if they weren't the entire thing would fall over.
Just
make sure you aren't deduping while rsnapshot is running. On most systems
you cannot hardlink across volumes.

So, you are dedupe across daily.1 then daily.2 and so on or across all
backup leves at
hardlink myserver/daily.1
hardlink myserver/daily.2
or
hardlink myserver/*

I dedupe across the entire volume, so that if hosts share files (say they
are comparable Ubuntu installs or something), they don't duplicate.
-scott

See above for some of the risks. I *hope* whatever restoration
techniques you use are intelligent enough to break, or set, hardlinks
as you need, because there are dangers.

[Gandalf Corvotempesta]

Post by Nico Kadel-Garcia
"cp -al" is lightning fast: it's talking only to the local filesystem,
not to any seprate "rsync" process which may have to intelligently
parse and complain file systems. Don't discard this unless you have to.

rsnapsnot man page advice using link_dest.
BTW i can try to use "cp -al" and see if next rsync would last less.

Post by Nico Kadel-Garcia
Try hardlinking *one* backup of one host. You may find considerable
amounts of file duplication, such as duplicate copies of multiple
source trees for the same person working on the same host. I've had
this happen when working on kernels or other bulky source trees. It
can also be dangerous as futz to restore from this, because you can
hardcode links *inside* source trees which *should* be broken, and
where copying to one modifies both source trees. Been there, done
that, even seen it with identical files inside mysql databases where
they multiple databases had the same MyISAM tables copied into them
directly. Hilarity ensued.....

Don't see the issue.
If identical MyISAM tables are copied, they can be hardlinked.
On next backup, if tables would differ, the new MyISAM files are
backupped and not hardlinked

Post by Nico Kadel-Garcia
See above for some of the risks. I *hope* whatever restoration
techniques you use are intelligent enough to break, or set, hardlinks
as you need, because there are dangers.

Isn't rsync smart enough to restore the file *content* and not the file
hardlink ?
How can I detect an hardlink coming from the server from an hardlink
made by rsnapshot during the backup?
Hardlink coming from servers should be restored "as-is", hardlink made
by rsnapshot should
be resolved and restored as single file.

On backup server they are both hard-links....

[Gandalf Corvotempesta]

Post by Scott Hess
The "cp -al" pass runs before the rsync, so on the backup server in
isolation. Using link-dest would push the "cp -al" I/O into the rsync
itself, so the rsync will likely take longer.

I don't think it's lightning fast.
I've tried yersterday and "cp -al" took more than 2 hours, plus the
rsync phase (about 1 hours)
rsync with link-dest took 1 hours and 20 minutes.

I'll see this night, yesterday backup run "cp -al" from daily.0 to
.sync and after all, cp -al .sync to daily.0

Probably, becase using link-dest there was no .sync directory.
Let's see this night.

[Gandalf Corvotempesta]

Post by Scott Hess
Do you want to run as fast as possible on the system being backed up?
Because in that case it will be faster to back up a single server at a
time, so that backups don't contend with each other on I/O.
Do you want to run with as little load as possible? Because in that
case running as fast as possible is likely to run at a higher load
than spreading things out.

In my use-case, i've seen that running 1 backup per time would require
the whole night and half day
Running 5 backups in parallels, starting at midnight, will end at about
7:30-8:00

Post by Scott Hess
Your last couple lines make it sound like you're having an issue where
the overall backup is taking too long. Rather than trying to
parallelize 98 backups,

I don't parallelize 98 backups. I parallelize 5 to 10 backups (still
trying to guess the best value).
I have 98 backups to do with this server, not 98 backups in parallel.

Post by Scott Hess
The "cp -al" pass runs before the rsync, so on the backup server in
isolation. Using link-dest would push the "cp -al" I/O into the rsync
itself, so the rsync will likely take longer.

Running "cp -al" before the rsync would lead to a delay running rsync,
thus i'll end with backups running in the morning.

Post by Scott Hess
Don't know what you're getting at. If sync is failing, you aren't
backing up current data, but you won't lose material amounts of data
until sync has been failing for weeks, don't let yourself get into
that situation.

Not really.
standard rsnapshot process will start rotating all backups. rotation
will remove the expired ones.
I need 2 weekly and 14 daily backups.
On daily.13, when rshapshot move that to weekly.0, the older weekly.1 is
removed.
If next sync will fail, weekly.1 is still gone.
With my scheduler, if sync fails, all rotation are stopped and thus i
preserve the whole backup tree.

Post by Scott Hess
I dedupe across the entire volume, so that if hosts share files (say
they are comparable Ubuntu installs or something), they don't duplicate.

Good idea. 90% of my servers are Debian 7.

[Scott Hess]

On Sun, May 1, 2016 at 9:21 AM, Gandalf Corvotempesta <

Post by Scott Hess
Do you want to run as fast as possible on the system being backed up?
Because in that case it will be faster to back up a single server at a
time, so that backups don't contend with each other on I/O.
Do you want to run with as little load as possible? Because in that case
running as fast as possible is likely to run at a higher load than
spreading things out.

In my use-case, i've seen that running 1 backup per time would require the
whole night and half day
Running 5 backups in parallels, starting at midnight, will end at about
7:30-8:00

So your goal is to get the backups done before a particular time, but you
don't actually care about load on the target server if that happens?

For most reasonable backup servers, you should have more than enough CPU to
run parallel backups, but you might want to pay attention to having enough
I/O capacity and memory. I'd be nervous about running parallel backups on
the same spindles, unless you have something which nicely spreads the load
across spindles.

The "cp -al" pass runs before the rsync, so on the backup server in

Post by Scott Hess
isolation. Using link-dest would push the "cp -al" I/O into the rsync
itself, so the rsync will likely take longer.

Running "cp -al" before the rsync would lead to a delay running rsync,
thus i'll end with backups running in the morning.

Then run it earlier?

Using sync_first, the "cp -al" phase happens during rotation (hourly,
daily, etc), while the rsync happens during sync.

Don't know what you're getting at. If sync is failing, you aren't backing

Post by Scott Hess
up current data, but you won't lose material amounts of data until sync has
been failing for weeks, don't let yourself get into that situation.

Not really.
standard rsnapshot process will start rotating all backups. rotation will
remove the expired ones.
I need 2 weekly and 14 daily backups.
On daily.13, when rshapshot move that to weekly.0, the older weekly.1 is
removed.
If next sync will fail, weekly.1 is still gone.
With my scheduler, if sync fails, all rotation are stopped and thus i
preserve the whole backup tree.

If everything was successful, you'd have deleted the oldest backup, so
obviously you don't really care about any data which is only present in
that backup. Having 14 backup directories rather than 13 isn't more
successful or less successful, having sync breaking means you are not
backing up current data, which is definitely a problem.

Put another way, if you have things like "rotate then sync" and the sync
fails, you should raise all the alarms you have and deal with that
IMMEDIATELY. If you have things like "sync then rotate" and the sync
fails, you should raise all the alarms you have an deal with that
IMMEDIATELY. There's really not much difference, here, IMHO. If someone
wants to restore something from yesterday and you find that you have a
complete set of 14 daily backups that are from 6 weeks ago, nobody is going
to be happy about that.

-scott

[Gandalf Corvotempesta]

Post by Scott Hess
So your goal is to get the backups done before a particular time, but
you don't actually care about load on the target server if that happens?

Yes and no.
Having backup running up to 07:00 in the morning, will result in low
load on server and almost 0 I/O contention with other services.
Running during the day, even with "nice 19" and "ionice idle" will
result in backup taking the whole day and still high load.

Post by Scott Hess
For most reasonable backup servers, you should have more than enough
CPU to run parallel backups, but you might want to pay attention to
having enough I/O capacity and memory. I'd be nervous about running
parallel backups on the same spindles, unless you have something which
nicely spreads the load across spindles.

I'm using hardware RAID-10
I/O is spread across 12 disks (6+6)

Post by Scott Hess
Then run it earlier?

From 08:00 to 23:00 i can't.

Post by Scott Hess
Put another way, if you have things like "rotate then sync" and the
sync fails, you should raise all the alarms you have and deal with
that IMMEDIATELY. If you have things like "sync then rotate" and the
sync fails, you should raise all the alarms you have an deal with that
IMMEDIATELY. There's really not much difference, here, IMHO. If
someone wants to restore something from yesterday and you find that
you have a complete set of 14 daily backups that are from 6 weeks ago,
nobody is going to be happy about that.

Let's assume I'm on vacation or i'm unable to deal immediatly with the
issue for whatever reason.
In the first case, i'm still loosing restore points. rotation still
happens, older backups are deleted and so on.
In the second case, new backups are not made due to the issue, but the
older one are still there.

But, yes, due to the nature of rsnapshot, having 13 days or 14 doesn't
make any difference.

One question: is daily.13 a "standalone" backup? Can I move it to
another server, to a dvd, or something else and still have the whole
backup available ? Can I remove everything up to daily.13
(daily.0....12) and still have the daily.13 available and complete?

In bacula, removing the "full" backup means breaking everything else
after it. Probably this is why I'm trying to stop the rotation if backup
is failing. Rotating/Purging bacula volume could lead to all backup
broken, in example:

full
incr
incr
incr
incr
incr
differential
incr
incr
incr
incr
full not executed properly and backup stopped here for some days

the last full is not completed, but Bacula is still expiring the older
ones based on retention period.
When the first full is purged, you have lost ALL backups for that server.
If you did a mistake in rotation period, you could loose all backups
with just 1 failure. Probably, this can't happen with rsnapshot as all
backups are hardlinked and not based on the previous one.

[Scott Hess]

On Sun, May 1, 2016 at 10:35 AM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Scott Hess
For most reasonable backup servers, you should have more than enough CPU
to run parallel backups, but you might want to pay attention to having
enough I/O capacity and memory. I'd be nervous about running parallel
backups on the same spindles, unless you have something which nicely
spreads the load across spindles.

I'm using hardware RAID-10
I/O is spread across 12 disks (6+6)

Read I/O across 12, write I/O across 6, that probably gives you headroom to
run a certain number of parallel rsync processes without too much
backpressure between them. You may want to monitor timestamps on your
.sync directories, though, to make certain they don't suddenly start rising
all at the same time.

Then run it earlier?

Post by Gandalf Corvotempesta
From 08:00 to 23:00 i can't.

You can't run the sync pass (the one full of rsyncs), or you can't run the
rotation (cp -al and rm -rf)?

If this is driven by a need to maintain 14 days of backups, then why not
run the rotation a little earlier and retain 15 days? When using
sync_first, the rotation only operates on the backup server, no rsync to
remote servers.

Put another way, if you have things like "rotate then sync" and the sync

Post by Gandalf Corvotempesta

Post by Scott Hess
fails, you should raise all the alarms you have and deal with that
IMMEDIATELY. If you have things like "sync then rotate" and the sync
fails, you should raise all the alarms you have an deal with that
IMMEDIATELY. There's really not much difference, here, IMHO. If someone
wants to restore something from yesterday and you find that you have a
complete set of 14 daily backups that are from 6 weeks ago, nobody is going
to be happy about that.

Let's assume I'm on vacation or i'm unable to deal immediatly with the
issue for whatever reason.
In the first case, i'm still loosing restore points. rotation still
happens, older backups are deleted and so on.
In the second case, new backups are not made due to the issue, but the
older one are still there.
But, yes, due to the nature of rsnapshot, having 13 days or 14 doesn't
make any difference.

I guess in the end the easiest way to have the system grind to a stop if
sync fails is the way you should go with.

BUT, keep in mind that with that many systems to backup, it seems likely
that sync is going to fail sometimes for some systems, so the question is
what to do when one system's sync is failing but the rest are fine?

One question: is daily.13 a "standalone" backup? Can I move it to another

Post by Gandalf Corvotempesta
server, to a dvd, or something else and still have the whole backup
available ? Can I remove everything up to daily.13 (daily.0....12) and
still have the daily.13 available and complete?

Each snapshot should be a complete copy of the directory structure, with
all unique files uniquely present, and all files shared with the previous
backup hardlinked. So in my snapshot root, I can do:

# ls -li daily.?/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.0/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.1/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.2/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.3/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.4/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.5/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.6/nbackup/bin/ls

They're all the same file (other refs are in 4x hourly.?, 5x weekly.?, and
.sync). I could copy or move daily.3 to a different disk, and the files in
all the others would still be present. There are no incrementals.

-scott

[Gandalf Corvotempesta]

Post by Scott Hess
I guess in the end the easiest way to have the system grind to a stop if
sync fails is the way you should go with.
BUT, keep in mind that with that many systems to backup, it seems likely
that sync is going to fail sometimes for some systems, so the question is
what to do when one system's sync is failing but the rest are fine?

If you read the code that i've posted, you can see that I'm looking
for "sync: completed successfully" for every
host. In case of failure, only that host is "freezed". That's why i'm
using a wrapper script to run rsnapshot in parallel
and not directly rsnapshot

After each backup (the script is run for each host), i'll look for
"sync: completed successfully" in log file.
If found, i'll start rotations. If not, i'll exit and send an email.
But just for that single host.

Post by Scott Hess
Each snapshot should be a complete copy of the directory structure, with all
unique files uniquely present, and all files shared with the previous backup
# ls -li daily.?/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.0/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.1/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.2/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.3/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.4/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.5/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.6/nbackup/bin/ls
They're all the same file (other refs are in 4x hourly.?, 5x weekly.?, and
.sync). I could copy or move daily.3 to a different disk, and the files in
all the others would still be present. There are no incrementals.

That's clear.
What I would like to know (or better, i would like to have a confirm) is:

can I copy "daily.3" to somewhere else? In that case, daily.3 would be
a total copy of my backupped server, like a "full" backup, right?
can I delete EVERYTHING except daily.3 and still having the daily.3
fully available ?

Like my example with "Bacula": deleting a full in bacula will result
in all subsequent backups unavailable. With hardlinks this should
never happens, as
hardlinked files are removed only when link count reach to 0. Deleting
everything but 1 directory, will result in link count >= 1 and thus
the whole backup is still available.

[Nico Kadel-Garcia]

On Mon, May 2, 2016 at 3:43 AM, Gandalf Corvotempesta

Post by Gandalf Corvotempesta

Post by Scott Hess
I guess in the end the easiest way to have the system grind to a stop if
sync fails is the way you should go with.
BUT, keep in mind that with that many systems to backup, it seems likely
that sync is going to fail sometimes for some systems, so the question is
what to do when one system's sync is failing but the rest are fine?

If you read the code that i've posted, you can see that I'm looking
for "sync: completed successfully" for every
host. In case of failure, only that host is "freezed". That's why i'm
using a wrapper script to run rsnapshot in parallel
and not directly rsnapshot
After each backup (the script is run for each host), i'll look for
"sync: completed successfully" in log file.
If found, i'll start rotations. If not, i'll exit and send an email.
But just for that single host.

Post by Scott Hess
Each snapshot should be a complete copy of the directory structure, with all
unique files uniquely present, and all files shared with the previous backup
# ls -li daily.?/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.0/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.1/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.2/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.3/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.4/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.5/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.6/nbackup/bin/ls
They're all the same file (other refs are in 4x hourly.?, 5x weekly.?, and
.sync). I could copy or move daily.3 to a different disk, and the files in
all the others would still be present. There are no incrementals.

That's clear.
can I copy "daily.3" to somewhere else? In that case, daily.3 would be
a total copy of my backupped server, like a "full" backup, right?
can I delete EVERYTHING except daily.3 and still having the daily.3
fully available ?

Yup. If you're a weasel, you can even move it aside and put it back
later, as long as you put it bck in the right order with any newly
rotated backups.

Copying a large daily.3 snapshot aside, and making sure it's
consistent, can be tricky if the copy rotates under you. This is why
I've long wanted to change the numbering scheme from "daily.0",
"daily.1", etc. to "daily.20160401010203", "daily.20160402113433", to
use full UTC compatbile YYYYMMDDhhmmss date stamped names. But I've
never gotten the traction to write and submit a patch.,

Post by Gandalf Corvotempesta
Like my example with "Bacula": deleting a full in bacula will result
in all subsequent backups unavailable. With hardlinks this should
never happens, as
hardlinked files are removed only when link count reach to 0. Deleting
everything but 1 directory, will result in link count >= 1 and thus
the whole backup is still available.

Not a problem with rsnapshot and hardlinked copies.

[Patrick O'Callaghan]

Post by Nico Kadel-Garcia
Copying a large daily.3 snapshot aside, and making sure it's
consistent, can be tricky if the copy rotates under you. This is why
I've long wanted to change the numbering scheme from "daily.0",
"daily.1", etc. to "daily.20160401010203", "daily.20160402113433", to
use full UTC compatbile YYYYMMDDhhmmss date stamped names. But I've
never gotten the traction to write and submit a patch.,

That would be a great idea. I always have to stop and remember if daily.0
is the most recent or the least recent. And having a timestamp would add
useful information that's otherwise buried in log files.

poc

[Rolf Muth]

Am Mittwoch 4 Mai 16 um 15:45:10 Uhr schrieb "Nico Kadel-Garcia"

Post by Nico Kadel-Garcia
...
I've long wanted to change the numbering scheme from "daily.0",
"daily.1", etc. to "daily.20160401010203", "daily.20160402113433", to
use full UTC compatbile YYYYMMDDhhmmss date stamped names. But I've
never gotten the traction to write and submit a patch.,

For such purpose I wrote do-timed-folders and do-del-timed-folders ...

http://www.heise.de/download/do-rsnapshots-1184971.html

[Tapani Tarvainen]

Post by Nico Kadel-Garcia
Copying a large daily.3 snapshot aside, and making sure it's
consistent, can be tricky if the copy rotates under you.

On solution is to use filesystem snapshot, if you're using LVM
or a filesystem that supports snapshots by itself (zfs, brtfs):
create snapshot, copy daily.3 (or whatever) out, destroy the
snapshot.

Post by Nico Kadel-Garcia
This is why I've long wanted to change the numbering scheme from
"daily.0", "daily.1", etc. to "daily.20160401010203",
"daily.20160402113433", to use full UTC compatbile YYYYMMDDhhmmss
date stamped names.

That would have its advantages, but it'd not be a trivial change.

[Gandalf Corvotempesta]

Post by Scott Hess
Each snapshot should be a complete copy of the directory structure, with all
unique files uniquely present, and all files shared with the previous backup
# ls -li daily.?/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.0/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.1/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.2/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.3/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.4/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.5/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.6/nbackup/bin/ls
They're all the same file (other refs are in 4x hourly.?, 5x weekly.?, and
.sync). I could copy or move daily.3 to a different disk, and the files in
all the others would still be present. There are no incrementals.

Let's assume that everything is going very bad, a bad sysadmin deleted
multiple backups and so on.
I'll end with having only a single, older, backup, in example: daily.5.
No ".sync" or anything else. Only daily.5

"daily.5" should be totally available as all hardlinked files are
still retained (due to link count > 0 thus files are still there)
What would happen by copying daily.5 to brand new ".sync" dir and
running rsnapshot again ? Only differences between daily.5 and current
backups are transfered, right?
Thus, in this case, i'll still able to do an incremental backup
starting from a very old "snapshot". Is this true?

This could be very usefull, with larger servers. If a full backup
would be 300GB and incremental would be just some hundreds of MB, the
ability to
run an incremental from an older backup will allow me to not transfer
300GB but just some MB.

[Scott Hess]

I'm thinking that at some point you're just going to have to dig in there
and figure out how rsync works. You've thrown out a number of hypothetical
situations on this thread, and at this point I'm worried that in three
months you'll come back and say "But you said _this_ would work, and you
said _that_ would work, but we lost our backups!"

The basic operation of rsnapshot is not super complicated, you can see what
it's doing right in the log files. I have in the past manually constructed
a new backup target out of pieces of other backup targets, because I wanted
to move a volume between servers and couldn't afford the space to backup
the volume in two places. It's not that hard. If you're worried about a
particular case, spin up a Linux server, build a trivial example structure
or three to backup, setup cron to run backups every five minutes for a few
hours, then get in there and experiment.

-scott


On Thu, May 5, 2016 at 2:52 AM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Scott Hess
Each snapshot should be a complete copy of the directory structure, with

all

Post by Scott Hess
unique files uniquely present, and all files shared with the previous

backup

Post by Scott Hess
# ls -li daily.?/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.0/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.1/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.2/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.3/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.4/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.5/nbackup/bin/ls
564308 -rwxr-xr-x 17 root root 110080 Mar 10 11:10 daily.6/nbackup/bin/ls
They're all the same file (other refs are in 4x hourly.?, 5x weekly.?,

and

Post by Scott Hess
.sync). I could copy or move daily.3 to a different disk, and the files

in

Post by Scott Hess
all the others would still be present. There are no incrementals.

Let's assume that everything is going very bad, a bad sysadmin deleted
multiple backups and so on.
I'll end with having only a single, older, backup, in example: daily.5.
No ".sync" or anything else. Only daily.5
"daily.5" should be totally available as all hardlinked files are
still retained (due to link count > 0 thus files are still there)
What would happen by copying daily.5 to brand new ".sync" dir and
running rsnapshot again ? Only differences between daily.5 and current
backups are transfered, right?
Thus, in this case, i'll still able to do an incremental backup
starting from a very old "snapshot". Is this true?
This could be very usefull, with larger servers. If a full backup
would be 300GB and incremental would be just some hundreds of MB, the
ability to
run an incremental from an older backup will allow me to not transfer
300GB but just some MB.

[Gandalf Corvotempesta]

Post by Scott Hess
I'm thinking that at some point you're just going to have to dig in there
and figure out how rsync works. You've thrown out a number of hypothetical
situations on this thread, and at this point I'm worried that in three
months you'll come back and say "But you said _this_ would work, and you
said _that_ would work, but we lost our backups!"

Absolutely not. This is not the case.

I'm trying to replace a Bacula backup environment (too many things to
check: huge databases (mine is 980GB), too many backup leves, too
prone to failures, ....) with
a smarter system like rsnapshot, where everything could be summarized
with: "rsync + cp + mv". No backup leves, no databases, files always
available for restore (and even for looking, like "cat my_lost_file")

I would like to copy the weekly backups to a tape library. Obviously I
need to resolve all hardlinks when copying or i'll end up with
inconsistent data in case of restore.
My question is: how can I detect an hardlink made by rsnapshot from an
hardlink that was on the source server ? rsnapshot's hardlink must be
dereferenced when copying to tape, original hardlink must preserved to
restore the system in the original state.

For example:

on source server:
$ ln file1 file2
$ echo 'test' > file3

when running multiple backups, file3 would be hardlinked between
".sync" and "daily.N". "file1" and "file" are already hardlinked.

When copying, I have to copy the *content* of "file3", dereferencing
it, but still preserving the hardlink between file1 and file2

I hope this is clear.

[Martin Schröder]

2016-05-06 10:53 GMT+02:00 Gandalf Corvotempesta

Post by Gandalf Corvotempesta
My question is: how can I detect an hardlink made by rsnapshot from an
hardlink that was on the source server ? rsnapshot's hardlink must be

You can not: rsnapshot does not preserve hard links _in_ the data to backup.

Best
Martin

[Gandalf Corvotempesta]

Post by Martin Schröder
You can not: rsnapshot does not preserve hard links _in_ the data to backup.

So, in my example, "file1" and "file2" would be copied as files and
not hardlinked together ?
Thus, all hardlinks that I can see in a backup directory, are from
rsnapshot and not from the source server.

Doing a restore would result in much more used space than the original
server, as all hardlinked files are resolved and not preserved. Right
?

[Martin Schröder]

2016-05-06 11:52 GMT+02:00 Gandalf Corvotempesta

Post by Gandalf Corvotempesta

Post by Martin Schröder
You can not: rsnapshot does not preserve hard links _in_ the data to backup.

So, in my example, "file1" and "file2" would be copied as files and
not hardlinked together ?
Thus, all hardlinks that I can see in a backup directory, are from
rsnapshot and not from the source server.

AFAIK yes.

Post by Gandalf Corvotempesta
Doing a restore would result in much more used space than the original
server, as all hardlinked files are resolved and not preserved. Right
?

s/would/could/, but yes.

Best
Martin

[Patrick O'Callaghan]

On 6 May 2016 at 10:52, Gandalf Corvotempesta <

Post by Christopher Barry

Post by Martin Schröder
You can not: rsnapshot does not preserve hard links _in_ the data to

backup.
So, in my example, "file1" and "file2" would be copied as files and
not hardlinked together ?
Thus, all hardlinks that I can see in a backup directory, are from
rsnapshot and not from the source server.
Doing a restore would result in much more used space than the original
server, as all hardlinked files are resolved and not preserved. Right
?

Yes. I suspect this is a problem with all backup systems that rely on hard
links to eliminate duplicates (including Apple's Time Machine AFAIK). They
work reasonably well as long as the source filesystem doesn't have a lot of
hard links in it, but restores will produce additional data copies unless
you run a separate dedupe process.

poc

[Nico Kadel-Garcia]

On Fri, May 6, 2016 at 6:17 AM, Patrick O'Callaghan

Post by Patrick O'Callaghan
On 6 May 2016 at 10:52, Gandalf Corvotempesta

Post by Gandalf Corvotempesta

Post by Martin Schröder
You can not: rsnapshot does not preserve hard links _in_ the data to backup.

So, in my example, "file1" and "file2" would be copied as files and
not hardlinked together ?
Thus, all hardlinks that I can see in a backup directory, are from
rsnapshot and not from the source server.
Doing a restore would result in much more used space than the original
server, as all hardlinked files are resolved and not preserved. Right
?

Yes. I suspect this is a problem with all backup systems that rely on hard
links to eliminate duplicates (including Apple's Time Machine AFAIK). They
work reasonably well as long as the source filesystem doesn't have a lot of
hard links in it, but restores will produce additional data copies unless
you run a separate dedupe process.

A "restore" of an individual snapshot would have few internal
snapshots, anywhere. Restoring multiple snapshots..... gets a bit
adventuresome to try and recreate the hardlink infrastructure.

[Nico Kadel-Garcia]

Post by Martin Schröder
2016-05-06 10:53 GMT+02:00 Gandalf Corvotempesta

Post by Gandalf Corvotempesta
My question is: how can I detect an hardlink made by rsnapshot from an
hardlink that was on the source server ? rsnapshot's hardlink must be

You can not: rsnapshot does not preserve hard links _in_ the data to backup.
Best
Martin

I beg your pardon? That is the "-H" option for rsync. It's not enabled
by default, but it's certainly available in the rsync_short_args.

[Martin Schröder]

Post by Nico Kadel-Garcia
I beg your pardon? That is the "-H" option for rsync. It's not enabled
by default, but it's certainly available in the rsync_short_args.

And how would it differentiate between hardlinks because of multiple
versions and because they are in the original data?

If we have three generations of file1 and file2 (which are hardlinked in
the original data), do file1 and file2 have three or six links?

Best
Martin

[Tapani Tarvainen]

Post by Martin Schröder

Post by Nico Kadel-Garcia
I beg your pardon? That is the "-H" option for rsync. It's not enabled
by default, but it's certainly available in the rsync_short_args.

And how would it differentiate between hardlinks because of multiple
versions and because they are in the original data?

Links in original data appear in the same directory tree
(.../daily.0/... &c), links created by rsnapshot are outside it.
Nothing else is needed to distinguish them.

Post by Martin Schröder
If we have three generations of file1 and file2 (which are hardlinked in
the original data), do file1 and file2 have three or six links?

Six, of course.

But when you restore from there, it'll only create two - those
that are in the directory tree you're restoring from.

Suggestion: try it.

[Tapani Tarvainen]

Post by Martin Schröder
rsnapshot does not preserve hard links _in_ the data to backup.

With default settings it does not, but you can explicitly ask it
to do so with, e.g.,

rsync_short_options=-aH

in the configuration file.

[Patrick O'Callaghan]

Post by Tapani Tarvainen

Post by Martin Schröder
rsnapshot does not preserve hard links _in_ the data to backup.

With default settings it does not, but you can explicitly ask it
to do so with, e.g.,
rsync_short_options=-aH
in the configuration file.

However I recommend reading the rsync man page on the -H option. It lists
several cases in which the result may not be what you expect (not sure if
any of them apply to rsnapshot).

poc

[Martin Schröder]

Post by Patrick O'Callaghan
However I recommend reading the rsync man page on the -H option. It lists
several cases in which the result may not be what you expect (not sure if
any of them apply to rsnapshot).

"If you specify a --link-dest directory that contains hard links, the
linking of the destination files against the --link-dest files can
cause some paths in the destination to become linked together due to
the --link-dest associations."

That would apply to rsnapshot, right?

Best
Martin

[Patrick O'Callaghan]

Post by Martin Schröder

Post by Patrick O'Callaghan
However I recommend reading the rsync man page on the -H option. It lists
several cases in which the result may not be what you expect (not sure if
any of them apply to rsnapshot).

"If you specify a --link-dest directory that contains hard links, the
linking of the destination files against the --link-dest files can
cause some paths in the destination to become linked together due to
the --link-dest associations."
That would apply to rsnapshot, right?

I would imagine so, but it's not clear is whether this is a bug or a
feature.

Note that none of this addresses the problem of *restoring* hard links. A
restore would have to use rsync -H again (or a dedupe process) to recover
the exact same state, and I suspect someone could find a counterexample
even then. Whether this matters or not depends on what we mean by "restore".

poc

[David Cantrell]

Post by Gandalf Corvotempesta
Absolutely not. This is not the case.
I'm trying to replace a Bacula backup environment (too many things to
check: huge databases (mine is 980GB), too many backup leves, too
prone to failures, ....) with
a smarter system like rsnapshot, where everything could be summarized
with: "rsync + cp + mv". No backup leves, no databases, files always
available for restore (and even for looking, like "cat my_lost_file")
I would like to copy the weekly backups to a tape library. Obviously I
need to resolve all hardlinks when copying or i'll end up with
inconsistent data in case of restore.
My question is: how can I detect an hardlink made by rsnapshot from an
hardlink that was on the source server ? rsnapshot's hardlink must be
dereferenced when copying to tape, original hardlink must preserved to
restore the system in the original state.

Hard links will be preserved within a snapshot if you tell rsync to use
-S. Apart from that rsnapshot will never create a hard link between two
files within a snapshot.

Unfortunately the only way to tell the difference between those and the
hard links between snapshots is to grovel over the filesystem and
compare every single file's inode number and path.

Consider this:
inode number filename
1 snapshot_root/weekly.0/a
1 snapshot_root/weekly.0/b
2 snapshot_root/weekly.0/c
3 snapshot_root/weekly.0/d
1 snapshot_root/weekly.1/a
1 snapshot_root/weekly.1/b
2 snapshot_root/weekly.1/c
4 snapshot_root/weekly.1/d

What this means is that on your source, a and b are hardlinks to each
other. You want to preserve that hard link. c has only one link on the
source, but hasn't changed recently so has extra links in your backups.
d also has only one link on the source, and has changed recently so only
has one link in your backups.

[David Cantrell]

Post by David Cantrell
Hard links will be preserved within a snapshot if you tell rsync to use
-S.

That should of course be -H. -S is to tell it to create sparse files
when possible.

[Christopher Barry]

On Sat, 30 Apr 2016 12:21:44 +0200

Post by Gandalf Corvotempesta

Post by Scott Hess
Meta-comment: Often I/O bandwidth or seeks are the limiting factor,
and running things in parallel will convert 5 serial 10-minute jobs
into 5 parallel 60-minute jobs. Make sure you're testing for that
in your case. Especially the cp -al and rm -rf phases are unlikely
to enjoy competing with each other for resources, whereas rsync
phases could plausibly timeshare with each other.

This is something that I've not considered, you are right.
What if i'll use "link-dest" and not "cp -al" ? it should avoid the cp phase.
My biggest issue is to finish all rsync phase as fast as possible, to
keep load on backupped server low. I'm running rsync in the middle of
the night, running backup serially (I have to backup 98 servers) will
result having some rsync running during the day. This is not what I
want.
"cp -al" is run on backup server, thus I don't care about load.
As improvement, I can run "rsnapshot sync" in parallel, and after all
syncs, I can run the daily rotation, even sequeantially.
But with "link-dest" the whole "cp" phase should be avoided, right ?

Post by Scott Hess
Only the first level benefits from sync. I always run the
longer-period rsnapshot calls first, because when combined with
use_lazy_deletes, those runs can be relatively tightly constrained
in runtime (though I leave a gap to my next sync, to let all the
lazy deletes clear out).

Ok, but rotating before sync could lead to deleted backups (based on
retention) even if the following sync is not completed.

Post by Scott Hess
How and where? Why not just have the script log somewhere
directly?

Yes, is the best solution.

Post by Scott Hess
Running hardlink every time is probably going to take a long time
and not do much most of the time. If you are frequently generating
duplicate data across hosts (like from a script or something), you
might either explore whether you actually need to backup that data,
or whether you can dedupe in targeted directories. I will dedupe
periodically either when I'm doing some other maintenance or when I
notice disk usage seems up.

I don't want to dedup across hosts (at the moment).
daily.3 has almost 90% of shared files with daily.2
I've tried to run "hardlink" across all backups for 1 host and saved 12GB

Post by Scott Hess
I just dedupe across my entire volume. The premise of rsnapshot
means that hardlinks are fine, if they weren't the entire thing
would fall over. Just make sure you aren't deduping while rsnapshot
is running. On most systems you cannot hardlink across volumes.

So, you are dedupe across daily.1 then daily.2 and so on or across all
backup leves at
hardlink myserver/daily.1
hardlink myserver/daily.2
or
hardlink myserver/*

The most important thing is to understand what truly needs to be backed
up. Looking at your exclusion lists (in a prior thread element) leads me
to believe that you might be backing up a bunch of stuff that's really
OS package data, and not your own specific unique data. Understand that
you will never restore an OS from this backup. Indeed, having any OS
data could create major issues for you during a restore operation later.

That said, if you are pulling a bunch of similar nodes to a central
host, maybe you should rethink what you really need. Adding another
disk to each box that only handles backups seriously ups the security
of the data, reduces backup time and resource requirements, while
eliminating network bandwidth consumption. From that backup, the
'previous' node numerically (current == node0008, previous == node0007)
could then pull current's backup, niced way down (bwlimit in rsync),
over the network.

Now, the data exists in three places. In this model there is no
centralized server or point of failure. 3 separate physical disks
scattered between 2 boxes would need to fail before you'd lose data.

Centralization is a double-edged sword. Sure, single point of
management, but if the central backup server craps out, everything is
at risk.

In almost every analysis of ensuring redundancy, a distributed system
wins.

[Gandalf Corvotempesta]

Post by Christopher Barry
The most important thing is to understand what truly needs to be
backed up. Looking at your exclusion lists (in a prior thread element)
leads me to believe that you might be backing up a bunch of stuff
that's really OS package data, and not your own specific unique data.
Understand that you will never restore an OS from this backup. Indeed,
having any OS data could create major issues for you during a restore
operation later.

Why?
Almost all of my server are virtual.
In the past, when I had to restore the whole server, I did a clone from
another, and restored the whole backup, even the OS.

One of Linux advantages is that everything is a file, thus everything
could be backupped and restored.

Post by Christopher Barry
That said, if you are pulling a bunch of similar nodes to a central
host, maybe you should rethink what you really need. Adding another
disk to each box that only handles backups seriously ups the security
of the data, reduces backup time and resource requirements, while
eliminating network bandwidth consumption. From that backup, the
'previous' node numerically (current == node0008, previous ==
node0007) could then pull current's backup, niced way down (bwlimit in
rsync), over the network. Now, the data exists in three places. In
this model there is no centralized server or point of failure. 3
separate physical disks scattered between 2 boxes would need to fail
before you'd lose data. Centralization is a double-edged sword. Sure,
single point of management, but if the central backup server craps
out, everything is at risk. In almost every analysis of ensuring
redundancy, a distributed system wins.

In my environment I have 6 backup server, not 1.
What I've posted here is a simplification, just a POC on what I'm
working on: parallelism and dedup, not an exact mirror of my environemnt.

I'm backing up tons of small VM, about 80-90 for each backup servers.
Most of them are almost identical (for example, varnish node 1,2,3,4,5,6
are all the same, DNS1,2,3,4 are identical, MX1,2,3,4 are identical and
so on).
Backupping them totally, in my environment, is easier than choose what
to backup and what to ignore, because I have the same VM template with
the same rsync configuration cloned multiple times and rsnapshot
automatically detect the new server (my hostlist in
/etc/rsnapshot.d/hosts/*.conf is created dynamically) and run the backup
every night.

Filtering out some servers is a waste of time. It's much easier to run
something like:
$ ping -c1 ${ip}
across my subnets once a day and create the configuration file
automatically if host is up

[Christopher Barry]

On Sun, 1 May 2016 18:44:40 +0200

Post by Gandalf Corvotempesta

Post by Christopher Barry
The most important thing is to understand what truly needs to be
backed up. Looking at your exclusion lists (in a prior thread
element) leads me to believe that you might be backing up a bunch of
stuff that's really OS package data, and not your own specific
unique data. Understand that you will never restore an OS from this
backup. Indeed, having any OS data could create major issues for you
during a restore operation later.

Why?

because you're unwittingly asking for pain.

Post by Gandalf Corvotempesta
Almost all of my server are virtual.
In the past, when I had to restore the whole server, I did a clone
from another, and restored the whole backup, even the OS.
One of Linux advantages is that everything is a file, thus everything
could be backupped and restored.

open files will bite you. versions will bite you.

Post by Gandalf Corvotempesta

Post by Christopher Barry
That said, if you are pulling a bunch of similar nodes to a central
host, maybe you should rethink what you really need. Adding another
disk to each box that only handles backups seriously ups the
security of the data, reduces backup time and resource requirements,
while eliminating network bandwidth consumption. From that backup,
the 'previous' node numerically (current == node0008, previous ==
node0007) could then pull current's backup, niced way down (bwlimit
in rsync), over the network. Now, the data exists in three places.
In this model there is no centralized server or point of failure. 3
separate physical disks scattered between 2 boxes would need to fail
before you'd lose data. Centralization is a double-edged sword.
Sure, single point of management, but if the central backup server
craps out, everything is at risk. In almost every analysis of
ensuring redundancy, a distributed system wins.

In my environment I have 6 backup server, not 1.

and it sounds like you need them, given your use case and methodology.

Post by Gandalf Corvotempesta
What I've posted here is a simplification, just a POC on what I'm
working on: parallelism and dedup, not an exact mirror of my
environemnt.
I'm backing up tons of small VM, about 80-90 for each backup servers.
Most of them are almost identical (for example, varnish node
1,2,3,4,5,6 are all the same, DNS1,2,3,4 are identical, MX1,2,3,4 are
identical and so on).
Backupping them totally, in my environment, is easier than choose what
to backup and what to ignore, because I have the same VM template with
the same rsync configuration cloned multiple times and rsnapshot
automatically detect the new server (my hostlist in
/etc/rsnapshot.d/hosts/*.conf is created dynamically) and run the
backup every night.

Just a very 'sledgehammer' approach IMHO.


Initially I assumed we were dealing with a lot of baremetal servers.
Either way, my point is you're approach is not really scalable.

Consider your use of VMs here. You copy the same template OS image a
bunch of times, then change the configuration data inside each copy to
'personalize' them. Sans data, these images are likely 99.95%
identical. Apparently, you're also storing the node's data inside these
image files as well. All you really care about is what makes this node
unique.

A better way to have a bunch of identical varnish, or dns, or mail, or
whatever servers would be to PXE boot a single read-only generic vm
image as many times as required, likely using iPXE[1], with a writable
disk file as an overlay fs for configuration and/or data per node. This
writable overlay is the node's 'personality' if you will. You'd get the
node's identity via dhcp of course. You might also attach to and/or
mount additional needed unique data space from elsewhere using iscsi or
nfs, or some other method. The writable config images and other mounted
space is the data you'd backup with rsnapshot, never *any* of the OS
image itself this way.

You'd backup that base image just once somewhere. Update the image,
deploy it, and you've just updated *all* vms after they reboot. Try out
a completely new image with one of the existing config images, and
easily roll back if there's an issue. The bootable primary image is
read-only, so to recover from a compromise requires only a simple clean
reboot. Any nefarious code or data is easily viewable in the config
overlay image for forensics.

One good use of centralization though is a log server to catch all the
nodes' logs in a single place. That makes log analysis simple for all
nodes, and keeps the config images light.

You need parallel and de-duplication now because your fundamental
understanding of how best to deploy a lot of similar VM nodes is
insufficient. My suggestion was for you to rethink the underlying
problem, not how to best make up for a flawed design later. I'm not
trying to be confrontational Gandalf, just sharing hard-won knowledge
from a lifetime of systems administration. You are designing a fix to a
problem that is really a symptom of a non-scalable, non-optimal
methodology. I am merely trying to help you view this from a different
perspective. One that I will agree can be difficult to see while you're
embroiled in keeping your head above water.

I wish you only good luck in however you choose to solve your problems.


-C

[1] http://ipxe.org/

Post by Gandalf Corvotempesta
Filtering out some servers is a waste of time. It's much easier to run
$ ping -c1 ${ip}
across my subnets once a day and create the configuration file
automatically if host is up
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications
Manager Applications Manager provides deep performance insights into
multiple tiers of your business applications. It resolves application
problems quickly and reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Nico Kadel-Garcia]

On Sun, May 1, 2016 at 3:35 PM, Christopher Barry

Post by Christopher Barry
On Sun, 1 May 2016 18:44:40 +0200

Post by Christopher Barry
The most important thing is to understand what truly needs to be
backed up. Looking at your exclusion lists (in a prior thread
element) leads me to believe that you might be backing up a bunch of
stuff that's really OS package data, and not your own specific
unique data. Understand that you will never restore an OS from this
backup. Indeed, having any OS data could create major issues for you
during a restore operation later.

Why?

because you're unwittingly asking for pain.

I've done it, but it took real caution. Database files that may be in
the midst of an atomic transation, and not yet written to the
filesystem, are a real risk, for relational databases in particular.

[Gandalf Corvotempesta]

Post by Nico Kadel-Garcia
I've done it, but it took real caution. Database files that may be in
the midst of an atomic transation, and not yet written to the
filesystem, are a real risk, for relational databases in particular.

I'm using a pre-xfer script to run a mysqldump on each host.
I don't restore relationals dbs directly from files, but from an sql
dump (taken with read lock and flush tables)

[Gandalf Corvotempesta]

Post by Christopher Barry
open files will bite you. versions will bite you.

If I have to restore a whole system, there are no open files to restore.
Obviously I restore by using an external system, for example a boot cd with
my (empty) disks mounted on /mnt

Post by Christopher Barry
You need parallel and de-duplication now because your fundamental
understanding of how best to deploy a lot of similar VM nodes is
insufficient. My suggestion was for you to rethink the underlying
problem, not how to best make up for a flawed design later. I'm not
trying to be confrontational Gandalf, just sharing hard-won knowledge
from a lifetime of systems administration. You are designing a fix to a
problem that is really a symptom of a non-scalable, non-optimal
methodology. I am merely trying to help you view this from a different
perspective. One that I will agree can be difficult to see while you're
embroiled in keeping your head above water.

your approach is absolutely correct, but you miss some points:

1) the current architecture wasn't planned by me but by someone else
that is not working here anymore.
2) my CEO would not agree to change the whole architecture (running
stable from many many year) to just use a backup software
3) based on point 2, it would be easier, for CEO, to change the backup
software, or, maybe, change the current sysadmin (me) than rethinking
the whole environment.

[Helmut Hullen]

Hallo, Patrick,

Post by Patrick O'Callaghan

Post by Nico Kadel-Garcia
Copying a large daily.3 snapshot aside, and making sure it's
consistent, can be tricky if the copy rotates under you. This is why
I've long wanted to change the numbering scheme from "daily.0",
"daily.1", etc. to "daily.20160401010203", "daily.20160402113433",
to use full UTC compatbile YYYYMMDDhhmmss date stamped names. But
I've never gotten the traction to write and submit a patch.,

That would be a great idea. I always have to stop and remember if
daily.0 is the most recent or the least recent. And having a
timestamp would add useful information that's otherwise buried in log
files.

Sorry - the backups _have_ a time stamp. They don't need the same (or
another) date string in the directory name.


If you can't tell your file manager to show this time stamp then a kind
of copy (including the hard links) needs few place and doesn't disturb
the "rsnapshot" way of rotating backups.

Viele Gruesse!
Helmut

[Patrick O'Callaghan]

Post by Helmut Hullen

Post by Patrick O'Callaghan
That would be a great idea. I always have to stop and remember if
daily.0 is the most recent or the least recent. And having a
timestamp would add useful information that's otherwise buried in log
files.

Sorry - the backups _have_ a time stamp. They don't need the same (or
another) date string in the directory name.

Where?

poc

[Scott Hess]

Post by Patrick O'Callaghan

Post by Helmut Hullen

Post by Patrick O'Callaghan
That would be a great idea. I always have to stop and remember if
daily.0 is the most recent or the least recent. And having a
timestamp would add useful information that's otherwise buried in log
files.

Sorry - the backups _have_ a time stamp. They don't need the same (or
another) date string in the directory name.

Where?

On Linux, ls -latr is what you want.
-l long output
-a all files (including .sync)
-t sort by time
-r reversed

-scott

[Patrick O'Callaghan]

Post by Scott Hess

Post by Patrick O'Callaghan

Post by Helmut Hullen

Post by Patrick O'Callaghan
That would be a great idea. I always have to stop and remember if
daily.0 is the most recent or the least recent. And having a
timestamp would add useful information that's otherwise buried in log
files.

Sorry - the backups _have_ a time stamp. They don't need the same (or
another) date string in the directory name.

Where?

On Linux, ls -latr is what you want.
-l long output
-a all files (including .sync)
-t sort by time
-r reversed

I thought you might say that. Unfortunately it's *not* a timestamp. It's
the latest access (or, with other options, modification) time of a set of
files, as seen from the rsnapshot server. There are some problems with this:

1) It's only visible to users with access to the server's filesystem (via
login or remote mount).

2) It doesn't indicate when the backup was run, only when the most recent
file modification happened (and that's assuming you can access all the
files and not just your own). They are not the same thing.

poc

[Scott Hess]

Post by Patrick O'Callaghan

On Wed, May 4, 2016 at 9:46 AM, Patrick O'Callaghan <

Post by Patrick O'Callaghan

Post by Helmut Hullen

Post by Patrick O'Callaghan
That would be a great idea. I always have to stop and remember if
daily.0 is the most recent or the least recent. And having a
timestamp would add useful information that's otherwise buried in log
files.

Sorry - the backups _have_ a time stamp. They don't need the same (or
another) date string in the directory name.

Where?

On Linux, ls -latr is what you want.
-l long output
-a all files (including .sync)
-t sort by time
-r reversed

I thought you might say that. Unfortunately it's *not* a timestamp. It's
the latest access (or, with other options, modification) time of a set of
1) It's only visible to users with access to the server's filesystem (via
login or remote mount).

How do users get access to the in-the-directory-name timestamps if they
don't have access to the filesystem where those directories live?

2) It doesn't indicate when the backup was run, only when the most recent

Post by Patrick O'Callaghan
file modification happened (and that's assuming you can access all the
files and not just your own). They are not the same thing.

Rsnapshot sets the directory timestamp using touch, I think as the last
thing it does. It's intentional, not some sort of side effect. I'm not
sure how renaming the directory would somehow be stronger than that. If
the actual timestamp changes after that, it's because someone is messing
around with the files, which in my experience should be avoided at all cost.

For my rsnapshot root, the timestamps are ~40s after the time when the cron
job is scheduled to run. They were ~20m until I changed to sync&&hourly,
with a sync pass run about an hour ahead of time.

Personally, I wouldn't mind having a .timestamp file in the snapshot, which
would contain a printed form of the timestamp rsnapshot used for the touch,
and perhaps with the same timestamp as the directory using touch -r. That
file would never need to be renamed after the snapshot is taken, so it
would be like a two-line patch. Something like that would be more reliable
for scripting, as the script could either use the filesystem timestamp or
parse the printed timestamp, and the printed timestamp could include high
precision without needing config options.

-scott

[Patrick O'Callaghan]

Post by Patrick O'Callaghan
I thought you might say that. Unfortunately it's *not* a timestamp. It's

Post by Patrick O'Callaghan
the latest access (or, with other options, modification) time of a set of
1) It's only visible to users with access to the server's filesystem (via
login or remote mount).

How do users get access to the in-the-directory-name timestamps if they
don't have access to the filesystem where those directories live?

That's a fair point. Nevertheless, being able to list a set of directory
names requires very little rights compared to listing all files with their
access times. The only requirement is read access to the top-level
directory where the backups live.

Post by Patrick O'Callaghan
2) It doesn't indicate when the backup was run, only when the most recent

Post by Patrick O'Callaghan
file modification happened (and that's assuming you can access all the
files and not just your own). They are not the same thing.

Rsnapshot sets the directory timestamp using touch, I think as the last
thing it does. It's intentional, not some sort of side effect. I'm not
sure how renaming the directory would somehow be stronger than that. If
the actual timestamp changes after that, it's because someone is messing
around with the files, which in my experience should be avoided at all cost.

Perhaps, but that behaviour is not documented as far as I know.

Post by Patrick O'Callaghan
For my rsnapshot root, the timestamps are ~40s after the time when the
cron job is scheduled to run. They were ~20m until I changed to
sync&&hourly, with a sync pass run about an hour ahead of time.
Personally, I wouldn't mind having a .timestamp file in the snapshot,
which would contain a printed form of the timestamp rsnapshot used for the
touch, and perhaps with the same timestamp as the directory using touch
-r. That file would never need to be renamed after the snapshot is taken,
so it would be like a two-line patch. Something like that would be more
reliable for scripting, as the script could either use the filesystem
timestamp or parse the printed timestamp, and the printed timestamp could
include high precision without needing config options.

That would work, though with slightly more access rights than already
mentioned (+x permission on the backup directories and +r on the .timestamp
files).

Of course if user backup directories are generally readable by their
owners, the permissions question is not an issue. That may be the normal
case.

poc

[Christopher Barry]

On Wed, 4 May 2016 10:30:58 -0700

On Wed, May 4, 2016 at 10:12 AM, Patrick O'Callaghan

Post by Patrick O'Callaghan

On Wed, May 4, 2016 at 9:46 AM, Patrick O'Callaghan <

Post by Patrick O'Callaghan

Post by Helmut Hullen

Post by Patrick O'Callaghan
That would be a great idea. I always have to stop and remember
if daily.0 is the most recent or the least recent. And having a
timestamp would add useful information that's otherwise buried
in log files.

Sorry - the backups _have_ a time stamp. They don't need the same
(or another) date string in the directory name.

Where?

On Linux, ls -latr is what you want.
-l long output
-a all files (including .sync)
-t sort by time
-r reversed

I thought you might say that. Unfortunately it's *not* a timestamp.
It's the latest access (or, with other options, modification) time
of a set of files, as seen from the rsnapshot server. There are some
1) It's only visible to users with access to the server's filesystem
(via login or remote mount).

How do users get access to the in-the-directory-name timestamps if they
don't have access to the filesystem where those directories live?
2) It doesn't indicate when the backup was run, only when the most recent

Post by Patrick O'Callaghan
file modification happened (and that's assuming you can access all
the files and not just your own). They are not the same thing.

Rsnapshot sets the directory timestamp using touch, I think as the last
thing it does. It's intentional, not some sort of side effect. I'm
not sure how renaming the directory would somehow be stronger than
that. If the actual timestamp changes after that, it's because
someone is messing around with the files, which in my experience
should be avoided at all cost.
For my rsnapshot root, the timestamps are ~40s after the time when the
cron job is scheduled to run. They were ~20m until I changed to
sync&&hourly, with a sync pass run about an hour ahead of time.
Personally, I wouldn't mind having a .timestamp file in the snapshot,
which would contain a printed form of the timestamp rsnapshot used for
the touch, and perhaps with the same timestamp as the directory using
touch -r. That file would never need to be renamed after the snapshot

in cron, do a:

date > /etc/rsnapshot/.timestamp && rsnapshot <increment>

where /etc/rsnapshot is one of the backed up directories in every
backup.

and bada-bing, Bob's your Uncle. :)

is taken, so it would be like a two-line patch. Something like that
would be more reliable for scripting, as the script could either use
the filesystem timestamp or parse the printed timestamp, and the
printed timestamp could include high precision without needing config
options.
-scott

[Scott Hess]

On Wed, May 4, 2016 at 4:20 PM, Christopher Barry <

Post by Christopher Barry

Post by Scott Hess
Personally, I wouldn't mind having a .timestamp file in the snapshot,
which would contain a printed form of the timestamp rsnapshot used for
the touch, and perhaps with the same timestamp as the directory using
touch -r. That file would never need to be renamed after the snapshot

date > /etc/rsnapshot/.timestamp && rsnapshot <increment>
where /etc/rsnapshot is one of the backed up directories in every
backup.
and bada-bing, Bob's your Uncle. :)

Derp.

Like:

/usr/bin/rsnapshot sync && date -R -r .sync >.sync/.timestamp && touch -r
.sync .sync/.timestamp

Then next rotation will just carry it along, forever. Create
.sync/.timestamp ahead of time so that the first creation doesn't screw up
the timestamp on .sync.

Like I said, two-line change :-).

Or cmd_postexec, which runs after backup for the lowest interval, and the
backup finishes by updating the directory timestamp.

-scott

[Nico Kadel-Garcia]

Post by Scott Hess
On Wed, May 4, 2016 at 4:20 PM, Christopher Barry

Post by Christopher Barry

Post by Scott Hess
Personally, I wouldn't mind having a .timestamp file in the snapshot,
which would contain a printed form of the timestamp rsnapshot used for
the touch, and perhaps with the same timestamp as the directory using
touch -r. That file would never need to be renamed after the snapshot

date > /etc/rsnapshot/.timestamp && rsnapshot <increment>
where /etc/rsnapshot is one of the backed up directories in every
backup.
and bada-bing, Bob's your Uncle. :)

Which does you *zero* good if you've got a tar, an rsync, or simple
NFS mount cp going on from the "daily.0" snapshot and it rotates out
from under you in the midst of the replication, This is an old
problem. Stuffing a legibble timestamp inside the numbered snapshot
does not help much when the path to the version of the file changes
out from under you.

Post by Scott Hess
Derp.
/usr/bin/rsnapshot sync && date -R -r .sync >.sync/.timestamp && touch -r
.sync .sync/.timestamp
Then next rotation will just carry it along, forever. Create
.sync/.timestamp ahead of time so that the first creation doesn't screw up
the timestamp on .sync.
Like I said, two-line change :-).

Which does not solve the underlying problem I referred to.

[Ken Woods]

Nico, the horse is dead.
Seriously.

Post by Nico Kadel-Garcia

Post by Scott Hess
On Wed, May 4, 2016 at 4:20 PM, Christopher Barry

Post by Christopher Barry

Post by Scott Hess
Personally, I wouldn't mind having a .timestamp file in the snapshot,
which would contain a printed form of the timestamp rsnapshot used for
the touch, and perhaps with the same timestamp as the directory using
touch -r. That file would never need to be renamed after the snapshot

date > /etc/rsnapshot/.timestamp && rsnapshot <increment>
where /etc/rsnapshot is one of the backed up directories in every
backup.
and bada-bing, Bob's your Uncle. :)

Which does you *zero* good if you've got a tar, an rsync, or simple
NFS mount cp going on from the "daily.0" snapshot and it rotates out
from under you in the midst of the replication, This is an old
problem. Stuffing a legibble timestamp inside the numbered snapshot
does not help much when the path to the version of the file changes
out from under you.

Post by Scott Hess
Derp.
/usr/bin/rsnapshot sync && date -R -r .sync >.sync/.timestamp && touch -r
.sync .sync/.timestamp
Then next rotation will just carry it along, forever. Create
.sync/.timestamp ahead of time so that the first creation doesn't screw up
the timestamp on .sync.
Like I said, two-line change :-).

Which does not solve the underlying problem I referred to.
------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Scott Hess]

Post by Nico Kadel-Garcia

Post by Scott Hess
On Wed, May 4, 2016 at 4:20 PM, Christopher Barry

Post by Christopher Barry

Post by Scott Hess
Personally, I wouldn't mind having a .timestamp file in the snapshot,
which would contain a printed form of the timestamp rsnapshot used for
the touch, and perhaps with the same timestamp as the directory using
touch -r. That file would never need to be renamed after the snapshot

date > /etc/rsnapshot/.timestamp && rsnapshot <increment>
where /etc/rsnapshot is one of the backed up directories in every
backup.
and bada-bing, Bob's your Uncle. :)

Which does you *zero* good if you've got a tar, an rsync, or simple
NFS mount cp going on from the "daily.0" snapshot and it rotates out
from under you in the midst of the replication, This is an old
problem. Stuffing a legibble timestamp inside the numbered snapshot
does not help much when the path to the version of the file changes
out from under you.

Write up a patch. Pull the thread and find out how long it is. Maybe it's
a lot shorter than the nay-sayers think it is likely to be. Maybe it's a
lot longer than the people who request the feature think it should be.
Either way we'd learn something.

-scott

[Rolf Muth]

Am Mittwoch 4 Mai 16 um 15:45:10 Uhr schrieb "Nico Kadel-Garcia"

Post by Nico Kadel-Garcia
...
I've long wanted to change the numbering scheme from "daily.0",
"daily.1", etc. to "daily.20160401010203", "daily.20160402113433", to
use full UTC compatbile YYYYMMDDhhmmss date stamped names. But I've
never gotten the traction to write and submit a patch.,

Dafür habe ich do-timed-folders und do-del-timed-folders geschrieben...

http://www.heise.de/download/do-rsnapshots-1184971.html

[Helmut Hullen]

Hallo, Patrick,

Post by Patrick O'Callaghan

Post by Scott Hess
Rsnapshot sets the directory timestamp using touch, I think as the
last thing it does. It's intentional, not some sort of side effect.

[...]

Post by Patrick O'Callaghan
Perhaps, but that behaviour is not documented as far as I know.

It is, in every log file.

Viele Gruesse!
Helmut

[Helmut Hullen]

Hallo, Gandalf,

daily.5. No ".sync" or anything else. Only daily.5
"daily.5" should be totally available as all hardlinked files are
still retained (due to link count > 0 thus files are still there)

Take a look at the "rsnapshot.log".

There you can see that you can rename each backup to some other name
which "rsnapshot" doesn't change, and there you can see that yoy can
make a "hard linked" copy with the simple command "rsync ..." - just
copy such a logged line to the CLI and adjust the source directory and
the target directory.

By the way: hard linked files and directories need to be one the same
disc/partition. Copying such a directory to another partition doesn't
hard link from the source partition.

Viele Gruesse!
Helmut

[Helmut Hullen]

Hallo, Tapani,

Post by Tapani Tarvainen

Post by Martin Schröder
rsnapshot does not preserve hard links _in_ the data to backup.

With default settings it does not, but you can explicitly ask it
to do so with, e.g.,
rsync_short_options=-aH
in the configuration file.

On a tape?

Viele Gruesse!
Helmut

[Tapani Tarvainen]

Post by Helmut Hullen
Hallo, Tapani,

Post by Tapani Tarvainen

Post by Martin Schröder
rsnapshot does not preserve hard links _in_ the data to backup.

With default settings it does not, but you can explicitly ask it
to do so with, e.g.,
rsync_short_options=-aH
in the configuration file.

On a tape?

:-)

Obviously not, but then it would not be rsnapshot that'd
write it to the tape, would it?

Though... I have seen a system that _could_ have hard links
on a tape, along with everything else one could have on a disk:
recovery system on some ancient HP-UX box.
It was rather amusing to watch it swap on tape. :-)
(OK, it stopped being amusing after a few hours.)

And actually I think 'tar' could write and read hard links
to and from tape, although I don't recall ever trying that.
So if you use the -H option with rsnapshot and then archive
the backup tree with tar, hard links could be preserved
in restore.

[Nico Kadel-Garcia]

On Fri, May 6, 2016 at 8:48 AM, Tapani Tarvainen

Post by Tapani Tarvainen

Post by Helmut Hullen
Hallo, Tapani,

Post by Tapani Tarvainen

Post by Martin Schröder
rsnapshot does not preserve hard links _in_ the data to backup.

With default settings it does not, but you can explicitly ask it
to do so with, e.g.,
rsync_short_options=-aH
in the configuration file.

On a tape?

:-)
Obviously not, but then it would not be rsnapshot that'd
write it to the tape, would it?
Though... I have seen a system that _could_ have hard links
recovery system on some ancient HP-UX box.
It was rather amusing to watch it swap on tape. :-)
(OK, it stopped being amusing after a few hours.)
And actually I think 'tar' could write and read hard links
to and from tape, although I don't recall ever trying that.
So if you use the -H option with rsnapshot and then archive
the backup tree with tar, hard links could be preserved
in restore.

tar, and dump, do hardlinks just fine.

[David Cantrell]

Post by Helmut Hullen
On a tape?

There's nothing preventing backup software from keeping track of hard
links when writing to tape. Even Ye Olde tar knows about them:

$ ls -il foo bar
45432431 -rw-r--r-- 2 dc staff 0 9 May 14:00 bar
45432431 -rw-r--r-- 2 dc staff 0 9 May 14:00 foo
$ tar cf baz.tar foo bar
$ tar tvf baz.tar
-rw-r--r-- 0 dc staff 0 9 May 14:00 foo
hrw-r--r-- 0 dc staff 0 9 May 14:00 bar link to foo

[Patrick O'Callaghan]

Post by David Cantrell

Post by Helmut Hullen
On a tape?

There's nothing preventing backup software from keeping track of hard
$ ls -il foo bar
45432431 -rw-r--r-- 2 dc staff 0 9 May 14:00 bar
45432431 -rw-r--r-- 2 dc staff 0 9 May 14:00 foo
$ tar cf baz.tar foo bar
$ tar tvf baz.tar
-rw-r--r-- 0 dc staff 0 9 May 14:00 foo
hrw-r--r-- 0 dc staff 0 9 May 14:00 bar link to foo

Once again: keeping track of hard links is trivial. Distinguishing between
hard links present in the original data and those created by the backup
system is expensive. Tar doesn't create any new links that weren't already
there in the data, so it's not the same as rsnapshot.

poc

[Helmut Hullen]

Hallo, Gandalf,

Post by Gandalf Corvotempesta
So, in my example, "file1" and "file2" would be copied as files and
not hardlinked together ?

Surely - a tape doesn't use hard links between backups.

Viele Gruesse!
Helmut

[Helmut Hullen]

Hallo, Nico,

Post by Nico Kadel-Garcia

Post by Martin Schröder
rsnapshot does not preserve hard links _in_ the data to backup.

[...]

Post by Nico Kadel-Garcia
tar, and dump, do hardlinks just fine.

And that's far away from "rsnapshot".

Viele Gruesse!
Helmut

[Nico Kadel-Garcia]

Post by Helmut Hullen
Hallo, Nico,

Post by Nico Kadel-Garcia

Post by Martin Schröder
rsnapshot does not preserve hard links _in_ the data to backup.

[...]

Post by Nico Kadel-Garcia
tar, and dump, do hardlinks just fine.

And that's far away from "rsnapshot".
Viele Gruesse!
Helmut

One of our previous commenters referred to writing the rsnapshot
copies to tape, and whether internal hardlinks would be preserved.
Both technologies work just fine for just that purpose.

In fact, "dump" is deprecated these days for lots of reasons. If you
need to backup rsnapshot copies to tape, I've had good success with
using the "AMANDA" software, and backup up the "weekly" and moving
aside a properly date-labeled "cp -al" copy of the "daily"
specifically to prevent accidental rsnapshot rotation during the
backup.

I'm also reaching back into the "WayBack" machine about this stuff,
that was way back when I first encountered rsnapshot and when I ported
AMANDA to SunOS....

[Gandalf Corvotempesta]

Post by Nico Kadel-Garcia
One of our previous commenters referred to writing the rsnapshot
copies to tape, and whether internal hardlinks would be preserved.
Both technologies work just fine for just that purpose.

Ok. I admit, I've created much confusion.
Let me try to explain (keep in mind that english is not my native language)

Actually i'm backingup tons of hosts with bacula. I would like to
replace bacula
with something better (for reasons that I don't repeat) and easier to
maitain.

i'm trying rsnapshot with a subset of these hosts and is working very well.
Now, I have 4 issues to address:

1) how to restore hardlinks that was present on source host? This is not
a big
deal, just a question because I'm unable to differentiate between
existing hardlinks
and hardlinks created by rsnapshot during the "cp -al" phase

2) I have to copy one or more backups to a tape library, for disaster
recovery.
Obviously, tape will be used only when everything is going bad, they are
a very
last resort. There is no need (for me) to preserve hardlinks. If i have
to restore from
tapes, everything else is lost, so having the exact hardlink preserved
it doesn't matter.

3) as with bacula there are many, many, many points to check and to
administer (mysql db,
full level retention, and so on), many things could go wrong.
In example if you loose/corrupt 1 full, you loose ALL backups for that host.
I would like to be sure that this is impossible with rsnapshot, as all
backup levels are indipendent.
Deleting everything except 1 directory doesn't loose all backups but
only the deleted ones.

4) hardlinks must be on the same filesystem. This is clear. So,
something like this:
$ cp daily.4 /mnt/other_file_system
would *resolve* all hardlinks and copy the whole backup (without
hardlinks) to the other filesystem, right?
This is something I could try on my one, next Monday at work.
The same should be when using tar over tape. I have to save the file
content, not a bad hardlink pointer.

[Ken Woods]

Post by Nico Kadel-Garcia
and moving
aside a properly date-labeled

You're going to push this forever, aren't you? It's so small minded and not scalable.

My snapshots can take weeks to run over infinitBand. What date would you propose is "proper"--when it completes or starts?

[Helmut Hullen]

Hallo, Gandalf,

Post by Gandalf Corvotempesta
4) hardlinks must be on the same filesystem. This is clear. So,
$ cp daily.4 /mnt/other_file_system
would *resolve* all hardlinks and copy the whole backup (without
hardlinks) to the other filesystem, right?

"That depends!" ...

That depends on the fileystem of the target.

rsync -axH $Source/. $Target

can rebuild/copy all hardlinks, if the filesystem fits.

And newer versions of "cp" also can do this job.

It's necessary if you need a full backup for a bigger (or a new) disk
...

Viele Gruesse!
Helmut