[rsnapshot-discuss] rsnapshot hangs mid-backup

[Ken Rossman]

I know this type of issue has been discussed briefly before on this list,
but I have yet to see a resolution, or much in the way of further analysis
of the issue.

I have a situation where I have a Linux server backing up a Linux client
(both are CentOS6.5+), and rsnapshot gets a good bit into the backup, then
just stops.

I can watch approximately where this occurs by including the -V switch
(lists the files being backed up to the terminal). I am attempting to
only back up a handful of files, so I am using a combination of includes
and excludes, with the resulting echo to the terminal looking like the
following:

‹‹‹‹‹

Setting locale to POSIX "C"
echo 64591 > /var/run/rsnapshot.pid
/usr/bin/rm -rf /pool/backups/rsnapshot/.snapshots/daily.6/
mv /pool/backups/rsnapshot/.snapshots/daily.5/ \
/pool/backups/rsnapshot/.snapshots/daily.6/
mv /pool/backups/rsnapshot/.snapshots/daily.4/ \
/pool/backups/rsnapshot/.snapshots/daily.5/
mv /pool/backups/rsnapshot/.snapshots/daily.3/ \
/pool/backups/rsnapshot/.snapshots/daily.4/
mv /pool/backups/rsnapshot/.snapshots/daily.2/ \
/pool/backups/rsnapshot/.snapshots/daily.3/
mv /pool/backups/rsnapshot/.snapshots/daily.1/ \
/pool/backups/rsnapshot/.snapshots/daily.2/
/usr/bin/cp -al /pool/backups/rsnapshot/.snapshots/daily.0 \
/pool/backups/rsnapshot/.snapshots/daily.1
/usr/bin/rsync -av --delete --numeric-ids --relative --delete-excluded \
--exclude=dev/ --exclude=dev/shm/ --exclude=proc/ --exclude=sys/ \
--exclude=run/ --exclude=home/ --exclude=cores/ --exclude=media/ \
--exclude=opt/ --exclude=root/ --exclude=usr/ --exclude=var/ \
--exclude=20151202/ --include=nwsys/release/conf/.syscfg \
--include=nwsys/release/conf/*.conf** \
--include=nwsys/release/conf/*.cfg** \
--include=nwsys/release/conf/cronfile** \
--include=nwsys/release/conf/storage/** --include=nwtext/client/** \
--include=etc/nagios/nrpe.cfg --include=etc/httpd/conf/httpd.conf \
--include=etc/httpd/conf.d/** --exclude="20151130/" --rsh=/usr/bin/ssh
\
***@xen-1.ma.newsedge.com:/ \
/pool/backups/rsnapshot/.snapshots/daily.0/xen-1.ma.newsedge.com
receiving incremental file list

:
:

pool/nwsys/symsearch/20151124.vlist
pool/nwsys/symsearch/20151127.vlist
pool/nwsys/symsearch/20151130.vlist
pool/nwsys/symsearch/slam.conf
pool/nwsys/symsearch/symsearch.db
pool/nwsys/symsearch/incoming/
pool/nwsys/symsearch/incoming/tmp/
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/



[and it hangs at the above pointŠ forever]


‹‹‹‹‹



I¹ve done a bit of checking to see if maybe rsync is out of some resource
on either side, and it does not appear to be (and if it were, would it not
abort?)

I am at a loss to explain this, and/or to go any further along in the
debugging phase, until I understand a bit more of the nature of this issue
‹ and hence, maybe would have a clue as to what resources to monitor.

Note that this is a REMOTE backup, using ssh as a transportŠ

Can anyone shed some light on this issue, or perhaps give me some things
to try to get to the bottom of what¹s actually going on?

Thanks!

Ken Rossman
***@acquiremedia.com

[Ken Rossman]

Forgot to mention, I am running rsnapshot 1.3.1. Perhaps there are some
updates I can apply?

Post by Ken Rossman
I know this type of issue has been discussed briefly before on this list,
but I have yet to see a resolution, or much in the way of further analysis
of the issue.
I have a situation where I have a Linux server backing up a Linux client
(both are CentOS6.5+), and rsnapshot gets a good bit into the backup, then
just stops.
I can watch approximately where this occurs by including the -V switch
(lists the files being backed up to the terminal). I am attempting to
only back up a handful of files, so I am using a combination of includes
and excludes, with the resulting echo to the terminal looking like the
‹‹‹‹‹
Setting locale to POSIX "C"
echo 64591 > /var/run/rsnapshot.pid
/usr/bin/rm -rf /pool/backups/rsnapshot/.snapshots/daily.6/
mv /pool/backups/rsnapshot/.snapshots/daily.5/ \
/pool/backups/rsnapshot/.snapshots/daily.6/
mv /pool/backups/rsnapshot/.snapshots/daily.4/ \
/pool/backups/rsnapshot/.snapshots/daily.5/
mv /pool/backups/rsnapshot/.snapshots/daily.3/ \
/pool/backups/rsnapshot/.snapshots/daily.4/
mv /pool/backups/rsnapshot/.snapshots/daily.2/ \
/pool/backups/rsnapshot/.snapshots/daily.3/
mv /pool/backups/rsnapshot/.snapshots/daily.1/ \
/pool/backups/rsnapshot/.snapshots/daily.2/
/usr/bin/cp -al /pool/backups/rsnapshot/.snapshots/daily.0 \
/pool/backups/rsnapshot/.snapshots/daily.1
/usr/bin/rsync -av --delete --numeric-ids --relative --delete-excluded \
--exclude=dev/ --exclude=dev/shm/ --exclude=proc/ --exclude=sys/ \
--exclude=run/ --exclude=home/ --exclude=cores/ --exclude=media/ \
--exclude=opt/ --exclude=root/ --exclude=usr/ --exclude=var/ \
--exclude=20151202/ --include=nwsys/release/conf/.syscfg \
--include=nwsys/release/conf/*.conf** \
--include=nwsys/release/conf/*.cfg** \
--include=nwsys/release/conf/cronfile** \
--include=nwsys/release/conf/storage/** --include=nwtext/client/** \
--include=etc/nagios/nrpe.cfg --include=etc/httpd/conf/httpd.conf \
--include=etc/httpd/conf.d/** --exclude="20151130/" --rsh=/usr/bin/ssh
\
/pool/backups/rsnapshot/.snapshots/daily.0/xen-1.ma.newsedge.com
receiving incremental file list
pool/nwsys/symsearch/20151124.vlist
pool/nwsys/symsearch/20151127.vlist
pool/nwsys/symsearch/20151130.vlist
pool/nwsys/symsearch/slam.conf
pool/nwsys/symsearch/symsearch.db
pool/nwsys/symsearch/incoming/
pool/nwsys/symsearch/incoming/tmp/
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/
[and it hangs at the above pointŠ forever]
‹‹‹‹‹
I¹ve done a bit of checking to see if maybe rsync is out of some resource
on either side, and it does not appear to be (and if it were, would it not
abort?)
I am at a loss to explain this, and/or to go any further along in the
debugging phase, until I understand a bit more of the nature of this issue
‹ and hence, maybe would have a clue as to what resources to monitor.
Note that this is a REMOTE backup, using ssh as a transportŠ
Can anyone shed some light on this issue, or perhaps give me some things
to try to get to the bottom of what¹s actually going on?
Thanks!
Ken Rossman

[David Cantrell]

Post by Ken Rossman
I have a situation where I have a Linux server backing up a Linux client
(both are CentOS6.5+), and rsnapshot gets a good bit into the backup, then
just stops.

It looks like it's rsync that's hanging.

Post by Ken Rossman
I can watch approximately where this occurs by including the -V switch
...
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/
[and it hangs at the above pointŠ forever]
Can anyone shed some light on this issue, or perhaps give me some things
to try to get to the bottom of what¹s actually going on?

What is rsync doing on the remote host? Use lsof to see what files it is
trying to access when it appears to be hung.

There may be several rsync processes, so check them all. And then once
you find the file it's reading, check what type that file is and whether
it is readable. Can you copy that file to somewhere else on the same
machine?

[Ken Rossman]

Post by David Cantrell

Post by Ken Rossman
I have a situation where I have a Linux server backing up a Linux client
(both are CentOS6.5+), and rsnapshot gets a good bit into the backup, then
just stops.

It looks like it's rsync that's hanging.

Post by Ken Rossman
I can watch approximately where this occurs by including the -V switch
...
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/
[and it hangs at the above pointŠ forever]
Can anyone shed some light on this issue, or perhaps give me some things
to try to get to the bottom of what¹s actually going on?

What is rsync doing on the remote host? Use lsof to see what files it is
trying to access when it appears to be hung.

I will do that shortly...

Post by David Cantrell
There may be several rsync processes, so check them all.

There's actually a pagefull of them. I'm not sure I am the only one trying
to rsync to/from that particular server, so I need a way to identify which
ones of those procs belong to what I am doing.

Post by David Cantrell
And then once you find the file it's reading, check what type that file
is and whether it is readable. Can you copy that file to somewhere else
on the same machine?

Thanks!!! I will check all of this...

Ken Rossman
***@aquiremedia.com
***@gmail.com

[Nico Kadel-Garcia]

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
I have a situation where I have a Linux server backing up a Linux client
(both are CentOS6.5+), and rsnapshot gets a good bit into the backup, then
just stops.

It looks like it's rsync that's hanging.

Post by Ken Rossman
I can watch approximately where this occurs by including the -V switch
...
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/
[and it hangs at the above pointŠ forever]
Can anyone shed some light on this issue, or perhaps give me some things
to try to get to the bottom of what¹s actually going on?

What is rsync doing on the remote host? Use lsof to see what files it is
trying to access when it appears to be hung.

I will do that shortly...

Post by David Cantrell
There may be several rsync processes, so check them all.

There's actually a pagefull of them. I'm not sure I am the only one trying
to rsync to/from that particular server, so I need a way to identify which
ones of those procs belong to what I am doing.

Post by David Cantrell
And then once you find the file it's reading, check what type that file
is and whether it is readable. Can you copy that file to somewhere else
on the same machine?

Thanks!!! I will check all of this...
Ken Rossman

"netstate" on the remote host should be helpful to determine which
underlying unencrypted rsync or encrypted SSH underlying process is
associated with the connection from your rsync client host.

[David Keegel]

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
I have a situation where I have a Linux server backing up a Linux client
(both are CentOS6.5+), and rsnapshot gets a good bit into the backup, then
just stops.

It looks like it's rsync that's hanging.

Post by Ken Rossman
I can watch approximately where this occurs by including the -V switch
...
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/
[and it hangs at the above pointŠ forever]

Does rsync always hang at the same point?

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
Can anyone shed some light on this issue, or perhaps give me some things
to try to get to the bottom of what¹s actually going on?

What is rsync doing on the remote host? Use lsof to see what files it is
trying to access when it appears to be hung.

If you want some indication of progress during the rsync of each file,
you could try adding
<tab>+rsync_long_args=--progress
to the end of the relevant backup line. That will give you a lot of
output.

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell
There may be several rsync processes, so check them all.

There's actually a pagefull of them. I'm not sure I am the only one trying
to rsync to/from that particular server, so I need a way to identify which
ones of those procs belong to what I am doing.

You might want to pick a time when you are not running rsnapshot (or
anything else that should be using rsync) and look at what rsync
processes are still running. Are they previous "hung" rsnapshots?
If they are unnecessary/useless rsyncs, you might want to kill them.

Post by Nico Kadel-Garcia
"netstate" on the remote host should be helpful to determine which
underlying unencrypted rsync or encrypted SSH underlying process is
associated with the connection from your rsync client host.

Perhap Nico meant something like
netstat -ntp | grep :22
to match the ssh underlying processes with network connections.

To look at what file descriptors are open for all running rsync
processes, you could run
for p in $(/sbin/pidof rsync)
do ls -l /proc/$p/fd/*
done

[Ken Rossman]

I have learned a lot in the past couple of days...

the aforementioned "hang" condition was due to my still not completely
understanding how rsnapshot/rsync 'include/exclude's work, and having
the rsnapshot stray into NFS mount territory.

I have since set:

# stay within one filesystem
one_fs 1

and that seems to have let me go further... backs up a large chunk of
data now in fact, and does not hang.

HOWEVER, the underlying root issue (for me anyway) is still not being
able to get includes/excludes right.

REDEFINITION OF BASE ISSUE:

- I need to back up a handful of files on a system using rsync
(rsnapshot, really).

- rsnapshot at least looks like it's going and getting a huge list of
files, outside the "include" parameters... not sure if the -V
parameter lists EVERY file that rsnapshot/rsync looks at, whether or
not it is subsequently actually copied...

Can anyone point me to a CLEAR and BRIEF tutorial on how to work with
includes and excludes in either/both rsnapshot/rsync?

I'm still very fuzzy on how this works in reality -- order of precedence, etc...

Thanks,
Ken

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
I have a situation where I have a Linux server backing up a Linux client
(both are CentOS6.5+), and rsnapshot gets a good bit into the backup, then
just stops.

It looks like it's rsync that's hanging.

Post by Ken Rossman
I can watch approximately where this occurs by including the -V switch
...
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/
[and it hangs at the above pointŠ forever]

Does rsync always hang at the same point?

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
Can anyone shed some light on this issue, or perhaps give me some things
to try to get to the bottom of what¹s actually going on?

What is rsync doing on the remote host? Use lsof to see what files it is
trying to access when it appears to be hung.

If you want some indication of progress during the rsync of each file,
you could try adding
<tab>+rsync_long_args=--progress
to the end of the relevant backup line. That will give you a lot of
output.

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell
There may be several rsync processes, so check them all.

There's actually a pagefull of them. I'm not sure I am the only one trying
to rsync to/from that particular server, so I need a way to identify which
ones of those procs belong to what I am doing.

You might want to pick a time when you are not running rsnapshot (or
anything else that should be using rsync) and look at what rsync
processes are still running. Are they previous "hung" rsnapshots?
If they are unnecessary/useless rsyncs, you might want to kill them.

Post by Nico Kadel-Garcia
"netstate" on the remote host should be helpful to determine which
underlying unencrypted rsync or encrypted SSH underlying process is
associated with the connection from your rsync client host.

Perhap Nico meant something like
netstat -ntp | grep :22
to match the ssh underlying processes with network connections.
To look at what file descriptors are open for all running rsync
processes, you could run
for p in $(/sbin/pidof rsync)
do ls -l /proc/$p/fd/*
done
--
___________________________________________________________________________
http://www.cyber.com.au/~djk/ Linux & Unix Systems Administration

[Scott Hess]

IMHO, being able to reason about include/exclude is something I will never
achieve except for brief moments, and that state of mind is soon lost.

Instead, I snag rsync command-lines from the logs, then use --dry-run while
I experiment with different include/exclude settings. This approach works
well both for figuring out what includes and excludes to use in the first
place, and for debugging things later. If you're brave, add
--delete-excluded .

I have found that I prefer to code my rsnapshot.conf with:
exclude_file /etc/rsnapshot.excludes
and put my excludes in that file. That makes it easy to put comments in.
I believe I found that putting exclude= arguments at the end of a backup
line in rsnapshot.conf caused the exclude_file global setting to be
ignored. So for a few of my backups the config has explicit
exclude=/apath,exclude_file=/etc/rsnapshot.excludes arguments.

The initial fairly-generic portion of my file looks like:

~~~ /etc/rsnapshot.excludes
; Giant files which churn.
slocate.db
dlocatedb
dlocatedb.old
mlocate.db

; Note: /X/ skips directory X at root of sync'ed directory. /X/*
; syncs the directory but not files within.

; Common directories with transient churn.
/tmp/*
/var/tmp/*
/var/cache/*

; Generates tons of errors if one_fs isn't set.
/sys/*
/proc/*

; Anything named nobackup - including directories.
; TODO: nobackup/ instead?
nobackup
~~~

The /X/ comment leads me to believe that saying /tmp/* syncs the /tmp
directory itself so the filesystem structure is right, but the files within
are skipped. I'm probably wrong, though :-).

-scott

Post by Ken Rossman
I have learned a lot in the past couple of days...
the aforementioned "hang" condition was due to my still not completely
understanding how rsnapshot/rsync 'include/exclude's work, and having
the rsnapshot stray into NFS mount territory.
# stay within one filesystem
one_fs 1
and that seems to have let me go further... backs up a large chunk of
data now in fact, and does not hang.
HOWEVER, the underlying root issue (for me anyway) is still not being
able to get includes/excludes right.
- I need to back up a handful of files on a system using rsync
(rsnapshot, really).
- rsnapshot at least looks like it's going and getting a huge list of
files, outside the "include" parameters... not sure if the -V
parameter lists EVERY file that rsnapshot/rsync looks at, whether or
not it is subsequently actually copied...
Can anyone point me to a CLEAR and BRIEF tutorial on how to work with
includes and excludes in either/both rsnapshot/rsync?
I'm still very fuzzy on how this works in reality -- order of precedence, etc...
Thanks,
Ken

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
I have a situation where I have a Linux server backing up a Linux

client

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
(both are CentOS6.5+), and rsnapshot gets a good bit into the

backup, then

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
just stops.

It looks like it's rsync that's hanging.

Post by Ken Rossman
I can watch approximately where this occurs by including the -V

switch

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
...
pool/nwsys/wndp/20151202.files
pool/nwsys/wndp/raw/
pool/nwsys/wndpqa/20151202.files
pool/nwsys/wndpqa/raw/
[and it hangs at the above pointÅ  forever]

Does rsync always hang at the same point?

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
Can anyone shed some light on this issue, or perhaps give me some

things

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell

Post by Ken Rossman
to try to get to the bottom of what¹s actually going on?

What is rsync doing on the remote host? Use lsof to see what files

it is

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell
trying to access when it appears to be hung.

If you want some indication of progress during the rsync of each file,
you could try adding
<tab>+rsync_long_args=--progress
to the end of the relevant backup line. That will give you a lot of
output.

Post by Nico Kadel-Garcia

Post by Ken Rossman

Post by David Cantrell
There may be several rsync processes, so check them all.

There's actually a pagefull of them. I'm not sure I am the only one

trying

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman
to rsync to/from that particular server, so I need a way to identify

which

Post by David Keegel

Post by Nico Kadel-Garcia

Post by Ken Rossman
ones of those procs belong to what I am doing.

You might want to pick a time when you are not running rsnapshot (or
anything else that should be using rsync) and look at what rsync
processes are still running. Are they previous "hung" rsnapshots?
If they are unnecessary/useless rsyncs, you might want to kill them.

Post by Nico Kadel-Garcia
"netstate" on the remote host should be helpful to determine which
underlying unencrypted rsync or encrypted SSH underlying process is
associated with the connection from your rsync client host.

Perhap Nico meant something like
netstat -ntp | grep :22
to match the ssh underlying processes with network connections.
To look at what file descriptors are open for all running rsync
processes, you could run
for p in $(/sbin/pidof rsync)
do ls -l /proc/$p/fd/*
done
--

___________________________________________________________________________

Post by David Keegel
http://www.cyber.com.au/~djk/ Linux & Unix Systems Administration

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple
OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Ken Rossman]

Still "no joy" getting includes/excludes correct. I either get nothing (or
fragments... the tops of directories & no files), or I get too much
stuff. Yeah, I know this is a pretty common issue, but I've been
wrestling with it for the past couple of weeks, and still haven't gotten it
right.

My preference would be to put all include and exclude patterns into a
separate file (or if need be, two separate files, includes in one file,
excludes in another), then "call" that file from the main rsnapshot.conf.

For now, I really ONLY want to back up the following files/dirs (and this
is in a filter file format, if I understand those correctly):

----------

### INCLUDES ###
+ /etc/nagios/nrpe.cfg
+ /etc/httpd/conf/httpd.conf
+ /etc/httpd/conf.d/**

+ /pool/nwsys/release/conf/.syscfg
+ /pool/nwsys/release/conf/*.conf
+ /pool/nwsys/release/conf/*.cfg
+ /pool/nwsys/release/conf/cronfile**
+ /pool/nwsys/release/conf/storage/**
+ /pool/nwtext/client/**

### EXCLUDES ###
- /etc/**
- /pool/**

----------

If I run with the above file, I get very little (which I assume is because
of the EXCLUDES section). If I take those out, I get much more than I
explicitly asked for.

Thanks to those who have already replied and tried to help. Not sure
what's different about what I am doing vs what others are doing...

I am running rsnapshot 1.3.1 and rsync 3.0.6 (protocol version 30)

Thanks,
Ken

[Scott Hess]

Post by Ken Rossman
### EXCLUDES ###
- /etc/**
- /pool/**

My guess is that given these excludes, rsync is excluding everything
directly in the /etc/ and /pool/ directories, and thus never reaching the
items you list in includes. rsync's -vv option will output things like
"hiding directory x/y because of pattern /x/**".

Search rsync's man page for the section which says:
The exclude patterns actually short-
circuit the directory traversal stage when rsync finds the files
to
send. If a pattern excludes a particular parent directory, it can
ren-
der a deeper include pattern ineffectual because rsync did not
descend
through that excluded section of the hierarchy.
It has an example of something which doesn't work, and how it could instead
be phrased. It looks pretty similar to your case.

-scott