[rsnapshot-discuss] Returned 255 while processing - Rolling back is lengthy

[Thierry Lavallee]

I admit we have a connection problem here. We are investigating for
months and can't find if it is the ISP or remote server.

This being said, I would like to ensure Rsnapshot has a viable
connection before making all its ninja moves. And having to Roll back
for over 8 hours.

We run our cron like this:
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync &&
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf daily


This is yesterday night log with comments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[18/Jul/2015:03:00:03] /usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync: started
[18/Jul/2015:03:00:03] echo 27184 > /var/run/rsnapshot_media02new.pid
[18/Jul/2015:03:00:03] mkdir -m 0755 -p /media/backupmedia02b/home/.sync/
[18/Jul/2015:03:00:03] /usr/bin/rsync -ax --delete --numeric-ids
--delete-excluded --rsh="/usr/bin/ssh -i /root/.ssh/id_rsa"
--link-dest=/media/backupmedia02b/home/daily.0/
***@media02.remote-server.com:/backup/cpbackup/daily/
/media/backupmedia02b/home/.sync/

#The sync is final here or not?
#Be cause if it did, I don't understand why Rsnapshot is trying to
connect AGAIN to the remote server and sync again. See below.


[18/Jul/2015:03:49:51] /usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync: ERROR:
/usr/bin/rsync returned 255 while processing
***@media02.remote-server.com:/backup/cpbackup/daily/
[18/Jul/2015:03:49:51] Rolling back ""
[18/Jul/2015:03:49:51] /bin/rm -rf /media/backupmedia02b/home/.sync/
[18/Jul/2015:06:43:24] /bin/cp -al /media/backupmedia02b/home/daily.0
/media/backupmedia02b/home/.sync

#.... Still waiting for this to complete over all daily directories :/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks for your support
-- Thierry

------------------------------------------------------------------------------

[Thomas Fjellstrom]

I recently had that error. It started out kinda random, then just got steadily
worse till backups just didn't run fully very often. Turns out that machine
was infected with a DdoS client, and kept thousands of sockets open that were
hidden from netstat. Basically the user was nearly out of sockets and ssh
almost never succeeded in connecting more than a few times.

Post by Thierry Lavallee
I admit we have a connection problem here. We are investigating for
months and can't find if it is the ISP or remote server.
This being said, I would like to ensure Rsnapshot has a viable
connection before making all its ninja moves. And having to Roll back
for over 8 hours.
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync &&
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf daily
This is yesterday night log with comments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[18/Jul/2015:03:00:03] /usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync: started
[18/Jul/2015:03:00:03] echo 27184 > /var/run/rsnapshot_media02new.pid
[18/Jul/2015:03:00:03] mkdir -m 0755 -p /media/backupmedia02b/home/.sync/
[18/Jul/2015:03:00:03] /usr/bin/rsync -ax --delete --numeric-ids
--delete-excluded --rsh="/usr/bin/ssh -i /root/.ssh/id_rsa"
--link-dest=/media/backupmedia02b/home/daily.0/
/media/backupmedia02b/home/.sync/
#The sync is final here or not?
#Be cause if it did, I don't understand why Rsnapshot is trying to
connect AGAIN to the remote server and sync again. See below.
[18/Jul/2015:03:49:51] /usr/bin/rsnapshot -c
/usr/bin/rsync returned 255 while processing
[18/Jul/2015:03:49:51] Rolling back ""
[18/Jul/2015:03:49:51] /bin/rm -rf /media/backupmedia02b/home/.sync/
[18/Jul/2015:06:43:24] /bin/cp -al /media/backupmedia02b/home/daily.0
/media/backupmedia02b/home/.sync
#.... Still waiting for this to complete over all daily directories :/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thanks for your support
-- Thierry
----------------------------------------------------------------------------
-- _______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Thierry Lavallee]

Thanks for the heads up Thomas. Fortunately this system is scanned by
ClamAV daily and nothing was reported.
Unless I missed parts of your post and that ClamAV would not recognize? :/

This being said, rsnapshot is a really heavy bulldozer to stop, as the
roll back is huge and time consuming. One failed SSH connection and here
we go for a 8 hours rollback within which timeframe we cannot attempt
another backup. Quite a drag.

And I would hope for the remote connection to be completely established
_*before*_ the whole ninja moves are made And in case the connection
breaks, that it is able to start from there on the next try. ;)

I though that using .sync was to solve this situation.
Seems not.

Am I missing something?
thanks!

Post by Thomas Fjellstrom
I recently had that error. It started out kinda random, then just got steadily
worse till backups just didn't run fully very often. Turns out that machine
was infected with a DdoS client, and kept thousands of sockets open that were
hidden from netstat. Basically the user was nearly out of sockets and ssh
almost never succeeded in connecting more than a few times.

Post by Thierry Lavallee
I admit we have a connection problem here. We are investigating for
months and can't find if it is the ISP or remote server.
This being said, I would like to ensure Rsnapshot has a viable
connection before making all its ninja moves. And having to Roll back
for over 8 hours.
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync &&
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf daily
This is yesterday night log with comments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[18/Jul/2015:03:00:03] /usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync: started
[18/Jul/2015:03:00:03] echo 27184 > /var/run/rsnapshot_media02new.pid
[18/Jul/2015:03:00:03] mkdir -m 0755 -p /media/backupmedia02b/home/.sync/
[18/Jul/2015:03:00:03] /usr/bin/rsync -ax --delete --numeric-ids
--delete-excluded --rsh="/usr/bin/ssh -i /root/.ssh/id_rsa"
--link-dest=/media/backupmedia02b/home/daily.0/
/media/backupmedia02b/home/.sync/
#The sync is final here or not?
#Be cause if it did, I don't understand why Rsnapshot is trying to
connect AGAIN to the remote server and sync again. See below.
[18/Jul/2015:03:49:51] /usr/bin/rsnapshot -c
/usr/bin/rsync returned 255 while processing
[18/Jul/2015:03:49:51] Rolling back ""
[18/Jul/2015:03:49:51] /bin/rm -rf /media/backupmedia02b/home/.sync/
[18/Jul/2015:06:43:24] /bin/cp -al /media/backupmedia02b/home/daily.0
/media/backupmedia02b/home/.sync
#.... Still waiting for this to complete over all daily directories :/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thanks for your support
-- Thierry
----------------------------------------------------------------------------
-- _______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[Thomas Fjellstrom]

Post by Thierry Lavallee
Thanks for the heads up Thomas. Fortunately this system is scanned by
ClamAV daily and nothing was reported.
Unless I missed parts of your post and that ClamAV would not recognize? :/

Hm, when I checked it with online scanners, I think clamav detected it, so it
probably would be checked unless your db is out of date. It was a kindof new
thing. I've only seen that very vague error when rsync/ssh couldn't open a
port on the backup machine. Maybe try running the rsync manually to see what
happens it might give you a more useful error message. Maybe even try repeated
ssh connections to see if they fail (it would take several outbound
connections to trigger the problem for me, so the first few hosts would
succeed while a few later ones may fail, but not always...).

Maybe your max file ulimit is too low? See what lsof or netstat have to say
about open ports.

Post by Thierry Lavallee
This being said, rsnapshot is a really heavy bulldozer to stop, as the
roll back is huge and time consuming. One failed SSH connection and here
we go for a 8 hours rollback within which timeframe we cannot attempt
another backup. Quite a drag.

I set up a wrapper script for rsnapshot that only does one host at a time into
a shared directory, and it does the sync separately after all of the "daily"
commands are done. Some of my backup sources aren't always up, or my internet
may not be entirely reliable all the time, and some backups would fail,
rolling back the entire thing. I didn't think that was ideal, so i wrote the
script to do that.

Post by Thierry Lavallee
And I would hope for the remote connection to be completely established
_*before*_ the whole ninja moves are made And in case the connection
breaks, that it is able to start from there on the next try. ;)
I though that using .sync was to solve this situation.
Seems not.
Am I missing something?
thanks!

I don't know really. I haven't messed with it other than my silly script to
avoid full rollbacks.

Post by Thierry Lavallee

Post by Thomas Fjellstrom
I recently had that error. It started out kinda random, then just got
steadily worse till backups just didn't run fully very often. Turns out
that machine was infected with a DdoS client, and kept thousands of
sockets open that were hidden from netstat. Basically the user was nearly
out of sockets and ssh almost never succeeded in connecting more than a
few times.

Post by Thierry Lavallee
I admit we have a connection problem here. We are investigating for
months and can't find if it is the ISP or remote server.
This being said, I would like to ensure Rsnapshot has a viable
connection before making all its ninja moves. And having to Roll back
for over 8 hours.
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync &&
/usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf daily
This is yesterday night log with comments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[18/Jul/2015:03:00:03] /usr/bin/rsnapshot -c
/root/scripts/backup/rsnapshot.media02_cpbackup.conf sync: started
[18/Jul/2015:03:00:03] echo 27184 > /var/run/rsnapshot_media02new.pid
[18/Jul/2015:03:00:03] mkdir -m 0755 -p /media/backupmedia02b/home/.sync/
[18/Jul/2015:03:00:03] /usr/bin/rsync -ax --delete --numeric-ids
--delete-excluded --rsh="/usr/bin/ssh -i /root/.ssh/id_rsa"
--link-dest=/media/backupmedia02b/home/daily.0/
/media/backupmedia02b/home/.sync/
#The sync is final here or not?
#Be cause if it did, I don't understand why Rsnapshot is trying to
connect AGAIN to the remote server and sync again. See below.
[18/Jul/2015:03:49:51] /usr/bin/rsnapshot -c
/usr/bin/rsync returned 255 while processing
[18/Jul/2015:03:49:51] Rolling back ""
[18/Jul/2015:03:49:51] /bin/rm -rf /media/backupmedia02b/home/.sync/
[18/Jul/2015:06:43:24] /bin/cp -al /media/backupmedia02b/home/daily.0
/media/backupmedia02b/home/.sync
#.... Still waiting for this to complete over all daily directories :/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thanks for your support
-- Thierry
-------------------------------------------------------------------------
--- -- _______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[David Cantrell]

Post by Thierry Lavallee
And I would hope for the remote connection to be completely established
_*before*_ the whole ninja moves are made And in case the connection
breaks, that it is able to start from there on the next try. ;)

Not possible I'm afraid. The remote connection isn't set up by
rsnapshot, it's set up by rsync, so as soon as you set it up, the sync
starts.

There may be fancy tricks you could play with ssh to set up the
connection in advance, keep it alive while everything else happens
(don't forget that cp -al can be Quite Time Consuming, and network
connections that pass no traffic tend to time out), and then re-use the
connection for rsync. But you need to consider:

* rsync may not be configured to use ssh - it might be using the plain
old rsync protocol, and presumably you'd want to establish that
connection too before doing all the moves and copies;
* you may be backing up multiple hosts over ssh and multiple hosts over
plain old rsync, so you have to keep track of all those connections,
not just one

We definitely don't want rsync-over-ssh to behave differently to
plain-old-rsync, or for *one* rsync over ssh to behave differently to
multiple rsyncs over ssh.

[Thierry Lavallee]

Feels like if the rsync diff was done to a temporary directory named
after a temporary random key, until it is complete without error, we
would be in a position to synchronize and put everything in place before
the ninja rm and mv occurs, hence protecting the main repository from
potential connection errors, crazy roll backs.

In fact I thought the sync procedure was doing this.

PS: and why not send an email on error with the complete log for the
session? ;)

Post by David Cantrell

Post by Thierry Lavallee
And I would hope for the remote connection to be completely established
_*before*_ the whole ninja moves are made And in case the connection
breaks, that it is able to start from there on the next try. ;)

Not possible I'm afraid. The remote connection isn't set up by
rsnapshot, it's set up by rsync, so as soon as you set it up, the sync
starts.
There may be fancy tricks you could play with ssh to set up the
connection in advance, keep it alive while everything else happens
(don't forget that cp -al can be Quite Time Consuming, and network
connections that pass no traffic tend to time out), and then re-use the
* rsync may not be configured to use ssh - it might be using the plain
old rsync protocol, and presumably you'd want to establish that
connection too before doing all the moves and copies;
* you may be backing up multiple hosts over ssh and multiple hosts over
plain old rsync, so you have to keep track of all those connections,
not just one
We definitely don't want rsync-over-ssh to behave differently to
plain-old-rsync, or for *one* rsync over ssh to behave differently to
multiple rsyncs over ssh.

[Gordon Messmer]

Post by Thierry Lavallee
In fact I thought the sync procedure was doing this.

As far as I can tell, rollback only happens if you're using link_dest
(or a backup script rather than a filesystem). Is there a reason you're
using link_dest with sync_first?

Post by Thierry Lavallee
PS: and why not send an email on error with the complete log for the
session? ;)

If you run rsnapshot from cron, all of its output will be emailed to the
job owner. I can't think of any reason why rsnapshot should support job
log emails when that's already handled.

[David Keegel]

Rsnapshot only does rollbacks if you are using link_dest (and rsync
fails). So I'd suggest turning off link_dest if you want to avoid
the possibility of rollbacks.

Post by Thierry Lavallee
This being said, rsnapshot is a really heavy bulldozer to stop, as the
roll back is huge and time consuming. One failed SSH connection and
here we go for a 8 hours rollback within which timeframe we cannot
attempt another backup. Quite a drag.
And I would hope for the remote connection to be completely established
before the whole ninja moves are made And in case the connection
breaks, that it is able to start from there on the next try. ;)
I though that using .sync was to solve this situation.
Seems not.
Am I missing something?
thanks!