[rsnapshot-discuss] Problems Getting Rsnapreport to Work

Discussion:

Nico Kadel-Garcia

2016-12-22 13:39:02 UTC

On Tue, Dec 20, 2016 at 11:31 AM, Mark Phillips

I am trying to get rsnapreport to work and I am getting some weird results.
It must be a config problem, but I am not sure what to change in the config
file.
I have this in my cron file -
50 23 * * * /usr/bin/rsnapshot daily 2>&1 | tee -a
/var/log/rsnapshot.log | /usr/local/bin/rsnapreport.pl | /usr/bin/mail -s

First step is to segregate those dumps to be different
rsnapshot.conf.[hostname] configurations for different backup targets,
in different rsnapshot directories, because as is if *one* backup host
fails then rsnapshot fails for all of them. It's reasonable to have a
shared template among them, but I urge you to do this split first. It
will also make debugging much easier for individual hosts.

Second: use fully qualified hostnnames for the host targets, to avoid
confusion about which host you're actually backing up. I've seen too
many sites that have "www.example.com", "www.office.example.com",
"www.offsite.example.com" to trust the resolution of short hostnames
to avoid confusion.

Third: try the relevant rsync commands from the command line of your
rsnapshot server, speaking directly to "swordfish" or other hosts that
reported issues. I'm particularly wondering if there is some funky,
malformed rsync wrapper enabled in the authorized_keys associated with
the rsnapshot process.

My rsnapshot.config has verbose 5 and loglevel 4. I tried verbose 3 and all
I got was errors no stats data.
Also in rsnapshot.config my backup points are
# SWORDFISH (Internent connected machine)
# TSUNAMI (local network machine)
# LOCALHOST
backup /etc/ localhost/
backup /home/ localhost/
backup /root/ localhost/
backup /opt/ localhost/
backup /var/www/ localhost/
backup /var/log/ localhost/
backup /var/local/ localhost/
SOURCE TOTAL FILES FILES TRANS TOTAL MB
MB TRANS LIST GEN TIME FILE XFER TIME
--------------------------------------------------------------------------------------------------------------------
/home/ 59 N/A 0.00
0.00 0.001 seconds 0.000 seconds
/opt/ 1 N/A 0.00
0.00 0.001 seconds 0.000 seconds
/root/ 432 N/A 0.00
0.00 0.001 seconds 0.000 seconds
/var/local/ 2 N/A 0.00
0.00 0.001 seconds 0.000 seconds
/var/log/ 406 N/A 0.00
0.00 0.001 seconds 0.000 seconds
/var/www/ 159 N/A 0.00
0.00 0.001 seconds 0.000 seconds
is 2 N/A 0.00
0.00 0.001 seconds 0.000 seconds
swordfish:/home/ 17 N/A 0.00
0.00 0.001 seconds 0.000 seconds
swordfish:/opt/ 125 N/A 0.00
0.00 0.001 seconds 0.000 seconds
swordfish:/root/ 47 N/A 0.00
0.00 0.001 seconds 0.000 seconds
tsunami:/opt/ 41 N/A 0.00
0.00 0.001 seconds 0.000 seconds
tsunami:/root/ 34 N/A 0.00
0.00 0.001 seconds 0.000 seconds
tsunami:/var/log 469 N/A 0.00
0.00 0.001 seconds 0.000 seconds
tsunami:/var/www 11 N/A 0.00
0.00 0.001 seconds 0.000 seconds
ERRORS
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
is NO STATS DATA
/etc/ NO STATS DATA
The rsnapshot.log. Does not show any errors. I believe the "is" entries come
from the "is uptodate" entry on most of the entries in the log.
[2016-12-20T02:11:54]
home/mark/AndroidStudioProjects/RocketFinder/app/build/intermediates/classes/debug/biz/phillipsmarketing/android/rocketfinder/database/dao/EngineDAO.class
is uptodate
Are the ERRORS entries normal for the rsnapreport output? I don't think the
errors are real as they seem to be related to the "is" entries. Is there a
config file entry that will remove these entries?
Why do the following entries have NO STATS DATA, whereas the other
And then on the localhost, but not the other localhost entries?
/etc/ NO STATS DATA
What am I missing in my config file that may be contributing to these issues
in the report?
I would sent the rsnapshot.log just for last night, but it is 250 MB! I have
attached my config file and excludes file.
Thanks,
Mark
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Nico Kadel-Garcia

2016-12-29 01:09:07 UTC

Permalink

On Fri, Dec 23, 2016 at 11:30 AM, Mark Phillips

Nico,
Thanks for your suggestions. I have some results and questions inline below.
Mark

Sorry for late response, I was off doing Xmas things.

Post by Nico Kadel-Garcia
On Tue, Dec 20, 2016 at 11:31 AM, Mark Phillips

If I understand your suggestion, I would create three
rsnapshot.conf.[hostname] files, each with different rsnapshot roots, and
run them out of cron like this
rsnapshot -c rsnapshot.conf.swordfish daily
rsnapshot -c rsnapshot.conf.tsunami daily
rsnapshot -c rsnapshot.conf.localhost.daily
/media/backup/rsnapshot/daily.0/tsunami
/media/backup/rsnapshot/daily.0/localhost
/media/backup/rsnapshot/daily.0/swordfish
I think what you are proposing would change this to be
/media/backup/rsnapshot/tsunami/daily.0
/media/backup/rsnapshot/localhost/daily.0
/media/backup/rsnapshot/swordfish/daily.0

Yup. You could "cp -al" the content from the old location to the new
locations, if you have enough inodes left That's the whole point of
rsnapshot, it copies *hard links* effectively.

In effect, I am abandoning my previous backups and starting over again? I
guess I could get a new drive and do this, and just save the old drive in
case I need it.
Is there a way to make the changes you recommend without starting over with
the backups as I described above?

See above.

Post by Nico Kadel-Garcia
Second: use fully qualified hostnnames for the host targets, to avoid
confusion about which host you're actually backing up. I've seen too
many sites that have "www.example.com", "www.office.example.com",
"www.offsite.example.com" to trust the resolution of short hostnames
to avoid confusion.

Sure, that is easy. I am able to ssh from localhost (the real host name is
orca, but since it is running rsnapshot it is called localhost) to swordfish
and tsunami just using those names. And, there are only three machines on
the local network - localhost, tsunami, and a windoze laptop that is not
part of the backup process. Swordfish is remote, so I could use the IP
address for both swordfish and tsunami.

Post by Nico Kadel-Garcia
Third: try the relevant rsync commands from the command line of your
rsnapshot server, speaking directly to "swordfish" or other hosts that
reported issues. I'm particularly wondering if there is some funky,
malformed rsync wrapper enabled in the authorized_keys associated with
the rsnapshot process.

I tried the following command as you suggested, which I took from the
rsnapshot log file for /etc/ on swordfish. I changed the destination for the
backup to test.0 and test.1 and added everything after 2>&1 to see what
would happen.
--delete-excluded --stats
--exclude-from=/root/backup_scripts/rsnapshot_excludes.txt
--rsh=/usr/bin/ssh --link-dest=/media/backup/rsnapshot/test.1
rsnapshot.log | /usr/local/bin/rsnapreport.pl | /usr/bin/mail -s "Orca rsync
I have attached the log file for the command, which shows no errors. The
email provided this output
SOURCE TOTAL FILES FILES TRANS TOTAL MB
MB TRANS LIST GEN TIME FILE XFER TIME
--------------------------------------------------------------------------------------------------------------------
2 N/A 0.00
0.00 0.001 seconds 0.000 seconds
and there are some errors from the rsnapreport.pl -
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl line 94, <> line 2321.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl line 100, <> line 2325.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl line 103, <> line 2326.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl line 106, <> line 2330.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl line 109, <> line 2331.
I assume these are due to running the report against the rsync output
directly versus running the rsync output through rsnapshot and then to
rsnapreport.

Yeah, you don't need the resnapreport.pl for what I was suggesting.

It seems that rsync is not having any issues with swordfish as shown in the
attached log. Or, did I run the test you had in mind incorrectly? I could
run the other rsync commands for the failing directories on swordfish and
tsunami, but those directories have GB of data, so this one runs the fastest
and does not report any errors.
Finally, since I have been reading more information about rsnapshot, what is
the advantage of running sync first, then daily, weekly, monthly, etc? It
seems it would use more disk space. Anyway, lots of references to rsnapshot

Not a lot more. sync first is done with hardlinks, to a staging
location, to avoid stepping on the existing "daily" or shortest
snapshots except by adding hardlinks to the sync directory. Then all
the snapshots get *rotated* around, and the obsolete ones deleted.

run the sync command first, then the rotations and I do not understand the
pros and cons of that approach versus just running a daily, weekly, monthly
as I am doing in my cron script.
Thanks for your suggestions!
Mark

"sync" is most handy if there's a chance of your most recent snapshot
failing, and not wanting to accidentally screw it up before you've
made a successful and complete copy of it, with rsync done on top. It
prevents daily.0 from getting screwed up and no longer being useful
for restoration, especially while the backup is in progress.

Bob Hutchinson

2017-01-07 13:39:53 UTC

Permalink

Nico,
Thanks for your suggestions. I have some results and questions inline below.
Mark
On Tue, Dec 20, 2016 at 11:31 AM, Mark Phillips

First step is to segregate those dumps to be different
rsnapshot.conf.[hostname] configurations for different backup targets,
in different rsnapshot directories, because as is if *one* backup host
fails then rsnapshot fails for all of them. It's reasonable to have a
shared template among them, but I urge you to do this split first. It
will also make debugging much easier for individual hosts.
If I understand your suggestion, I would create three
rsnapshot.conf.[hostname] files, each with different rsnapshot roots,
and run them out of cron like this
rsnapshot -c rsnapshot.conf.swordfish daily
rsnapshot -c rsnapshot.conf.tsunami daily
rsnapshot -c rsnapshot.conf.localhost.daily
/media/backup/rsnapshot/daily.0/tsunami
/media/backup/rsnapshot/daily.0/localhost
/media/backup/rsnapshot/daily.0/swordfish
I think what you are proposing would change this to be
/media/backup/rsnapshot/tsunami/daily.0
/media/backup/rsnapshot/localhost/daily.0
/media/backup/rsnapshot/swordfish/daily.0
In effect, I am abandoning my previous backups and starting over again?
I guess I could get a new drive and do this, and just save the old drive
in case I need it.
Is there a way to make the changes you recommend without starting over
with the backups as I described above?

Once you have set up the cron as suggest by others (I would place the
commands to be run in shell scripts and run them from cron, more
convenient and you can check if the remote server is up using fping or
somesuch)

You could use the mv command, something like this:
# create temporary folders
mkdir -p /media/backup/rsnapshot2/tsunami

# move stuff
mv /media/backup/rsnapshot/daily.0/tsunami/ \
/media/backup/rsnapshot2/tsunami/daily.0

# same for the others

# remove the old (it should be empty now)
rm -fr /media/backup/rsnapshot

# move temporary folder into place
mv /media/backup/rsnapshot2 /media/backup/rsnapshot

# this should do it without using more space
# mv is fast as long as source and target are on the same filesystem

Second: use fully qualified hostnnames for the host targets, to avoid
confusion about which host you're actually backing up. I've seen too
many sites that have "www.example.com <http://www.example.com>",
"www.office.example.com <http://www.office.example.com>",
"www.offsite.example.com <http://www.offsite.example.com>" to trust
the resolution of short hostnames
to avoid confusion.
Sure, that is easy. I am able to ssh from localhost (the real host name
is orca, but since it is running rsnapshot it is called localhost) to
swordfish and tsunami just using those names. And, there are only three
machines on the local network - localhost, tsunami, and a windoze laptop
that is not part of the backup process. Swordfish is remote, so I could
use the IP address for both swordfish and tsunami.
Third: try the relevant rsync commands from the command line of your
rsnapshot server, speaking directly to "swordfish" or other hosts that
reported issues. I'm particularly wondering if there is some funky,
malformed rsync wrapper enabled in the authorized_keys associated with
the rsnapshot process.
I tried the following command as you suggested, which I took from the
rsnapshot log file for /etc/ on swordfish. I changed the destination for
the backup to test.0 and test.1 and added everything after 2>&1 to see
what would happen.
--delete-excluded --stats
--exclude-from=/root/backup_scripts/rsnapshot_excludes.txt
--rsh=/usr/bin/ssh --link-dest=/media/backup/rsnapshot/test.1
rsnapshot.log | /usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> |
I have attached the log file for the command, which shows no errors. The
email provided this output
SOURCE TOTAL FILES FILES TRANS TOTAL MB
MB TRANS LIST GEN TIME FILE XFER TIME
--------------------------------------------------------------------------------------------------------------------
2 N/A 0.00
0.00 0.001 seconds 0.000 seconds
and there are some errors from the rsnapreport.pl <http://rsnapreport.pl> -
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 94, <> line 2321.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 100, <> line
2325.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 103, <> line
2326.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 106, <> line
2330.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 109, <> line
2331.
I assume these are due to running the report against the rsync output
directly versus running the rsync output through rsnapshot and then to
rsnapreport.
It seems that rsync is not having any issues with swordfish as shown in
the attached log. Or, did I run the test you had in mind incorrectly? I
could run the other rsync commands for the failing directories on
swordfish and tsunami, but those directories have GB of data, so this
one runs the fastest and does not report any errors.
Finally, since I have been reading more information about rsnapshot,
what is the advantage of running sync first, then daily, weekly,
monthly, etc? It seems it would use more disk space. Anyway, lots of
references to rsnapshot run the sync command first, then the rotations
and I do not understand the pros and cons of that approach versus just
running a daily, weekly, monthly as I am doing in my cron script.
Thanks for your suggestions!
Mark
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

--
-----------------
Bob Hutchinson
Midwales dot com
-----------------

Mark Phillips

2017-01-14 18:22:23 UTC

Permalink

Bob,

Thanks for the suggestion for moving the rsnapshot backups around. However,
I think there is a missing step, which is to also move the .rsnapshot
folder as well to the temporary location before removing the original
rsnapshot folder. See
http://rsnapshot.org/rsnapshot/docs/docbook/rest.html#how-it-works.

I forgot this step, so in removingthe original rsnapshot folder I think I
also deleted the .rsnapshot folder. At least I cannot find it on my backup
drive, nor on the server that runs the backups. And, the initial daily
backup I ran after moving the backup folders around is taking many hours
longer than it use to, so I assume it is rebuilding the .snapshot folder.

To summarize,

You could use the mv command, something like this:
# create temporary folders
mkdir -p /media/backup/rsnapshot2/tsunami

# move stuff
mv /media/backup/rsnapshot/daily.0/tsunami/ \
/media/backup/rsnapshot2/tsunami/daily.0

# same for the others

# move the root folder to the temporary folder (added step)
mv /media/backup/rsnapshot/.snapshot /media/backup/rsnapshot2/

# remove the old (it should be empty now)
rm -fr /media/backup/rsnapshot

# move temporary folder into place
mv /media/backup/rsnapshot2 /media/backup/rsnapshot

# this should do it without using more space
# mv is fast as long as source and target are on the same filesystem

Mark

Nico,
Thanks for your suggestions. I have some results and questions inline

below.

Mark
On Tue, Dec 20, 2016 at 11:31 AM, Mark Phillips

I am trying to get rsnapreport to work and I am getting some weird

results.

It must be a config problem, but I am not sure what to change in

the config

file.
I have this in my cron file -
50 23 * * * /usr/bin/rsnapshot daily 2>&1 | tee -a
/var/log/rsnapshot.log | /usr/local/bin/rsnapreport.pl <

http://rsnapreport.pl> | /usr/bin/mail -s

First step is to segregate those dumps to be different
rsnapshot.conf.[hostname] configurations for different backup

targets,

in different rsnapshot directories, because as is if *one* backup

host

fails then rsnapshot fails for all of them. It's reasonable to have a
shared template among them, but I urge you to do this split first. It
will also make debugging much easier for individual hosts.
If I understand your suggestion, I would create three
rsnapshot.conf.[hostname] files, each with different rsnapshot roots,
and run them out of cron like this
rsnapshot -c rsnapshot.conf.swordfish daily
rsnapshot -c rsnapshot.conf.tsunami daily
rsnapshot -c rsnapshot.conf.localhost.daily
/media/backup/rsnapshot/daily.0/tsunami
/media/backup/rsnapshot/daily.0/localhost
/media/backup/rsnapshot/daily.0/swordfish
I think what you are proposing would change this to be
/media/backup/rsnapshot/tsunami/daily.0
/media/backup/rsnapshot/localhost/daily.0
/media/backup/rsnapshot/swordfish/daily.0
In effect, I am abandoning my previous backups and starting over again?
I guess I could get a new drive and do this, and just save the old drive
in case I need it.
Is there a way to make the changes you recommend without starting over
with the backups as I described above?

Once you have set up the cron as suggest by others (I would place the
commands to be run in shell scripts and run them from cron, more
convenient and you can check if the remote server is up using fping or
somesuch)
# create temporary folders
mkdir -p /media/backup/rsnapshot2/tsunami
# move stuff
mv /media/backup/rsnapshot/daily.0/tsunami/ \
/media/backup/rsnapshot2/tsunami/daily.0
# same for the others
# remove the old (it should be empty now)
rm -fr /media/backup/rsnapshot
# move temporary folder into place
mv /media/backup/rsnapshot2 /media/backup/rsnapshot
# this should do it without using more space
# mv is fast as long as source and target are on the same filesystem

that

reported issues. I'm particularly wondering if there is some funky,
malformed rsync wrapper enabled in the authorized_keys associated

with

the rsnapshot process.
I tried the following command as you suggested, which I took from the
rsnapshot log file for /etc/ on swordfish. I changed the destination for
the backup to test.0 and test.1 and added everything after 2>&1 to see
what would happen.
--delete-excluded --stats
--exclude-from=/root/backup_scripts/rsnapshot_excludes.txt
--rsh=/usr/bin/ssh --link-dest=/media/backup/rsnapshot/test.1
rsnapshot.log | /usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> |
I have attached the log file for the command, which shows no errors. The
email provided this output
SOURCE TOTAL FILES FILES TRANS TOTAL MB
MB TRANS LIST GEN TIME FILE XFER TIME
------------------------------------------------------------

--------------------------------------------------------

2 N/A 0.00
0.00 0.001 seconds 0.000 seconds
and there are some errors from the rsnapreport.pl <http://rsnapreport.pl>

Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 94, <> line

2321.

Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 100, <> line
2325.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 103, <> line
2326.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 106, <> line
2330.
Use of uninitialized value $source in hash element at
/usr/local/bin/rsnapreport.pl <http://rsnapreport.pl> line 109, <> line
2331.
I assume these are due to running the report against the rsync output
directly versus running the rsync output through rsnapshot and then to
rsnapreport.
It seems that rsync is not having any issues with swordfish as shown in
the attached log. Or, did I run the test you had in mind incorrectly? I
could run the other rsync commands for the failing directories on
swordfish and tsunami, but those directories have GB of data, so this
one runs the fastest and does not report any errors.
Finally, since I have been reading more information about rsnapshot,
what is the advantage of running sync first, then daily, weekly,
monthly, etc? It seems it would use more disk space. Anyway, lots of
references to rsnapshot run the sync command first, then the rotations
and I do not understand the pros and cons of that approach versus just
running a daily, weekly, monthly as I am doing in my cron script.
Thanks for your suggestions!
Mark
------------------------------------------------------------

------------------

Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

--
-----------------
Bob Hutchinson
Midwales dot com
-----------------
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss