[rsnapshot-discuss] Limit number of rsync processes

[Colin Walker]

First, apologies if this this a topic that you guys have already covered as this is my first post.

I am currently using rsnapshot to backup my server.

The server has 4 CPUs and for the 5-10 mins that the backup is running the server load average goes high and the server performance drops to noticeable levels.

[Dieter Bloms]

Hi,

Post by Colin Walker
I am currently using rsnapshot to backup my server.
The server has 4 CPUs and for the 5-10 mins that the backup is running the server load average goes high and the server performance drops to noticeable levels.

[Colin Walker]

You are correct that I am backing up my local box to the same box. Using it as a sort of version control. Does that explain the 4 processes? Is it just a co-incidence I have 4 CPUs?

I did set bwlimit to 5000 and didn't see a huge improvement. Does bwlimit affect internal disk I/O?

Ionice sounds interesting. Do you have any examples of how to integrate that within rsnapshot?

Thanks in advance. Colin.

Sent from my iPhone

Hi,

Post by Colin Walker
I am currently using rsnapshot to backup my server.
The server has 4 CPUs and for the 5-10 mins that the backup is running the server load average goes high and the server performance drops to noticeable levels.

[Yedidyah Bar-David]

Hi all,

Post by Colin Walker
You are correct that I am backing up my local box to the same box. Using it as a sort of version control. Does that explain the 4 processes?

I think it does. If you want it to not ssh to itself, you should change
your 'backup' lines in rsnapshot.conf to not use '***@host:' e.g.:
instead of
backup ***@localhost:/somedir/ somedir/
use
backup /somedir/ somedir/

Post by Colin Walker
Is it just a co-incidence I have 4 CPUs?

Yes...

Post by Colin Walker
I did set bwlimit to 5000 and didn't see a huge improvement. Does bwlimit affect internal disk I/O?

It affects only bandwidth used for actual copying. If the load is mainly
caused by the first stage when rsync scans the directory trees, bwlimit
won't help you.

Post by Colin Walker
Ionice sounds interesting. Do you have any examples of how to integrate that within rsnapshot?

Never tried it, but you can try changing cmd_rsync like:
cmd_rsync ionice -c 3 rsync

[Yedidyah Bar-David]

backup /dir/ dir/ in my rsnapshot.conf
I dont believe that my files change to such a degree that it would take 5 minutes to back them up internally. I exclude all logs, mp4, pdf etc etc...

As I wrote before, probably most of the time is taken by scanning the
directory trees to see what needs to be copied.

I am still a little unclear why there are 4 processes spawned? Does it create 2 processes for the source and destination servers, hence if on the same box, you get 4?

No idea. I now verified that I get 3.

A quick google search finds:
http://lists.samba.org/archive/rsync/2009-June/023338.html

[Colin Walker]

Ty Didi and Deiter.

All sorted. Many thanks for your assistance.

Sent from my iPhone

Post by Yedidyah Bar-David

backup /dir/ dir/ in my rsnapshot.conf
I dont believe that my files change to such a degree that it would take 5 minutes to back them up internally. I exclude all logs, mp4, pdf etc etc...

As I wrote before, probably most of the time is taken by scanning the
directory trees to see what needs to be copied.

I am still a little unclear why there are 4 processes spawned? Does it create 2 processes for the source and destination servers, hence if on the same box, you get 4?

No idea. I now verified that I get 3.
http://lists.samba.org/archive/rsync/2009-June/023338.html
--
Didi

[Dieter Bloms]

Hi,

Post by Colin Walker
You are correct that I am backing up my local box to the same box.
Using it as a sort of version control. Does that explain the 4
processes? Is it just a co-incidence I have 4 CPUs?
I did set bwlimit to 5000 and didn't see a huge improvement. Does
bwlimit affect internal disk I/O?
Ionice sounds interesting. Do you have any examples of how to
integrate that within rsnapshot?

ionice -c 3 rsnapshop ....

[David Cantrell]

Post by Colin Walker
I did set bwlimit to 5000 and didn't see a huge improvement. Does bwlimit affect internal disk I/O?

bwlimit is in kilobytes per second IIRC, so that's 40Mbits per second,
which is quite a lot. And given that it's the rate at which to transfer
from one process to another, then if both ends are on the same machine,
that's actually *80* Mbps of disk I/O. Which is a lot, especially when
you consider that it'll be really *thrashy* I/O and so the disk will
actually spend more time moving the heads back and forth than reading
and writing data.

[David Cantrell]

Post by Colin Walker
The server has 4 CPUs and for the 5-10 mins that the backup is running the server load average goes high and the server performance drops to noticeable levels.