[rsnapshot-discuss] rsnashot high load

Discussion:

Balogh László

2016-01-31 10:45:49 UTC

Hi All,

I'm new in this thread. I'm using rsnapshot for backing up my linux
system about two months ago. It's working very well, i have only one
problem.

I've set up an incremental backup, i have three hourly (00, 08 and 16h),
7 daily, 4 weekly and 6 monthly:
retain hourly 3
retain daily 7
retain weekly 4
retain monthly 6

Interesting is, that i have problems at the 08h hourly backup. When the
rsnapshot is running at that time (08h) the system load gets a very high
level, above 15, sometimes about 30. At the other times the load is also
a little bit high, but the system is reachable. But when its backing up
at 08h after a few minutes the load gets incredible high, so the system
is not reacheble, or takes minutes to type a character.
There is no other job at 08h which could affect rsnaphot job.

I googled for the sollution and tried what i found:

1. In /etc/default/rsync file i configured:
RSYNC_NICE='10'
RSYNC_IONICE='-c3'

2. I'm running in cron with option ionice -c 3:
5 0,8,16 * * * ionice -c 3
/usr/bin/rsnapshot hourly

Nothing helps. :(

I also had checked my hard drives, which should be ok:
***@mcllserver:/mnt/sdc1# smartctl -a /dev/sda | grep 0x00
SMART capabilities: (0x0003) Saves SMART data before entering
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 174 173 021 Pre-fail
Always - 4258
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 13
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 098 098 000 Old_age
Always - 1867
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 13
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 7
193 Load_Cycle_Count 0x0032 196 196 000 Old_age
Always - 14075
194 Temperature_Celsius 0x0022 115 111 000 Old_age
Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0

***@mcllserver:/mnt/sdc1# smartctl -a /dev/sdb | grep 0x00
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 186 180 021 Pre-fail
Always - 5683
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 35
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age
Always - 12113
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 32
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2
194 Temperature_Celsius 0x0022 121 111 000 Old_age
Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0

***@mcllserver:/mnt/sdc1# smartctl -a /dev/sdc | grep 0x00
<--------------- BACKUP DRIVE!
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 386
3 Spin_Up_Time 0x0027 168 168 021 Pre-fail
Always - 4566
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 48
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age
Always - 19282
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 48
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 37
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 113 108 000 Old_age
Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0

***@mcllserver:/mnt/sdc1# smartctl -a /dev/sdd | grep 0x00
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 36
3 Spin_Up_Time 0x0027 169 167 021 Pre-fail
Always - 6541
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 115
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 061 061 000 Old_age
Always - 29141
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 113
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 104
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 115 106 000 Old_age
Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0

I tried manually rsync some large folders, but the load seems to be ok
(same source, same destination drive/folder), goes to load 2, but not
higher.

I'm using Ubuntu server 14.04.3 LTS in 24/7 mode.

Backup destiantion is a separate local hard drive mounted (mount output
of the backup destination drive)
/dev/sdc1 on /mnt/sdc1 type ext4
(rw,noatime,commit=120,errors=remount-ro)

Has anyone any idea what should i try to get rid of that high load?

Thanks
Regards
Laszlo

---
Ezt az e-mailt az Avast víruskeresõ szoftver átvizsgálta.
https://www.avast.com/antivirus

David Keegel

2016-01-31 11:59:46 UTC

Permalink

László,

There are a number of things which could make your system slow; you
need to work out why it is slow.

I'd check whether your system is running low on available memory (RAM)
and therefore paging/swapping around 08h. Running rsync -aH will use
a lot of RAM for processing a large directory (with lots of files).

If it happens while you are around, you could run top to get a better
idea whether you are running short of memory (mem free small, mem buffers
small, swap used large, cpu %wait high), have enough processes using a
lot of CPU to consume all available CPU resources (cpu %idle=0) or
something else is making your I/O slow (again cpu %wait high and
processes in "D" state but without the other shortage of memory
indicators).

It's curious that you have a slowness problem at 08h but not 16h.
I'm wondering if some overnight cron jobs are running for a lot
longer than you expect, or many users are logging in around 08h.
Looking at the top processes listing in top should give you an
idea about those possibilities.

Ideally, start top before 08:00 to see a difference between before and
after rsnapshot starts, and look for processes running before 08:00.

If you don't have time to investigate properly, and just want to
try a short cut, you could try removing -H from rsync_short_args
(or --hard-links from rsync_long_args) and see if slowness stops.

Post by Balogh LÃ¡szlÃ³
Hi All,
I'm new in this thread. I'm using rsnapshot for backing up my linux
system about two months ago. It's working very well, i have only one
problem.
I've set up an incremental backup, i have three hourly (00, 08 and
retain hourly 3
retain daily 7
retain weekly 4
retain monthly 6
Interesting is, that i have problems at the 08h hourly backup. When the
rsnapshot is running at that time (08h) the system load gets a very
high level, above 15, sometimes about 30. At the other times the load
is also a little bit high, but the system is reachable. But when its
backing up at 08h after a few minutes the load gets incredible high, so
the system is not reacheble, or takes minutes to type a character.
There is no other job at 08h which could affect rsnaphot job.
RSYNC_NICE='10'
RSYNC_IONICE='-c3'
5 0,8,16 * * * ionice -c 3
/usr/bin/rsnapshot hourly
Nothing helps. :(
SMART capabilities: (0x0003) Saves SMART data before entering
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 174 173 021 Pre-fail
Always - 4258
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 13
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 098 098 000 Old_age
Always - 1867
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 13
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 7
193 Load_Cycle_Count 0x0032 196 196 000 Old_age
Always - 14075
194 Temperature_Celsius 0x0022 115 111 000 Old_age
Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 186 180 021 Pre-fail
Always - 5683
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 35
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age
Always - 12113
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 32
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2
194 Temperature_Celsius 0x0022 121 111 000 Old_age
Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
<--------------- BACKUP DRIVE!
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 386
3 Spin_Up_Time 0x0027 168 168 021 Pre-fail
Always - 4566
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 48
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age
Always - 19282
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 48
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 37
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 113 108 000 Old_age
Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 36
3 Spin_Up_Time 0x0027 169 167 021 Pre-fail
Always - 6541
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 115
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 061 061 000 Old_age
Always - 29141
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 113
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 104
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 115 106 000 Old_age
Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
I tried manually rsync some large folders, but the load seems to be ok
(same source, same destination drive/folder), goes to load 2, but not
higher.
I'm using Ubuntu server 14.04.3 LTS in 24/7 mode.
Backup destiantion is a separate local hard drive mounted (mount output
of the backup destination drive)
/dev/sdc1 on /mnt/sdc1 type ext4
(rw,noatime,commit=120,errors=remount-ro)
Has anyone any idea what should i try to get rid of that high load?
Thanks
Regards
Laszlo

--
___________________________________________________________________________
David Keegel <***@cyber.com.au> Cyber IT Solutions Pty. Ltd.
http://www.cyber.com.au/~djk/ Linux & Unix Systems Administration

Ken Woods

2016-01-31 19:27:09 UTC

Permalink

Does "sar" exist in unbuntu?

Post by David Keegel
László,
There are a number of things which could make your system slow; you
need to work out why it is slow.
I'd check whether your system is running low on available memory (RAM)
and therefore paging/swapping around 08h. Running rsync -aH will use
a lot of RAM for processing a large directory (with lots of files).
If it happens while you are around, you could run top to get a better
idea whether you are running short of memory (mem free small, mem buffers
small, swap used large, cpu %wait high), have enough processes using a
lot of CPU to consume all available CPU resources (cpu %idle=0) or
something else is making your I/O slow (again cpu %wait high and
processes in "D" state but without the other shortage of memory
indicators).
It's curious that you have a slowness problem at 08h but not 16h.
I'm wondering if some overnight cron jobs are running for a lot
longer than you expect, or many users are logging in around 08h.
Looking at the top processes listing in top should give you an
idea about those possibilities.
Ideally, start top before 08:00 to see a difference between before and
after rsnapshot starts, and look for processes running before 08:00.
If you don't have time to investigate properly, and just want to
try a short cut, you could try removing -H from rsync_short_args
(or --hard-links from rsync_long_args) and see if slowness stops.

--
___________________________________________________________________________
http://www.cyber.com.au/~djk/ Linux & Unix Systems Administration
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Ken Rossman

2016-01-31 20:25:55 UTC

Permalink

install either "sysstat" or "atsar" and you'll have 'sar'.

Post by Ken Woods
Does "sar" exist in unbuntu?

LÃ¡szlÃ³,
There are a number of things which could make your system slow; you
need to work out why it is slow.
I'd check whether your system is running low on available memory (RAM)
and therefore paging/swapping around 08h. Running rsync -aH will use
a lot of RAM for processing a large directory (with lots of files).
If it happens while you are around, you could run top to get a better
idea whether you are running short of memory (mem free small, mem buffers
small, swap used large, cpu %wait high), have enough processes using a
lot of CPU to consume all available CPU resources (cpu %idle=0) or
something else is making your I/O slow (again cpu %wait high and
processes in "D" state but without the other shortage of memory
indicators).
It's curious that you have a slowness problem at 08h but not 16h.
I'm wondering if some overnight cron jobs are running for a lot
longer than you expect, or many users are logging in around 08h.
Looking at the top processes listing in top should give you an
idea about those possibilities.
Ideally, start top before 08:00 to see a difference between before and
after rsnapshot starts, and look for processes running before 08:00.
If you don't have time to investigate properly, and just want to
try a short cut, you could try removing -H from rsync_short_args
(or --hard-links from rsync_long_args) and see if slowness stops.

the

Post by Balogh LÃ¡szlÃ³
rsnapshot is running at that time (08h) the system load gets a very
high level, above 15, sometimes about 30. At the other times the load
is also a little bit high, but the system is reachable. But when its
backing up at 08h after a few minutes the load gets incredible high,

Post by Balogh LÃ¡szlÃ³
the system is not reacheble, or takes minutes to type a character.
There is no other job at 08h which could affect rsnaphot job.
RSYNC_NICE='10'
RSYNC_IONICE='-c3'
5 0,8,16 * * * ionice -c 3
/usr/bin/rsnapshot hourly
Nothing helps. :(
SMART capabilities: (0x0003) Saves SMART data before entering
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 174 173 021 Pre-fail
Always - 4258
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 13
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 098 098 000 Old_age
Always - 1867
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 13
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 7
193 Load_Cycle_Count 0x0032 196 196 000 Old_age
Always - 14075
194 Temperature_Celsius 0x0022 115 111 000 Old_age
Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 186 180 021 Pre-fail
Always - 5683
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 35
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age
Always - 12113
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 32
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2
194 Temperature_Celsius 0x0022 121 111 000 Old_age
Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
<--------------- BACKUP DRIVE!
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 386
3 Spin_Up_Time 0x0027 168 168 021 Pre-fail
Always - 4566
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 48
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age
Always - 19282
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 48
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 37
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 113 108 000 Old_age
Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
SMART capabilities: (0x0003) Saves SMART data before entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 36
3 Spin_Up_Time 0x0027 169 167 021 Pre-fail
Always - 6541
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 115
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 061 061 000 Old_age
Always - 29141
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 113
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 104
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 115 106 000 Old_age
Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
I tried manually rsync some large folders, but the load seems to be ok
(same source, same destination drive/folder), goes to load 2, but not
higher.
I'm using Ubuntu server 14.04.3 LTS in 24/7 mode.
Backup destiantion is a separate local hard drive mounted (mount

output

Post by Balogh LÃ¡szlÃ³
of the backup destination drive)
/dev/sdc1 on /mnt/sdc1 type ext4
(rw,noatime,commit=120,errors=remount-ro)
Has anyone any idea what should i try to get rid of that high load?
Thanks
Regards
Laszlo

___________________________________________________________________________

http://www.cyber.com.au/~djk/ Linux & Unix Systems Administration

------------------------------------------------------------------------------

Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

Balogh László

2016-01-31 20:33:29 UTC

Permalink

Yes it exists. And you've right, that could be a good tool to check, but
unfortunately the logging option was disabled, so sar -u gives nothing.
But i enabled it, so we can check tomorrow after 8-9am what will be in
the sar log.

Thanks for the tip!

Post by Ken Woods
Does "sar" exist in unbuntu?

Post by Balogh LÃ¡szlÃ³
Hi All,
I'm new in this thread. I'm using rsnapshot for backing up my linux
system about two months ago. It's working very well, i have only one
problem.
I've set up an incremental backup, i have three hourly (00, 08 and
retain hourly 3
retain daily 7
retain weekly 4
retain monthly 6
Interesting is, that i have problems at the 08h hourly backup. When the
rsnapshot is running at that time (08h) the system load gets a very
high level, above 15, sometimes about 30. At the other times the load
is also a little bit high, but the system is reachable. But when its
backing up at 08h after a few minutes the load gets incredible high, so
the system is not reacheble, or takes minutes to type a character.
There is no other job at 08h which could affect rsnaphot job.
RSYNC_NICE='10'
RSYNC_IONICE='-c3'
5 0,8,16 * * * ionice -c 3
/usr/bin/rsnapshot hourly
Nothing helps. :(
SMART capabilities: (0x0003) Saves SMART data before
entering
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 174 173 021 Pre-fail
Always - 4258
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 13
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 098 098 000 Old_age
Always - 1867
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 13
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 7
193 Load_Cycle_Count 0x0032 196 196 000 Old_age
Always - 14075
194 Temperature_Celsius 0x0022 115 111 000 Old_age
Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
SMART capabilities: (0x0003) Saves SMART data before
entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 186 180 021 Pre-fail
Always - 5683
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 35
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age
Always - 12113
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 35
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 32
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2
194 Temperature_Celsius 0x0022 121 111 000 Old_age
Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
<--------------- BACKUP DRIVE!
SMART capabilities: (0x0003) Saves SMART data before
entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 386
3 Spin_Up_Time 0x0027 168 168 021 Pre-fail
Always - 4566
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 48
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age
Always - 19282
10 Spin_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 48
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 37
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 113 108 000 Old_age
Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
SMART capabilities: (0x0003) Saves SMART data before
entering
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 36
3 Spin_Up_Time 0x0027 169 167 021 Pre-fail
Always - 6541
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 115
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 061 061 000 Old_age
Always - 29141
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 113
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 104
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 10
194 Temperature_Celsius 0x0022 115 106 000 Old_age
Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age
Offline - 0
I tried manually rsync some large folders, but the load seems to be ok
(same source, same destination drive/folder), goes to load 2, but not
higher.
I'm using Ubuntu server 14.04.3 LTS in 24/7 mode.
Backup destiantion is a separate local hard drive mounted (mount output
of the backup destination drive)
/dev/sdc1 on /mnt/sdc1 type ext4
(rw,noatime,commit=120,errors=remount-ro)
Has anyone any idea what should i try to get rid of that high load?
Thanks
Regards
Laszlo

---
Ezt az e-mailt az Avast vÃruskeresÅ szoftver Ã¡tvizsgÃ¡lta.
https://www.avast.com/antivirus

jungle Boogie

2016-02-04 00:06:26 UTC

Permalink

Yes it exists. And you've right, that could be a good tool to check, but unfortunately the logging option was disabled, so sar -u gives nothing. But i enabled it, so we can check tomorrow after 8-9am what will be in the sar log. Thanks for the tip!

What have you found out?

--
-------
inum: 883510009027723
sip: ***@sip2sip.info
xmpp: jungle-***@jit.si

Balogh László

2016-02-04 07:56:03 UTC

Permalink

Hi!

I made some investigations, and had found out the main cause of the
problem was, that the memory was used nearly 100%, and swap is also
involved. Had found out that during the rsnapshot process memory is
going full, and after rsnapshot finishes it does not clear data from
memory. So, next time it runs again the load was going extremely high.
Some days ago i added to cron a new event, which is flushing memory
cache and emptying swap right after rsnapshot is finishing (i know
clearing cache and swap is dangerous, but till now i did not get any
problem with it). From that time i did not get any high load warning
(i'm using Nagios to check systems) and rsnapshot is finishing in about
half an hour (before it was working 2 hours with high system load).
Memory usage is now normal. When rsnapshot is running it goes to full,
but after clearing cache and swap it goes back to normal, and staying at
that level all the time, so no other application gets to use the whole
memory. This means the problem is in rsnapshot and not in other app. I
have 8GB RAM in my system.
So, now it seems to be ok with this small modification. Will see after a
while. I don't know why rsnapshot is using so much of memory, and why it
does not clear its data after finishing.

BR
Laszlo

Post by jungle Boogie

Yes it exists. And you've right, that could be a good tool to check, but unfortunately the logging option was disabled, so sar -u gives nothing. But i enabled it, so we can check tomorrow after 8-9am what will be in the sar log. Thanks for the tip!

What have you found out?

---
Ezt az e-mailt az Avast víruskereső szoftver átvizsgálta.
https://www.avast.com/antivirus

l***@gmail.com

2016-02-04 12:04:04 UTC

Permalink

Post by Balogh LÃ¡szlÃ³
Hi!
I made some investigations, and had found out the main cause of the
problem was, that the memory was used nearly 100%, and swap is also
involved. Had found out that during the rsnapshot process memory is
going full, and after rsnapshot finishes it does not clear data from
memory. So, next time it runs again the load was going extremely high.
Some days ago i added to cron a new event, which is flushing memory
cache and emptying swap right after rsnapshot is finishing (i know
clearing cache and swap is dangerous, but till now i did not get any
problem with it). From that time i did not get any high load warning
(i'm using Nagios to check systems) and rsnapshot is finishing in about
half an hour (before it was working 2 hours with high system load).
Memory usage is now normal. When rsnapshot is running it goes to full,
but after clearing cache and swap it goes back to normal, and staying at
that level all the time, so no other application gets to use the whole
memory. This means the problem is in rsnapshot and not in other app. I
have 8GB RAM in my system.
So, now it seems to be ok with this small modification. Will see after a
while. I don't know why rsnapshot is using so much of memory, and why it
does not clear its data after finishing.

Is your version of rsync too old? This FAQ may shed some light on your
first question.

https://rsync.samba.org/FAQ.html#4

Regards,
ST
--

Balogh László

2016-02-04 13:11:41 UTC

Permalink

My rsync version:
rsync version 3.1.0 protocol version 31

rsnapshot version:
1.3.1

Br
Laszlo

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
Hi!
I made some investigations, and had found out the main cause

of the

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
problem was, that the memory was used nearly 100%, and swap is also
involved. Had found out that during the rsnapshot process

memory is

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
going full, and after rsnapshot finishes it does not clear

data from

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
memory. So, next time it runs again the load was going

extremely high.

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
Some days ago i added to cron a new event, which is flushing

memory

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
cache and emptying swap right after rsnapshot is finishing (i know
clearing cache and swap is dangerous, but till now i did not

get any

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
problem with it). From that time i did not get any high load

warning

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
(i'm using Nagios to check systems) and rsnapshot is finishing in about
half an hour (before it was working 2 hours with high system

load).

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
Memory usage is now normal. When rsnapshot is running it goes to full,
but after clearing cache and swap it goes back to normal, and

staying

Post by l***@gmail.com
at

Post by Balogh LÃ¡szlÃ³
that level all the time, so no other application gets to use

the whole

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
memory. This means the problem is in rsnapshot and not in

other app. I

Post by l***@gmail.com

Post by Balogh LÃ¡szlÃ³
have 8GB RAM in my system.
So, now it seems to be ok with this small modification. Will

see after

Post by l***@gmail.com
a

Post by Balogh LÃ¡szlÃ³
while. I don't know why rsnapshot is using so much of memory,

and why

Post by l***@gmail.com
it

Post by Balogh LÃ¡szlÃ³
does not clear its data after finishing.

Is your version of rsync too old? This FAQ may shed some light on your
first question.
https://rsync.samba.org/FAQ.html#4
Regards,
ST
--

------------------------------------------------------------------------------

Post by l***@gmail.com
Site24x7 APM Insight: Get Deep Visibility into Application

Performance

Post by l***@gmail.com
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!

http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140

Post by l***@gmail.com
_______________________________________________
rsnapshot-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

---------------------------------------

Patrick O'Callaghan

2016-02-04 12:20:11 UTC

Permalink

Post by Balogh LÃ¡szlÃ³
I made some investigations, and had found out the main cause of the
problem was, that the memory was used nearly 100%, and swap is also
involved. Had found out that during the rsnapshot process memory is
going full, and after rsnapshot finishes it does not clear data from
memory.

I don't understand. Each run of rsnapshot (from cron) is a separate
process. When it finishes all its memory resources are returned to the
system, as happens with every other process. Why do you think this is not
happening?

Is there any possibility that you have two overlapping rsnapshots at
particular times (i.e. a second one starts before the first one had time to
finish)? This would account for the increased load at certain hours of the
day.

poc

Nico Kadel-Garcia

2016-02-04 12:26:27 UTC

Permalink

On Thu, Feb 4, 2016 at 7:20 AM, Patrick O'Callaghan

I don't understand. Each run of rsnapshot (from cron) is a separate process.
When it finishes all its memory resources are returned to the system, as
happens with every other process. Why do you think this is not happening?
Is there any possibility that you have two overlapping rsnapshots at
particular times (i.e. a second one starts before the first one had time to
finish)? This would account for the increased load at certain hours of the
day.
poc

This. And rsync before rsync version 3 could be really, really
resource consoming for very large numbers of files, especially
preserving hardlinks.

David Keegel

2016-02-04 23:18:57 UTC

Permalink

Post by Balogh LÃ¡szlÃ³
Hi!
I made some investigations, and had found out the main cause of the
problem was, that the memory was used nearly 100%, and swap is also
involved.

Okay, that makes sense.

Post by Balogh LÃ¡szlÃ³
Had found out that during the rsnapshot process memory is going full,

I guess you mean a rsync (or cp) child process started by rsnapshot
is using a lot of memory.

Post by Balogh LÃ¡szlÃ³
and after rsnapshot finishes it does not clear data from memory.

I do not understand, this does not make sense to me. Please explain
how you determined this. Can you show evidence?

Post by Balogh LÃ¡szlÃ³
So, next time it runs again the load was going extremely high.

Do you have a lockfile set in rsnapshot configuration?
I recommend it to make sure your system does not run more than one
rsnapshot at a time (apart from use_lazy_deletes).

Post by Balogh LÃ¡szlÃ³
Some days ago i added to cron a new event, which is flushing memory
cache and emptying swap right after rsnapshot is finishing (i know
clearing cache and swap is dangerous, but till now i did not get any

Please tell us what you added exactly. I don't know what you mean
about "flushing memory cache and emptying swap".

Post by Balogh LÃ¡szlÃ³
problem with it). From that time i did not get any high load warning
(i'm using Nagios to check systems) and rsnapshot is finishing in about
half an hour (before it was working 2 hours with high system load).

Have you checked rsnapshot log and/or email from cron to see if rsnapshot
is completing successfully?

I am concerned that your new event in cron might cause problems for
rsnapshot backups.

Post by Balogh LÃ¡szlÃ³
Memory usage is now normal. When rsnapshot is running it goes to full,
but after clearing cache and swap it goes back to normal, and staying at
that level all the time, so no other application gets to use the whole
memory. This means the problem is in rsnapshot and not in other app. I
have 8GB RAM in my system.

I would be interested in memory usage in the hour or so before you
run rsnapshot, in case something else is also using lots of memory.

(Laszlo sent me 400KB of images on Jan 31 which make me suspect that.)

Post by Balogh LÃ¡szlÃ³
So, now it seems to be ok with this small modification. Will see after a
while. I don't know why rsnapshot is using so much of memory, and why it
does not clear its data after finishing.
BR
Laszlo

Balogh László

2016-02-05 06:53:22 UTC

Permalink

This post might be inappropriate. Click to display it.

Ken Woods

2016-02-05 16:08:43 UTC

Permalink

Post by Balogh LÃ¡szlÃ³
have 8GB RAM in my system.

âIs this a delayed message from 1997?
â

â

Balogh László

2016-02-05 16:25:12 UTC

Permalink

Post by Ken Woods

Post by Balogh LÃ¡szlÃ³
have 8GB RAM in my system.

âIs this a delayed message from 1997?
â

Why do you ask that?

---
Ezt az e-mailt az Avast vÃruskeresÅ szoftver Ã¡tvizsgÃ¡lta.
https://www.avast.com/antivirus