rsync slows down (random)

6 posts / 0 new
Last post
Philippe Ferreira
Offline
Last seen: 8 years 2 months ago
Joined: 26.11.2008 - 16:51
rsync slows down (random)

Sometimes, on directories containing a mix of big and small files, rsync can suddenly slows down on big files. It can slow down up to 15kB/s, even on a LAN! It generally happens when --ignore-times is used. Thank you.

itefix
Offline
Last seen: 9 hours 13 min ago
Joined: 01.05.2008 - 21:33
Actually this may happen as

Actually this may happen as rsync's quick check mechanism is turned off by --ignore-times option:

-I, --ignore-times
Normally rsync will skip any files that are already the same size and have the same modification timestamp. This option turns off this "quick check" behavior, causing all files to be updated

By specifying that option, you tell rsync to use rsync algorithm all the time whenever rsync daemon is involved. That may result in extra work as each updated file must be checked by block by block.

Philippe Ferreira
Offline
Last seen: 8 years 2 months ago
Joined: 26.11.2008 - 16:51
I agree with you, however I'm

I agree with you, however I'm using "--ignore-times" on purpose. I have to check all files block by block. It's a transfer of SQL Server database files, so the timestamps are not updated everytime... The problem is the following: how do you explain a speed of 15kB/s on a 10GB file, over a 8Mbit/s WAN and even over a Gigabit LAN?

itefix
Offline
Last seen: 9 hours 13 min ago
Joined: 01.05.2008 - 21:33
Well, that explains the

Well, that explains the problem: An SQL Server manages contents of the database files itself. It may move chunks of data from one position to another as a result of garbage collection or disk optimization for example. That may result in a completely different image in comparison to a previous one, causing rsync to use excessive amount of time for block-to-block sync.

Philippe Ferreira
Offline
Last seen: 8 years 2 months ago
Joined: 26.11.2008 - 16:51
I don't think this is the

I don't think this is the problem, because this generally happens on the SQL LOG file, which is very sequential. Moreover, when this slow down occurs, the speed is very stable (between 15kB/s and 20kB/s), until the end of the file... So this is a big problem. I'll try to make this problem reproductible, making a copy of both versions of files (local and remote) when this will occur again. Thank you.

itefix
Offline
Last seen: 9 hours 13 min ago
Joined: 01.05.2008 - 21:33
When you mention a 10GB file,

When you mention a 10GB file, I was thinking of database files themselves :-) I have informed you about the possible scenarios. I am not sure what else I can contribute here. If those slowdowns happens randomly, it is even more difficult for me to pinpoint a potential problem. It could be anything. You can try to run rsync with three -v options to see what happens.