writefd_unbuffered failed to write 4092

12 posts / 0 new
Last post
osiristrading
Offline
Last seen: 9 years 6 months ago
Joined: 09.03.2012 - 10:23
writefd_unbuffered failed to write 4092

Good day

We transfer several terabytes of virtual servers to a remote site (5-6 at the same time due to the latency to the remote site).

Out of the 5-6 sometimes 1 or 2 fail, and we get these message in the log file on the server side:

2012/03/28 13:42:12 rsync: writefd_unbuffered failed to write 4092 bytes to socket : Connection reset by peer (104)

2012/03/28 15:06:41 rsync: writefd_unbuffered failed to write 4092 bytes to socket : Connection reset by peer (104)

 

On the client side rsync.exe gets stuck on

sending incremental file list

 

The connection between the sites, although 100ms, is extremely stable. I have a ping going the whole time between the two nodes and there is almost zero packet loss. Our command line for the client is as follows:

c:\cwrsync\bin\rsync --timeout=1800 --block-size=32768 -v -r --inplace --ignore-times --progress --include='*/' --include='*.vhd*' --include='*.VHD*'  --exclude='*' /cygdrive/v/%%i rsync://xx.xx.xx.xx/HPV

(ip address is anonymized by tk)

The VHD files are quite large, ranging from 40GB to 150GB, but the failure is not size dependent, it happens to all of them.

Any ideas how to fix/work around this?

Thanks

Jason

 

 

itefix
Offline
Last seen: 7 hours 28 min ago
Joined: 01.05.2008 - 21:33
Can you run rsync --version

Can you run rsync --version at the receiver side (is it a unix machine?) and publish results here ? Do you observe any resource hits (cpu, disk use, net utilization) at both sides ?

osiristrading
Offline
Last seen: 9 years 6 months ago
Joined: 09.03.2012 - 10:23
The server side is running

The server side is running Windows Server 2008 R2.

 

C:\cwrsyncserver\bin>rsync --version

rsync  version 3.0.9  protocol version 30

Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.

Web site: http://rsync.samba.org/

Capabilities:

    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,

    no socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,

    append, ACLs, no xattrs, iconv, symtimes

 

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you

are welcome to redistribute it under certain conditions.  See the GNU

General Public Licence for details.

 
 
There is a fair amount of disk usage on the server side when the copy first starts (~700MB/sec disk IO) while each file is read entirely, but CPU usage remains low. I changed my scripts to start each rsync.exe process on the client 30 seconds apart and got far fewer errors - not sure if this information helps at all? 
 
Both the client and server are installed from the latest bundle in the downloads section.
itefix
Offline
Last seen: 7 hours 28 min ago
Joined: 01.05.2008 - 21:33
Spreading start times by 30

Spreading start times by 30 seconds seems to help to reduce the problem. That may indicate that there are some resource-bound issues here. I would suggest to use the option --bwlimit :

--bwlimit=KBPS
This option allows you to specify a maximum transfer rate in kilobytes per second. This option is most effective when using rsync with large files (several megabytes and up). Due to the nature of rsync transfers, blocks of data are sent, then if rsync determines the transfer was too fast, it will wait before sending the next data block. The result is an average transfer rate equaling the specified limit. A value of zero specifies no limit.

As you have several concurrent operations with large files, it may help to control how much bandwidth each connection will use. Example: If your available total bandwidth is 5 MB / sec, it may be worth to experiment with bwlimit values of 1500, 2000, 2500. The real value is totally dependent on your environment though.

If it doesn't help, I can think to deliver an rsync version using socketpairs. It may be a better option for pure rsync transfers.

osiristrading
Offline
Last seen: 9 years 6 months ago
Joined: 09.03.2012 - 10:23
HiThank you, I will try that.

Hi

Thank you, I will try that. I assume this limits the total of all incoming connections to this speed?

Also, what do you mean by a version using socketpairs? 

THanks

Jason

itefix
Offline
Last seen: 7 hours 28 min ago
Joined: 01.05.2008 - 21:33
Bandwidth limit applies per

Bandwidth limit applies per connection if you use it at the sender side. You can also specify it at the receiver side (rsync server) to limit the total throughput.

Rsync packaged in Cwrsync uses no socketpairs as it seems a more stable alternative if remote shell (ssh) is involved. This is default for Cygwin environment. However, there is no remote shell in your setup. Rsync compiled with socketpairs options can be a viable alternative in your scenario. I recall similar situations which that helped. Worth to try if bwlimit doesn't help.

osiristrading
Offline
Last seen: 9 years 6 months ago
Joined: 09.03.2012 - 10:23
Thanks, I'll let you know as

Thanks, I'll let you know as soon as I'm done testing. I have to let the current sync (3.5TB) finish, then start again.

osiristrading
Offline
Last seen: 9 years 6 months ago
Joined: 09.03.2012 - 10:23
OK I've done some testing

OK I've done some testing with BW limits, and there is no change, I still get the errors about 10-15% of the time, sometimes more. The errors only occur during the initial read (i.e. when client says "sending incremental file list"). 

Is there a way to tell the client to retry if there is a protocol error? Currently the client just waits until the --timeout parameter has been reached.

itefix
Offline
Last seen: 7 hours 28 min ago
Joined: 01.05.2008 - 21:33
Ok. It seems that the

Ok. It seems that the receiving side is getting quite busy. As the problem occurs during creating/sending the file list, it may be worth to specify files to be copied explicitly by using --files-from option:

--files-from=FILE
Using this option allows you to specify the exact list of files to transfer (as read from the specified FILE or - for standard input). It also tweaks the default behavior of rsync to make transferring just the specified files and directories easier
See rsync documentation for more information.
osiristrading
Offline
Last seen: 9 years 6 months ago
Joined: 09.03.2012 - 10:23
There are perhaps 30 files at

There are perhaps 30 files at most being transferred. Surely this cannot be a problem?

itefix
Offline
Last seen: 7 hours 28 min ago
Joined: 01.05.2008 - 21:33
Sorry for a late response. As

Sorry for a late response. As stops are occuring during sending incremental file lists, I was wondering if that might help to send an explicit list of files. An other scenario can be to produce 30 explicit rsync commands for each VM.

Do you see any resource hits at the receiver side ?

athwds
Offline
Last seen: 8 years 5 months ago
Joined: 18.06.2013 - 18:03
I've had this error for

I've had this error for weeks.   It turns out my destination HD was full! 

rsync should really have a better error message!