Support for large files

7 posts / 0 new
Last post
itefix
Offline
Last seen: 1 hour 16 min ago
Joined: 01.05.2008 - 21:33
Support for large files

Hmm. 4 GB is a kind of well-known limit :-) Cygwin 1.5.x supports large files and recent versions of rsync on cygwin is configured with 64-bit support.
I suspect then gcc compiler I use for building rsync executables(v3.3.3).

According to notes at gcc web site, large file support comes with gcc 3.4.

I have recently upgraded my gcc to v3.4.1 and built rsync 2.6.5 again.

File signatures:
MD5: c64673f0259237bb4b6ed00b1bb2374e *rsync-265-with-gcc-341.zip
SHA1: 6f82b1d9346b4cd71f478b7b57a7e65a6ce8e9bb *rsync-265-with-gcc-341.zip

Give a try and let me know if it helps.

Rgrds Tev

itefix
Offline
Last seen: 1 hour 16 min ago
Joined: 01.05.2008 - 21:33
Re: Support for large files

OK, I'll test in the next day or so and advise.
Thank you.

Vaughan

itefix
Offline
Last seen: 1 hour 16 min ago
Joined: 01.05.2008 - 21:33
Re: Support for large files

Hi,
rsync fell over on the same file, in the same place I think (or certainly close to it).

Bytes transferred of the file was: 4,294,705,152

Error message:
rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" : Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(584)

Vaughan

itefix
Offline
Last seen: 1 hour 16 min ago
Joined: 01.05.2008 - 21:33
Re: Support for large files

Hi,
Unfortunately I am also seeing similar errors on much smaller files (600k):

rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" : Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(570)

Files vary, and the io.c(value) changes, but the substance of the error is consistent. I have tried disabling all switches apart from -a, and it makes no difference.

This is with the v2 client and the newer version that you compiled, I will try v1.3, and see if that makes any difference.

I can also post the sending_file log for an example file if that might be useful.

itefix
Offline
Last seen: 1 hour 16 min ago
Joined: 01.05.2008 - 21:33
Re: Support for large files

Hi,
The 1.3 rsync.exe has identical symptoms. I don't understand it. I had successfully rsync'ed this directory tree a number of times previously.

Trying to think of the things that I have changed. I guess I'll go back to 1.3 server and see if that makes any difference.

Vaughan

itefix
Offline
Last seen: 1 hour 16 min ago
Joined: 01.05.2008 - 21:33
Re: Support for large files

Hi,
I did a test. cwRsync 2.0.0 client and server are installed on windows 2003 machines. I tried to copy a 6.22 GB image file from client to server. First run took about 1h 20m and is completed successfully:

F:\image\data\w2003-mf>cwrsync (batch file running rsync --progress -v test3.dat xxxxx::test with cygwin=nontsec)
test3.dat
6684591156 100% 1.27MB/s 1:23:28 (1, 100.0% of 1)

sent 6685407238 bytes received 38 bytes 1334279.47 bytes/sec
total size is 6684591156 speedup is 1.00

There was about 2.5 GB free space on server side. I then tried to run the same command again. I got following error message:

F:\image\data\w2003-mf>cwrsync
test3.dat
rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" : Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(584)

I observed that rsync on server side used temporary disk space and the problem above appeared because there were no free space on disk. I then run rsync again with --inplace option and the operation is completed successfully:

F:\image\data\w2003-mf>cwrsync
opening tcp connection to xxxxx port 873
opening connection using --server -vv --inplace . test
test3.dat
6684591156 100% 23.07MB/s 0:04:36 (1, 100.0% of 1)
total: matches=81767 tag_hits=81767 false_alarms=3 data=0

sent 327158 bytes received 654174 bytes 1454.90 bytes/sec
total size is 6684591156 speedup is 6811.75

itefix
Offline
Last seen: 1 hour 16 min ago
Joined: 01.05.2008 - 21:33
Re: Support for large files

Hi Tev,
Well this sounds promising for a few reasons:

1. I went back to v1.3 on both the client and the server, and it didn't make any difference.

2. At the site where I was trying to send the 5GB file, there isn't heaps of free space, so this is consistent with your observations.

3. At the site where I am seeing the problem with the 600K file, there isn't much free space on the server either (doh!), it's more apparent here, and I should have thought to check for that.

I am sure you're right about this. Thanks.

Vaughan

Topic locked