cwRsync - Rsync for Windows

Try to remove /etc/fstab and use the option --no-perms  option instead. The problem can also be related to real-time anti-virus scanning. Consider file/folder exclusion if possible.

From Windows Developer network:

“In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters. A local path is structured in the following order: drive letter, colon, backslash, name components separated by backslashes, and a terminating null character. For example, the maximum path on drive D is "D:\some 256-character path string<NUL>" where "<NUL>" represents the invisible terminating null character for the current system codepage. (The characters < > are used here for visual clarity and cannot be part of a valid path string.)

The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the "\\?\" prefix. For example, "\\?\D:\very long path".”

Even if the underlying APIs may have support for really long paths, the problem occurs due to user interface tools like Windows Explorer, Cmd prompt and PowerShell which don't support Unicode variants of the Windows APIs. The result is then path names up to 256 chars. As of version 1.7, Cygwin uses unicode Windows APIs, you can use tools like bash shell or nano editor for management of files/paths with long names.

Log example for cwRsync transfer with long path names:

...
2014/03/25 22:08:42 [1784] >f+++++++++ backup/0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
/0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
/0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
/0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789/testfile
...

Assuming that you want to pull directory c:\backup on Windows machine to /var/backup on Linux machine:

On Windows machine :

• Install Cwrsync Server

• Start RsyncServer service

• Add a new module to the configuration file rsyncd.conf :

[backup]
path = /cygdrive/c/backup
read only = true
transfer logging = yes

On Linux machine :

• Use the command below to initiate rsync:

rsync -vrt windows_machine::backup /var/backup

NB! Keep in mind that the recipe above is suitable for operations within a secure network.

As of version 2.6.7, rsync server can kick scripts before and after transfers. You can use the recipe below to implement this very handy functionality in cwrsync server:

1. Install Copssh and Cwrsync Server.
2. Start a windows command shell from cwrsync start menu and make a copy of bash.exe as sh.exe in the bin directory.
   
   

   copy bin\bash.exe bin\sh.exe

3. Create two script files in the bin directory for pre-exec operations:
   
   

   c:\program files\ICW\bin\pre-exec.sh

   
   
   

   # Shell script for your pre-exec operations
   cmd /c "c:\program files\ICW\bin\pre-exec.cmd"
   exit 0

   
   
   

   
   c:\program files\ICW\bin\pre-exec.cmd

   
   
   

   
   @ECHO OFF
   .....

4. Create two script files for post-exec operations as described above.
5. Update your rsyncd.conf file:
   
   

   
   
   

   
   # Module definitions
   [test]
   path = .....
   ......
   pre-xfer exec = /bin/pre-exec.sh
   post-xfer exec = /bin/post-exec.sh

Use of vocals would be inappropriate (!) as the word rsync has no vocals in it, and this package is also dependent on cygwin:

CYGWIN + RSYNC

Assuming that you want to synchronize c:\documents and settings\user on WINCLIENT to c:\backup on WINSERVER :

On WINSERVER:
 Install cwRsync Server and Copssh into the same directory. 

• Start RsyncServer and OpenSSHD services

• Activate a user (test1 in our example below) via Copssh control panel and specify c:\backup as home directory.

• Add a new module to rsyncd.conf :

[backup]
path = /cygdrive/c/backup
read only = false
transfer logging = yes

on WINCLIENT :

• Download cwRsync client
• Open a command prompt and initiate an SSH tunnel (Port 5000 will be used for local port forwarding to Rsync daemon port 873 at WINSERVER)

bin\ssh -L 5000:WINSERVER:873 test1@WINSERVER

• Open a new command prompt and access Rsync daemon via port 5000:

bin\rsync -vrt "/cygdrive/c/documents and settings/user/" rsync://localhost:5000/backup

• Close the SSH tunnel upon the completion of Rsync transfers.
  
  

cwRsync itself has no support for it. However, after an rsync operation, you can use Robocopy (standard in more recent Windows versions), to transfer all windows specific file information (NTFS security, timestamps, attributes, ownership, auditing info):

ROBOCOPY source destination /XO /XN /XC /E /COPY:ATSOU

options /XO (exclude older), /XN (exclude newer) and /XC (exclude changed) assure that only existing files are targeted. Option /E is for recursive directory operation including empty ones. Option /COPY:ATSOU instructs Robocopy to copy only attributes (A), timestamps (T), NTFS security information (S), Ownership O) and auditing (U).

Assuming that you want to synchronize from directory c:\srcdir on machine WINCLIENT to directory c:\destdir on machine WINSERVER :

On WINSERVER :

• Install cwRsync Server

• Start RsyncServer service

• Make sure that the service account has write access to c:\destdir. You can use Prep a Dir for Upload wizard from start menu.

• Add a new module to rsyncd.conf :

[wintest]
path = /cygdrive/c/destdir
read only = false
transfer logging = yes

On WINCLIENT :

• Download cwRsync client.

• Use the command below to initiate rsync communication:

".\bin\rsync" -av /cygdrive/c/srcdir/ WINSERVER::wintest

NB! Keep in mind that the recipe above is suitable for operations within a secure network.

Assuming that you want to synchronize from directory /home/user on *nix machine to directory c:\backup on Windows machine :

On Windows:
 

• Install Copssh and cwRsync server. Use the same service account.

• Start RsyncServer and OpenSSHD services

• Activate a user
• Import your public key via Copssh Control Panel:

•  Make sure that the user and the service account has write access to c:\backup
• Add a new module to rsyncd.conf :

[backup]
path = /cygdrive/c/backup
read only = false
transfer logging = yes

On Linux:

• Use shell script below after having it updated according to your needs:

NB! You must forward ssh port 22 to Windows machine at the edge of your secure network (your router/firewall). I recommend strongly to tighten security further by using options available in rsync and openssh (host limitations, secrets file ...).

Rsync server uses port 873 by default. You can add port directive to the configuration file rsyncd.conf to specify an alternative port.

port new-port-number

Don't forget to restart the service.

Here is a simple recipe ( NB! This is not a very secure solution, since we use null passphrase during generation of public key pairs, use it at your own risk):

• cwRsync client. Generate key pairs :

ssh-keygen -q -t rsa -f cwrsync -N ""

Files cwrsync and cwrsync.pub will be generated.

• Transfer cwrsync.pub to your *nix machine.

• *nix machine. Run following commands:

mkdir -p ~$user/.ssh
cat cwrsync.pub > ~$user/.ssh/authorized_keys
chmod 755 ~$user ~$user/.ssh
chmod 644 ~$user/.ssh/authorized_keys
chown $user ~$user ~$user/.ssh ~$user/.ssh/authorized_keys

• cwRsync client. Update your rsync commands in your batch file by specifying option -e "ssh -i cwrsync".

cwRsync has all ssh binaries you need to establish ssh communication.

Use batch file example in the cwRsync client package to set up communication correctly.

It's trying to fire up ssh because your target has a single colon in it.

server:/path/to/stuff means "fire up ssh or rsh and make me a tunnel to
server"

server::module/path/to/stuff means "try to access an rsync daemon on
server and access path/to/stuff on module"

/path/to/stuff means "access /path/to/stuff on the local machine"

Hope that helps.

by Jim Salter

Windows paths may contain a colon (:) as a part of drive designation and backslashes (example c:\, g:\). However, in rsync syntax, a colon in a path means searching for a remote host.

Solution: use absolute path 'a la unix', replace backslashes (\) with slashes (/) and put -/cygdrive/- in front of the drive letter.

Example :

C:\WORK\*

• • -->

/cygdrive/c/work/*

If you're setting up cwrsync to run as a Service on a Windows machine ensure the .conf file has the line:

strict modes = false

otherwise rsync passwords will not work in Windows.

YES: Cygwin 1.7 binaries used by Cwrsync allow co-existence of multiple Cygwin implementations.

If you run into permissions problems or your directories' security ACLs are populated by some unwanted groups/users, you need to make sure that:

• File ../etc/fstab  exists with at least the content below (with Unix line endings even if the file has only one line!):

none /cygdrive cygdrive binary,posix=0,user,noacl 0 0
  

That will instruct Cygwin not to touch permissions.

NB! You may experience very slow transfer rates and/or building the file list may take long time. You can remove /etc/fstab and use the option --no-perms  option instead. The problem can also be related to real-time anti-virus scanning. Consider file/folder exclusion if possible.

More information:

The Cygwin Mount Table

The cygdrive path prefix

Assuming that you want to pull directory c:\backup on machine WIN_SOURCE to directory /var/backup on machine LINUX_DEST :

On WIN_SOURCE :

- Install cwRsync Server (without openssh component)
- Start RsyncServer service
- Add a new module to rsyncd.conf :

[backup]

 path = /cygdrive/c/backup

 read only = true

 transfer logging = yes

On LINUX_DEST :

- Use the command below to initiate rsync:

• rsync -av WIN_SOURCE::backup /var/backup

NB! Keep in mind that the recipe above is suitable for operations within a secure network.