EzDevInfo.com

Rsync creates a directory with the same name inside of destination directory

When I run this command

rsync -avzp --del -e "ssh -p myport" user@hostname:/var/www/tests /var/www/tests

files get synchronized but instead of saving files in /var/www/tests, Rsync creates one more directory "tests" inside of existing "tests":

/var/www/tests/tests

and puts files there. How to tell Rsync not to create a new directory?

Nice rsync on remote machine

When using rsync+ssh to access a remote machine, is there a way to "nice" the rsync process on the remote machine (to lower its priority)?

Editing the question to clarify:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
backups  16651 86.2  0.1   3576  1636 ?        Rs   11:06   0:06 rsync --ser...

(rsync line snipped)

This is a backup cron job that normally runs at 4am, but when I happen to be awake (and committing, or using Bugzilla hosted on that same machine), it kills server performance, so I wanted a quick "hack" to try and fix it a bit.

Using RSYNC with Amazon S3

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools)

Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and cost. I'm considering using rsync with S3 and came across s3rsync. I was just wondering if anybody had any experience using this, or any other utility?

rsync - Exclude files that are over a certain size?

I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a lot about a small, such as firefox cookie files, or .bashrc. Is there some invocation to rsync that will exclude files that are over a certain size? That way I could copy all files that are less than 10MB first, then do all files. That way I can do a fast backup of the most important files, then a longer backup of everything else.

Perform rsync while following sym links

I have a directory that contains symbolic links to other directories located on different media on my system:

/opt/lun1/2011
/opt/lun1/2010
/opt/lun2/2009
/opt/lun2/2008
/opt/lun3/2007

But the symbolic links show up as:

/files/2011
/files/2010
/files/2009
/files/2008
/files/2007

How can I perform an rsync that follows the symbolic links?

e.g.:

rsync -XXX /files/ user@server:/files/

Sync LVM snapshots to backup server

I have a number of Xen virtual machines running on a number of Linux servers. These VMs store their disk images in Linux LVM volumes with device names along the lines of /dev/xenVG/SERVER001OS and so on. I'd like to take regular backups of those disk images so I can restore the VMs in case we need to (the LVM devices are already mirrored with DRBD between two physical machines each, I'm just being extra paranoid here).

How do I go about this? Oviously the first step is to snapshot the LVM device, but how do I then transfer data to a backup server in the most efficient manner possible? I could simply copy the whole device, something along the lines of:

dd if=/dev/xenVG/SERVER001OS | ssh administrator@backupserver "dd of=/mnt/largeDisk/SERVER001OS.img"

...but that would take a lot of bandwidth. Is there an rsync-like tool for synching contents of whole disk blocks between remote servers? Something like:

rsync /dev/xenVG/SERVER001OS backupServer:/mnt/largeDisk/SERVER001OS.img

If I understand rsync's man page correctly, the above command won't actually work (will it?), but it shows what I'm aiming for. I understand the --devices rsync option is to copy devices themselves, not the contents of those devices. Making a local copy of the VM image before syncing it with the remote server isn't an option as there isn't the disk space.

Is there a handy utility that can synch between block devices and a backup file on a remote server? I can write one if I have to, but an existing solution would be better. Have I missed an rsync option that does this for me?

How to repeatedly call rsync until files are sucessfully transferred

I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with

rsync: connection unexpectedly closed

Rsync is called with --partial, so I'd like to be able to call rsync in a loop until files are fully transfered. There doesn't seem to be a flag to tell rsync to retry.

What would be the best way to script it? A bash for loop?

Does Rsync Allow Files To Be Synced Both Ways?

I have backed up a linux web server using rsync with cygwin. I now have a perfect copy of the server on my windows laptop. If i delete or modify a file on my laptop and run rsync again with cygwin will it delete/update the same file on the server? Im under the impression that if i delete/modify on the server and run rsync on my laptop it will delete/modify the local file on my laptop but does this work in reverse?

Rsync daemon: is it really useful?

Are there any practical benefits in using rsyncd compared to rsync over ssh? Does it really increase speed, stability, anything?

Complete restore of linux system

I am familiar with using rsync to back up various files from my system but what is the best way to completely restore a machine.

What I have tried in the past is:

1. Do a basic format/reinstall from the Fedora install disks
2. Make sure networking is enabled
3. Copy everything from rsync backup over the top of the newly installed system

This way sort of works but I do not think every package that was installed works 100% afterwards.

I want to be able to restore my system with the minimum amount of effort and everything work the same as at the moment the backup was taken. Also if possible install to other machines and essentailly have two machines with the same packages and data.

How can I use rsync to duplicate a directory tree, creating hardlinks to files?

From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory structure starting at the root data folder, creating hard links to all original the files (some of them are rather big), and I can overwrite in the destination tree only the files that need migrating. In the end, I can safely switch from the old files to the new files with two mv operations.

However, I can't seem to get rsync to do this. I tried

rsync -a --link-dest=$DATA $DATA $DATA/../upgrade_tmp

but instead of creating hard links to files, rsync copies them entirely. Is there a problem using the same source and link-dest directory?

Rsync : copying over timestamps only

Currently I have two directories A/ and B/ which are identical in every respect, with the exception of the timestamps. Therefore if I run the command :

rsync --dry-run -crvv A/ B/

then all files are marked "uptodate", whereas the command :

rsync --dry-run -rvv A/ B/

shows that all files are to be copied over from A/ to B/.

My question is this : given that I know the files are identical (in respect to contents), then is there any way (via rsync or otherwise) to set the timestamps for files in B/ to be identical to the timestamps of the files in A/, without copying over all the files from A/ to B/ ?

Thanks

Why is rsync skipping the main directory?

I'm trying to use rync locally (on a windows machine) to a remote server (my osx box) in order to test a remote deploy build script. I've done rsync before just fine between 2 linux servers, but I'm having problems now. Here is the output:

$ rsync -v -e ssh myuser@10.86.83.11:/Library/WebServer/sites/staging/app1/ ./export

skipping directory /Library/WebServer/sites/staging/app1/.

sent 8 bytes  received 13 bytes  3.82 bytes/sec
total size is 0  speedup is 0.00

$ 

or

$ rsync -avz -e ssh myuser@10.86.83.11:/Library/WebServer/sites/staging/app1/ ./export

receiving file list ... done
./

sent 26 bytes  received 68 bytes  17.09 bytes/sec
total size is 0  speedup is 0.00


$ 

remote app1 directory is empty while local export directory has 4 sub directories and then a bunch of files in each of those

How can one efficiently use S3 to back up files incrementally?

I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP.

There look to be a few approaches.

s3rsync (but this just bolts on rsync to s3). Straightforward. Not sure I want to depend on something 3rd party. I wish s3 just supported rsync.

There also are some rsync 'clones' like duplicity that claim to support s3 without said bolt-on. But how can it do this? Are they keeping an index file locally? I'm not sure how that can be as efficient.

I obviously want to use s3 because it's cheap and reliable, but there are things that rsync is the tool for, like backing up a giant directory of images.

What are the options here? What do I lose by using duplicity + s3 instead of rsync + s3rsync + s3?

Options to efficiently synchronize 1 million files with remote servers?

At a company I work for we have such a thing called "playlists" which are small files ~100-300 bytes each. There's about a million of them. About 100,000 of them get changed every hour. These playlists need to be uploaded to 10 other remote servers on different continents every hour and it needs to happen quick in under 2 mins ideally. It's very important that files that are deleted on the master are also deleted on all the replicas. We currently use Linux for our infrastructure.

I was thinking about trying rsync with the -W option to copy whole files without comparing contents. I haven't tried it yet but maybe people who have more experience with rsync could tell me if it's a viable option?

What other options are worth considering?

Update: I have chosen the lsyncd option as the answer but only because it was the most popular. Other suggested alternatives are also valid in their own way.