File Transfer (scp, rsync, sftp)

Category: Intermediate Linux Commands
Type: Linux Commands
Generated on: 2025-07-10 03:11:22
For: System Administration, Development & Technical Interviews

File Transfer Cheatsheet (scp, rsync, sftp)

This cheatsheet provides a practical guide to using scp, rsync, and sftp for file transfer in Linux environments. It’s designed for both sysadmins and developers.

1. Command Overview

scp (Secure Copy): Copies files securely between hosts on a network, using SSH for authentication and encryption. Good for simple file transfers, single files, and small sets of files. Less efficient for large or frequently updated datasets.
rsync (Remote Sync): A powerful and versatile file synchronization tool. Efficiently transfers only the differences between files, making it ideal for backups, mirroring, and continuous synchronization. Supports local and remote transfers.
sftp (Secure File Transfer Protocol): Provides an interactive, shell-like interface for transferring files securely over SSH. Useful for browsing remote directories, managing permissions, and performing multiple operations.

2. Basic Syntax

scp:

scp [options] source_file target_file
scp [options] source_files... target_directory

rsync:
Terminal window
```
rsync [options] source destination
```
sftp:
Terminal window
```
sftp [user@]host
```

3. Practical Examples

scp:

Copy a single file from local to remote:

scp local_file.txt user@remote_host:/path/to/destination/
# Expected Output:  (Shows progress bar and completion)

Copy a single file from remote to local:

scp user@remote_host:/path/to/remote_file.txt /local/destination/
# Expected Output:  (Shows progress bar and completion)

Copy an entire directory from local to remote (recursively):

scp -r local_directory user@remote_host:/path/to/destination/
# Expected Output:  (Shows progress bar for each file transferred)

Copy a file between two remote servers (using your local machine as an intermediary):

scp user1@remote_host1:/path/to/file user2@remote_host2:/path/to/destination/
# Expected Output:  (Shows progress bar and completion.  Transfers through the local machine.)

rsync:

Synchronize a local directory with a remote directory:

rsync -avz /local/directory/ user@remote_host:/path/to/remote/directory/
# Expected Output:  Lists files being transferred and their sizes.

Synchronize a remote directory with a local directory:

rsync -avz user@remote_host:/path/to/remote/directory/ /local/directory/
# Expected Output:  Lists files being transferred and their sizes.

Delete files on the destination that don’t exist on the source (mirroring):

rsync -avz --delete /local/directory/ user@remote_host:/path/to/remote/directory/
# Expected Output: Lists files being transferred and any files being deleted.
# WARNING: `--delete` can be destructive.  Double-check your source and destination!

Dry run (see what would be transferred without actually transferring anything):

rsync -avzn /local/directory/ user@remote_host:/path/to/remote/directory/
# Expected Output: Lists files that *would* be transferred.  No actual transfer occurs.

sftp:
Terminal window
```
sftp user@remote_host
# Expected Output:  sftp> prompt
```
Inside the sftp prompt:
- ls: List files in the remote directory.
- cd: Change directory on the remote server.
- lcd: Change directory on the local machine.
- get remote_file local_file: Download a file.
- put local_file remote_file: Upload a file.
- mkdir remote_directory: Create a directory on the remote server.
- rm remote_file: Delete a file on the remote server.
- rmdir remote_directory: Delete a directory on the remote server.
- exit: Exit the sftp session.

4. Common Options

scp:
- -r: Recursive copy (for directories).
- -p: Preserve modification times, access times, and modes.
- -q: Quiet mode (suppress progress bars and diagnostic messages).
- -v: Verbose mode (show debugging information).
- -C: Enable compression.
- -P port: Specify a different SSH port.
- -i identity_file: Specify an identity (private key) file for authentication.
rsync:
- -a: Archive mode (recursive, preserves permissions, times, owner, group, symlinks). Equivalent to -rlptgoD.
- -v: Verbose mode.
- -z: Enable compression.
- -n: Dry run (don’t actually transfer files).
- --delete: Delete extraneous files from destination directories. USE WITH CAUTION!
- -e ssh: Specify the remote shell to use (usually ssh). Useful for specifying different SSH options. Example: -e "ssh -p 2222"
- --exclude='PATTERN': Exclude files matching a pattern.
- --include='PATTERN': Include files matching a pattern (overrides exclude).
- --progress: Show progress during transfer.
- --bwlimit=KBPS: Limit I/O bandwidth; KBytes per second. Example: --bwlimit=1000
sftp: (Options are primarily internal commands within the sftp shell.)
- -b batchfile: Execute a series of commands from a file.
- -o ssh_option: Pass options directly to the underlying SSH client (e.g., -o Port=2222).

5. Advanced Usage

scp:
- Using a different SSH port:
  Terminal window
```
scp -P 2222 local_file.txt user@remote_host:/path/to/destination/
```
- Using a specific SSH key:
  Terminal window
```
scp -i ~/.ssh/my_private_key local_file.txt user@remote_host:/path/to/destination/
```
- Chaining scp commands for complex transfers: While not ideal, sometimes necessary when dealing with multiple hops. Consider using rsync or sftp scripting for more complex scenarios.

rsync:

Incremental backups with hard links (preserves disk space):

rsync -avz --link-dest=/path/to/previous/backup /source/directory/ /path/to/new/backup/
# This creates a new backup, but hard links to files from the previous backup that haven't changed.

Excluding multiple files and directories:

rsync -avz --exclude='*.log' --exclude='tmp/*' --exclude='cache' /source/directory/ user@remote_host:/path/to/destination/

Using --include and --exclude for fine-grained control:

rsync -avz --include='*.txt' --include='important/*' --exclude='*' /source/directory/ user@remote_host:/path/to/destination/
# This will only transfer .txt files and the "important" directory.  Everything else is excluded.

Using rsync with cron for automated backups:

0 2 * * * rsync -avz --delete /source/directory/ user@backup_server:/path/to/backup/ > /var/log/backup.log 2>&1
# This runs a backup every day at 2 AM, deleting extraneous files on the backup server.

sftp:

Scripting sftp for automated tasks:

# Create a script (e.g., sftp_script.txt)
echo "get remote_file.txt local_file.txt" > sftp_script.txt
echo "put local_file2.txt remote_file2.txt" >> sftp_script.txt
echo "exit" >> sftp_script.txt

sftp -b sftp_script.txt user@remote_host

Using sftp in interactive mode for file management: Useful for quickly browsing and managing files on a remote server without needing to open a full SSH session.

6. Tips & Tricks

scp:
- Use tab completion for paths, especially on remote servers.
- For large transfers, consider using scp -C (compression) to potentially improve transfer speed, but test if it helps as it can sometimes increase overhead.
- If you’re constantly transferring to the same server, configure SSH keys for passwordless authentication.
rsync:
- Always test your rsync commands with the -n (dry run) option first to ensure they do what you expect. This is especially important when using --delete.
- Use --progress to monitor the transfer progress, especially for large files.
- If you are experiencing slow transfer speeds, check the network connection and consider using --bwlimit to avoid overwhelming the network.
- For extremely large datasets, consider using a dedicated backup solution like borgbackup or duplicity.
sftp:
- Use the ! command to execute local shell commands from within the sftp shell (e.g., !ls -l).
- Use the help command to view available sftp commands.
- Consider using a GUI-based SFTP client like FileZilla for more visual file management.

7. Troubleshooting

“Permission denied (publickey)”: This usually means your SSH key isn’t configured correctly on the remote server. Ensure your public key is in ~/.ssh/authorized_keys on the remote server. Double-check file permissions.
“Connection refused”: The remote server is not accepting SSH connections. Verify that the SSH daemon is running and that the firewall allows connections on port 22 (or the configured SSH port).
“No such file or directory”: Double-check the paths to the source and destination files or directories. Use absolute paths to avoid ambiguity.
rsync is slow: Check the network connection, enable compression (-z), and consider using --bwlimit if you are overwhelming the network. Also, ensure you’re not transferring unnecessary data (use --exclude).
rsync is deleting files unexpectedly: Double-check your command, especially the --delete option. Use the -n (dry run) option to test your command before running it. Verify you understand the difference between trailing slashes on directory names. /source/ will copy the contents of source, whereas /source will copy the source directory itself.
sftp “Couldn’t resolve hostname”: The hostname in the sftp command is incorrect or the server is unreachable. Verify the hostname and network connectivity.

8. Related Commands

ssh: Secure Shell (used for remote login and command execution). scp and sftp rely on ssh for secure communication.
tar: Tape Archive (used for creating archive files). Often used in conjunction with scp or rsync to transfer a single large archive.
nc (netcat): A general-purpose network utility. Can be used for transferring files, but is less secure than scp, rsync, or sftp.
wget / curl: Command-line tools for downloading files from web servers (HTTP/HTTPS).
find: Used for locating files based on various criteria. Useful for creating lists of files to be transferred by scp or rsync. Example: find . -name "*.log" -print0 | rsync -avz --files-from=- --from0 . user@remote_host:/path/to/logs/

This cheatsheet provides a comprehensive overview of scp, rsync, and sftp. Remember to consult the man pages (man scp, man rsync, man sftp) for the most up-to-date and detailed information. Always test your commands in a non-production environment before running them in production.