CI/CD Pipeline Tools

Category: DevOps and System Tools
Type: Linux Commands
Generated on: 2025-07-10 03:22:00
For: System Administration, Development & Technical Interviews

CI/CD Pipeline Tools Cheatsheet (Linux Commands - DevOps and System Tools)

This cheatsheet covers Linux commands frequently used in CI/CD pipelines for DevOps and System Administration. It focuses on automation, scripting, and interacting with various tools.

1. Command Overview

This cheatsheet focuses on commands essential for building, testing, and deploying software in automated CI/CD pipelines. They facilitate tasks like:

Environment Setup: Creating isolated environments (containers, VMs).
Code Management: Interacting with Git repositories.
Build Automation: Compiling and packaging code.
Testing: Running unit, integration, and end-to-end tests.
Deployment: Moving artifacts to target environments.
Configuration Management: Applying configurations to systems.
Monitoring and Logging: Gathering data and debugging issues.

2. Basic Syntax

command [options] [arguments]

3. Practical Examples

3.1. Git

Command Overview: Used for version control, cloning repositories, managing branches, and pushing/pulling changes.
Basic Syntax: git <command> [options] [arguments]

Practical Examples:

Clone a repository:

git clone https://github.com/example/my-project.git

Output:

Cloning into 'my-project'...
remote: Enumerating objects: 123, done.
remote: Counting objects: 100% (123/123), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 123 (delta 12, reused 87, pack reused 0)
Receiving objects: 100% (123/123), 22.52 KiB | 3.22 MiB/s, done.
Resolving deltas: 100% (12/12), done.

Check the status:

git status

Output:

On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

Add and commit changes:

git add .
git commit -m "Add initial commit"

Output:

[main (root-commit) 1234567] Add initial commit
 1 file changed, 1 insertion(+)
 create mode 100644 README.md

Push changes to remote:

git push origin main

Output:

Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 310 bytes | 310.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack reused 0
To https://github.com/example/my-project.git
   1234567..7890abc  main -> main

Common Options:
- -b <branch>: Create a new branch.
- -m "<message>": Commit message.
- --amend: Amend the last commit.
- --force: Force push (use with caution!).

Advanced Usage:

Tagging a release:

git tag -a v1.0 -m "Release version 1.0"
git push origin --tags

Rebasing a branch:

git checkout feature-branch
git rebase main
git push origin feature-branch --force # After resolving conflicts

Warning: Force pushing can overwrite remote history; use with caution.

Tips & Tricks:
- Use .gitignore to exclude files from version control.
- Use git stash to temporarily save uncommitted changes.
- Use git log to view commit history.
Troubleshooting:
- “fatal: refusing to merge unrelated histories”: Use git pull origin main --allow-unrelated-histories (use with caution).
- Conflicts: Resolve conflicts manually in the conflicted files.
Related Commands: gitk, git-gui, hub (GitHub CLI).

3.2. Docker

Command Overview: Used for containerization, building, running, and managing Docker images and containers.
Basic Syntax: docker <command> [options] [arguments]

Practical Examples:

Build a Docker image:
Terminal window
```
docker build -t my-image:latest .
```
Output: (Detailed build log showing each layer being built)

Run a container:

docker run -d -p 80:80 my-image:latest

Output: (Container ID)

1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p

List running containers:

docker ps

Output:

CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS                NAMES
1a2b3c4d5e6f   my-image:latest   "nginx -g 'daemon of…"   2 minutes ago   Up 2 minutes   0.0.0.0:80->80/tcp   my-container

Stop a container:
Terminal window
```
docker stop 1a2b3c4d5e6f
```
Output:
```
1a2b3c4d5e6f
```
Remove a container:
Terminal window
```
docker rm 1a2b3c4d5e6f
```
Output:
```
1a2b3c4d5e6f
```

Remove an image:

docker rmi my-image:latest

Output:

Untagged: my-image:latest
Deleted: sha256:abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890
Deleted: sha256:01234567890abcdef01234567890abcdef01234567890abcdef01234567890a

Common Options:
- -t <tag>: Tag an image.
- -d: Run in detached mode (background).
- -p <host_port>:<container_port>: Publish a container’s port to the host.
- -v <host_path>:<container_path>: Mount a volume.
- --name <container_name>: Name a container.
- -f <Dockerfile>: Specify a Dockerfile path.
Advanced Usage:
- Docker Compose: Define and manage multi-container applications.
  Terminal window
```
docker-compose up -d
```
- Docker Swarm: Orchestrate a cluster of Docker engines.
- Multi-stage builds: Optimize image size by using multiple FROM instructions in the Dockerfile.
- Using environment variables:
  Terminal window
```
docker run -e MY_VARIABLE=value my-image
```
Tips & Tricks:
- Use .dockerignore to exclude files from the Docker build context.
- Use smaller base images (e.g., alpine) to reduce image size.
- Cache Docker layers to speed up builds.
- Use Docker Hub or other registries to store and share images.
Troubleshooting:
- “Cannot connect to the Docker daemon”: Ensure Docker is running and the user has necessary permissions.
- Port conflicts: Check for existing processes using the same port.
- Image build failures: Review the Dockerfile for errors.
Related Commands: docker-compose, docker swarm, kubectl (if using Kubernetes).

3.3. Ansible

Command Overview: Used for configuration management, application deployment, and task automation.
Basic Syntax: ansible <host-pattern> -m <module> -a "<module_arguments>"

Practical Examples:

Ping all hosts:

ansible all -m ping

Output:

server1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
server2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Execute a shell command on all hosts:

ansible all -m shell -a "uptime"

Output:

server1 | SUCCESS | rc=0 >>
 10:22:00 up 1 day,  5:34,  1 user,  load average: 0.00, 0.01, 0.05
server2 | SUCCESS | rc=0 >>
 10:22:00 up 2 days,  10:12,  2 users,  load average: 0.12, 0.08, 0.06

Copy a file to all hosts:

ansible all -m copy -a "src=/path/to/local/file dest=/path/to/remote/file"

Output:

server1 | CHANGED => {
    "changed": true,
    "checksum": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "dest": "/path/to/remote/file",
    "gid": 0,
    "group": "root",
    "md5sum": "d41d8cd98f00b204e9800998ecf8427e",
    "mode": "0644",
    "owner": "root",
    "secontext": "unconfined_u:object_r:user_home_t:s0",
    "size": 0,
    "src": "/path/to/local/file",
    "state": "file",
    "uid": 0
}
server2 | CHANGED => {
    "changed": true,
    "checksum": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "dest": "/path/to/remote/file",
    "gid": 0,
    "group": "root",
    "md5sum": "d41d8cd98f00b204e9800998ecf8427e",
    "mode": "0644",
    "owner": "root",
    "secontext": "unconfined_u:object_r:user_home_t:s0",
    "size": 0,
    "src": "/path/to/local/file",
    "state": "file",
    "uid": 0
}

Run an Ansible playbook:
Terminal window
```
ansible-playbook my-playbook.yml
```
Output: (Detailed output showing each task being executed)

Common Options:
- -i <inventory>: Specify the inventory file.
- -u <user>: Specify the remote user.
- -k: Prompt for SSH password.
- --ask-become-pass: Prompt for the become (sudo) password.
- -e <extra_vars>: Pass extra variables.
Advanced Usage:
- Using roles: Organize playbooks into reusable components.
- Templating: Use Jinja2 templates to dynamically generate configuration files.
- Vault: Encrypt sensitive data in playbooks.
- Dynamic Inventory: Automatically discover hosts from cloud providers or other sources.
Tips & Tricks:
- Use ansible-lint to check playbooks for errors.
- Use ansible-doc to get documentation for modules.
- Use --check (or -C) to run playbooks in dry-run mode.
Troubleshooting:
- SSH connection errors: Verify SSH connectivity and authentication.
- Syntax errors in playbooks: Check YAML syntax and indentation.
- Module errors: Ensure the module is installed on the target host.
Related Commands: ansible-galaxy, ansible-vault, terraform.

3.4. Terraform

Command Overview: Used for infrastructure as code (IaC), provisioning and managing cloud resources.
Basic Syntax: terraform <command> [options] [arguments]
Practical Examples:
- Initialize a Terraform project:
  Terminal window
```
terraform init
```
  Output: (Downloads necessary providers and modules)
- Plan infrastructure changes:
  Terminal window
```
terraform plan
```
  Output: (Shows the changes that will be applied)
- Apply infrastructure changes:
  Terminal window
```
terraform apply
```
  Output: (Creates or modifies cloud resources)
- Destroy infrastructure:
  Terminal window
```
terraform destroy
```
  Output: (Deletes cloud resources) WARNING: Destructive operation!
Common Options:
- -auto-approve: Approve the changes automatically (use with caution!).
- -var <key=value>: Pass variables to the Terraform configuration.
- -target <resource>: Target a specific resource for apply or destroy.
Advanced Usage:
- Modules: Organize Terraform code into reusable components.
- Remote state: Store the Terraform state in a remote backend (e.g., AWS S3, Azure Storage).
- Workspaces: Manage multiple environments (e.g., development, staging, production) with the same Terraform configuration.
Tips & Tricks:
- Use terraform fmt to format Terraform code.
- Use terraform validate to validate Terraform configuration.
- Use terraform import to import existing resources into Terraform management.
Troubleshooting:
- Authentication errors: Verify cloud provider credentials.
- Resource conflicts: Check for naming conflicts or resource dependencies.
- State corruption: Restore the Terraform state from a backup or previous version.
Related Commands: aws, az, gcloud (cloud provider CLIs).

3.5. kubectl (Kubernetes CLI)

Command Overview: Interacts with a Kubernetes cluster, used for deploying, managing, and monitoring applications.
Basic Syntax: kubectl <command> [TYPE] [NAME] [flags]

Practical Examples:

Get all pods:

kubectl get pods

Output:

NAME                               READY   STATUS    RESTARTS   AGE
my-app-deployment-7b889d44d-abcde   1/1     Running   0          20m
my-app-deployment-7b889d44d-fghij   1/1     Running   0          20m

Describe a pod:
Terminal window
```
kubectl describe pod my-app-deployment-7b889d44d-abcde
```
Output: (Detailed information about the pod, including events, status, and configuration)

Create a deployment:

kubectl apply -f my-deployment.yaml

Output:

deployment.apps/my-app-deployment created

Scale a deployment:

kubectl scale deployment my-app-deployment --replicas=3

Output:

deployment.apps/my-app-deployment scaled

Delete a deployment:

kubectl delete deployment my-app-deployment

Output:

deployment.apps/my-app-deployment deleted

Common Options:
- -n <namespace>: Specify the namespace.
- -o <output_format>: Specify the output format (e.g., yaml, json).
- -f <file>: Specify a configuration file.
- --dry-run: Simulate the command without making changes.
Advanced Usage:
- Rolling updates: Update deployments with zero downtime.
- Services: Expose applications to the outside world.
- Ingress: Manage external access to services.
- Helm: Package and manage Kubernetes applications.
Tips & Tricks:
- Use kubectl explain to get documentation for Kubernetes resources.
- Use aliases to shorten frequently used commands (e.g., alias k=kubectl).
- Use kubectl autoscale to automatically scale deployments based on resource usage.
Troubleshooting:
- Connection refused: Verify Kubernetes cluster is running and accessible.
- Permission denied: Check RBAC (Role-Based Access Control) configuration.
- Pod errors: Inspect pod logs and events for errors.
Related Commands: helm, minikube, kind.

3.6 rsync

Command Overview: Used for fast, versatile, incremental file transfer and synchronization. Often used to deploy code and configuration.
Basic Syntax: rsync [options] source destination
Practical Examples:
- Copy a file to a remote server:
  Terminal window
```
rsync -avz /path/to/local/file user@remote_host:/path/to/remote/directory
```
  Output: (Shows the files being transferred)
- Synchronize a directory to a remote server:
  Terminal window
```
rsync -avz /path/to/local/directory/ user@remote_host:/path/to/remote/directory/
```
  Output: (Shows the files being transferred and updated)
- Synchronize a directory locally:
  Terminal window
```
rsync -avz /path/to/source/directory/ /path/to/destination/directory/
```
  Output: (Shows the files being transferred)
Common Options:
- -a: Archive mode (preserves permissions, timestamps, etc.).
- -v: Verbose mode (shows more details).
- -z: Compress data during transfer.
- -r: Recursive (for directories).
- -u: Update only (skip files that are newer on the destination).
- -delete: Delete files on the destination that don’t exist on the source (use with caution!).
- --exclude=PATTERN: Exclude files matching the pattern.
- --include=PATTERN: Include files matching the pattern (overrides exclude).
Advanced Usage:
- Using SSH keys for authentication: Avoid password prompts.
- Scheduling rsync with cron: Automate backups and synchronization.
- Using --link-dest for incremental backups: Create hard links to unchanged files.
Tips & Tricks:
- Use trailing slashes on directory paths to control whether the directory itself is copied.
- Use --dry-run to test rsync commands before running them.
- Use --progress to show a progress bar during the transfer.
Troubleshooting:
- Permission errors: Check file permissions on the source and destination.
- Connection refused: Verify SSH connectivity.
- Files not being updated: Check timestamps and update flags.
Related Commands: scp, cp, tar.

3.7 sed

Command Overview: A powerful stream editor used for text manipulation, search, and replace. Useful for configuration file updates and data transformations.
Basic Syntax: sed [options] 'command' file
Practical Examples:
- Replace a string in a file:
  Terminal window
```
sed 's/old_string/new_string/g' input.txt > output.txt
```
  Output: (The contents of output.txt will have the string replaced)
- Replace a string in a file in-place:
  Terminal window
```
sed -i 's/old_string/new_string/g' input.txt
```
  Warning: This modifies the original file!
- Delete lines matching a pattern:
  Terminal window
```
sed '/pattern_to_delete/d' input.txt > output.txt
```
  Output: (The contents of output.txt will have the matching lines removed)
- Insert a line after a matching pattern:
  Terminal window
```
sed '/pattern_to_match/a New line to insert' input.txt > output.txt
```
  Output: (The contents of output.txt will have the new line added)
- Print only lines matching a pattern:
  Terminal window
```
sed -n '/pattern_to_match/p' input.txt
```
  Output: (Only lines containing the pattern are printed to the terminal)
Common Options:
- -i: Edit the file in-place.
- -n: Suppress default output (used with p command).
- -e 'command': Execute multiple commands.
- -f script_file: Execute commands from a script file.
Advanced Usage:
- Using regular expressions: Perform more complex pattern matching.
- Using backreferences: Refer to captured groups in the replacement string.
- Using multiple commands: Combine multiple sed commands to perform complex transformations.
- Address ranges: Apply commands to specific line ranges.
Tips & Tricks:
- Always test sed commands on a copy of the file before using -i.
- Use single quotes to protect special characters in the command.
- Use \ to escape special characters in the pattern.
Troubleshooting:
- Unexpected results: Double-check the regular expression and replacement string.
- “Invalid command code”: Check for syntax errors in the sed command.
- File not being modified: Ensure the user has write permissions to the file.
Related Commands: awk, grep, tr.

3.8 awk

Command Overview: A powerful text processing tool used for data extraction, transformation, and reporting.
Basic Syntax: awk [options] 'pattern { action }' file
Practical Examples:
- Print the first column of a file:
  Terminal window
```
awk '{print $1}' input.txt
```
  Output: (Prints the first word of each line)
- Print lines where the second column is greater than 10:
  Terminal window
```
awk '$2 > 10 {print}' input.txt
```
  Output: (Prints lines where the second field’s value exceeds 10)
- Print the sum of the third column:
  Terminal window
```
awk '{sum += $3} END {print sum}' input.txt
```
  Output: (Prints the total sum of the third column’s values)
- Print lines containing a specific pattern:
  Terminal window
```
awk '/pattern_to_match/' input.txt
```
  Output: (Prints lines matching the provided pattern)
- Use a custom field separator:
  Terminal window
```
awk -F',' '{print $1, $2}' input.csv
```
  Output: (Prints the first and second columns separated by commas)
Common Options:
- -F <separator>: Specify the field separator.
- -v <variable=value>: Assign a value to a variable.
- -f <script_file>: Execute commands from a script file.
Advanced Usage:
- Using regular expressions: Perform more complex pattern matching.
- Using built-in variables: Access variables like NR (record number), NF (number of fields), and FS (field separator).
- Using functions: Define custom functions to perform complex calculations and transformations.
- Conditional statements: Use if, else, and else if to control the flow of execution.
Tips & Tricks:
- Use single quotes to protect special characters in the command.
- Use BEGIN and END blocks to perform setup and cleanup tasks.
- Use associative arrays to store and retrieve data.
Troubleshooting:
- Incorrect output: Double-check the field separator and the pattern.
- Syntax errors: Check for missing braces, semicolons, or parentheses.
- Variables not being initialized: Ensure variables are initialized before being used.
Related Commands: sed, grep, cut.

3.9 curl

Command Overview: A command-line tool for transferring data with URLs. Used to interact with APIs, download files, and test web services.
Basic Syntax: curl [options] URL
Practical Examples:
- Download a file:
  Terminal window
```
curl -O https://example.com/file.txt
```
  Output: (Shows download progress)
- Send a GET request:
  Terminal window
```
curl https://api.example.com/data
```
  Output: (Prints the response body to the terminal)
- Send a POST request with data:
  Terminal window
```
curl -X POST -d "key1=value1&key2=value2" https://api.example.com/data
```
  Output: (Prints the response body to the terminal)
- Send a JSON POST request:
  Terminal window
```
curl -X POST -H "Content-Type: application/json" -d '{"key1": "value1", "key2": "value2"}' https://api.example.com/data
```
  Output: (Prints the response body to the terminal)
- Set a custom header:
  Terminal window
```
curl -H "Authorization: Bearer <token>" https://api.example.com/data
```
  Output: (Prints the response body to the terminal)
Common Options:
- -O: Save the downloaded file with the same name as the remote file.
- -o <file>: Save the downloaded file with a custom name.
- -X <method>: Specify the HTTP method (e.g., GET, POST, PUT, DELETE).
- -H <header>: Set a custom header.
- -d <data>: Send data with the request.
- -u <user:password>: Authenticate with a username and password.
- -k: Allow insecure connections (use with caution!).
- -v: Verbose mode (shows more details).
Advanced Usage:
- Using cookies: Send and receive cookies.
- Following redirects: Automatically follow HTTP redirects.
- Using proxies: Connect to the internet through a proxy server.
- Uploading files: Send files to a server.
Tips & Tricks:
- Use jq to parse JSON responses.
- Use --data-urlencode to URL-encode data.
- Use --limit-rate to limit the download speed.
Troubleshooting:
- Connection refused: Verify the server is running and accessible.
- Authentication errors: Check credentials and authorization headers.
- Incorrect data: Double-check the data being sent in the request.
Related Commands: wget, httpie, netcat.

3.10 jq

Command Overview: A lightweight and flexible command-line JSON processor. Used to filter, transform, and manipulate JSON data.
Basic Syntax: jq [options] 'filter' [file]
Practical Examples:
- Pretty-print a JSON file:
  Terminal window
```
jq '.' input.json
```
  Output: (Prints the JSON data in a human-readable format)
- Extract a specific field:
  Terminal window
```
jq '.name' input.json
```
  Output: (Prints the value of the name field)
- Extract all values from an array:
  Terminal window
```
jq '.[].name' input.json
```
  Output: (Prints all values from the name field in the array)
- Filter an array based on a condition:
  Terminal window
```
jq '.[] | select(.age > 30)' input.json
```
  Output: (Prints the objects where the age property is greater than 30)
- Transform data:
  Terminal window
```
jq '[.[] | {name: .name, age: .age + 1}]' input.json
```
  Output: (Prints a new array with each object having its age incremented by one)
Common Options:
- -r: Output raw strings (without quotes).
- -c: Compact output (one JSON object per line).
- -s: Slurp all input into a single array.
- -f <script_file>: Execute commands from a script file.
Advanced Usage:
- Using functions: Define custom functions to perform complex transformations.
- Using regular expressions: Perform more complex pattern matching.
- Piping data: Combine jq with other commands to process data in a pipeline.
Tips & Tricks:
- Use online jq playgrounds to experiment with filters.
- Use jq --help to see all available options and functions.
- Use descriptive variable names in filters.
Troubleshooting:
- Syntax errors: Check for missing dots, parentheses, or brackets.
- Incorrect output: Double-check the filter and the input data.
- No output: Ensure the filter matches the input data.
Related Commands: yq (YAML processor), sed, awk.

4. Common Options

Many commands share common options:

-h or --help: Display help information.
-v or --verbose: Increase verbosity.
-q or --quiet: Suppress output.
--version: Display version information.

5. Advanced Usage

Chaining commands: Use pipes (|) to pass the output of one command to another.
Terminal window
```
cat file.txt | grep "error" | awk '{print $1}'
```
Using xargs: Build and execute command lines from standard input.
Terminal window
```
ls *.txt | xargs rm  # Remove all .txt files
```
Writing shell scripts: Combine multiple commands into a script for automation.

6. Tips & Tricks

Command history: Use the up and down arrow keys to navigate command history.
Tab completion: Use the Tab key to complete commands, file names, and options.
Aliases: Create aliases for frequently used commands. Edit ~/.bashrc or ~/.zshrc to add aliases.
Environment variables: Use environment variables to store configuration settings.

7. Troubleshooting

“Command not found”: Ensure the command is installed and in your PATH.
“Permission denied”: Check file permissions and user privileges.
Unexpected output: Double-check command syntax and options.
Refer to man pages: Use man <command> to access the command’s manual page.

8. Related Commands

System Administration: systemctl, journalctl, top, htop, ps, df, du.
Networking: ping, traceroute, netstat, ss, ifconfig, ip.
File Management: find, grep, tar, gzip, bzip2, zip, unzip.
Security: ssh, scp, chmod, chown, sudo.

This cheatsheet provides a starting point for using Linux commands in CI/CD pipelines. Experiment with different commands and options to find the best solutions for your specific needs. Remember to always test commands in a safe environment before running them in production.