Command line (Unix)
Commands tips and examples:

Configuration
Conserver la configuration d'iptables lors des dรฉmarrages
# Sauvegarde de la configuration d'iptables dans un fichier
iptables-save > /etc/iptables.conf
# Crรฉation d'un fichier sh dans le dossier ...
echo "#!/bin/sh" > /etc/network/if-up.d/iptables
# ... qui restaurera les rรจgles sauvegardรฉ dans le fichier /etc/iptables.conf
echo "iptables-restore < /etc/iptables.conf" >> /etc/network/if-up.d/iptables
# Rendre executable le script bash
chmod +x /etc/network/if-up.d/iptables
Il est nรฉcรฉssaire de relancer ce script ร chaque modifications d'iptables
Gestion des services
(must be executable: chmod +x /etc/init.d/blah
)
#! /bin/sh
# /etc/init.d/blah
## BEGIN INIT INFO
# Provides: blah
# Required-Start: $local_fs $syslog $remote_fs dbus
# Required-Stop: $local_fs $syslog $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start blah
## END INIT INFO
# Some things that run always
touch /var/lock/blah
# Carry out specific functions when asked to by the system
case "$1" in
start)
echo "Starting script blah"
echo "Could do more here"
;;
stop)
echo "Stopping script blah"
echo "Could do more here"
;;
*)
echo "Usage: /etc/init.d/blah {start|stop}"
exit 1
;;
esac
exit 0
Define when service have to be start or stop:
update-rc.d blah defaults
To remove from the startup sequence
update-rc.d -f blah remove
https://help.ubuntu.com/community/UbuntuBootupHowto
http://www.debian-administration.org/articles/28
man update-rc.d
http://en.wikipedia.org/wiki/Runlevel
http://www.debuntu.org/how-to-managing-services-with-update-rc-d/
List all runlevel for the service
apache2
:ls -l /etc/rc?.d/*apache2
User prompt
#!/bin/bash
echo "Yes,No?"
read answer
echo "You're answer: $answer"
http://www.tldp.org/LDP/Bash-Beginners-Guide/html/Bash-Beginners-Guide.html#sect_08_02_01
User management
# Set deadline for user password
chage -E 2005-12-31 user1
# Create a new group
groupadd [group]
# Delete a group
groupdel [group]
# Rename a group from moon to sun
groupmod -n moon sun
# Check correct syntax and file format of `/etc/group` and groups existence
grpck
# Log into a new group to change default group of newly created files
newgrp - [group]
# Change password
passwd
# Change a user password (only by root)
passwd user1
# Check correct syntax and file format of `/etc/passwd` and users existence
pwck
# Create a new user `user1` belongs `admin` group
useradd -c "User Linux" -g admin -d /home/user1 -s /bin/bash user1
# Create a new user
useradd user1
# Delete a user (`-r` eliminates home directory)
userdel -r user1
# Change user attributes as description, group and other
usermod -c "User FTP" -g system -d /ftp/user1 -s /bin/nologin user1
# Show user rights on files
ls -l
# or, but show folders "." and ".."
ls -all
# User informations
id USERNAME
# Show last logs of users
lastlog
# Show last logged users
last
# Show current user
who -a
# Command history
~/.bash_history
~/.history
# for root user
/root/.bash_history
# Be sure no user can hide his command history (a = append only, i = immutable)
chattr +a /home/user/.bash_history
chattr +i /home/user/.profile
# See also
# http://www.akyl.net/securing-bashhistory-file-make-sure-your-linux-system-users-won%E2%80%99t-hide-or-delete-their-bashhistory
# http://zero202.free.fr/bash/html/ar01s01.html#id336074
# https://en.wikipedia.org/wiki/Chattr
chattr
chwon
chmod
Network
Note: ifconfig, route, mii-tool, nslookup commands are obsolete
# Show status of ethernet interface eth0
ethtool eth0
#Manually set ethernet interface speed
ethtool --change eth0 autoneg off speed 100 duplex full
# Show link status of `eth0`
mii-tool eth0
# Show configuration of an ethernet network card
ifconfig eth0
# Configure IP Address
ifconfig eth0 192.168.1.1 netmask 255.255.255.0
# Configure `eth0` in promiscuous mode to gather packets (sniffing)
ifconfig eth0 promisc
# Show status of wireless interface eth1
iwconfig eth1
# Manually set wireless interface speed
iwconfig eth1 rate 1Mb/s fixed
# List wireless networks in range
iwlist scan
# Disable an interface `eth0`
ifdown eth0
# Activate an interface `eth0`
ifup eth0
# List network interfaces and their status
ip link show
# Rename interface eth0 to wan
ip link set dev eth0 name wan
# Bring interface eth0 up (or down)
ip link set dev eth0 up
# List addresses for interfaces
ip addr show
# Add (or del) ip and mask (255.255.255.0)
ip addr add 1.2.3.4/24 brd + dev eth0
# List routing table
ip route show
# Set default gateway to 1.2.3.254
ip route add default via 1.2.3.254
# Show routing table
route -n
# Configure default gateway
route add -net 0/0 gw IP_Gateway
# Configure static route to reach network `192.168.0.0/16`
route add -net 192.168.0.0 netmask 255.255.0.0 gw 192.168.1.1
# Remove static route
route del 0/0 gw IP_gateway
# Lookup DNS ip address for name or vice versa
host example.com
# Machine name
hostname
# Lookup local ip address (equivalent to host `hostname`)
hostname -i
# Active interface `eth0` in dhcp mode
dhclient eth0
# List network services on a system and their PID
netstat -tupl
# List active network connections to/from system and their PID
netstat -tup
# Show routing table alike "route -n"
netstat -rn
# Show all HTTP traffic
tcpdump tcp port 80
# DNS record
dig example.com
# Lookup whois info for hostname or ip address
whois example.com
# Activate ip routing temporarily
echo "1" > /proc/sys/net/ipv4/ip_forward
# Reverse DNS
ping -a 8.8.8.8
nslookup -type=ptr 8.8.8.8
nslookup www.example.com
# Test if remote port is open
nc -vz example.com 80
nmap -p 80 example.com
telnet example.com 80
timeout 1 bash -c "</dev/tcp/example.com/80" && echo Port open || echo Port closed
/etc/hostname
/etc/hosts
Windows networking
Note: samba is the package that provides all this windows specific networking support
# Find windows machines. See also findsmb
smbtree
# Find the windows (netbios) name associated with ip address
nmblookup -A 1.2.3.4
# List shares on windows machine or samba server
smbclient -L windows_box
# Show remote shares of a windows host
smbclient -L ip_addr/hostname
# Mount a windows share
mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share
# Send popup to windows machine (off by default in XP sp2)
echo 'message' | smbclient -M windows_box
# Mount a windows network share
mount -t smbfs -o username=user,password=pass //WinClient/share /mnt/share
# Netbios name resolution
nbtscan ip_addr
# Netbios name resolution
nmblookup -A ip_addr
# Like wget can download files from a host windows via smb
smbget -Rr smb://ip_addr/share
HTTP
# Get headers
curl -s -D - https://example.com -o /dev/null
# Raw body
curl -s --ignore-content-length --raw https://example.com
# Get HTTP response
echo -en "GET /feed/ HTTP/1.1\r\nHost:example.com\r\nUser-Agent:Firefox\r\n\r\n" | nc infoheap.com 80
# Get the header Content-Type
echo -en "GET /feed/ HTTP/1.1\r\nHost:example.com\r\nUser-Agent:Firefox\r\n\r\n" | nc localhost 80 | head -n 30 | grep -a Content-Type
# Search if test is in first 1000 bytes (include header)
echo -en "GET /feed/ HTTP/1.1\r\nHost:example.com\r\nUser-Agent:Firefox\r\n\r\n" | nc localhost 80 | head -c1K | grep -a test
# Get body length
echo -e "GET / HTTP/1.1\r\nHost: example.com\r\nAccept-Encoding: gzip, deflate\r\n\r\n" | nc localhost 80 | sed ':a;N;$!ba;s/^.*\r\n\r\n//g' | wc -c
# Get body uncompressed length (should be gzipped)
echo -e "GET / HTTP/1.1\r\nHost: example.com\r\nAccept-Encoding: gzip, deflate\r\n\r\n" | nc 93.184.216.34 80 | sed ':a;N;$!ba;s/^.*\r\n\r\n//g' | gzip -d | wc -c
# Simple HTTP server
echo -e "GET / HTTP/1.1\r\nHost:example.com\r\nUser-Agent:Firefox\r\n\r\n" | nc example.com 80 > response.http
while true; do nc -l 8000 < response.http; done
# Create `.htpasswd` file
htpasswd -c /path/.htpasswd USERNAME
How to display request headers with command line curl - Stack Overflow
wget
Multi purpose download tool
# Store local browsable version of a page to the current dir
(cd dir/ && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html)
# Continue downloading a partially downloaded file (with the ability to stop the download and resume later)
wget -c http://www.example.com/large.file
# Download a set of files to the current directory
wget -r -nd -np -l1 -A '*.jpg' http://www.example.com/dir/
# FTP supports globbing directly
wget ftp://remote/file[1-9].iso/
# Process output directly
wget -q -O- http://www.pixelbeat.org/timeline.html | grep 'a href' | head
# Download url at 1AM to current dir
echo 'wget url' | at 01:00
# Do a low priority download (limit to 20KB/s in this case)
wget --limit-rate=20k url
# Check links in a file
wget -nv --spider --force-html -i bookmarks.html
# Efficiently update a local copy of a site (handy from cron)
wget -m http://www.example.com/
wget -m -U "" http://example.com/path/dir
SSH Tunnel
# Redirect local port to a remote machine
ssh -NfL 8080:127.0.0.1:0808 user@host
Backup external server
Use rsync
if possible, or lftp
, curl
, wget
, scp
, sftp
. See also unison
, bacula
or amanda
Copy files over SSH
scp /path/localfile user@host:/path/remotefile
scp /path/localfile user@host:/path/remotedir/
scp /path/localdir user@host:/path/remotedir
scp -i /home/username/.ssh/id_rsa -r -p username@www.mydomain.tld:/ /dst/dir
Copy files over SSH shell
# Copy content of a directory on remote directory via ssh
( cd /tmp/local/ && tar c . ) | ssh -C user@ip_addr 'cd /home/share/ && tar x -p'
# Copy a local directory on remote directory via ssh
( tar c /home ) | ssh -C user@ip_addr 'cd /home/backup-home && tar x -p'
Copy files with sftp
sftp
sftp -i /path/to/private/keyfile remote.host.tld
Copy files with lftp
lftp
/dst/dir.lftp
:
set ftp:list-options -a
#set cmd:fail-exit true
# To use keyfile
# http://www.adminschoice.com/how-to-configure-ssh-without-password
# ssh-keygen -t rsa -f keyfile -N ""
#set sftp:connect-program "ssh -a -x -i <keyfile>"
open -u user,pass sftp://sftp.dc0.gpaas.net
mirror --verbose --delete --parallel=4 --only-newer --exclude ^snapshots/ --exclude ^vhosts/ --exclude ^lamp0/\.config --exclude ^lamp0/var/admin/gandi.cnf$ --exclude ^lamp0/var/admin/logrotate.conf$ --exclude ^lamp0/var/cron/admin/logrotate$ --exclude ^lamp0/var/cron/admin/phpsess$ --exclude ^lamp0/db/ --exclude ^lamp0/var/log/.+ --exclude ^lamp0/var/php/www/.+ "${SRC_DIR}" "${DST_DIR}"
exit
# For paths in variables used in sed, see https://stackoverflow.com/a/29613573/470117
SRC_DIR=/src/dir; DST_DIR=/dst/dir; sed -e "s#\${SRC_DIR}#$SRC_DIR#" -e "s#\${DST_DIR}#$DST_DIR#" /dst/dir.lftp | lftp -f /dev/stdin
lftp -u username,password ftphost << EOF
mirror -Re --use-cache /home/ /backup/
mirror -Re --use-cache /etc/ /backup/
mirror -Re --use-cache /var/www/ /backup/
quit 0
EOF
--exclude
/ -x
use extended regular expression (see egrep
) \.nfo
. Folders path don't start with slash, but have a trailing slash: etc/bin/
Add --reverse
to restore backup
For lftp โ> Fatal error: Certificate verification: Not trusted
:
mkdir -p /root/.lftp
echo "set ssl:verify-certificate no" >> /root/.lftp/rc
https://linux.die.net/man/1/lftp
http://mewbies.com/lftp_basic_usage_tutorial.htm
Copy files from FTP with wget
wget
Use a .wgetrc
file to store configuration
/dst/dir.wget
:
# Credentials
ftp_user=user
ftp_password=password
# Other commands
exclude_directories=/snapshots,/vhosts,/lamp0/.config,/lamp0/db,/lamp0/var/log,/lamp0/var/php/www
#reject=.listing
#cut_dirs=0
quiet=on
no_parent=on
mirror=on
add_hostdir=off
#retr_symlinks=on
cd /dst/dir
WGETRC=/dst/dir.wget wget ftp://www.mydomain.tld/
# Clean up obselete files
find . -name ".listing" -type f -exec bash -c 'cd $(dirname "{}"); find . -maxdepth 0 ! -name ".listing" -printf "%P\n" | fgrep -vf ".listing" | while read file; do rm $(basename "$file"); done;' \;
# find .-name ".listing" -type f -exec bash -c 'cd $(dirname "{}"); find . -maxdepth 0 ! -name ".listing" -printf "%P\n" | fgrep -vf ".listing" | xargs -r rm' \;
Note: if you set password in URL don't forget to encode special chars: mypass#gh647
to mypass%23gh647
Copy files with rsync
rsync
Network efficient file copier
Use the --dry-run
option for testing
# Synchronization between directories
rsync -rogpav --delete /home /tmp
# Rsync via SSH tunnel
rsync -rogpav -e ssh --delete /home ip_address:/tmp
# Synchronize a local directory with a remote directory via SSH and compression
rsync -az -e ssh --delete ip_addr:/home/public /home/local
# Synchronize a remote directory with a local directory via SSH and compression
rsync -az -e ssh --delete /home/local ip_addr:/home/public
# Copy as archive (oneway, for backup)
rsync -a username@www.mydomain.tld:/ /dst/dir
rsync -r -t -v /path/srcdir /path/destdir
rsync -r -t -v /path/srcdir remote:~/My\ documents
# Escape spaces, and keep special char interpreted at the destination. See "How to rsync over ssh when directory names have spaces - Unix & Linux Stack Exchange" https://unix.stackexchange.com/questions/104618/how-to-rsync-over-ssh-when-directory-names-have-spaces
DEST="~/My\ documents"; rsync -r -t -v /path/srcdir "remote:${DEST// /\\ }"
# Only get diffs. Do multiple times for troublesome downloads
rsync -P rsync://rsync.server.com/path/to/file file
# Locally copy with rate limit. It's like nice for I/O
rsync --bwlimit=1000 fromfile tofile
# Mirror web site (using compression and encryption)
rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html'
# Synchronize current directory with remote one
rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/
date=$(date +%Y%m%d-%H%M)
#[for loop over users]
older=( $backups/$user/*(N/om) )
rsync --archive --recursive \
--fuzzy --partial --partial-dir=$backups/$user/.rsync-partial \
--log-file=$tempfile --link-dest=${^older[1,20]} \
--files-from=$configdir/I-$user \
--exclude-from=$configdir/X-$user \
$user@$from:/ $backups/$user/$date/
# Copy folder to folder (if mounted)
rsync -ah --progress --exclude='$RECYCLE.BIN' --exclude='$Recycle.Bin' --exclude='.AppleDB' --exclude='.AppleDesktop' --exclude='.AppleDouble' --exclude='.com.apple.timemachine.supported' --exclude='.dbfseventsd' --exclude='.DocumentRevisions-V100*' --exclude='.DS_Store' --exclude='.fseventsd' --exclude='.PKInstallSandboxManager' --exclude='.Spotlight*' --exclude='.SymAV*' --exclude='.symSchedScanLockxz' --exclude='.TemporaryItems' --exclude='.Trash*' --exclude='.vol' --exclude='.VolumeIcon.icns' --exclude='Desktop DB' --exclude='Desktop DF' --exclude='hiberfil.sys' --exclude='lost+found' --exclude='Network Trash Folder' --exclude='pagefile.sys' --exclude='Recycled' --exclude='RECYCLER' --exclude='System Volume Information' --exclude='Temporary Items' --exclude='Thumbs.db' /src/dir/ /dst/dir/
# With exlude list `--exclude-from=exclude-list.txt`:
cat > exclude-list.txt << 'EOF'
# https://alexkaloostian.com/2015/01/22/what-are-all-these-hidden-items-on-my-mac-part-1/
# http://superuser.com/questions/180582/what-are-desktop-db-or-desktop-df-files-on-external-hd
# http://forum.mac4ever.com/pourquoi-t73534.html#p997828
# http://netatalk.sourceforge.net/wiki/index.php/Special_Files_and_Folders
# Apple OS
.AppleDouble
.com.apple.timemachine.donotpresent
.com.apple.timemachine.supported
.Spotlight-V100
.DocumentRevisions-V100
.DS_Store
.Trash
.Trashes
.VolumeIcon.icns
._.Trashes
.fseventsd
.dbfseventsd
.metadata_never_index
.LSOverride
.MobileBackups
.TemporaryItems
.file
.hotfiles.btree
.quota.ops.user
.quota.user
.quota.ops.group
.quota.group
.vol
.PKInstallSandboxManager
.PKInstallSandboxManager-SystemSoftware
# Max OS6-9
Desktop DB
Desktop DF
# AFP
.AppleDB
.AppleDesktop
Temporary Items
Network Trash Folder
.apdisk
# Sherlock files Mac OS 8.5 to OSX 10.3
TheFindByContentFolder
TheVolumeSettingsFolder
.FBCIndex
.FBCSemaphoreFile
.FBCLockFolder
# http://apple.stackexchange.com/questions/14980/why-are-dot-underscore-files-created-and-how-can-i-avoid-them
# https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats
# See also dot_clean command
#._*
# Linux
lost+found
# Windows
$RECYCLE.BIN
$Recycle.Bin
Thumbs.db
ehthumbs.db
ehthumbs_vista.db
pagefile.sys
hiberfil.sys
desktop.ini
Recycled
RECYCLER
System Volume Information
# Other
.SymAV*
.symSchedScanLockxz
# Package manager cache
node_modules
EOF
Resources for exclude metadata and system files:
https://github.com/github/gitignore
https://github.com/joeblau/gitignore.io/tree/master/data
https://gist.github.com/keithmorris/c652b2a4d88788e5416d
https://github.com/dotphiles/dotphiles/blob/master/git/gitignore
Math
# Quick math (Calculate ฯ). See also bc
echo '(1 + sqrt(5))/2' | bc -l
# Calculate ฯ the unix way
seq -f '4/%g' 1 2 99999 | paste -sd-+ | bc -l
# More complex (int) e.g. This shows max FastE packet rate
echo 'pad=20; min=64; (100*10^6)/((pad+min)*8)' | bc
# Python handles scientific notation
echo 'pad=20; min=64; print (100E6)/((pad+min)*8)' | python
# Plot FastE packet rate vs packet size
echo 'pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)' | gnuplot -persist
# Base conversion (decimal to hexadecimal)
echo 'obase=16; ibase=10; 64206' | bc
# Base conversion (hex to dec) ((shell arithmetic expansion))
echo $((0x2dec))
# Unit conversion (metric to imperial)
units -t '100m/9.58s' 'miles/hour'
# Unit conversion (SI to IEC prefixes)
units -t '500GB' 'GiB'
# Definition lookup
units -t '1 googol'
# Add a column of numbers. See also add and funcpy
seq 100 | (tr '\n' +; echo 0) | bc
Date and clock
# Display a calendar
cal -3
# Display a calendar for a particular month year
cal 9 1752
# System date
date
# Set system date and time `MonthDayhoursMinutesYear.Seconds`
date 041217002020.00
# save date changes on BIOS
clock -w
# What date is it this friday. See also day
date -d fri
# Exit a script unless it's the last day of the month
[ $(date -d '12:00 +1 day' +%d) = '01' ] || exit
# What day does xmas fall on, this year
date --date='25 Dec' +%A
# Convert seconds since the epoch (1970-01-01 UTC) to date
date --date='@2147483647'
# What time is it on west coast of US (use tzselect to find TZ)
TZ='America/Los_Angeles' date
# What's the local time for 9AM next Friday on west coast US
date --date='TZ="America/Los_Angeles" 09:00 next Fri'
Locale
# Print number with thousands grouping appropriate to locale
printf "%'d\n" 1234
# Use locale thousands grouping in ls. See also l
BLOCK_SIZE=\'1 ls -l
# Extract info from locale database
echo "I live in $(locale territory)"
# Lookup locale info for specific country. See also ccodes
LANG=en_IE.utf8 locale int_prefix
# List fields available in locale database
locale -kc $(locale | sed -n 's/\(LC_.\{4,\}\)=.*/\1/p') | less
Packages
RPM Packages ( Fedora, Red Hat and like):
# install a rpm package
rpm -ivh [package.rpm]
# install a rpm package ignoring dependencies requests
rpm -ivh --nodeeps [package.rpm]
# upgrade a rpm package without changing configuration files
rpm -U [package.rpm]
# upgrade a rpm package only if it is already installed
rpm -F [package.rpm]
# remove a rpm package
rpm -e [package]
# show all rpm packages installed on the system
rpm -qa
# show all rpm packages with the name "httpd"
rpm -qa | grep httpd
# obtain information on a specific package installed
rpm -qi [package]
# show rpm packages of a group software
rpm -qg "System Environment/Daemons"
# show list of files provided by a rpm package installed
rpm -ql [package]
# show list of configuration files provided by a rpm package installed
rpm -qc [package]
# show list of dependencies required for a rpm packet
rpm -q [package] --whatrequires
# show capability provided by a rpm package
rpm -q [package] --whatprovides
# show scripts started during installation / removal
rpm -q [package] --scripts
# show history of revisions of a rpm package
rpm -q [package] --changelog
# verify which rpm package belongs to a given file
rpm -qf /etc/httpd/conf/httpd.conf
# show list of files provided by a rpm package not yet installed
rpm -qp [package.rpm] -l
# import public-key digital signature
rpm --import /media/cdrom/RPM-GPG-KEY
# verify the integrity of a rpm package
rpm --checksig [package.rpm]
# verify integrity of all rpm packages installed
rpm -qa gpg-pubkey
# check file size, permissions, type, owner, group, MD5 checksum and last modification
rpm -V [package]
# check all rpm packages installed on the system - use with caution
rpm -Va
# verify a rpm package not yet installed
rpm -Vp [package.rpm]
- `rpm -ivh /usr/src/redhat/RPMS/`arch`/[package.rpm]`: install a package built from a rpm source
# extract executable file from a rpm package
rpm2cpio [package.rpm] | cpio --extract --make-directories *bin*
# build a rpm package from a rpm source
rpmbuild --rebuild [package.src.rpm]
# List all packages by installed size (Bytes) on rpm distros
rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' | sort -k1,1n
YUM packages tool (Fedora, RedHat and alike):
# download and install a rpm package
yum -y install [package]
# That will install an RPM, and try to resolve all the dependencies for you using your repositories.
yum localinstall [package.rpm]
# update all rpm packages installed on the system
yum -y update
# upgrade a rpm package
yum update [package]
# remove a rpm package
yum remove [package]
# list all packages installed on the system
yum list
# find a package on rpm repository
yum search [package]
# clean up rpm cache erasing downloaded packages
yum clean [package]
# remove all files headers that the system uses to resolve dependency
yum clean headers
# remove from the cache packages and headers files
yum clean all
DEB packages (Debian, Ubuntu and like):
# install / upgrade a deb package
dpkg -i [package.deb]
# remove a deb package from the system
dpkg -r [package]
# show all deb packages installed on the system
dpkg -l
# show all deb packages with the name "httpd"
dpkg -l | grep httpd
# obtain information on a specific package installed on system
dpkg -s [package]
# show list of files provided by a package installed on system
dpkg -L [package]
# show list of files provided by a package not yet installed
dpkg --contents [package.deb]
# verify which package belongs to a given file
dpkg -S /bin/ping
# List all packages by installed size (KBytes) on deb distros
dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' | sort -k1,1n
APT packages tool (Debian, Ubuntu and alike):
# returns list of packages which corresponds string "searched-packages"
apt-cache search [package]
# install / upgrade a deb package from cdrom
apt-cdrom install [package]
# install / upgrade a deb package
apt-get install [package]
# update the package list
apt-get update
# upgrade all of the installed packages
apt-get upgrade
# remove a deb package from system
apt-get remove [package]
# verify correct resolution of dependencies
apt-get check
# clean up cache from packages downloaded
apt-get clean
Pacman packages tool (Arch, Frugalware and alike):
# Install package `name` with dependencies
pacman -S name
# Delete package `name` and all files of it
pacman -R name
Monitoring and debugging
fuser
lsof
ls -l /proc/[process id]/fd
ls -l /proc/*/fd
for p in [0-9]*; do ls -l /proc/$p/fd ;done
strace ls
someexe > /dev/null & sudo dtruss -f -t open -p $!
inotifywait -m -r -e OPEN /path/to/traced/directory
# Displays status of RAM in megabytes
free -m
# Force closure of the process and finish it
kill -9 process_id
# force a process to reload configuration
kill -1 process_id
# Show history reboot
last reboot
# Display kernel loaded
lsmod
# Display a list of files opened by processes
lsof -p process_id
# Displays a list of open files in a given path system
lsof /home/user1
# Displays linux tasks
ps -eafw
# Displays linux tasks in a hierarchical mode
ps -e -o pid,args --forest
# Shows a tree system processes
pstree
# Monitoring reliability of a hard-disk through SMART
smartctl -A /dev/hda
# Check if SMART is active on a hard-disk
smartctl -i /dev/hda
# Display system calls made and received by a process
strace -c ls >/dev/null
# Display library calls
strace -f -e open ls >/dev/null
# Show events inherent to the process of booting kernel
tail /var/log/dmesg
# Show system events
tail /var/log/messages
# Display linux tasks using most cpu
top
# Display interrupts in real-time
watch -n1 'cat /proc/interrupts'
# Show shared libraries required by ssh program
ldd /usr/bin/ssh
# See also
auditctl
inotify
# Get process details for process ID 3117
/proc/3117
# Monitor messages in a log file
tail -f /var/log/messages
# Summarise/profile system calls made by command
strace -c ls >/dev/null
# Show file opens
echo exit | strace bash -li |& grep '^open'
# List system calls made by command
strace -f -e open ls >/dev/null
# Monitor what's written to stdout and stderr
strace -f -e trace=write -e write=1,2 ls >/dev/null
# List library calls made by command
ltrace -f -e getenv ls >/dev/null
# List opened files and ports
# http://en.wikipedia.org/wiki/Lsof
lsof
# List paths that process id has open
lsof -p $$
# List processes that have specified path open
lsof ~
# Show network traffic except ssh. See also tcpdump_not_me
tcpdump not port 22
# List processes in a hierarchy
ps -e -o pid,args --forest
# List processes by % cpu usage
ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d'
# List processes by mem (KB) usage. See also ps_mem.py
ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS
# List all threads for a particular process
ps -C firefox-bin -L -o pid,tid,pcpu,state
# List elapsed wall time for particular process IDs
ps -p 1,$$ -o etime=
# Show system reboot history
last reboot
# Show amount of (remaining) RAM (-m displays in MB)
free -m
# Watch changeable data continuously
watch -n.1 'cat /proc/interrupts'
# Monitor udev events to help configure rules
udevadm monitor
# Active processes
ps -A
# Active processes - Tree processes
/bin/ps acxfwwwe
# Active processes - Live display
top
# List active network connections (TCP) with corresponding processes
netstat -ape
System
System information
See also sysinfo
.
# Show name and version of distribution
head -n1 /etc/issue
# Show kernel version and system architecture
uname -a
# show architecture of machine(2)
uname -m
# show used kernel version
uname -r
# show architecture of machine
arch
# show information CPU info
cat /proc/cpuinfo
# show interrupts
cat /proc/interrupts
# verify memory use
cat /proc/meminfo
# show file(s) swap
cat /proc/swaps
# show version of the kernel
cat /proc/version
# show network adpters and statistics
cat /proc/net/dev
# show mounted file system(s)
cat /proc/mounts
# Show RAM total seen by the system
grep MemTotal /proc/meminfo
# Show CPU(s) info
grep "model name" /proc/cpuinfo
# Show PCI info
lspci -tv
# Show USB info
lsusb -tv
# Show state of cells in laptop battery
grep -F capacity: /proc/acpi/battery/BAT0/info
# Display SMBIOS/DMI information
dmidecode -q | less
Power management
Aka shutdown, restart and logout of a system
# shutdown system(2)
init 0
# leaving session
logout
# reboot(2)
reboot
# shutdown system(1)
shutdown -h now
# planned shutdown of the system at 16:30
shutdown -h 16:30 &
# cancel a planned shutdown of the system
shutdown -c
# reboot
shutdown -r now
# shutdown system
telinit 0
Interactive
See also linux keyboard shortcuts
# Line editor used by bash, python, bc, gnuplot, ...
readline
# Virtual terminals with detach capability, ...
screen
# Visual file manager that can browse rpm, tar, ftp, ssh, ...
mc
# Interactive/scriptable graphing
gnuplot
# Web browser
links
# open a file or url with the registered desktop application
xdg-open .
Shell
Command-line interface (CLI), not for graphical user interface (GUI)
The default shell is defined for a specific user by the file /etc/passwd
# List available shells
cat /etc/shells
# Clear shell cache
hash -r
# Change shell
chsh
chsh --list-shells
In /etc/bashrc
or ~/.bashrc
# If id command returns zero, youโve root access.
if [ $(id -u) -eq 0 ];
then you are root, set red colour prompt
PS1="\\[$(tput setaf 1)\\]\\u@\\h:\\w\\[$(tput sgr0)\\]"
else normal
PS1="[\\u@\\h:\\w] $"
fi
Shells:
Bash (Unix shell) - Wikipedia - Bourne-Again shell
Shell config
# Display current shell prompt setting
echo $PS1
# Change current shell prompt setting
export PS1="..."
export PS1="\[\033[0m\]\[\033[0;30;47m\] \u\[\033[0m\]\[\033[0;30;47m\]@\h\[\033[0m\] \w\n\[\033[1;32m\] \@ \$ \[\033[0m\]"
export PS1="[${LOGNAME}@$(hostname)]"
Profile files, defined by (in order, but also if the shell is interactive and/or a login):
(global system)
/etc/profile
(global system)
/etc/bashrc
(user)
~/.bash_profile
(user)
~/.bash_login
(user)
~/.profile
(user)
~/.bashrc
(not used by some OS like macOS for login shell, it's recommended toif [ -s ~/.bashrc ]; then source ~/.bashrc; fi
in~/.profile
for that case)(user)
~/.bash_logout
au logout
(and /etc/bash.bashrc
, /etc/bash.bashrc.local
?)
To reload a configuration file (here the file ~/.bashrc
)
source ~/.bashrc
# or use dot command:
. ~/.bashrc
initialisation at the session level (
.bash_profile
) and initialisation at each shell level (.bashrc
)
By convention, the prompt ends with $
for users and by #
for root
terminal - Why doesn't .bashrc run automatically? - Ask Different - add
if [ -s ~/.bashrc ]; then source ~/.bashrc; fi
to~/.bash_profile
Suggestion: disambiguate shell setup steps ยท Issue #351 ยท Schniz/fnm - About macOS that doesn't source
~/.bashrc
Unix shell - Wikipedia - Configuration files
Command alias
# quick dir listing
alias l='ls -l --color=auto'
Programmable bash completions
See complete
buildin command
Terminal history auto complete
In ~/.inputrc
or /etc/inputrc
:
# History auto complete with "start with" filter
"\e[5~": history-search-backward
"\e[6~": history-search-forward
set show-all-if-ambiguous on
set completion-ignore-case on
http://www.gnu.org/software/bash/manual/html_node/Readline-Init-File.html
https://stackoverflow.com/questions/1030182/how-do-i-change-bash-history-completion-to-complete-whats-already-on-the-line
On OSX Page up and Page down in Terminal are by default use to manipulat buffer instead to send key stroke
You can change it in Preferences โ Settings โ Keyboard
:
Page down :
\033[6~
Page up :
\033[5~
End :
\033OF
(works in vi, vim, etc.) ou\033[4~
,\033[F
,\005
Home :
\033OH
(works in vi, vim, etc.) ou\033[1~
,\033[H
,\001
Use esc to type these strings (add \033
in input field) or past (via the context menu)
Shell variables
env
See also arrays
Colors and control sequences
For ls
colors:
export CLICOLOR=1
export TERM=xterm-color
export LSCOLORS=DxFxcxdxBxegedbxHxacHd
key=effect;foreground colour;background colour
separated by :
.
Ex.: a symlink bright pink ln=01;95;40
give:
export LS_COLORS="di=01;33;40:ln=01;95;40"
To known color config:
man ls
#
# Dont forget to call ~/.bashrc in your ~/.profile script
#
export CLICOLOR=1
alias ll='ls -l'
alias la='ls -A'
alias vi='vim'
alias l='ls -CF'
function cyan_red_prompt {
local CYAN="\[\033[0;36m\]"
local GRAY="\[\033[0;37m\]"
local RED="\[\033[0;31m\]"
PS1="${CYAN}[\u@\h ${RED}\w${CYAN}]${GRAY} "
}
cyan_red_prompt
Mouse position and click
Aka mouse tracking escape sequences
Command arguments
usage() {
cat <<EOT
Usage: ${basename $0} [-a value] [-h] value...
Options:
-h Print this help
-a value Print something
EOT
}
while getopts ":ha:" opt; do
case $opt in
a)
echo "-a with value: $OPTARG"
;;
h)
usage
exit 0
;;
\?)
echo "Invalid option: -$OPTARG" >&2
usage
exit 1
;;
:)
echo "Option -$OPTARG requires an argument." >&2
usage
exit 1
;;
esac
done
shift $((OPTIND-1))
others=($@)
echo "${others[@]}"
echo "$@"
Command alias
alias NICKNAME='COMMAND -withargs'
A ajouter dans ~/.bashrc
Shell script file
Note: take care editing bash scripts
Create a shell script file.sh
the execute the following command to allow it to be executable:
chmod +x /path/to/file.sh
Start with the shebang #!/bin/sh
, which indicate the script interpreter (sh
, php
, python
, perl
, node
, etc.) or for portability, use env
:
#!/usr/bin/env sh
echo Hello world!
Or detect the right interpreter with an additional line, here to use nodejs
or node
. :
command in bash is noop.
#!/usr/bin/env sh
':' //; exec "$(command -v nodejs || command -v node)" "$0" "$@"
console.log('Hello world!');
To include shell script into an other:
# Relative to execution path:
#source /path/to/script.sh
# So use this instead:
source $(dirname $0)/script.sh
# Or:
CURRENT_DIR=`dirname $0`
$CURRENT_DIR/script.sh
Shell script syntax
Use $( )
, and don't use `` anymore: Bash FAQ 82.
Avoid using UPPERCASE variable names. That namespace is generally reserved by the shell for special purposes (like PATH
), so using it for your own variables is a bad idea.
Escape special char in argument:
command $'\t'
Wildcard files:
for file in *.ext1; do echo "${file} ${file/.ext1/.ext2}"; done
ls file_*{.png,.jpg}
File Wildcards
Brace expansion
sudo mv /path/file{,.old}
sudo ln {/path1,/path2}/file
Strings:
echo "The current user (\$USER) \"$USER\" use the shell (\$SHELL) \"$SHELL\""
echo 'The current user ($USER) "'"$USER"'" use the shell ($SHELL) "'"$SHELL"'"'
Shell parameter expansion:
# Remove trailing slash for all arguments `a/ b/c/ d /` output `a b/c d`
echo ${@%/}
# myvar=a/b/// output `a/b`
shopt extglob
echo ${myvar%%+(/)}
somevar=$(echo "$value" | sed 's:/*$::')
echo $somebar
dir=/a/b///
echo $(realpath -s --canonicalize-missing $dir)
# Construct command parts as an array
mycommand=(/some/command "$PATH_NAME")
# Print the command
printf '%s\n' "${mycommand[*]}"
# Execute command
"${mycommand[@]}"
Redirection
File pointer and file redirection
named pipe (
mkfifo
)
while read item; do
echo $item
done < sql.res
Bash support process substitution:
cat <(echo "tmp file content")
Is similar to: echo hello > file; cat file; rm file
$(cat file)
is same as $(< file)
cat <<< hello
diff <(cd dir1; ls) <(cd dir2; ls)
(
echo open ip_address
echo user username password
echo put -O $today $file_gnome_curr
echo bye
) | lftp -f /dev/stdin >> lftp.log 2>&1
Here-doc & here-string:
cat > /path/to/file <<EOF
Some thing.
Blahblah!
$somevar
EOF
cat > /path/to/file <<< "Some thing."
cat <<EOF
test1
test2
EOF
(cat <<'EOF'
file1
file2
EOF
) | while read -r file; do
cp "from/$file" "to/$file";
done;
cat <<EOF >> greetings.txt
line 1
line 2
EOF
# Here-doc to var
# Use `IFS=` otherwise any leading and trailing spaces will be trimmed
# Use `<<'EOF'` instead `<<EOF` to ignore variables tokens
IFS= read -r -d '' MYVAR <<'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF
File descriptors (fd) :
0 is
stdin
IN1 is
stdout
OUT2 is
stderr
ERROR
2>&1
redirects fd 2 to 1 (stderr
to stdout
)
# Empty a file (ex: clean a log file)
> /var/log/apache2/error.log
# Append to a file
echo "Appended text" >> /path/file
# Log all infos (and erros) of following commands to a file `/tmp/log.txt`
( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /tmp/log.txt
man -P less\ +/^REDIRECTION bash
http://www.tldp.org/LDP/Bash-Beginners-Guide/html/Bash-Beginners-Guide.html#sect_08_02_03_02
/usr/include/unistd.h
http://en.wikipedia.org/wiki/File_descriptor
Pipe:
Network with redirection
Use instead specialized tools like netcat
or socat
/dev/tcp/<host>/<port>
and/dev/udp/<host>/<port>
{ echo -e "GET / HTTP/1.0\r\nHost: www.example.com\r\n\r" >&3; cat <&3 ; } 3<> /dev/tcp/www.example.com/80
bash - /dev/tcp listen instead of nc listen - Unix & Linux Stack Exchange - port listing is not possible
GitHub - caquino/redis-bash: REDIS-BASH - Bash library to access Redis - Example of what can be made with network redirection
Fork process
Execute in background
If a command is terminated by the control operator &, the shell executes the command in the background in a subshell
echo '' > .mycmd.pid
mycmd & echo "$!" >> .mycmd.pid
# ... later
cat .mycmd.pid | xargs -n 1 kill -9
Add a bin folder to global executables
Pour รฉviter de tapper le chemin entier vers les commandes / executables
PATH=$PATH:/path/bindir
export PATH
PATH=$PATH:/path/bindir command-use-path
PATH=$PATH:/path/bindir bash -c 'echo $PATH'
PATH=$PATH:/path/bindir bash -c 'command1 | command2'
(export PATH=$PATH:/path/bindir; command1 | command2)
Pour l'ajouter de faรงon permanente, l'รฉcrire dans ~/.profile
ou ~/.bashrc
Automatic error detection
Instead || exit 1
after each important command.
Dotfiles
Tree navigation
# Enter to directory `/home`
cd /home
# Go back one level
cd ..
# Go back two levels
cd ../..
# Go to $HOME directory
cd
# Go to home directory
cd ~user1
# Go to previous directory
cd -
# Get the current working directory
pwd
# Go to dir, execute command and return to current dir
(cd dir && command)
# Put current dir on stack so you can popd back to it
pushd .
Files and folders operations
File name:
Using
sed
or other external processes to do simple string operations like stripping extensions and prefixes is inefficient. Instead, use parameter expansions which are part of the shell (no external process means it will be faster). Some helpful articles on the subject are listed below:Bash FAQ 73: Parameter expansions Bash FAQ 100: String manipulations
# Modify timestamp of a file or directory - (YYMMDDhhmm)
touch -t 0712250000 file1
# Create directories `dir1` and `dir2` (in same parent dir)
mkdir dir1 dir2
# Try to create a directory tree
mkdir -p /tmp/dir1/subdir1
# Extract music from m3u (ignore lines start with #, whitespace or are empty)
# Playlists entries could be relative
cd "/path/to/playlists"
cat "playlist1.m3u" "playlist1.m3u" "playlist3.m3u" | grep "^[^# \t]" | tr -s '\n' | while read -r line; do cp --parent -v "$line" /destination; done
# Get extensions of all files
find . -type f -printf "%f\n" | grep -o -E '(\.[^\.]*)$' | sort | uniq
# File checksum SHA256, used in chef cookbooks
openssl dgst -sha256 path/to/myfile
# File checksumMD5
openssl dgst -md5 path/to/myfile
# Read or write BSD-style checksum file
# https://unix.stackexchange.com/a/476480/60668
sha256sum -c SHA256SUMS path/to/myfile
sha256sum --tag path/to/myfile
# Encrypt a file with GNU Privacy Guard
gpg -c file1
# Decrypt a file with GNU Privacy Guard
gpg file1.gpg
# Create hardlink aka hardware link
# Note: It's not possible to create hardlink with a directory (due to the risk of loops in tree)
ln /path/to/source_file /path/to/target_file
ln -T /etc/apache2/sites-available/example.com /etc/apache2/sites-enabled/example.com
# Remove file link
unlink /etc/apache2/sites-enabled/websiteA.exemple
# Create a symbolic link to file or directory
ln -s file1 lnk1
File listing and searching
Find support shell pattern for -path
and -ipath
parameters:
*/file.ext
*.ext
*/file.*
# Find all files that have been modified in the past 7 days
find . -type f -mtime -7
# Search binary files are not used in the last 100 days
find /usr/bin -type f -atime +100
# Find all JPEGs that have been modified more than 30 days ago
find . -name \*.jpg -mtime +30
# Move all JPEGs from the current folder (recursively) that are greater than 40k into the folder /tmp/2
find . -name \*.jpg -size +40k -exec mv {} /tmp/2 +
# search files and directories belonging to `user1`
find / -user user1
# List files by date. See also newest and find_mm_yyyy
ls -lrt
# Print in 9 columns to width of terminal
ls /usr/bin | pr -T9 -W$COLUMNS
# Search 'expr' in this dir and below. See also findrepo
find -name '*.[ch]' | xargs grep -E 'expr'
# Search all regular files for 'example' in this dir and below
find -type f -print0 | xargs -r0 grep -F 'example'
# Search all regular files for 'example' in this dir
find -maxdepth 1 -type f | xargs grep -F 'example'
# Process each item with multiple commands (in while loop)
find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done
# Get filename (without the extension)
find ./ -name "tile_*.png" -exec bash -c 'filename="$1";echo "${filename%.*}"' _ {} \;
# search files with `.bin` extension within directory `/home/user1`
find /home/user1 -name \*.bin
# Search files with `.rpm` extension and modify permits
find / -name *.rpm -exec chmod 755 '{}' \;
# Search files with `.rpm` extension ignoring removable partitions as cdrom, pen-drive, etc.โฆ
find / -xdev -name \*.rpm
# Find files with the `.ps` extension - first run `updatedb` command
locate \*.ps
#Search cached index for names. This re is like glob *file*.txt
locate -r 'file[^/]*\.txt'
# Find files not readable by all (useful for web site)
find -type f ! -perm -444
# Find dirs not accessible by all (useful for web site)
find -type d ! -perm -111
# Search for empty files and folder (or file with only whitespaces)
find -empty
# List all folders and files, ordered by size
du -ak . | sort -nr | less
# List top 10 largest files and directories
du -a /var | sort -n -r | head -n 10
# Quickly search (sorted) dictionary for prefix
look reference
# Append to each files
find -type f -exec sh -c 'echo "something" >> "$1"' _ {} \;
find -type f -print0 | while IFS= read -r -d '' file; do echo "something" >> "$file"; done
find -type f -exec bash -c 'for file; do echo "something" >> "$file"; done' _ {} \+
# See also https://serverfault.com/questions/343705/how-do-i-append-a-specific-number-of-null-bytes-to-a-file/
# Use a simple shell loop, to process each of the images.
mkdir thumbnails
for f in *.jpg
do convert $f -thumbnail 200x90 thumbnails/$f.gif
done
# Use find to substitute filenames into a 'convert' command.
# This also provides the ability to recurse though directories by removing
# the -prune option, as well as doing other file checks (like image type,
# or the disk space used by an image).
find * -prune -name '*.jpg' \
-exec convert '{}' -thumbnail 200x90 thumbnails/'{}'.gif \;
# Use xargs -- with a shell wrapper to put the argument into a variable
# This can be combined with either "find" or "ls" to list filenames.
ls *.jpg | xargs -n1 sh -c 'convert $0 -thumbnail 200x90 thumbnails/$0.gif'
# An alternative method on Linux (rather than plain Unix)
# This does not need a shell to handle the argument.
ls *.jpg | xargs -r -I FILE convert FILE -thumbnail 200x90 FILE_thumb.gif
unix - Appending new lines to multiple files - Super User - How to use
echo "test" >> {}
within find exec
Find missing files
Find duplicates files
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate | awk '{$1=""; print substr($0,2)}'
Remove duplicates
#!/bin/bash
gawk '
{
cmd="md5 -r " q FILENAME q
cmd | getline cksm
close(cmd)
sub(/ .*$/,"",cksm)
if(a[cksm]++){
cmd="rm " q FILENAME q
system(cmd)
close(cmd)
}
nextfile
}' q='"' *
Rdfind โ redundant data find:
rdfind -makehardlinks true
fdupes
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
BTRFS Deduplication - btrf Wiki
See also diff -q --binary
or cmp -s $1 $2 && echo "identical" || echo "different"
centos - What's the quickest way to find duplicated files? - Unix & Linux Stack Exchange
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
Replace the use of md5sum with
diff
orcmp
: How do I do a binary diff on two identically sized files under Linux? - Super User
Then replace files by hardlink
Find files by content
# Find multiple exec a executed only if the previous one exit with 0 https://stackoverflow.com/questions/5119946/find-exec-with-multiple-commands#comment34296391_6043896
find -type f -iname "*.properties" -exec grep -q -i -E '^abtest=false$' {} \; -print
find -type f -iname "*.properties" -exec grep -q -i -E '^abtest=' {} \; -exec sed -i 's/^abtest=.*/abtest=true/i' {} \; -print
# Use test to inverse the grep exit value https://stackoverflow.com/a/30495279/470117 (-v/--invert-match option is not useful for that case), for that we need also sh interpreter to use builtin test command
# For append mode (redirection), need to use sh interpreter directly, else {} will be interpreted directly by Bash. See https://superuser.com/questions/1327969/appending-new-lines-to-multiple-files/1327980#1327980
# -exec sh -c 'echo "command name: $0, first arg: $1"' test {} \;
find -iname "*.properties" -exec sh -c 'grep -q -i -E "^abtest=" $1; test $? -eq 1' match {} \; -exec sh -c 'echo -e "\n\nabtest=true" >> $1' append {} \; -print
find -iname "*.properties" -type f -print0 | xargs -0 grep -EHi '^abtest='
# Same as (flexibity of find vs glob include/exclude filters):
grep -EHir --include="*.properties" '^abtest=' /path/to/dir
grep -ir --include=\*.{php,js,css} "/api/v1/"
# Example search in all .less files that contains `url("<url>")` or `url('<url>')` or `url(<url>)`
find "$wd" -iname "*.less" \( -not -ipath "*/node_modules/*" \) -type f -print0 | xargs -0 grep -EliZ 'url\(('"'"'|"|)(.*?)\1\)' | xargs -0 -n1 echo "Do something with that file:"
Find corrupted files
Find files with 100 consecutive zero bytes (partially downloaded, etc.)
find . -type f -exec egrep "\x00{100,}" {} \; -exec echo {} \;
# If not work try with grep -P
Note: If you get grep: invalid repetition count(s)
, use (\x00\x00\x00\x00){250,}
instead of \x00{1000,}
(because repetition as limitation)
Delete files and directories
Aka remove files and dirs
# Remove `file1` and `file2`
rm -f file1 file2
# Remove `dir1` and `dir2` and their contents recursively
rm -rf dir1 dir2
# Remove directory `dir1`
rmdir dir1
# Remove all files start with "title_" and ends with ".png"
find . -name 'tile_*.png' -delete
# Remove all files with exceptions:
find . -type f -not \( -name '*.php' -or -name '*.iso' \) -exec rm {} \;
# Remove all files with a list of exceptions:
find . -type f -printf "%P\n" ! -name "list.txt" | fgrep -vf list.txt | xargs -r rm
# Remove empty dir
find <path> -type d -empty -delete
# Can also use (not recommanded)
find <path> -type d -empty -print0 | xargs -0 -I{} rmdir "{}"
find <path> -type d -print0 | xargs -0 -r rmdir -p --ignore-fail-on-non-empty
# Handle trailing dot rmdir error: rmdir -p "./test" -> rmdir: failed to remove directory '.': Invalid argument
find -mindepth 1 -type d -printf '%P\0' | xargs -0 -r rmdir -p --ignore-fail-on-non-empty
# Find all dirs in reverse order (depth first), remove empty dir to top
find -mindepth 1 -type d -print0 | tac -s $'\0' | xargs -0 -r rmdir --ignore-fail-on-non-empty
# Delete all Unix hidden files
# All files or folders that start with `.`, like `.htaccess` or `.DS_STORE`
rm -rf .[^.]*
# All dir `.git`
find . -name ".git" -type d -prune -exec rm -rf {} +
# All dir `node_modules` (which can recusively contains `node_modules` dirs; Use `-prune` for that)
find . -name "node_modules" -type d -prune -exec rm -rf {} +
http://www.cyberciti.biz/faq/linux-bash-delete-all-files-in-directory-except-few/
Copy or move files and directories
Aka duplicate and rename
# Rename (move) a file or directory
mv dir1 new_dir
# Change file extension
for j in /path/dir/*.html
do
n=${j/.html}
mv "$j" "$n.php"
done
# Change file extension
for f in *; do mv "$f" "$f.bin"; done;
# Rename `Ref_00001.jpg` to `anim001.jpg`
find . -name 'Ref_00*.jpg' -print0 | awk 'BEGIN {RS = "\0"; ORS = "\0"} {print; gsub(/\/Ref_00/,"/anim"); print }' | xargs -0 -n2 mv
# Or use bash+sed directly, but less performant
for i in 'Ref_00'*.jpg; do mv "$i" "$(echo $i | sed 's/^Ref_00/anim/')"; done;
# To test, replace find by: `echo -en "Ref_00001.jpg\0Ref_00002.jpg\0Ref_00002.jpg"` and replace xargs with `xargs -0 -n2 echo`
# Rename `sprite1.png` to `sprite0.png`, `sprite2.png` to `sprite1.png`, etc.
find . -maxdepth 1 -type f -exec mv "{}" "{}.tmp" \; -print0 | awk 'BEGIN {RS = "\0"; ORS = "\0"} {match($0,/^(.*sprite)([0-9]+)(.*)$/,a); print a[1]a[2]a[3]".tmp"; print a[1]a[2]-1a[3] }' | xargs -0 -n2 mv
# To test, replace find by: `echo -en "sprite1.png\0sprite2.png\0sprite3.png"` and replace xargs with `xargs -0 -n2 echo`
# Use Perl rename (util-linux rename.ul, perl prename, p5-file-rename)
# Rename *.JPG to *.jpg
rename 's/\.JPG/\.jpg/' *.JPG
# Strip spaces
rename 's/ //' *.jpg
# Lower case
rename 'y/A-Z/a-z/' *
# Copying a file
cp file1 file2
# Copy all files in dir1 to dir2 (recursively)
cp -R dir1/. dir2/
# Copy all files of a directory within the current work directory
cp dir/* .
# Copy a directory within the current work directory
cp -a /tmp/dir1 .
# Copy a directory
cp -a dir1 dir2
# Find and copy all files with `.txt` extension from a directory to another
find /home/user1 -name '*.txt' | xargs cp -av --target-directory=/home/backup/ --parents
# local copy preserving permissions and links from a directory to another
tar cf - . | (cd /tmp/backup ; tar xf - )
On macOS install it with port port install p5-file-rename
(but should use the cmd rename-5.22
where 22
is installed version via port instead of rename
) or brew brew install rename
Archives and compressed files
# Decompress a file called `file1.bz2`
bunzip2 file1.bz2
# Compress a file called `file1`
bzip2 file1
# Decompress a file called `file1.gz`
gunzip file1.gz
# Compress a file called `file1`
gzip file1
# Compress with maximum compression
gzip -9 file1
# Create an archive rar called `file1.rar`
rar a file1.rar test_file
# Compress `file1`, `file2` and `dir1` simultaneously
rar a file1.rar file1 file2 dir1
# Decompress rar archive
rar x file1.rar
# Create a uncompressed tarball
tar -cvf archive.tar file1
# Create an archive containing `file1`, `file2` and `dir1`
tar -cvf archive.tar file1 file2 dir1
# Show contents of an archive
tar -tf archive.tar
# Extract a tarball
tar -xvf archive.tar
# Extract a tarball into / tmp
tar -xvf archive.tar -C /tmp
# Create a tarball compressed into bzip2
tar -cvfj archive.tar.bz2 dir1
# Create a tarball compressed into gzip
tar -cvfz archive.tar.gz dir1
# Decompress a compressed tar archive in bzip2
tar -xvfj archive.tar.bz2
# Decompress a compressed tar archive in gzip
tar -xvfz archive.tar.gz
# Make a incremental backup of directory `/home/user`
tar -Puf backup.tar /home/user
# Decompress rar archive
unrar x file1.rar
# Decompress a zip archive
unzip file1.zip
# Create an archive compressed in zip
zip file1.zip file1
# Compress in zip several files and directories simultaneously
zip -r file1.zip file1 file2 dir1
Create archive from file list:
#!/bin/bash
# Create temp list
list=$(mktemp "/tmp/archive_list.XXXXXXXXXX")
# Write stdin file / folder list
cat > "$list"
# Get destination folder based on first file / folder
first=$(head -n 1 "$list")
folder=$(dirname "$first")
# Temporary file and final file
final="$folder/Archive.zip"
file="$final.tmp"
# Create archive file (zip, ultra)
/opt/local/bin/7z a -tzip -mx=9 "$file" @"$list"
# Move to final destination
mv "$file" "$final"
# Remove temp list
rm "$list"
Split files
FAT32 maximum file size is "4 GiB minus 1 byte" (4294967295 bytes)
To write file with a size larger than that limit, you need to split it.
In destination folder, execute following command:
split -b 4294967295 example.dmg example.dmg.part.
(the part example.dmg.part.
will be used to name output files like example.dmg.part.aa
, example.dmg.part.ab
and so on)
cat example.dmg.part.* > example.dmg
Read write race condition
cat tmp | head -1 >new && mv new tmp
{ cat tmp | head -1; } >tmp
Choose incremental filename
Test if dirB/file.ext
exist else choose dest file name as dirB/file-1.ext
, if already exist choose dirB/file-2.ext
and so one
source=dirA/file.ext
dest_dir=dirB
file=$(basename file.ext)
basename=${file%.*}
ext=${file##*.}
if [[ ! -e "$dest_dir/$basename.$ext" ]]; then
# file does not exist in the destination directory
mv "$source" "$dest_dir"
else
num=2
while [[ -e "$dest_dir/$basename-$num.$ext" ]]; do
(( num++ ))
done
mv "$source" "$dest_dir/$basename-$num.$ext"
fi
File attributes
# Allows write opening of a file only append mode
chattr +a file1
# Allows that a file is compressed / decompressed automatically by the kernel
chattr +c file1
# Makes it an immutable file, which can not be removed, altered, renamed or linked
chattr +i file1
# All attributes (see details in the manpage of `chattr`):
# - append only (a)
# - compressed (c)
# - no dump (d)
# - extent format (e)
# - immutable (i)
# - data journalling (j)
# - secure deletion (s)
# - no tail-merging (t)
# - undeletable (u)
# - no atime updates (A)
# - synchronous directory updates (D)
# - synchronous updates (S)
# - top of directory hierarchy (T)
# show specials attributes
lsattr
File type
# Get file type if known by libmagic
file --mime /path/file
# Get information about files in the current folder
find . -maxdepth 1 -type f -exec file "{}" \;
Supported formats are listed by:
files in
/etc/magic
/etc/magic.mime
/usr/share/misc/magic.mgc
/usr/share/file/magic.mgc
/opt/local/share/misc/magic.mgc
files in
/opt/local/share/misc/magic
~/.magic.mgc
~/.magic
The extension of compiled magic: *.mgc
# Test if a file exist
filename=$1
if [ -f $filename ]
then
echo "$filename exists"
else
echo "$filename does NOT exist"
fi
Files permissions
# Change group of files
chgrp group1 file1
# Set permissions reading (r), write (w) and (x) access to users owner (u) group (g) and others (o)
chmod ugo+rwx directory1
# Remove permits reading (r), write (w) and (x) access to users group (g) and others (or
chmod go-rwx directory1
# Set SUID bit on a binary file - the user that running that file gets same privileges as owner
chmod u+s /bin/file1
# Disable SUID bit on a binary file
chmod u-s /bin/file1
# Set SGID bit on a directory - similar to SUID but for directory
chmod g+s /home/public
# Disable SGID bit on a directory
chmod g-s /home/public
# Set STIKY bit on a directory - allows files deletion only to legitimate owners
chmod o+t /home/public
# Disable STIKY bit on a directory
chmod o-t /home/public
# Change owner of a file
chown user1 file1
# Change user owner of a directory and all the files and directories contained inside
chown -R user1 directory1
# Change user and group ownership of a file
chown user1:group1 file1
# View all files on the system with SUID configured
find / -perm -u+s
Name, extension and parent folder
Note: Don't use parameter substitution, it's not work with all cases
folder=/tmp
dirname "$folder"
# /
basename "$folder"
# tmp
echo "${folder%/*}"
#
echo "${folder%/*.*}"
# /tmp
echo "${folder##*/}"
# tmp
echo "${folder##*.}"
# /tmp
folder2=/tmp/folder
dirname "$folder2"
# /tmp
basename "$folder2"
# folder
echo "${folder2%/*}"
# /tmp
echo "${folder2%/*.*}"
# /tmp/folder
echo "${folder2##*/}"
# folder
echo "${folder2##*.}"
# /tmp/folder
file=/tmp/file.png
dirname "$file"
# /tmp
basename "$file"
# file.png
echo "${file%/*}"
# /tmp
echo "${file%/*.*}"
# /tmp
echo "${file##*/}"
# file.png
echo "${file##*.}"
# png
extension=$([[ "$file" = *.* ]] && echo "${file##*.}" || echo '')
Read file content or stream
# View the contents of a file starting from the first row
cat file1
# View first two lines of a file
head -2 file1
# Pagers (moving down the file content one screen at a time)
# View content of a file along, allowing only forward navigation through the file
more file1
# View content of a file along, allowing forward and backward navigation through the file
less file1
# View the contents of a file starting from the last line
tac file1
# View last two lines of a file
tail -2 file1
# View in real time what is added to a file
tail -f /var/log/messages
# View from line 20 to 30 of a file
sed -n '20,30 p' file1
# View from line 20 to end of a file
sed -n '20,$ p' file1
# View from the begining to line 20 of a file
sed -n '1,20 p' file1
# Divide into 5 columns
ls /tmp | pr -T5 -W$COLUMNS
Create or truncate a file
# Create or tuncate a file
> /var/log/apache2/error.log
# Create an empty file or change the modification date
touch file.ext
# Create a large test file (taking no space)
dd bs=1 seek=2TB if=/dev/null of=file.ext
File comparaisons
Note: you can export LANG=C for speed. Also these assume no duplicate lines within a file
# Union of unsorted files
sort file1 file2 | uniq
# Intersection of unsorted files
sort file1 file2 | uniq -d
# Difference of unsorted files
sort file1 file1 file2 | uniq -u
# Symmetric Difference of unsorted files
sort file1 file2 | uniq -u
# Union of sorted files
join -t'\0' -a1 -a2 file1 file2
# Intersection of sorted files
join -t'\0' file1 file2
# Difference of sorted files
join -t'\0' -v2 file1 file2
# Symmetric Difference of sorted files
join -t'\0' -v1 -v2 file1 file2
Use diff and patch
Compare 2 folders:
Sans -q
mais avec -u
pour faire un patch
diff -rq /path/dirA /path/dirB
Et sans prendre en compte les fichiers metadonnรฉes d'OSX et de Windows, le tout ordonnรฉ :
diff -uqr /path/dirA /path/dirB | grep -v -e 'DS_Store' -e 'Thumbs' | sort
Comparer 2 files:
diff -u /path/fileA /path/fileB
Apply a patch:
patch -i my.patch
Use heredoc
patch -i -p1 - <<EOF
diff -ur a/Makefile.in b/Makefile.in
--- a/Makefile.in 2016-03-13 01:08:14.000000000 +0100
+++ b/Makefile.in 2016-03-13 01:09:13.000000000 +0100
@@ -1,3 +1,4 @@
+vpath = src
top_builddir = ..
srcdir = @srcdir@
top_srcdir = @top_srcdir@
EOF
Use -p1
to strip path a/
and b/
Folder bind
Aka folder link, like symlink but for folder (standart implementation don't allow to link folder)
# Mount a folder at a specific point (here `/path/destdir`)
mount --bind /path/srcdir /path/destdir
# Unmount
umount /path/destdir
The mount point is registerd in /etc/fstab
as fs-type
: bind
Archives and compression
# Encrypt file
gpg -c file
# Decrypt file
gpg file.gpg
# Make compressed archive of dir/
tar -c dir/ | bzip2 > dir.tar.bz2
# Find all files with `.log` extension and make an bzip archive
find /var/log -name '*.log' | tar cv --files-from=- | bzip2 > log.tar.bz2
# Extract archive (use gzip instead of bzip2 for tar.gz files)
bzip2 -dc dir.tar.bz2 | tar -x
# Make encrypted archive of dir/ on remote machine
tar -c dir/ | gzip | gpg -c | ssh user@remote 'dd of=dir.tar.gz.gpg'
# Make archive of subset of dir/ and below
find dir/ -name '*.txt' | tar -c --files-from=- | bzip2 > dir_txt.tar.bz2
# Make copy of subset of dir/ and below
find dir/ -name '*.txt' | xargs cp -a --target-directory=dir_txt/ --parents
# Copy (with permissions) copy/ dir to /where/to/ dir
( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p )
# Copy (with permissions) contents of copy/ dir to /where/to/
( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p )
# Copy (with permissions) copy/ dir to remote:/where/to/ dir
( tar -c /dir/to/copy ) | ssh -C user@remote 'cd /where/to/ && tar -x -p'
# Backup harddrive (raw) to remote machine
dd bs=1M if=/dev/sda | gzip | ssh user@remote 'dd of=sda.gz'
Incremental listing
Incrementaly add/changed/delete files (list made from cron
or similar)
find . -type f -mtime -7 -exec ls -l {} \; > list.txt
find . -type f -exec ls -i {} + | sort -k2 > list.txt
tar --create --file=archive.1.tar --listed-incremental=/var/log/usr.snar .
tar -cf /dev/null -g /var/log/usr.snar .
date +%s
/backupdevice/folder
date +%s%N > /backupdevice/folder.index
Change encoding
Use recode
that obsoletes iconv, dos2unix, unix2dos
# Show available conversions (aliases on each line)
recode -l
#iconv -l
# Windows "ansi" to local charset (auto does CRLF conversion)
recode windows-1252.. file_to_change.txt
# Windows utf8 to local charset
recode utf-8/CRLF.. file_to_change.txt
# Latin9 (western europe) to utf8
recode iso-8859-15..utf8 file_to_change.txt
#iconv -f iso-8859-15 -t utf-8 <infile> -o <outfile>
# Base64 encode / decode
recode ../b64 < file.png
# Quoted printable decode
recode /qp.. < file.qp > file.txt
# Text to HTML
recode ..HTML < file.txt > file.html
# Lookup table of characters
recode -lf windows-1252 | grep euro
# Convert a text file format from MSDOS to UNIX (line-ending CRLF to LF)
recode ibmpc..latin1 file file2
#dos2unix file file2
#cat file | tr -d '\r' > file2 && mv -v file2 file
#sed -i s/\r//g file
# convert a text file format from UNIX to MSDOS (line-ending LF to CRLF)
recode latin1..ibmpc file file2
#unix2dos file file2
#sed -i 's/$/\r/' file
# Show what a code represents in latin-9 charmap
echo -n 0x80 | recode latin-9/x1..dump
# Show latin-9 encoding
echo -n 0x20AC | recode ucs-2/x2..latin-9/x
# Show utf-8 encoding
echo -n 0x20AC | recode ucs-2/x2..utf-8/x
Temporary file
temp_file=$(mktemp)
# Remove the file when the script exit
trap "rm -f $temp_file" 0
# do something with the file
echo "Hello World!" > $temp_file
cat $temp_file
# Remove the file now (optional)
rm -f "$temp_file"
# Do something else
tmp - How create a temporary file in shell script? - Unix & Linux Stack Exchange - See example with file descriptor, which is cleaned automatically
Mktemp with suffix/extension | Gawen's blog - Use temp dir instead of temp file
Prefix a list of paths
If sed
pattern come from variable, some escaping is needed: sed "s/$BRE/$REPL/"
Instead use awk with variables:
prefix=/some/path/to/
# awk sub() can be use to replace with regex
find -printf '%P\0' | awk -v prefix="$prefix" 'BEGIN {RS = "\0"; ORS = "\0"} {print prefix $0}' | do_something_with_null_sep_list
Change file in-place
file=test.json
tmp=$(mktemp -u "${file}.XXXXXXXXXX")
address=abcde
jq --arg a "$address" '.address = $a' "$file" > "$tmp" && mv "$tmp" "$file"
contents="$(jq '.address = "abcde"' test.json)" && echo -E "${contents}" > test.json
# Note "contents" cannot contain a literal null character. Bash variables cannot store literal nulls.
Or sponge
Do not use echo "$( jq '.address = "abcde"' test.json )" > test.json
, this will not always work. Large files it will cause issues, also with whitespaces, non-printable and escapment sequences. Never redirect a file to itself, it is always a bad idea.
See also Temporary file
Relative path
function relative_path_from_to() {
# strip trailing slashes
path1=${1%\/}
path2=${2%\/}
# common part of both paths
common=$(printf '%s\x0%s' "${path1}" "${path2}" | sed 's/\(.*\/\).*\x0\1.*/\1/')
# how many directories we have to go up to the common part
up=$(grep -o "/" <<< ${path1#$common} | wc -l)
# create a prefix in the form of ../../ ...
prefix=""; for ((i=0; i<=$up; i++)); do prefix="$prefix../"; done
# return prefix plus second path without common
printf "$prefix${2#$common}"
}
relative_path_from_to /path/test1/file /path/test2/file
# > ../../test2/file
# require the path of both file and dir to exist, resolve symlink
realpath --relative-to=DIR FILE
Text operations
Aka text processing
# Echo multiline string (use the newline separator `\n`)
echo -e "a\nb"
# Add line to begin of file, aka prepend text
echo "firstline" | cat - file.txt > file.txt.tmp && mv file.txt.tmp file.txt
# Convert from lower case in upper case
tr '[:lower:]' '[:upper:]'
# Sort contents of two files
sort file1 file2
# Sort contents of two files omitting lines repeated
sort file1 file2 | uniq
# Sort contents of two files by viewing only unique line
sort file1 file2 | uniq -u
# Sort contents of two files by viewing only duplicate line
sort file1 file2 | uniq -d
# Sort IPV4 ip addresses
sort -t. -k1,1n -k2,2n -k3,3n -k4,4n
# Filter non printable characters
tr -dc '[:print:]'
# cut fields separated by blanks
tr -s '[:blank:]' '\t'
# Count lines
wc -l
# Number row of a file
cat -n file1
# Compare contents of two files by deleting only unique lines from `file1`
comm -1 file1 file2
# Compare contents of two files by deleting only unique lines from `file2`
comm -2 file1 file2
# Compare contents of two files by deleting only the lines that appear on both files
comm -3 file1 file2
# Compare with awk: https://stackoverflow.com/questions/15065818/compare-files-with-awk
# Find differences between two files
diff file1 file2
# Look up words "Aug" on file `/var/log/messages`
grep Aug /var/log/messages
# Look up words that begin with "Aug" on file `/var/log/messages`
grep ^Aug /var/log/messages
# Select from file `/var/log/messages` all lines that contain numbers
grep [0-9] /var/log/messages
# Search string "Aug" at directory `/var/log` and below
grep Aug -R /var/log/*
# Find text in files
grep -rn "texttofind" *
grep "string text to find in all files" . -R
# Highlight occurances of regular expression in dictionary
grep --color reference /usr/share/dict/words
# Merging contents of two files for columns
paste file1 file2
# Merging contents of two files for columns with `+` delimiter on the center
paste -d '+' file1 file2
# Find differences between two files and merge interactively alike `diff`
sdiff file1 file2
# Find in files with a multile regex
# See https://stackoverflow.com/a/7167115/470117
find ./path/to/files -type f -exec grep -Pazo '(?s)<control>.*?</control>' {} \+
# Split string into an array
mapfile -d '' array < <(echo -en "a\0b\0c")
# Join array with single char separator
echo "$(IFS=, ; echo "${array[*]}")"
# Join array with multi chars separator
printf "\"%s\"," "${array[@]}"
Sed
Aka stream editor
Note: sed uses stdin and stdout. Newer versions support inplace editing with the -i option
Remove multiline comment:
# Remove in place all multiline comments
# Note: see also m regex flag
sed -Ei -e '1h;2,$H;$!d;g' -e 's|/\*.*?\*/||g' file.ext
# Single line only (sed match by default on line basis)
sed -i '/<!--.*-->/ d' file
# Note: the , implied the multiple lines.
sed -i '/<!--/,/-->/ d' file
See also:
Search an replace in files
#/bin/bash
# Search files that match some pre conditions then apply remplacements
# find . -iname "*.aspx" -type f -exec grep -q -i -E "Page Language=\"VB\"" {} \; -exec grep -q -i -E "<Fnac:HtmlFooter" {} \; -exec ./substitute.sh {} \; -exec unix2dos -q {} \; -print
REGEXP='<Fnac:HtmlFooter\s+ID="HtmlFooter"\s+runat="server"(\s+Omniture="Default")?\s+OmnitureEVar2="([^"]*)"(\s+OmniturePageName="([^"]*)")?(\s+TagCommander="Default")?\s+TagCommanderIdentifier="1"\s+TagCommanderTemplateType="([^"]*)"\s+TagCommanderTemplateName="([^"]*)"\s+/>'
# All chars must be escaped, only newlines are kept for readability
REPLACEMENT=$(cat <<EOF | sed ':a;N;$!ba;s/\n/\\n/g'
<%
' Html.SetTrackingValues(Tracker.Omniture, new {...});
TrackingHelpers.SetTrackingValues(Tracker.TagCommander, New With
{
Key .template_type = "\6",
Key .template_name = "\7"
})
TrackingHelpers.SetTrackingValues(Tracker.Omniture, New With
{
Key .eVar2 = "\2",
Key .pageName = "\4"
})
%>
<%=WebFormsMvcUtilities.Partial("~/Shared/_Tracking.cshtml", New List(Of Tracker)({ Tracker.TagCommander, Tracker.Omniture }))%>
</body>
</html>
EOF
)
sed -Ei 's|<%@ Register TagPrefix="fnac" TagName="HtmlFooter" Src="~/Nav/Core/Common/HtmlControls/HtmlFooter.ascx" %>||' "$@"
sed -Ei -e '1h;2,$H;$!d;g' -e "s|$REGEXP|$REPLACEMENT|" "$@"
# Replace text in file, need a temp file
# Note: if `old-text` contains slashes `/`, back slash them: `../..` give `..\/..`
sed "s/old-text/new-text/g" file.txt > file.txt.tmp && mv file.txt.tmp file.txt
# Replace string1 with string2
sed 's/string1/string2/g'
# Modify anystring1 to anystring2
sed 's/\(.*\)1/\12/g'
# Remove comments and blank lines
sed '/^ *#/d; /^ *$/d'
# Concatenate lines with trailing \
sed ':a; /\\$/N; s/\\\n//; ta'
# Remove trailing spaces from lines
sed 's/[ \t]*$//'
# Escape shell metacharacters active within double quotes
sed 's/\([`"$\]\)/\\\1/g'
# Right align numbers
seq 10 | sed "s/^/ /; s/ *\(.\{7,\}\)/\1/"
# Duplicate a column
seq 10 | sed p | paste - -
# Print 1000th line
sed -n '1000{p;q}'
# Print lines 10 to 20
sed -n '10,20p;20q'
# Extract title from HTML web page
sed -n 's/.*<title>\(.*\)<\/title>.*/\1/ip;T;q'
# Delete a particular line
sed -i 42d ~/.ssh/known_hosts
# Replace "string1" with "string2"
sed 's/string1/string2/g'
# Remove all blank lines
sed '/^$/d'
# Remove comments and blank lines
sed '/ *#/d; /^$/d'
# Eliminates the first line
sed -e '1d'
# View only lines that contain the word `string1`
sed -n '/string1/p'
# Remove empty characters at the end of each row
sed -e 's/ *$//'
# Remove only the word "string1" from text and leave intact all
sed -e 's/string1//g'
# Print from 1th to 5th row of example.txt
sed -n '1,5p'
# Print row number 5 of example.txt
sed -n '5p;5q'
# Replace more zeros with a single zero
sed -e 's/00*/0/g'
AWK
AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed.
# Print all fields
awk '{print NR": "$0; for(i=1;i<=NF;++i) print "\t"i": "$i}'
# Remove all even lines
awk 'NR%2==1'
# View the first column of a line
echo a b c | awk '{print $1}'
# View the first and third column of a line
echo a b c | awk '{print $1,$3}'
#!/usr/bin/env awk -f
{print $0}
Awk-Batteries/csv.awk at master ยท Nomarian/Awk-Batteries - parse CSV
gawk -f "$AWK/Units/csv.awk" -e '{print NF}'
Disk and file system operations
# Mounting a Filesystem:
# force umount when the device is busy
fuser -km /mnt/hda2
# mount disk called hda2 - verify existence of the directory `/mnt/hda2`
mount /dev/hda2 /mnt/hda2
# mount a floppy disk
mount /dev/fd0 /mnt/floppy
# mount a cdrom / dvdrom
mount /dev/cdrom /mnt/cdrom
# mount a cdrw / dvdrom
mount /dev/hdc /mnt/cdrecorder
# mount a cdrw / dvdrom
mount /dev/hdb /mnt/cdrecorder
# mount a file or iso image
mount -o loop file.iso /mnt/cdrom
# mount a Windows FAT32 file system
mount -t vfat /dev/hda5 /mnt/hda5
# mount a usb pen-drive or flash-drive
mount /dev/sda1 /mnt/usbdisk
# mount a windows network share
mount -t smbfs -o username=user,password=pass //WinClient/share /mnt/share
# unmount disk called hda2 - exit from mount point `/mnt/hda2` first
umount /dev/hda2
# run umount without writing the file /etc/mtab - useful when the file is read-only or the hard disk is full
umount -n /mnt/hda2
# Show all partitions registered on the system
cat /proc/partitions
# List mounted filesystems on the system (and align output)
mount | column -t
# Show info about disk sda
hdparm -i /dev/sda
# displays the characteristics of a hard-disk
hdparm -i /dev/hda
# perform test reading on a hard-disk
hdparm -tT /dev/sda
# Do a read speed test on disk sda
hdparm -tT /dev/sda
# Test for unreadable blocks on disk sda
badblocks -s /dev/sda
# How long has this disk (system) been powered on in total
smartctl -A /dev/sda | grep Power_On_Hours
# Check bad blocks on disk hda1
badblocks -v /dev/hda1
# Repair / check integrity of dos filesystems on disk hda1
dosfsck /dev/hda1
# Repair / check integrity of ext2 filesystem on disk hda1
e2fsck /dev/hda1
# Repair / check integrity of ext3 filesystem on disk hda1
e2fsck -j /dev/hda1
# Repair / check integrity of linux filesystem on disk hda1
fsck /dev/hda1
# Repair / check integrity of ext2 filesystem on disk hda1
fsck.ext2 /dev/hda1
# Repair / check integrity of ext3 filesystem on disk hda1
fsck.ext3 /dev/hda1
# Repair / check integrity of fat filesystem on disk hda1
fsck.vfat /dev/hda1
# Repair / check integrity of dos filesystem on disk hda1
fsck.msdos /dev/hda1
Backup the filesystem
# Note: only supported by ext2/ext3 filesystems
# Make a full backup of directory `/home`
dump -0aj -f /tmp/home0.bak /home
# Make a incremental backup of directory `/home`
dump -1aj -f /tmp/home0.bak /home
# Restoring a backup interactively
restore -if /tmp/home0.bak
Format a filesystem:
# Format a floppy disk
fdformat -n /dev/fd0
# Create a filesystem type linux ext2 on hda1 partition
mke2fs /dev/hda1
# Create a filesystem type linux ext3 (journal) on hda1 partition
mke2fs -j /dev/hda1
# Create a filesystem type linux on hda1 partition
mkfs /dev/hda1
# Create a FAT32 filesystem
mkfs -t vfat 32 -F /dev/hda1
Filesystem swap:
# Create a swap filesystem
mkswap /dev/hda3
# Activating a new swap partition
swapon /dev/hda3
# Activate two swap partitions
swapon /dev/hda2 /dev/hdb3
Disk space
See also FSlint
# Show files by size, biggest last
ls -lSr | more
# Show top disk users in current dir. See also dutop
du -s * | sort -k1,1rn | head
# Sort paths by easy to interpret disk usage
du -hs /home/* | sort -k1,1h
# Show size of the files and directories sorted by size
du -sk * | sort -rn
# Estimate space used by directory `dir1`
du -sh dir1
# Show free space on mounted filesystems aks disk usage
df -h
# Show free inodes on mounted filesystems
df -i
# Show disks partitions sizes and types (run as root)
fdisk -l
Clone disk
# Make a copy of a local hard disk on remote host via ssh
dd bs=1M if=/dev/hda | gzip | ssh user@ip_addr 'dd of=hda.gz'
# Copy content of the harddrive to a file
dd if=/dev/sda of=/tmp/file1
# Make a copy of MBR (Master Boot Record) to floppy
dd if=/dev/hda of=/dev/fd0 bs=512 count=1
# Restore MBR from a copy saved to floppy
dd if=/dev/fd0 of=/dev/hda bs=512 count=1
Aka duplicate disks
For supported partitions (on source drive), use Disk Utility
For not supported partitions
Clone exact disk (boot sector, all partitions & data, etc.) to an other drive (to a file, use a path to a file instead of /dev/sdX
, see this page for other options like split the image and dd
man page for partial copy etc.)
For disk use /dev/sdX
(or /dev/hdX
, or BSD /dev/diskX
or even /dev/rdiskX
for better speed)
sudo dd if=/dev/sdX of=/dev/sdY bs=64K conv=noerror,sync status=progress
Change bs
to find the optimal value. 64K seem a good value, but it's related to hardware and software (USB2.0 to USB2.0 RAID). Try to use the same value as the disk cache 8388608
size, or the block size 4096
.
With bs=64K
:
USB 2.0 to USB 2.0: 20 to 11 MB/s
SATA 3.0 (3.0 Gb/s) to SATA 3.0 (3.0 Gb/s): 130 MB/s
Check disk ID (here X and Y) in XXXXX
CDs
And DVDs, CD images
# Save copy of data cdrom
gzip < /dev/cdrom > cdrom.iso.gz
# Create cdrom image from contents of dir
mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz
# Mount the cdrom image at /mnt/dir (read only)
mount -o loop cdrom.iso /mnt/dir
# Clear a CDRW
cdrecord -v dev=/dev/cdrom blank=fast
# Burn cdrom image (use dev=ATAPI -scanbus to confirm dev)
gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom -
# Rip audio tracks from CD to wav files in current dir
cdparanoia -B
# Make audio CD from all wavs in current dir (see also cdrdao)
cdrecord -v dev=/dev/cdrom -audio -pad *.wav
# perform an md5sum on a device, like a CD
dd if=/dev/hdc | md5sum
# mount an ISO image
mount -o loop cd.iso /mnt/iso
Command operations
# Show full path to a binary / executable
which command
# Show location of a binary file, source or man
whereis command
# See how long a command takes (stopwatch). Ctrl-d to stop. See also sw
time command
# Show commands pertinent to string. See also threadsafe
# Display a list of commands that pertain to keywords of a program, useful when you know what your program does, but you don't know the name of the command
apropos somekeyword
# Find the location of a command
command -v git 2> /dev/null
Command help
Aka man, command documentation
# displays description of what a program does
whatis somekeyword
man mv
# find any related commands
man -k mv
# make a pdf of a manual page
man -t ascii | ps2pdf - > ascii.pdf
Section numbers of the man
manual:
executable programs or shell commands
system calls (functions provided by the kernel)
library calls (functions within program libraries)
special files (usually found in /dev)
file formats and conventions eg /etc/passwd
games
miscellaneous (including macro packages and conventions), e.g.
man(7)
,groff(7)
system administration commands (usually only for root)
kernel routines (non standard)
Some commands support also (or instead) an option that show the help, could be:
--help
-h
help
-?
/?
SSH
Aka Secure SHell
# Run command on $HOST as $USER (default command=shell)
ssh $USER@$HOST command
# Run GUI command on $HOSTNAME as $USER
ssh -f -Y $USER@$HOSTNAME xeyes
# Copy with permissions to $USER's home directory on $HOST
scp -p -r $USER@$HOST: file dir/
# Use faster crypto for local LAN. This might saturate GigE
scp -c arcfour $USER@$LANHOST: bigfile
# Forward connections to $HOSTNAME:8080 out to $HOST:80
ssh -g -L 8080:localhost:80 root@$HOST
# Forward connections from $HOST:1434 in to imap:143
ssh -R 1434:imap:143 root@$HOST
# Install public key for $USER@$HOST for password-less log in
ssh-copy-id $USER@$HOST
To disallow SSH remote root login, in /etc/ssh/sshd_config
update/add:
PermitRootLogin no
SSH Keyfile
# If `-f filename` is not defined, will generate 2 files (by default `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`)
ssh-keygen -t rsa -b 4096 -f ~/.ssh/user@host_rsa
ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/your_email@example.com_rsa
ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/your_email@example.com-hostname.ext_rsa
In ~/.ssh/config
:
# username@hostname
Host hostname
RSAAuthentication yes
IdentityFile ~/.ssh/username@hostname_rsa
User username
SSH Configurations - Tower Help - Connecting & Authenticating
Convert PDF 1.4
Force version
gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dColorConversionStrategy=/LeaveColorUnchanged -dDownsampleMonoImages=false -dDownsampleGrayImages=false -dDownsampleColorImages=false -dAutoFilterColorImages=false -dAutoFilterGrayImages=false -dColorImageFilter=/FlateEncode -dGrayImageFilter=/FlateEncode -o airbus_ar_2016_pdf1.4.pdf airbus_ar_2016.pdf
Email local spool
/var/spool/mail/$USER
mail -s "Hello" localusername </dev/null
UUID
# UUID (but violates the UUID specification)
od -x /dev/urandom | head -1 | awk '{OFS="-"; print $2$3,$4,$5,$6,$7$8$9}'
# UUID v4
od -x /dev/urandom | head -1 | awk '{OFS="-"; srand($6); sub(/./,"4",$5); sub(/./,substr("89ab",rand()*4,1),$6); print $2$3,$4,$5,$6,$7$8$9}'
Last updated
Was this helpful?