test-your-sysadmin-skills

My favorite Linux distribution:

• Arch Linux, which offers a nice minimalist base system on which one can build a custom operating system. The beauty of it too is that it has the Arch User Repository (AUR), which when combined with its official binary repositories allows it to probably have the largest repositories of any distribution. Its packaging process is also very simple, which means if one wants a package not in its official repositories or the AUR, it should be easy to make it for oneself.
• Linux Mint, which is also built from Ubuntu LTS releases, but features editions featuring a few different desktop environments, including Cinnamon, MATE and Xfce. Mint is quite polished and its aesthetics are rather appealing, I especially like its new icon theme, although I do quite dislike its GTK+ theme (too bland to my taste). I’ve also found a bug in its latest release Mint 19, that is getting quite irritating as I asked for with it over a fortnight ago on their forums and I have received no replies so far and it is a bug that makes my life on it more difficult.
• Kali Linux, is a Debian-based Linux distribution aimed at advanced Penetration Testing and Security Auditing. Kali contains several hundred tools which are geared towards various information security tasks, such as Penetration Testing, Security research, Computer Forensics and Reverse Engineering.

Useful resources:

• List of Linux distributions
• What is your favorite Linux distro and why?

GNU isn't really an OS. It's more of a set of rules or philosophies that govern free software, that at the same time gave birth to a bunch of tools while trying to create an OS. So GNU tools are basically open versions of tools that already existed, but were reimplemented to conform to principals of open software. GNU/Linux is a mesh of those tools and the Linux kernel to form a complete OS, but there are other GNUs, e.g. GNU/Hurd.

Unix and BSD are "older" implementations of POSIX that are various levels of "closed source". Unix is usually totally closed source, but there are as many flavors of Unix as there are Linux (if not more). BSD is not usually considered "open", but it was considered to be very open when it was released. Its licensing also allowed for commercial use with far fewer restrictions than the more "open" licenses of the time allowed.

Linux is the newest of the four. Strictly speaking, it's "just a kernel"; however, in general, it's thought of as a full OS when combined with GNU Tools and several other core components.

The main governing differences between these are their ideals. Unix, Linux, and BSD have different ideals that they implement. They are all POSIX, and are all basically interchangeable. They do solve some of the same problems in different ways. So other then ideals and how they choose to implement POSIX standards, there is little difference.

For more info I suggest your read a brief article on the creation of GNU, OSS, Linux, BSD, and UNIX. They will be slanted towards their individual ideas, but those articles should give you a better idea of the differences.

Useful resources:

• What is the difference between Unix, Linux, BSD and GNU? (original)
• The Great Debate: Is it Linux or GNU/Linux?

CLI is an acronym for Command Line Interface or Command Language Interpreter. The command line is one of the most powerful ways to control your system/computer.

In Unix like systems, CLI is the interface by which a user can type commands for the system to execute. The CLI is very powerful, but is not very error-tolerant.

The CLI allows you to do manipulations with your system’s internals and with code in a much more fine-tuned way. It offers greater flexibility and control than a GUI regardless of what OS is used. Many programs that you might want to use in your software that are hosted on say Github also require running some commands on the CLI in order to get them running.

My favorite tools

• screen - free terminal multiplexer, I can start a session and My terminals will be saved even when you connection is lost, so you can resume later or from home
• ssh - the most valuable over-all command to learn, I can use it to do some amazing things:
  • mount a file system over the internet with sshfs
  • forward commands: runs against a rsync server with no rsync deamon by starting one itself via ssh
  • run in batch files: I can redirect the output from the remote command and use it within local batch file
• vi/vim - is the most popular and powerful text editor, it's universal, it's work very fast, even on large files
• bash-completion - contains a number of predefined completion rules for shell

BASH is my favorite. It’s really a preferential kind of thing, where I love the syntax and it just "clicks" for me. The input/output redirection syntax (>>, << 2>&1, 2>, 1>, etc) is similar to C++ which makes it easier for me to recognize.

I also like the ZSH shell, because is much more customizable than BASH. It has the Oh-My-Zsh framework, powerful context based tab completion, pattern matching/globbing on steroids, loadable modules and more.

Useful resources:

• Comparison of command shells

• man [commandname] can be used to see a description of a command (ex.: man less, man cat)

• -h or --help some programs will implement printing instructions when passed this parameter (ex.: python -h and python --help)

• w - a lot of great information in there with the server uptime
• top - you can see all running processes, then order them by CPU, memory utilization and more
• netstat - to know on what port and IP your server is listening on and what processes are using those
• df - reports the amount of available disk space being used by file systems
• history - tell you what was previously run by the user you are currently connected to

Useful resources:

• First 5 Commands When I Connect on a Linux Server (original)

In the order of output:

-rwxrw-r--    1    root   root 2048    Jan 13 07:11 db.dump

• file permissions,
• number of links,
• owner name,
• owner group,
• file size,
• time of last modification,
• file/directory name

File permissions is displayed as following:

• first character is - or l or d, d indicates a directory, a - represents a file, l is a symlink (or soft link) - special type of file
• three sets of characters, three times, indicating permissions for owner, group and other:
  • r = readable
  • w = writable
  • x = executable

In your example -rwxrw-r--, this means the line displayed is:

For a summary of logged-in users, including each login of a username, the terminal users are attached to, the date/time they logged in, and possibly the computer from which they are making the connection, enter:

# It uses /var/run/utmp and /var/log/wtmp files to get the details.
who

For extensive information, including username, terminal, IP number of the source computer, the time the login began, any idle time, process CPU cycles, job CPU cycles, and the currently running command, enter:

# It uses /var/run/utmp, and their processes /proc.
w

Also important for displays a list of last logged in users, enter:

# It uses /var/log/wtmp.
last

Useful resources:

• 4 Ways to Identify Who is Logged-In on Your Linux System

The most significant advantage of executing the running process in the background is that you can do any other task simultaneously while other processes are running in the background. So, more processes can be completed in the background while you are working on different processes. It can be achieved by adding a special character & at the end of the command.

Generally applications that take too long to execute and doesn't require user interaction are sent to background so that we can continue our work in terminal.

For example if you want to download something in background, you can:

wget https://url-to-download.com/download.tar.gz &

When you run the above command you get the following output:

[1] 2203

Here 1 is the serial number of job and 2203 is PID of the job.

You can see the jobs running in background using the following command:

jobs

When you execute job in background it give you a PID of job, you can kill the job running in background using the following command:

kill PID

Replace the PID with the PID of the job. If you have only one job running you can bring it to foreground using:

fg

If you have multiple jobs running in background you can bring any job in foreground using:

fg %#

Replace the # with serial number of the job.

To be completed.

Running (everything) as root is bad because:

• Stupidity: nothing prevents you from making a careless mistake. If you try to change the system in any potentially harmful way, you need to use sudo, which ensures a pause (while you're entering the password) to ensure that you aren't about to make a mistake.

• Security: harder to hack if you don't know the admin user's login account. root means you already have one half of the working set of admin credentials.

• You don't really need it: if you need to run several commands as root, and you're annoyed by having to enter your password several times when sudo has expired, all you need to do is sudo -i and you are now root. Want to run some commands using pipes? Then use sudo sh -c "command1 | command2".

• You can always use it in the recovery console: the recovery console allows you to recover from a major mistake, or fix a problem caused by an app (which you still had to run as sudo). Ubuntu doesn't have a password for the root account in this case, but you can search online for changing that - this will make it harder for anyone that has physical access to your box to be able to do harm.

Useful resources:

• Why is it bad to log in as root? (original)
• What's wrong with always being root?
• Why you should avoid running applications as root

You'd use top/htop for both. Using free and vmstat command we can display the physical and virtual memory statistics respectively. With the help of sar command we see the CPU utilization & other stats (but sar isn't even installed in most systems).

Useful resources:

• How do I Find Out Linux CPU Utilization?
• 16 Linux server monitoring commands you really need to know

Linux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes.

These 3 numbers are not the numbers for the different CPUs. These numbers are mean values of the load number for a given period of time (of the last 1, 5 and 15 minutes).

Load average is usually described as "average length of run queue". So few CPU-consuming processes or threads can raise load average above 1. There is no problem if load average is less than total number of CPU cores. But if it gets higher than number of CPUs, this means some threads/processes will stay in queue, ready to run, but waiting for free CPU.

It is meant to give you an idea of the state of the system, averaged over several periods of time. Since it is averaged, it takes time for it to go back to 0 after a heavy load was placed on the system.

Some interpretations:

• if the averages are 0.0, then your system is idle
• if the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing
• if the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing
• if they are higher than your CPU count, then you might have a performance problem (it depends)

Useful resources:

• Linux Load Averages: Solving the Mystery (original)
• Linux load average - the definitive summary
• How CPU load averages work (and using them to triage webserver performance!)

The passwords are not stored anywhere on the system at all. What is stored in /etc/shadow are so called hashes of the passwords.

A hash of some text is created by performing a so called one way function on the text (password), thus creating a string to check against. By design it is "impossible" (computationally infeasible) to reverse that process.

Older Unix variants stored the encrypted passwords in /etc/passwd along with other information about each account.

Newer ones simply have a * in the relevant field in /etc/passwd and use /etc/shadow to store the password, in part to ensure nobody gets read access to the passwords when they only need the other stuff (shadow is usually protected more strongly than passwd).

For more info consult man crypt, man shadow, man passwd.

Useful resources:

• Where is my password stored on Linux?

To change all the directories e.g. to 755 (drwxr-xr-x):

find /opt/data -type d -exec chmod 755 {} \;

To change all the files e.g. to 644 (-rw-r--r--):

find /opt/data -type f -exec chmod 644 {} \;

Useful resources:

• How do I set chmod for a folder and all of its subfolders and files? (original)

It looks that at one point or another are overwriting the default PATH environment variable. The type of errors you have, indicates that PATH does not contain e.g. /bin, where the commands (including bash) reside.

One way to begin debugging your bash script or command would be to start a subshell with the -x option:

bash --login -x

This will show you every command, and its arguments, which is executed when starting that shell.

Also very helpful is show PATH variable values:

echo $PATH

If you run this:

PATH=/bin:/sbin:/usr/bin:/usr/sbin

most commands should start working - and then you can edit ~/.bash_profile instead of ~/.bashrc and fix whatever is resetting there. Default variable values for and other users is in file.

In most cases, you can stop a running script by using the CTRL + C keyboard combination. This sends an interrupt signal (SIGINT) to the script, which terminates its execution. If this does not work and the script is still running, you can try using the CTRL + \ combination, which sends a quit signal (SIGQUIT) to the script, which may terminate it immediately.

Alternatively, if you are using a terminal or command line interface, you can try using the kill command to send a signal to the script process. You can find the process ID (PID) of the script by using the ps or top command, and then use kill with the PID to stop the script.

In some cases, you may need to use the kill -9 command to force the script to stop, as the regular kill command may not work if the script is stuck or not responding. The -9 option sends a SIGKILL signal, which forces the process to stop immediately.

The grep utilities are a family of Unix tools, including egrep and fgrep.

grep searches file patterns. If you are looking for a specific pattern in the output of another command, grep highlights the relevant lines. Use this grep command for searching log files, specific processes, and more.

For match multiple strings:

grep -E "string1|string2" filename

or

grep -e "string1" -e "string2" filename

Useful resources:

• What is grep, and how do I use it? (original)

• head: to check the starting of a file.
• tail: to check the ending of the file. It is the reverse of head command.
• cat: used to view, create, concatenate the files.
• more: used to display the text in the terminal window in pager form.
• less: used to view the text in the backward direction and also provides single line movement.

Useful resources:

• Viewing text files from the shell prompt

• SIGHUP - is sent to a process when its controlling terminal is closed. It was originally designed to notify the process of a serial line drop (a hangup). Many daemons will reload their configuration files and reopen their logfiles instead of exiting when receiving this signal.
• SIGINT - is sent to a process by its controlling terminal when a user wishes to interrupt the process. This is typically initiated by pressing Ctrl+C, but on some systems, the "delete" character or "break" key can be used.
• SIGKILL - is sent to a process to cause it to terminate immediately (kill). In contrast to SIGTERM and SIGINT, this signal cannot be caught or ignored, and the receiving process cannot perform any clean-up upon receiving this signal.
• SIGTERM - is sent to a process to request its termination. Unlike the SIGKILL signal, it can be caught and interpreted or ignored by the process. This allows the process to perform nice termination releasing resources and saving state if appropriate. SIGINT is nearly identical to SIGTERM.

Useful resources:

• POSIX signals
• Introduction To Unix Signals Programming

In Unix and Unix-like operating systems, kill is a command used to send a signal to a process. By default, the message sent is the termination signal, which requests that the process exit. But kill is something of a misnomer; the signal sent may have nothing to do with process killing.

Useful resources:

• Mastering the "Kill" Command in Linux

rm only deletes the named files (and not directories). With -rf as you say:

• -r, -R, --recursive recursively deletes content of a directory, including hidden files and sub directories
• -f, --force ignore nonexistent files, never prompt

Useful resources:

• What is the difference between rm -r and rm -f?

To be completed.

# list of content
tar tf archive.tgz

# extract file
tar xf archive.tgz filename

Useful resources:

• List the contents of a tar or tar.gz file
• How to extract specific file(s) from tar.gz

If you want to execute each command only if the previous one succeeded, then combine them using the && operator:

cd /my_folder && rm *.jar && svn co path to repo && mvn compile package install

If one of the commands fails, then all other commands following it won't be executed.

If you want to execute all commands regardless of whether the previous ones failed or not, separate them with semicolons:

cd /my_folder; rm *.jar; svn co path to repo; mvn compile package install

In your case, I think you want the first case where execution of the next command depends on the success of the previous one.

You can also put all commands in a script and execute that instead:

#! /bin/sh
cd /my_folder \
&& rm *.jar \
&& svn co path to repo \
&& mvn compile package install

Useful resources:

• Execute combine multiple linux commands in one line (original)

chmod a+x /path/to/file

• a - for all users
• x - for execution permission
• r - for read permission
• w - for write permission

Useful resources:

• How to Set File Permissions Using chmod
• What does "chmod +x your_file_name" do and how do I use it?

To sync the contents of dir1 to dir2 on the same system, type:

rsync -av --progress --delete dir1/ dir2

• -a, --archive - archive mode
• --delete - delete extraneous files from dest dirs
• -v, --verbose - verbose mode (increase verbosity)
• --progress - show progress during transfer

Useful resources:

• How can I sync two local directories? (original
• Synchronizing folders with rsync

• manually backup of a file before editing (with brace expansion like this: cp filename{,.orig})
• manual copy of the directory structure where file is stored (e.g. cp, rsync or tar)
• make a backup of original file in your editor (e.g. set rules in your editor configuration file)
• the best solution is to use git (or any other version control) to keep track of configuration files (e.g. etckeeper for /etc directory)

Useful resources:

• Backup file with .bak before filename extension
• Is it a good idea to use git for configuration file version controlling?

find / -type f -size +20M

Useful resources:

• How can I find files that are bigger/smaller than x bytes?

sudo is in most modern Linux distributions where (but not always) the root user is disabled and has no password set. Therefore you cannot switch to the root user with su (you can try). You have to call sudo with root privileges: sudo su.

su just switches the user, providing a normal shell with an environment nearly the same as with the old user.

su - invokes a login shell after switching the user. A login shell resets most environment variables, providing a clean base.

Useful resources:

• su vs sudo -s vs sudo -i vs sudo bash
• Why do we use su - and not just su? (original)

find / -mmin -60 -type f

Useful resources:

• Get all files modified in last 30 days in a directory (orignal)

They are essential to investigate issues on the system. Log management is absolutely critical for IT security.

Servers, firewalls, and other IT equipment keep log files that record important events and transactions. This information can provide important clues about hostile activity affecting your network from within and without. Log data can also provide information for identifying and troubleshooting equipment problems including configuration problems and hardware failure.

It’s your server’s record of who’s come to your site, when, and exactly what they looked at. It’s incredibly detailed, showing:

• where folks came from
• what browser they were using
• exactly which files they looked at
• how long it took to load each file
• and a whole bunch of other nerdy stuff

Factors to consider:

• legal requirements for retention or destruction
• company policies for retention and destruction
• how long the logs are useful
• what questions you're hoping to answer from the logs
• how much space they take up

By collecting and analyzing logs, you can understand what transpires within your network. Each log file contains many pieces of information that can be invaluable, especially if you know how to read them and analyze them.

Useful resources:

• How long do you keep log files?

An incremental backup is a type of backup that only copies files that have changed since the previous backup.

Useful resources:

• What Is Incremental Backup?

A RAID (Redundant Array of Inexpensive Disks) is a technology that is used to increase the performance and/or reliability of data storage.

• RAID0: Also known as disk striping, is a technique that breaks up a file and spreads the data across all the disk drives in a RAID group. There are no safeguards against failure
• RAID1: A popular disk subsystem that increases safety by writing the same data on two drives. Called "mirroring," RAID 1 does not increase write performance, but read performance may equal up to the sum of each disks' performance. However, if one drive fails, the second drive is used, and the failed drive is manually replaced. After replacement, the RAID controller duplicates the contents of the working drive onto the new one
• RAID5: It is disk subsystem that increases safety by computing parity data and increasing speed by interleaving data across three or more drives (striping). Upon failure of a single drive, subsequent reads can be calculated from the distributed parity such that no data is lost
• RAID6: RAID 6 extends RAID 5 by adding another parity block. It requires a minimum of four disks and can continue to execute read and write of any two concurrent disk failures. RAID 6 does not have a performance penalty for read operations, but it does have a performance penalty on write operations because of the overhead associated with parity calculations
• RAID10: Also known as RAID 1+0, is a RAID configuration that combines disk mirroring and disk striping to protect data. It requires a minimum of four disks, and stripes data across mirrored pairs. As long as one disk in each mirrored pair is functional, data can be retrieved. If two disks in the same mirrored pair fail, all data will be lost because there is no parity in the striped sets

Useful resources:

• RAID

useradd -m -g initial_group username

-g/--gid: defines the group name or number of the user's initial login group. If specified, the group name must exist; if a group number is provided, it must refer to an already existing group.

If not specified, the behaviour of useradd will depend on the USERGROUPS_ENAB variable contained in /etc/login.defs. The default behaviour (USERGROUPS_ENAB yes) is to create a group with the same name as the username, with GID equal to UID.

Useful resources:

• How can I change a user's default group in Linux?

To be completed.

• Protecting Hardware
• Proper Cooling
• Organized Workspace
• Better Power Management
• Cleaner Environment

Useful resources:

• 5 Reasons to Rackmount Your PC

To be completed.

The most important things to understand about the OSI (or any other) model are:

• we can divide up the protocols into layers
• layers provide encapsulation
• layers provide abstraction
• layers decouple functions from others

Useful resources:

• OSI Model and Networking Protocols Relationship

VLANs and subnets solve different problems. VLANs work at Layer 2, thereby altering broadcast domains (for instance). Whereas subnets are Layer 3 in the current context.

Subnet - is a range of IP addresses determined by part of an address (often called the network address) and a subnet mask (netmask). For example, if the netmask is 255.255.255.0 (or /24 for short), and the network address is 192.168.10.0, then that defines a range of IP addresses 192.168.10.0 through 192.168.10.255. Shorthand for writing that is 192.168.10.0/24.

VLAN - a good way to think of this is "switch partitioning." Let's say you have an 8 port switch that is VLAN-able. You can assign 4 ports to one VLAN (say VLAN 1) and 4 ports to another VLAN (say VLAN 2). VLAN 1 won't see any of VLAN 2's traffic and vice versa, logically, you now have two separate switches. Normally on a switch, if the switch hasn't seen a MAC address it will "flood" the traffic to all other ports. prevent this.

POP and IMAP are both protocols for retrieving messages from a mail server to a mail client.

POP (Post Office Protocol) uses a one way push from mail server to client. By default this will send messages to the POP mail client and remove them from the mail server, though it is possible to configure the mail server to retain all messages. Any actions you take on the message in your mail client (labeling, deleting, moving to a folder) will not be reflected on the mail server, and thus inaccessible to other mail clients pulling from the mail server. POP uses little storage space on the mail server and can be seen as more secure since messages only exist on one mail client instead of the mail server and multiple clients.

IMAP (Internet Message Access Protocol) uses two way communication between mail server and client. Deleting or labeling a message in your mail client configured with IMAP will also delete or label the message on the mail server. IMAP allows for a similar experience when accessing mail across different clients or devices since messages can existing in the same state across multiple devices. IMAP can also save disk space on the mail client by selectively syncing messages, deleting older messages from the mail client since it can sync them from the mail server later as needed.

Choose IMAP if you need to access messages across multiple devices and you want to save disk space on your client device. Choose POP if you want to save disk space on your mail server, only access messages from one client device, and ensure that messages do not exist on multiple systems.

Using the commands netstat -nr, route -n or ip route show we can see the default route and routing tables.

Useful resources:

• How to check routes (routing table) in linux
• FreeBSD Set a Default Route/Gateway

Well, the most likely difference is that you still have to do an actual lookup of localhost somewhere.

If you use 127.0.0.1, then (intelligent) software will just turn that directly into an IP address and use it. Some implementations of gethostbyname will detect the dotted format (and presumably the equivalent IPv6 format) and not do a lookup at all.

Otherwise, the name has to be resolved. And there's no guarantee that your hosts file will actually be used for that resolution (first, or at all) so localhost may become a totally different IP address.

By that I mean that, on some systems, a local hosts file can be bypassed. The host.conf file controls this on Linux (and many other Unices).

If you use a Unix domain socket it'll be slightly faster than using TCP/IP (because of the less overhead you have). Windows is using TCP/IP as a default, whereas Linux tries to use a Unix Domain Socket if you choose localhost and TCP/IP if you take 127.0.0.1.

Useful resources:

• What is the difference between 127.0.0.1 and localhost?
• localhost vs. 127.0.0.1

ping uses ICMP, specifically ICMP echo request and ICMP echo reply packets. There is no 'port' associated with ICMP. Ports are associated with the two IP transport layer protocols, TCP and UDP. ICMP, TCP, and UDP are "siblings"; they are not based on each other, but are three separate protocols that run on top of IP.

ICMP packets are identified by the 'protocol' field in the IP datagram header. ICMP does not use either UDP or TCP communications services, it uses raw IP communications services. This means that the ICMP message is carried directly in an IP datagram data field. raw comes from how this is implemented in software, to create and send an ICMP message, one opens a raw socket, builds a buffer containing the ICMP message, and then writes the buffer containing the message to the raw socket.

The IP protocol value for ICMP is 1. The protocol field is part of the IP header and identifies what is in the data portion of the IP datagram.

However, you could use nmap to see whether ports are open or not:

nmap -p 80 example.com

Useful resources:

• Ping Port Number
• Is it possible to ping an address:port?

To troubleshoot communication problems between servers, it is better to ideally follow the TCP/IP stack:

1. Application Layer: are the services up and running on both servers? Are they correctly configured (eg. bind the correct IP and correct port)? Do application and system logs show meaningful errors?

2. Transport Layer: are the ports used by the application open (try telnet!)? Is it possible to ping the server?

3. Network Layer: is there a firewall on the network or on the OS correctly configured? Is the IP stack correctly configured (IP, routes, dns, etc.)? Are switches and routers working (check the ARP table!)?

4. Physical Layer: are the servers connected to a network? Are packets being lost?

To be completed.

Examples for resolve IP address to domain name:

# with host command:
host domain.com 8.8.8.8

# with dig command:
dig @9.9.9.9 google.com

# with nslookup command:
nslookup domain.com 8.8.8.8

You can (sometimes) resolve an IP Address back to a hostname. IP Address can be stored against a PTR record. You can then do:

dig A <hostname>

To lookup the IPv4 address for a host, or:

dig AAAA <hostname>

To lookup the IPv6 address for a host, or:

dig PTR ZZZ.YYY.XXX.WWW.in-addr.arpa.

To lookup the hostname for IPv4 address WWW.XXX.YYY.ZZZ (note the octets are reversed), or:

dig PTR b.a.9.8.7.6.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa.

Useful resources:

• How can I resolve a hostname to an IP address in a Bash script?
• How To Resolve IP Addresses To Domain Names?

Modern operating systems have turned off all potentially insecure services by default. On the other hand, some vendors of network devices still allow to establish communication using the telnet protocol.

Telnet uses most insecure method for communication. It sends data across the network in plain text format and anybody can easily find out the password using the network tool.

In the case of Telnet, these include the passing of login credentials in plain text, which means anyone running a sniffer on your network can find the information he needs to take control of a device in a few seconds by eavesdropping on a Telnet login session.

Useful resources:

• Telnet and SSH as a secure alternative
• How to telnet to an IP address on a specific port?

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP.

Useful resources:

• What is the difference between curl and wget? (original)

SSH stands for Secure Shell. It is a protocol that lets you drop from a server "A" into a shell session to a server "B". It allows you interact with your server "B".

An SSH connection to be established, the remote machine (server A) must be running a piece of software called an SSH daemon and the user's computer (server B) must have an SSH client.

The SSH daemon and SSH client listen for connections on a specific network port (default 22), authenticates connection requests, and spawns the appropriate environment if the user provides the correct credentials.

Useful resources:

• Understanding the SSH Encryption and Connection Process

An SSH key is an access credential in the SSH protocol. Its function is similar to that of user names and passwords, but the keys are primarily used for automated processes and for implementing single sign-on by system administrators and power users.

Instead of requiring a user's password, it is possible to confirm the client's identity by using asymmetric cryptography algorithms, with public and private keys.

If your SSH service only allows public-key authentication, an attacker needs a copy of a private key corresponding to a public key stored on the server.

If your SSH service allows password based authentication, then your Internet connected SSH server will be hammered day and night by bot-nets trying to guess user-names and passwords. The bot net needs no information, it can just try popular names and popular passwords. Apart from anything else this clogs your logs.

Useful resources:

• Key-Based Authentication (Public Key Authentication)
• SSH password vs. key authentication

Packet filtering is a firewall technique used to control network access by monitoring outgoing and incoming packets and allowing them to pass or halt based on the source and destination Internet Protocol (IP) addresses, protocols and ports.

Packet filtering is appropriate where there are modest security requirements. The internal (private) networks of many organizations are not highly segmented. Highly sophisticated firewalls are not necessary for isolating one part of the organization from another.

However it is prudent to provide some sort of protection of the production network from a lab or experimental network. A packet filtering device is a very appropriate measure for providing isolation of one subnet from another.

Operating at the network layer and transport layer of the TCP/IP protocol stack, every packet is examined as it enters the protocol stack. The network and transport headers are examined closely for the following information:

• protocol (IP header, network layer) - in the IP header, byte 9 (remember the byte count begins with zero) identifies the protocol of the packet. Most filter devices have the capability to differentiate between TCP, UPD, and ICMP.
• source address (IP header, network layer) - the source address is the 32-bit IP address of the host which created the packet.
• destination address (IP header, network layer) - the destination address is the 32-bit IP address of the host the packet is destined for.
• source port (TCP or UDP header, transport layer) - each end of a TCP or UDP network connection is bound to a port. TCP ports are separate and distinct from UDP ports. Ports numbered below 1024 are reserved – they have a specifically defined use. Ports numbered above 1024 (inclusive) are known as ephemeral ports. They can be used however a vendor chooses. For a list of "well known" ports, refer to RFP1700. The source port is a pseudo-randomly assigned ephemeral port number. Thus it is often not very useful to filter on the source port.
• destination port (TCP or UDP header, transport layer) - the destination port number indicates a port that the packet is sent to. Each service on the destination host listens to a port. Some well-known ports that might be filtered are 20/TCP and 21/TCP - ftp connection/data, 23/TCP - telnet, 80/TCP - http, and 53/TCP - DNS zone transfers.
• connection status (TCP header, transport layer) - the connection status tells whether the packet is the first packet of the network session. The ACK bit in the TCP header is set to “false” or 0 if this is the first packet in the session. It is simple to disallow a host from establishing a connection by rejecting or discarding any packets which have the ACK bit set to "false" or 0.

Hide the topology and characteristics of your back-end servers

The reverse proxy server can hide the presence and characteristics of the origin server. It acts as an intermediate between internet cloud and web server. It is good for security reason especially when you are using web hosting services.

Allows transparent maintenance of backend servers

Changes you make to servers running behind a reverse proxy are going to be completely transparent to your end users.

Load Balancing

The reverse proxy will then enforce a load balancing algorithm like round robin, weighted round robin, least connections, weighted least connections, or random, to distribute the load among the servers in the cluster.

When a server goes down, the system will automatically failover to the next server up and users can continue with their secure file transfer activities.

SSL offloading/termination

Handles incoming HTTPS connections, decrypting the requests and passing unencrypted requests on to the web servers.

IP masking

Using a single ip but different URLs to route to different back end servers.

Useful resources:

• The Benefits of a Reverse Proxy

Router describes the general technical function (layer-3 forwarding) or a hardware device intended for that purpose, while gateway describes the function for the local segment (providing connectivity to elsewhere). You could also state that "you set up a router as gateway". Another term is hop which describes the forwarding in between subnets.

The term default gateway is used to mean the router on your LAN which has the responsibility of being the first point of contact for traffic to computers outside the LAN.

It's just a matter of perspective, the device is the same.

Useful resources:

• Difference between router and gateway (orignal)

DNS records are basically mapping files that tell the DNS server which IP address each domain is associated with, and how to handle requests sent to each domain. Some DNS records syntax that are commonly used in nearly all DNS record configurations are A, AAAA, CNAME, MX, PTR, NS, SOA, SRV, TXT, and NAPTR.

• SOA - A Start Of Authority
• A - Address Mapping records
• AAAA - IP Version 6 Address records
• CNAME - Canonical Name records
• MX - Mail exchanger record
• NS - Name Server records
• PTR - Reverse-lookup Pointer records

Useful resources:

• List of DNS record types

The OSI model explains why it doesn't make sense to make routing, a layer 3 concept, decisions based on a physical, layer 2, mechanism.

Modern networking is broken into many different layers to accomplish your end to end communication. Your network card (what is addressed by the mac address - physical address) needs to only be responsible for communicating with peers on it's physical network.

The communication that you are allowed to accomplish with your MAC address is going to be limited to other devices that reside within physical contact to your machine. On the internet, for example, you are not physically connected to each machine. That's why we make use of TCP/IP (a layer 3, logical address) mechanism when we need to communicate with a machine that we are not physically connected to.

IP is an arbitrary numbering scheme imposed in a hierarchical fashion on a group of computers to logically distinguish them as a group (that's what a subnet is). Sending messages between those groups is done by routing tables, themselves divided into multiple levels so that we don't have to keep track of every single subnet.

It's also pretty easy to relate this to another pair of systems. You have a State Issued ID Number, why would you need a mailing address if that ID number is already unique to just you? You need the mailing address because it's an arbitrary system that describes where the unique destination for communications to you should go.

On the other hand, the distribution of MAC addresses across the network is random and completely unrelated to topology. Routes grouping would be impossible, every router would need to keep track of routes for every single device that relays traffic trough it. That is what layer 2 switches do, and that does not scale well beyond a certain number of hosts.

Useful resources:

• Why couldn't MAC addresses be used instead of IPv4|6 for networking? (original)

Whether you have a standard /24 VLAN for end users, a /30 for point-to-point links, or something in between and subnet that must contain up to 30 devices works out to be a /27 - or a subnet mask of 255.255.255.224.

Useful resources:

• How do you calculate the prefix, network, subnet, and host numbers?
• The slash after an IP Address - CIDR Notation
• Why are there 3 ranges of private IPv4 addresses?
• IP Calculator

• 1xx - Informational responses - communicates transfer protocol-level information
• 2xx - Success - indicates that the client’s request was accepted successfully
• 3xx - Redirection - indicates that the client must take some additional action in order to complete their request
• 4xx - Client side error - this category of error status codes points the finger at clients
• 5xx - Server side error - the server takes responsibility for these error status codes

Useful resources:

• HTTP Status Codes

DevOps is a cohesive team that engages in both Development and Operations tasks, or it's individual Operations and Development teams that work very closely together. It's more of a "way" of working collaboratively with other departments to achieve common goals.

It is a system that records changes to a file or set of files over time so that you can recall specific versions later. Version control systems consist of a central shared repository where teammates can commit changes to a file or set of file. Then you can mention the uses of version control.

Version control allows you to:

• revert files back to a previous state
• revert the entire project back to a previous state
• compare changes over time
• see who last modified something that might be causing a problem
• who introduced an issue and when

The seven rules of a great commit message:

• separate subject from body with a blank line
• limit the subject line to 50 characters
• capitalize the subject line
• do not end the subject line with a period
• use the imperative mood in the subject line
• wrap the body at 72 characters
• use the body to explain what and why vs. how

Useful resources:

• Getting Started - About Version Control (original)

• git init - create a new local repository
• git commit -m "message" - commit changes to head
• git status - list the files you've added with git add and also commit any files you've changed since then
• git push origin master - send changes to the master branch of your remote repository

• clone repository
• deploy stage (QA)
• testing environment (QA)
• deploy stage (PROD)

• docker ps - show running containers
• docker ps -a - show all containers
• docker images - show docker images
• docker logs <container-id|container-name> - get logs from container
• docker network ls - show all docker networks
• docker volumes ls - show all docker volumes
• docker exec -it <container-id|container-name> bash - execute bash in container with interactive shell

Security misconfiguration is a vulnerability when a device/application/network is configured in a way which can be exploited by an attacker to take advantage of it. This can be as simple as leaving the default username/password unchanged or too simple for device accounts etc.

To be completed.

To be completed.

BIOS: Full form of BIOS is Basic Input or Output System that performs integrity checks and it will search and load and then it will execute the bootloader.

Bootloader: Since the earlier phases are not specific to the operating system, the BIOS-based boot process for x86 and x86-64 architectures is considered to start when the master boot record (MBR) code is executed in real mode and the first-stage boot loader is loaded. In UEFI systems, a payload, such as the Linux kernel, can be executed directly. Thus no boot loader is necessary. Some popular bootloaders: GRUB, Syslinux/Isolinux or Lilo.

Kernel: The kernel in Linux handles all operating system processes, such as memory management, task scheduling, I/O, interprocess communication, and overall system control. This is loaded in two stages - in the first stage, the kernel (as a compressed image file) is loaded into memory and decompressed, and a few fundamental functions such as basic memory management are set up.

Init: Is the parent of all processes on the system, it is executed by the kernel and is responsible for starting all other processes.

• SysV init - init's job is "to get everything running the way it should be once the kernel is fully running. Essentially it establishes and operates the entire user space. This includes checking and mounting file systems, starting up necessary user services, and ultimately switching to a user-environment when system startup is completed.
• systemd - the developers of systemd aimed to replace the Linux init system inherited from Unix System V. Like init, systemd is a daemon that manages other daemons. All daemons, including systemd, are background processes. Systemd is the first daemon to start (during booting) and the last daemon to terminate (during shutdown).
• runinit - runinit is an init scheme for Unix-like operating systems that initializes, supervises, and ends processes throughout the operating system. It is a reimplementation of the daemontools process supervision toolkit that runs on the Linux, Mac OS X, *BSD, and Solaris operating systems.

Useful resources:

To be completed.

The problem with a load of 1.00 is that you have no headroom. In practice, many sysadmins will draw a line at 0.70.

The "Need to Look into it" Rule of Thumb: 0.70 If your load average is staying above > 0.70, it's time to investigate before things get worse.

The "Fix this now" Rule of Thumb: 1.00. If your load average stays above 1.00, find the problem and fix it now. Otherwise, you're going to get woken up in the middle of the night, and it's not going to be fun.

Rule of Thumb: 5.0. If your load average is above 5.00, you could be in serious trouble, your box is either hanging or slowing way down, and this will (inexplicably) happen in the worst possible time like in the middle of the night or when you're presenting at a conference. Don't let it get there.

Useful resources:

• Proper way of interpreting system load on a 4 core 8 thread processor
• Understanding Linux CPU Load - when should you be worried?

The real user ID is who you really are (the user who owns the process), and the effective user ID is what the operating system looks at to make a decision whether or not you are allowed to do something (most of the time, there are some exceptions).

When you log in, the login shell sets both the real and effective user ID to the same value (your real user ID) as supplied by the password file.

If, for instance, you execute setuid, and besides running as another user (e.g. root) the setuid program is also supposed to do something on your behalf.

After executing setuid, it will have your real ID (since you're the process owner) and the effective user id of the file owner (for example root) since it is setuid.

Let's use the case of passwd:

-rwsr-xr-x 1 root root 45396 may 25  2012 /usr/bin/passwd

When user2 wants to change their password, they execute /usr/bin/passwd.

The RUID will be user2 but the EUID of that process will be root.

user2 can use only passwd to change their own password, because internally passwd checks the RUID and, if it is not root, its actions will be limited to real user's password.

It's necessary that the EUID becomes root in the case of passwd because the process needs to write to /etc/passwd and/or /etc/shadow.

Useful resources:

• Difference between Real User ID, Effective User ID and Saved User ID? (original)

Using logrotate is the usual way of dealing with logfiles. But instead of adding content to /etc/logrotate.conf you should add your own job to /etc/logrotate.d/, otherwise you would have to look at more diffs of configuration files during release upgrades.

If it's actively being written to you don't really have much you can do by way of truncate. Your only options are to truncate the file:

: >/var/log/massive-logfile

It's very helpful, because it's truncate the file without disrupting the processes.

Useful resources:

• How to Use logrotate to Manage Log Files
• System logging

To be completed.

Useful resources:

• Linux Processes

To be completed.

Useful resources:

• top explained visually
• htop Explained Visually
• Explanation of everything you can see in htop/top on Linux

top works reasonably well, as long as you look at the right numbers.

• M Sorts by current resident memory usage
• T Sorts by total ( or cummulative) CPU usage
• P Sorts by current CPU usage (this is the default refresh)
• ? Displays a usage summary for all top commands

This is very important information to obtain when problem solving why a computer process is running slowly and making decisions on what processes to kill/software to uninstall.

Useful resources:

• How to find the process(es) which are hogging the machine

By using Infrastructure as a Code approach, there are multiple good ways:

1. Configuration Synchronization Change Management Model:

There are Configuration Management Tools (Ansible, Chef, Puppet, Saltstack, ...), that can be used to automatically update ntpd service on all servers. To keep systems stable, system packages on servers are usually auto-updated with only security updates. Major or minor versions of packages are usually version locked in configuration definitions to prevent misconfiguration of the service. Change is then deployed by changing ntpd version in configuration definition.

With this approach, it is important to be careful when deploying changes into infrastructure massively. The pipeline of deployment should include Unit, Integration and System tests, and eventually be first deployed into Staging environment to prove configuration. If tests prove configuration correctness, deployment should be done by incremental rollout with ability to rollback in case of errors or failure.

2. Immutable Servers Model:

In Immutable Server model, whole unit (server, container) is replaced by new updated image rather than making changes to running server (this eliminates configuration drift). With this approach you usually build server image with tools like Packer or Docker with Dockerfile. This image is then tested and deployed similarly as in option above (1.), but now using techniques such as Canary Release, which also has ability to incremental rollout and rollback.

Useful resources:

• Infrastructure as a Code - Chapter 8: Patterns for Updating and Changing Servers

To be completed.

Your console has two types of messages:

• generated by the kernel (via printk)
• generated by userspace (usually your init system)

Kernel messages are always stored in the kmsg buffer, visible via dmesg command. They're also often copied to your syslog. This also applies to userspace messages written to /dev/kmsg, but those are fairly rare.

Meanwhile, when userspace writes its fancy boot status text to /dev/console or /dev/tty1, it's not stored anywhere at all. It just goes to the screen and that's it.

dmesg is used to review boot messages contained in the kernel ring buffer. A ring buffer is a buffer of fixed size for which any new data added to it overwrites the oldest data in it.

It shows operations once the boot process has completed, such as command line options passed to the kernel; hardware components detected, events when a new USB device is added, or errors like NIC (Network Interface Card) failure and the drivers report no link activity detected on the network and so much more.

If system logging is done via the journal component you should use journalctl. It shows messages include kernel and boot messages; messages from syslog or various services.

Boot issues/errors calls for a system administrator to look into certain important files in conjunction with particular commands (handled differently by different versions of Linux):

• /var/log/boot.log - system boot log, it contains all that unfolded during the system boot

Swap space is a restricted amount of physical memory that is allocated for use by the operating system when available memory has been fully utilized. It is memory management that involves swapping sections of memory to and from physical storage.

If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory.

Workload increases your RAM demand. You are running a workload that requires more memory. Usage of the entire swap indicates that. Also, changing swappiness to 1 might not be a wise decision. Setting swappiness to 1 does not indicate that swapping will not be done. It just indicates how aggressive kernel will be in respect of swapping, it does not eliminate swapping. Swapping will happen if needs to be done.

• Increasing the size of the swap space - firstly, you'd have increased disk use. If your disks aren't fast enough to keep up, then your system might end up thrashing, and you'd experience slowdowns as data is swapped in and out of memory. This would result in a bottleneck.

• Adding more RAM - the real solution is to add more memory. There's no substitute for RAM, and if you have enough memory, you'll swap less.

For monitoring swap space usage:

• cat /proc/swaps - to see total and used swap size
• grep SwapTotal /proc/meminfo - to show total swap space
• free - to display the amount of free and used system memory (also swap)

On Linux and other Unix-like operating systems, new files are created with a default set of permissions. Specifically, a new file's permissions may be restricted in a specific way by applying a permissions "mask" called the umask. The umask command is used to set this mask, or to show you its current value.

Permanently change (set e.g. umask 02):

• ~/.profile
• ~/.bashrc
• ~/.zshrc
• ~/.cshrc

Useful resources:

• What is Umask and How To Setup Default umask Under Linux?

Underneath the file system files are represented by inodes (or is it multiple inodes not sure)

• a file in the file system is basically a link to an inode
• a hard link then just creates another file with a link to the same underlying inode

When you delete a file it removes one link to the underlying inode. The inode is only deleted (or deletable/over-writable) when all links to the inode have been deleted.

• a symbolic link is a link to another name in the file system

Once a hard link has been made the link is to the inode. deleting renaming or moving the original file will not affect the hard link as it links to the underlying inode. Any changes to the data on the inode is reflected in all files that refer to that inode.

Note: Hard links are only valid within the same file system. Symbolic links can span file systems as they are simply the name of another file.

Differences:

• Hardlink cannot be created for directories. Hard link can only be created for a file
• Softlink also termed a symbolic links or symlinks can link to a directory

Useful resources:

• What is the difference between a hard link and a symbolic link?

This is probably one of my most irksome things that people mess up all the time. The SUID/GUID bit and the sticky-bit are 2 completely different things.

If you do a man chmod you can read about the SUID and sticky-bits.

SUID/GUID

What the above man page is trying to say is that the position that the x bit takes in the rwxrwxrwx for the user octal (1st group of rwx) and the group octal (2nd group of rwx) can take an additional state where the x becomes an s. When this occurs this file when executed (if it's a program and not just a shell script) will run with the permissions of the owner or the group of the file.

So if the file is owned by root and the SUID bit is turned on, the program will run as root. Even if you execute it as a regular user. The same thing applies to the GUID bit.

Examples:

no suid/guid - just the bits rwxr-xr-x are set.

ls -lt b.pl
-rwxr-xr-x 1 root root 179 Jan  9 01:01 b.pl

suid & user's executable bit enabled (lowercase s) - the bits rwsr-x-r-x are set.

chmod u+s b.pl
ls -lt b.pl
-rwsr-xr-x 1 root root 179 Jan  9 01:01 b.pl

suid enabled & executable bit disabled (uppercase S) - the bits rwSr-xr-x are set.

LC_ALL is the environment variable that overrides all the other localisation settings. This sets all LC_ type variables at once to a specified locale.

The main reason to set LC_ALL=C before command is that fine to simply get English output (general change the locale used by the command).

On the other hand, also important is to increase the speed of command execution with LC_ALL=C e.g. grep or fgrep. Using the LC_ALL=C locale increased our performance and brought command execution time down.

For example, if you set LC_ALL=en_US.utf8 your system opened multiple files from the /usr/lib/locale directory. For LC_ALL=C a minimum amount of open and read operations is performed.

If you want to restore all your normal (original) locale settings for the session:

To be completed.

1) Main requirements - remember about this

• which users have access to the app filesystem
• permissions for web servers, e.g. Apache and app servers e.g. uwsgi
• permissions for specific directories like a uploads, cache and main app directory like a /var/www/app01/html
• correct umask value for users and suid/sgid (only for specific situations)
• permissions for all future files and directories
• permissions for cron jobs and scripts

2) Application directories

/var/www contains a directory for each website (isolation of the apps), e.g. /var/www/app01, /var/www/app02

mkdir /var/www/{app01,app02}

3) Application owner and group

Each application has a designated owner (e.g. u01-prod, u02-prod) and group (e.g. g01-prod, g02-prod) which are set as the owner of all files and directories in the website's directory:

chown -R u01-prod:g01-prod /var/www/app01
chown -R u02-prod:g02-prod /var/www/app02

To be completed.

Useful resources:

• What differences it will make, if i use “telinit 6” instead of “reboot” command to restart my computer?

Restart the system, type boot -s at the Boot: prompt to enter single-user mode.

At the question about the shell to use, hit Enter which will display a # prompt.

Enter mount -urw / to remount the root file system read/write, then run mount -a to remount all the file systems.

Run passwd root to change the root password then run exit to continue booting.

Single user mode should basically let you log in with root access & change just about anything. For example, you might use single-user mode when you are restoring a damaged master database or a system database, or when you are changing server configuration options (e.g. password recovery).

Useful resources:

• FreeBSD Reset or Recover Root Password
• Single User Mode Definition

For example:

# cat  >filename ... - overwrite file
# cat >>filename ... - append to file
cat > filename << __EOF__
data
__EOF__

To set the kernel parameters in Unix-like, first edit the file /etc/sysctl.conf after making the changes save the file and run the command sysctl -p, this command will make the changes permanently without rebooting the machine.

Useful resources:

• How to Change Kernel Runtime Parameters in a Persistent and Non-Persistent Way

/proc is a virtual file system that provides detailed information about kernel, hardware and running processes.

Since /proc contains virtual files, it is called virtual file system. These virtual files have unique qualities. Most of them are listed as zero bytes in size.

Virtual files such as /proc/interrupts, /proc/meminfo, /proc/mounts and /proc/partitions provide an up-to-the-moment glimpse of the system’s hardware. Others: /proc/filesystems file and the /proc/sys/ directory provide system configuration information and interfaces.

Useful resources:

• Linux Filesystem Hierarchy - /proc

To be completed.

There are three types of journaling available in ext3/ext4 file systems:

• Journal - metadata and content are saved in the journal
• Ordered - only metadata is saved in the journal. Metadata are journaled only after writing the content to disk. This is the default
• Writeback - only metadata is saved in the journal. Metadata might be journaled either before or after the content is written to the disk

An inode is a data structure on a filesystem on Linux and other Unix-like operating systems that stores all the information about a file except its name and its actual data. A data structure is a way of storing data so that it can be used efficiently.

A Unix file is stored in two different parts of the disk - the data blocks and the inodes. I won't get into superblocks and other esoteric information. The data blocks contain the "contents" of the file. The information about the file is stored elsewhere - in the inode.

A file's inode number can easily be found by using the ls command, which by default lists the objects (i.e. files, links and directories) in the current directory (i.e. the directory in which the user is currently working), with its -i option. Thus, for example, the following will show the name of each object in the current directory together with its inode number:

ls -i

df's -i option instructs it to supply information about inodes on each filesystem rather than about available space. Specifically, it tells df to return for each mounted filesystem the total number of inodes, the number of free inodes, the number of used inodes and the percentage of inodes used. This option can be used together with the -h option as follows to make the output easier to read:

df -hi

Finding files by inodes

If you know the inode, you can find it using the find command:

find . -inum 435304 -print

This problem may be more difficult to solve because several steps may be required - sometimes you have get test/file: Permission denied, test/file: No such file or directory or test/file: Input/output error.

That happens when the user can't do a stat() on the files (which requires execute permissions), but can read the directory entries (which requires read access on the directory). So you get a list of files in the directory, but can't get any information on the files because they can't be read. If you have a directory which has read permission but not execute, you'll see this.

Some processes like a rsync generates temporary files that get created and dropped fast which will cause errors if you try to call other simple file management commands like rm, mv etc.

Example of output:

?????????? ? ?        ?               ?            ? sess_kee6fu9ag7tiph2jae

1. change permissions: chmod 0777 sess_kee6fu9ag7tiph2jae and try remove

• LVM makes it quite easy to move file systems around
• you can extend a volume group onto a new physical volume
• move any number of logical volumes of an old physical one
• remove that volume from the volume group without needing to unmount any partitions
• you can also make snapshots of logical volumes for making backups
• LVM has built in mirroring support so you can have a logical volume mirrored across multiple physical volumes
• LVM even supports TRIM

Useful resources:

• What is LVM and what is it used for?

Use the lvextend command for resize LVM partition.

• extending the size by 500MB:

lvextend -L +500M /dev/vgroup/lvolume

• extending all available free space:

lvextend -l +100%FREE /dev/vgroup/lvolume

and resize2fs or xfs_growfs to resize filesystem:

• for ext filesystems:

resize2fs /dev/vgroup/lvolume

• for xfs filesystem:

xfs_growfs mountpoint_for_/dev/vgroup/lvolume

Useful resources:

• Extending a logical volume

Is a process that has completed execution (via the exit system call) but still has an entry in the process table: it is a process in the "Terminated state".

Processes marked defunct are dead processes (so-called "zombies") that remain because their parent has not destroyed them properly. These processes will be destroyed by init if the parent process exits.

Useful resources:

• What is a process, and why doesn't it get killed?

To be completed.

It would be better if you run command as it provides many security options like:

• You can set a password for root accounts
• You can remove root accounts that are accessible from outside the local host
• You can remove anonymous-user accounts
• You can remove the test database, which by default can be accessed by anonymous users

Useful resources:

• What is Purpose of using mysql_secure_installation?

Speaking of killing processes never use kill -9/SIGKILL unless absolutely mandatory. This kill can cause problems because of its brute force.

Always try to use the following simple procedure:

• first, send SIGTERM (kill -15) signal first which tells the process to shutdown and is generally accepted as the signal to use when shutting down cleanly (but remember that this signal can be ignored).
• next try to send SIGHUP (kill -1) signal which is commonly used to tell a process to shutdown and restart, this signal can also be caught and ignored by a process.

The far majority of the time, this is all you need - and is much cleaner.

Useful resources:

• When should I not kill -9 a process?
• SIGTERM vs. SIGKILL

strace is a powerful command line tool for debugging and troubleshooting programs in Unix-like operating systems such as Linux. It captures and records all system calls made by a process and the signals received by the process.

Strace Overview

strace can be seen as a light weight debugger. It allows a programmer/user to quickly find out how a program is interacting with the OS. It does this by monitoring system calls and signals.

Uses

Good for when you don't have source code or don't want to be bothered to really go through it. Also, useful for your own code if you don't feel like opening up GDB, but are just interested in understanding external interaction.

Example of attach to the process

strace -p <PID> - to attach a process to strace.

strace -e trace=read,write -p <PID> - by this you can also trace a process/program for an event, like read and write (in this example). So here it will print all such events that include read and write system calls by the process.

Other such examples

• -e trace=network - trace all the network related system calls.
• -e trace=signal - trace all signal related system calls.
• -e trace=ipc - trace all IPC related system calls.

To be completed.

Typical current algorithms are:

• MD5
• SHA-1 (also called SHA)

both should not be used for cryptographic/security purposes any more!!

• SHA-256
• SHA-512
• SHA-3 (KECCAK was announced the winner in the competition for a new federal approved hash algorithm in October 2012)

Useful resources:

• What is the algorithm used to encrypt Linux passwords?
• How to find the hashing algorithm used to obfuscate passwords?

Most Unix-like operating systems, including Linux and BSD, provide ways to limit and control the usage of system resources such as threads, files, and network connections on a per-process and per-user basis. These "ulimits" prevent single users from using too many system resources.

Hard limit is the maximum allowed to a user, set by the superuser or root. This value is set in the file /etc/security/limits.conf. The user can increase the soft limit on their own in times of needing more resources, but cannot set the soft limit higher than the hard limit.

Useful resources:

• Basic SELinux Troubleshooting in CLI

Server side

Setting LogLevel VERBOSE in file /etc/ssh/sshd_config is probably what you need, although there are higher levels:

SSH auth failures are logged in /var/log/auth.log, /var/log/secure or /var/log/audit/audit.log.

The following should give you only ssh related log lines (for example):

grep 'sshd' /var/log/auth.log

Next, the most simple command to list all failed SSH logins is the one shown below:

grep "Failed password" /var/log/auth.log

also useful is:

grep "Failed\|Failure" /var/log/auth.log

On newer Linux distributions you can query the runtime log file maintained by Systemd daemon via journalctl command ( or ). For example:

To be completed.

I want the DBA to ask questions like:

• How big will the database be? (whether we can add the database to an existing server)
• How critical is the database? (about clustering, disaster recovery, high availability)

For example:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 100 > /path/to/file

For example (with visudo command):

user1 ALL=(user2) NOPASSWD: /opt/scripts/bin/generate.sh

The command paths must be absolute! Then call sudo -u user2 /opt/scripts/bin/generate.sh from a user1 shell.

In a bash script, you have several ways to check if the running user is root.

As a warning, do not check if a user is root by using the root username. Nothing guarantees that the user with ID 0 is called root. It's a very strong convention that is broadly followed but anybody could rename the superuser another name.

I think the best way when using bash is to use $EUID because $UID could be changed and not reflect the real user running the script.

if (( $EUID != 0 )); then
  echo "Please run as root"
  exit
fi

In many Unix variants, nobody is the conventional name of a user account which owns no files, is in no privileged groups, and has no abilities except those which every other user has.

It is common to run daemons as nobody, especially servers, in order to limit the damage that could be done by a malicious user who gained control of them.

However, the usefulness of this technique is reduced if more than one daemon is run like this, because then gaining control of one daemon would provide control of them all. The reason is that nobody-owned processes have the ability to send signals to each other and even debug each other, allowing them to read or even modify each other's memory.

When should I use nobody account?

When permissions aren't required for a program's operations. This is most notable when there isn't ever going to be any disk activity.

A real world example of this is memcached (a key-value in-memory cache/database/thing), sitting on my computer and my server running under the nobody account. Why? Because it just doesn't need any permissions and to give it an account that did have write access to files would just be a needless risk.

A good example are also web servers. Imagine if Apache ran as root and someone found a way to send custom commands to the console through Apache would have access to your entire system.

The command you want is named tee:

foo | tee output.file

For example, if you only care about stdout:

ls -a | tee output.file

If you want to include stderr, do:

program [arguments...] 2>&1 | tee outfile

2>&1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.

Furthermore, if you want to append to the log file, use tee -a as:

program [arguments...] 2>&1 | tee -a outfile

You should use #!/usr/bin/env bash for portability: different *nixes put bash in different places, and using /usr/bin/env is a workaround to run the first bash found on the PATH.

Running ./script does exactly that, and requires execute permission on the file, but is agnostic to what type of a program it is. It might be a bash script, an sh script, or a Perl, Python, awk, or expect script, or an actual binary executable. Running bash script would force it to be run under sh, instead of anything else.

Useful resources:

• What is the preferred Bash shebang? (original)

Use nohup to make your process ignore the hangup signal:

nohup long-running-process &
exit

or you want to be using GNU Screen:

screen -d -m long-running-process
exit

Useful resources:

• 5 Ways to Keep Remote SSH Sessions and Processes Running After Disconnection

To find out the main purpose of an intermediate CA, you should first learn about Root CAs, Intermediate CAs, and the SSL Certificate Chain Trust.

Root CAs are primary CAs which typically don’t directly sign end entity/server certificates. They issue Root certificates which are usually pre-installed within all browsers, mobiles, and applications. The private key of these certificates is used to sign other subsequent certificates called intermediate certificates. Root CAs are usually kept "offline” and in a highly secure environment with stringently limited access.

Intermediates CAs are CAs that subordinate to the Root CA by one or more levels, being trusted by these to sign certificates on their behalf. The purpose of creating and using Intermediate CAs is primarily for security because if the intermediate private key is compromised, then the Root CA can revoke the intermediate certificate and create a new one with a new cryptographic key pair.

SSL Certificate Chain Trust is the list of SSL certificates, from the root certificate to the end entity/server certificate. For an SSL Certificate to be trusted, it must be issued by a trusted CAs which is included in the trusted CA list of the connecting device (browser, mobile, and application). Therefore, the connecting device will test the trustworthiness of each SSL Certificate in the Chain Trust until it matches the one issued by a trusted CA.

The Root-Intermediate CA structure is created by each major CA to protect against the disastrous effects of a root key compromise. If a root key is compromised, it would render the root and all subordinated certificates untrustworthy. For this reason, creating an Intermediate CA is a best practice to ensure a rigorous protection of the primary root key.

Useful resources:

• How certificate chains work

Solution 1:

systemctl reload postgresql

Solution 2:

su - postgres
/usr/bin/pg_ctl reload

Solution 3:

SELECT pg_reload_conf();

The best way is exec $SHELL -l because exec replaces the current process with a new one. Also good (but other) solution is . ~/.profile.

Useful resources:

• How to reload .bash_profile from the command line?

kill -9 $$

or

unset HISTFILE && exit

Useful resources:

• How do I close a terminal without saving the history?

toor is an alternative superuser account, where toor is root spelled backwards. It is intended to be used with a non-standard shell so the default shell for root does not need to change.

This is important as shells which are not part of the base distribution, but are instead installed from ports or packages, are installed in /usr/local/bin which, by default, resides on a different file system. If root's shell is located in /usr/local/bin and the file system containing /usr/local/bin) is not mounted, root will not be able to log in to fix a problem and will have to reboot into single-user mode in order to enter the path to a shell.

Some people use toor for day-to-day root tasks with a non-standard shell, leaving root, with a standard shell, for single-user mode or emergencies. By default, a user cannot log in using toor as it does not have a password, so log in as root and set a password for toor before using it to login.

Useful resources:

• The root account (and toor)

For example use fgrep:

fgrep * -R "string"

or:

grep -insr "pattern" *

• -i ignore case distinctions in both the PATTERN and the input files
• -n prefix each line of output with the 1-based line number within its input file
• -s suppress error messages about nonexistent or unreadable files.
• -r read all files under each directory, recursively.

Useful resources:

• How to grep a string in a directory and all its subdirectories files in LINUX?

You can do this with ldd command:

ldd /bin/ls

It's easy to get dragged down into bikeshedding about cloning environments and miss the real point:

• only production is production

and every time you deploy there you are testing a unique combination of deploy code + software + environment.

Every once in a while a good solution is regular cloning of the production servers to create testing servers. You can create instances with an exact copy of your production environment under a dev/test with snapshots, for example:

• generate a snapshot of production
• copy the snapshot to staging (or other)
• create a new disk using this snapshot

Sure, you can spin up clones of various system components or entire systems, and capture real traffic to replay offline (the gold standard of systems testing). But many systems are too big, complex, and cost-prohibitive to clone.

Before environment synchronization a good way is keeping track of every change that you make to the testing environment and provide a way for propagating this to the production environment, so that you do not skip any step and do it as smoothly as possible.

Also structure comparison tool or deploy scripts that update the testing environment from production environment is a good solution.

Presync tasks

First of all is informing developers and clients about not making changes on the test environment (if possible, disabling test domains that target this environment or set static pages with information about synchronization).

It is also important to make backup/snapshots of both environments.

Database servers

• sync/update system version (e.g. packages)
• create dump file from database on production db server
• import dump file on testing db server
• if necessary, syncs login permissions, roles, database permissions, open connections to the database and other

Web/App servers

• sync/update system version (e.g. packages)
• if necessary, updated kernel parameters, firewall rules and other
• sync/update configuration files of all running/important services
• sync/update user accounts (e.g. permissions) and their home directories
• deploy project from git/svn repository
• sync/update important directories existing in project, e.g. static, asset and other
• sync/update permissions for project directory
• remove/update all webhooks
• update cron jobs

To be completed.

If you can telnet to the port, this means that the service listening on the port is running and you can connect to it (it's not a networking problem). It is good to check this way for the IP address to which the domain is resolved and using the same domain to test connection.

First of all check if your site is online from a other location. It then lets you know if the site is down everywhere, or if only your network is unable to view it. It is also a good idea to check what the web browser returns.

If only IP connection working

• you can use whois to see what DNS servers serve up the hostname to the site: whois www.example.com
• you can use tools like dig or host to test DNS to see if the host name is resolving: host www.example.org dns.example.org
• you can also check global public dns servers: host www.example.com 9.9.9.9

If domain not resolved it's probably problem with DNS servers.

If domain resolved properly

• investigate the log files and resolve the issue regarding to the logs, it's the best way to show what's wrong
• check the http status code, usually it will be the response with the 5xx, maybe server is overload because clients making lot's of connection to the website or specific url? maybe your caching rules not working properly?
• check web/proxy server configuration (e.g. nginx -t -c </path/to/nginx.conf>), maybe another sysadmin has made some changes to the domain configuration?
• maybe something on the server has crashed? maybe run out of space or run out of memory?

To be completed.

To be completed.

HTTP/2 supports queries multiplexing, headers compression, priority and more intelligent packet streaming management. This results in reduced latency and accelerates content download on modern web pages.

Key differences with HTTP/1.1:

• it is binary, instead of textual
• fully multiplexed, instead of ordered and blocking
• can therefore use one connection for parallelism
• uses header compression to reduce overhead
• allows servers to "push" responses proactively into client caches

Useful resources:

• What is HTTP/2 - The Ultimate Guide

Modify NGINX configuration file for domain

Set correct client_max_body_size variable value:

client_max_body_size 20M;

Restart Nginx to apply the changes.

Read more