Networking is often overlooked topic by both developers and DevOps engineers. Yet it's at least to a certain extent integral part of every system and application. Being therefore able to perform basic troubleshooting and use common tools is very valuable skill and might come in handy next time you run into issues with DNS or routing, or when you just need to send HTTP request to some web application. So, let's go over following list of basic networking tools that every developer should know and explore examples that might even become your cheat sheet for all things networking.

The Everyday Basics

Sending a HTTP request is one of the things that everyone of us does every day. And doing so with cURL can be as simple as curl google.com, but curl command has lots of options that can make it a lot more useful. More often than not, sending plain request is not good enough and you will need to specify type of request or include some extra headers. You can do that with -X and -H arguments, respectively. Another common thing to do would be sending some data in request, which can be done with -d:


~ $ curl -X POST \  # POST Request
         -H "accept: application/json" \  # Set headers
         -H "Authorization: Bearer ..." \  # Can be used multiple times
         -d '{"text" : "Hello, World!"}' \  # Data in JSON format
         https://example.com/some/path

curl can be also helpful for debugging. You can turn on verbose output using -vvv or use -k to switch to insecure mode when you are dealing with website certificates.


~ $ curl -X GET -k \
         -vvv \
         -H 'Content-type: application/json' \
         -d '{"text" : "Hello, World!"}' \
         https://example.com/some/path

When using verbose mode, there's going to be lots of data coming in and sometimes it makes sense to rather use -I to return only headers instead of whole responses.

Downloading files is another super common task that can be easily done from terminal using wget. Same as with curl, wget has a lot of options, but some of the more common and very useful ones include - -O to change filename, -q (quiet) to suppress noisy output, -b to run in background or -c to resume previous download. Simple example:


~ $ wget -q https://example.com/1.0.5/file-linux-amd64 -O new-filename

Things don't always work as expected and some services might become unavailable. So, to check if a host is reachable, one can use ping <some-IP> or ping6 <some-IP> if using IPv6. In case you need to check not just whether a host is reachable but also whether specific port is open, then you can use telnet <host> <port>.

ping is a lot simpler command than the ones above, but it still has some useful arguments, e.g. -c <number> to specify number pings to make or -l <size> to specify size of payload to send.

Looking up your IP address is probably not something you need to do every day, but you need it, then hostname command is there for you. Especially with -I argument which displays all IPs of your machine. hostname command can also be used to set your machines hostname, but not persistently, so preferably use hostnamectl for that instead.

DNS is a Mess

General rule of thumb is that when something is not quite right with networking, then it's most likely because of DNS. Basic inspection and debugging of DNS can be done with either nslookup, host or dig. Out of these 3, I would recommend using dig even though its output is little hard to read, because nslookup is deprecated at this point and host doesn't provide as much data as dig.


~ $ dig github.com

; <<>> DiG 9.16.1-Ubuntu <<>> github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31813
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;github.com.			IN	A

;; ANSWER SECTION:
github.com.		59	IN	A	140.82.121.3  # IP of website

;; Query time: 15 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)  # Nameserver Address
;; WHEN: So okt 10 17:15:03 CEST 2020
;; MSG SIZE  rcvd: 55

~ $ dig -t NS github.com
...
;; ANSWER SECTION:
github.com.		233	IN	NS	dns1.p08.nsone.net.  # List of nameservers
... # More records
github.com.		233	IN	NS	dns3.p08.nsone.net.
...

~ $ dig -t NS github.com +short  # Query nameservers with short output
dns2.p08.nsone.net.
...
ns-1707.awsdns-21.co.uk.

~ $ dig -x 140.82.121.3  # DNS Reverse lookup
...
;; QUESTION SECTION:
;3.121.82.140.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
3.121.82.140.in-addr.arpa. 1570	IN	PTR	lb-140-82-121-3-fra.github.com.

The example above shows some of the basic queries one can perform with dig. Using this command might seem a little intimidating as its syntax is frankly plain stupid. So, let's go over each query and its output to make little more sense of it - first query - dig github.com does a basic DNS query for github.com. Its output shows 4 sections - header, question, answer and stats. We only really care about question and answer. The former describes what we asked, which in this case is DNS information about A Record for github.com. The later shows answer in form of IP address of website.

Second query above uses -t NS to do search for nameservers, which are displayed in the answer section. If you want just the answer without all the clutter around it, you can add +short as shown with the third query.

The last query above performs reverse DNS search. This is done by using -x argument while passing in IP address instead of hostname. If the information from these queries isn't enough for your debugging, then you can use +trace which will show whole delegation path.

If the dig command is not your cup of tea and you need to perform only simple queries or for example check DNS connectivity, you can also use previously mentioned host command:


~ $  host github.com
github.com has address 140.82.121.3
github.com mail is handled by 5 alt2.aspmx.l.google.com.

~ $ host 140.82.121.3
3.121.82.140.in-addr.arpa domain name pointer lb-140-82-121-3-fra.github.com.

In previous section I mentioned that you should not use hostname to set hostname as it will not persist across reboot. There are options for changing hostname persistently - one is to use hostnamectl set-hostname or you can also directly modify /etc/hostname, which is what hostnamectl does anyway.


~ $ hostnamectl set-hostname some-host.domain.com
~ $ hostnamectl status
   Static hostname: some-host.domain.com
         Icon name: computer-vm
           Chassis: vm
        Machine ID: gd99711b6414efb8bb8791f9f2d4d71a
           Boot ID: 8add50a8560541cb89d1ca2aede6961b
    Virtualization: kvm
  Operating System: Ubuntu 19.04
            Kernel: Linux 5.0.0-15-generic
      Architecture: x86-64

There are also other files related to DNS that you might want to check out when troubleshooting. First file that is checked when doing DNS search is /etc/hosts. If the entry is not found there, DNS search is performed using /etc/resolv.conf, which contains list IPs of nameservers for resolution. You can test hostname resolution manually using getent hosts example.com


~ $ cat /etc/resolv.conf
search some.example.com
nameserver 10.0.2.3 # IP of nameserver to query
nameserver 8.8.8.8

Problems with Connection?

If you're confident that the DNS is not culprit of your problem, then it's time to check connection related issues. One can do that using ip command. When looking for any issues with connection, we should start from hardware and work our way up the stack, therefore let's first run ip link show.


~ $ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:ee:9a:00:33:f5 brd ff:ff:ff:ff:ff:ff

~ $ ip -s link show ens3
3: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
    link/ether 52:ee:9a:00:33:f5 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    14624090276 14595993 0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    2059715701 6107955  0       0       0       0

This command displays all network interfaces available on your system. The important field in the output is state, which should show UP. When you find the interface being currently used, then you can also run ip -s link show <name> to see stats. The important fields in the output of said command are RX for received data and TX for transmitted.

Next, we can take a look at our IP addresses using ip addr show. This will show us both IPv4 and IPv6 addresses (if running in dual-stack mode) as well as hardware (MAC) address of the device (link/ether line):


~ $ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:ee:9a:00:33:f5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.189/24 brd 192.168.1.255 scope global noprefixroute dynamic ens3
       valid_lft 72719sec preferred_lft 72719sec
    inet6 fe80::b3ec:c89:50bf:9cdd/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Last but not least from the ip family of commands is ip route. This gives us IPv4 routing table and can be useful for finding IP of your router, which is the default gateway (first line in output):


~ $ ip route
default via 192.168.1.1 dev ens3 proto dhcp metric 600  # All other packets are sent through default gateway located at 192.168.1.1
192.168.1.0/24 dev ens3 proto kernel scope link src 192.168.1.189 metric 600  # All packets destined for 192.168.1.0/24 are sent directly through ens3 device

In case you need to dig deeper into routing issues or maybe if you are looking for specific router that's causing trouble, you can also use tracepath. tracepath shows path of your request, where each line is one hop (one router).


~ $ tracepath -b twitter.com
 1?: [LOCALHOST]                      pmtu 1500
 1:  router.asus.com (192.168.1.1)                         2.957ms 
 1:  router.asus.com (192.168.1.1)                         5.767ms 
 2:  192.168.100.1 (192.168.100.1)                         1.469ms 
 ...
 8:  ??? (104.244.42.129)                                 21.838ms reached
     Resume: pmtu 1500 hops 8 back 12

Apart from all the issues that you might encounter, there are some fairly basic connection configuration tasks which you might need to perform from time to time. There really are 2 tools that you can choose from - nmcli and nmtui. The former being classic CLI tool with all the possible options and the later being curses-based terminal GUI. I strongly recommend using nmtui as it's very easy to use, simple to navigate and fool-proof tool that will help you to avoid errors when configuring connections.

However, If you insist on using nmcli, please make sure to check out man nmcli-examples, which shows some common usage examples. I will just leave here one simple example for nmcli, to maybe deter you from touching it:


# Adding an ethernet connection profile with manual IP configuration
~ $ nmcli con add con-name my-con ifname ens3 type ethernet ip4 192.168.100.100/24 gw4 192.168.100.1 ip4 1.2.3.4 ip6 abbe::cafe

Regardless of whether you will use nmcli or nmtui, all the configs will be stored in /etc/sysconfig/network-scripts (on RHEL/CentOS, anyway). These configs can also be edited directly without any tool, but I do not recommend this as CLI options and file variables have often very different names, so it's hard to see what is what.

One word of caution for dealing with connections. I recommend to either modify these configs using physical console (not over SSH), because in case you mess up, you will not lose access to the machine. If that's not possible, you can alternatively setup deferred job (using at) that will revert your changes after hour or so (don't forget to cancel it if your config works).

Securing Your Server with Firewall

One way to improve security of your server is to make sure only necessary ports are open. To set that up, one might need to deal with iptables and netfilter, which can be a daunting task. You are therefore better off using some kind of frontend for these tools. If you are on Ubuntu you can go with ufw which stands for uncomplicated firewall:


~ $ ufw allow ssh/tcp  # http (80), https (443)
~ $ ufw logging on  # Turn on logging
~ $ ufw enable  # Enable firewall

~ $ ufw allow from 15.15.15.0/24 to any port 22    # Allow SSH connections from specific IP or Subnet
~ $ ufw allow from 15.15.15.0/24 to any port 5432  # Allow PostgreSQL connections from specific IP or Subnet

I think the examples above don't really need much of an explanation as it really is uncomplicated. If you are not using Ubuntu though, but for example RHEL/CentOS, you will be probably by default running firewalld with its firewall-cmd. This tool is not so simple, but still pretty easy to use:


~ $ cat /usr/lib/firewalld/services/postgresql.xml
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>PostgreSQL</short>
  <description>PostgreSQL Database Server</description>
  <port protocol="tcp" port="5432"/>
</service>

~ $ firewall-cmd --permanent --zone=internal --add-service=postgresql  # Add PostgreSQL service instead of specific port
~ $ firewall-cmd --reload  # Apply changes to runtime

firewalld provides set of files in /usr/lib/firewalld/services which describe predefined services. These files make it easier to allow/block various common services like DBs, SSH, HTTP, LDAP, etc. In the example above you can see contents of PostgreSQL service file which defines protocol and port used by PostgreSQL. Thanks to that file we can the above command to allow connection to PostgreSQL database (--add-service=postgresql).


~ $ cat /usr/lib/firewalld/zones/block.xml 
<?xml version="1.0" encoding="utf-8"?>
<zone target="%%REJECT%%">
  <short>Block</short>
  <description>Unsolicited incoming network packets are rejected. Incoming packets that are related to outgoing network connections are accepted.
    Outgoing network connections are allowed.</description>
</zone>

~ $ firewall-cmd --add-source=172.217.23.238/32 --zone=block --permanent  # Block traffic from specific IP address
~ $ firewall-cmd --reload

There is also a second set of files in /usr/lib/firewalld/zones/block.xml which describes zones like Public, DMZ, Block or Work. All traffic is diverted into zones based on criteria like source IP or source port. This classification simplifies management of firewall. In the example above you can see that we can simply block specific IP by diverting all its traffic to block zone.

You probably noticed that in both examples we first used command with --permanent and then --reload , that's because firewalld doesn't automatically apply changes to both runtime and persistent configs, therefore it's important to run both of these command to avoid troubleshooting issues that don't exist.

Scanning All Kinds of Stuff

Last but not least - it's pretty common networking task to run device scans in network to - for example - find connected machines in local network or to run port scan to see which ports are open or which application is listening and where.

For the former you can use nmap which can scan hosts, ports or whole subnets. This tool has a lot of options and really warrants article on its own, but to give you quick overview of the most common arguments and options:


~ $ nmap -sP ...  # Ping Scan - Scan network and find out which devices are up and running
~ $ nmap -sL ...  # List targets in network
~ $ nmap -O ...  # OS detection
~ $ nmap -p 22,21,80,8080 -sV ...  # Detect version of service running on port
~ $ nmap -A ...  #  OS detection, version detection, script scanning, and traceroute

~ $ nmap -p 22,21,80,8080 172.217.17.110  # IP of goggle.com; Also `-p "*"` works for scanning all ports
Nmap scan report for ams15s29-in-f14.1e100.net (172.217.17.110)
Host is up (0.0072s latency).

PORT     STATE    SERVICE
21/tcp   filtered ftp
22/tcp   filtered ssh
80/tcp   open     http
8080/tcp filtered http-proxy

Nmap done: 1 IP address (1 host up) scanned in 1.46 seconds

These examples cover basic usage of nmap and I personally rarely need anything except for the arguments and options shown above. One extra argument you might want to use though, is -F (fast scan) argument, which can be useful to speed up scans, as it will omit most of the ports to be scanned, leaving only the most common ones.

Other tool worth mentioning in this section is ss which is replacement for old netstat. It's used to investigate sockets (combination of IP, protocol and port). It's very handy when you need to find where some service listens. I generally use -plat arguments which shows all (a) listening (l) TCP (t) sockets and their respective processes (p):


~ $ ss -plat
State        Recv-Q    Send-Q     Local Address:Port        Peer Address:Port      Process
LISTEN       0         128              0.0.0.0:ssh              0.0.0.0:*
ESTAB        0         0           192.168.43.1:40790     192.168.43.151:ssh      users:(("ssh",pid=886049,fd=3)) 
LISTEN       0         32                     *:ftp                    *:*
LISTEN       0         128                 [::]:ssh                 [::]:*
LISTEN       0         128                    *:https                  *:*
LISTEN       0         128                    *:http                   *:*

Conclusion

Our jobs - regardless of whether you're developer, DevOps engineer or SRE - often have wide range of duties, so it's important to strive to have at least basic understanding of fundamentals of networking (among other things). It might seem like there's no immediate benefit to learning things like networking, so lot of people tend to ignore it. Learning these fundamentals is in my opinion very much worth the time though, especially in the long run. It should also go beyond just being able to run the above mentioned commands (even though that helps a lot), but rather understanding the concepts and internal workings of things like TCP/IP, DNS, DHCP, TLS, etc.

Blog Posts Tagged #networking

Blog Posts Tagged `#networking`