Encrypted traffic is on the rise. It's no longer possible to inspect the content of the communication. What does this mean for network traffic monitoring?

Encrypted traffic is on the rise. In 2016, NSS Labs found out that 97% of surveyed enterprises saw an increase in encrypted web traffic. Both Let's Encrypt and Google  currently report between 70% and 90% of traffic is HTTPS (HTTP encrypted with SSL/TLS). Google has been pressing for HTTPS for a while now. Back in 2014, they started taking into account whether the website uses HTTPS for page ranking, and in 2017 Google Chrome browser started showing "Not Secure" warning in the address bar when visiting websites without HTTPS. In August 2018, Scott Helme, a Security Researcher, analyzed 1 million most popular websites - over 51% are already using HTTPS. Gartner predicts 80% of traffic will be encrypted in 2019.

What does this all mean for network traffic monitoring?

Using encryption it is no longer possible to inspect the content of the communication. Let's take a look at the following screenshots from a popular packet capture tool - Wireshark. Let's visit the website - example.com. At first, without encryption (plain HTTP) - http://example.com. In Wireshark we can dive deep and inspect the HTML code of the website content being transmitted from the server to the client. Anybody can intercept our communication and take a look themselves: This is especially problematic with online banking, healthcare services and other sensitive online transactions. This is where HTTPS (HTTP with SSL/TLS encryption) comes in. How does it work? First, the client and server agree on the encryption algorithm they are going to use, and then the encrypted data is transmitted. This is a very simplified explanation and I hope more tech savvy readers will forgive me.

How will our communication with example.com look like if we use HTTPS? Here is a screenshot from Wireshark when we visit https://example.com: 

The contents are now no longer readable. This is sad news for anyone trying to spy on us. They can no longer get our credit card information or other sensitive data. This is why Let's Encrypt and other organizations push for HTTPS. It makes our online lives more secure.

Now, we can all agree that encryption is great, pat ourselves on the back for spreading the HTTPS gospel and go home, right?

Well, not so fast. What if you need to know what is going on in your network? What are you going to do when there is an issue with the critical business application? Does your network administrator still rely on legacy packet capture solution for troubleshooting and monitoring? What can you do when you can no longer inspect the contents of communication?

Briefly, consider an example of detecting BitTorrent traffic in a network. In the past it was only a matter of checking the TCP port number (the range for a BitTorrent is 6881-6889). However, nowadays client applications tend to randomize port numbers. It was too easy to block the said ports on the firewall. It's fairly straightforward to perform deep packet inspection (DPI) when packets are unencrypted. Examples of BitTorrent signatures are "GET /announce?info_hash", "GET / request" and "GET /torrents/". The DPI engine looks for these strings within the payload, and once found BitTorrent traffic is detected. Unfortunately, most of the client applications encrypt BitTorrent communication by default. Users can disable encryption, but they still accept encrypted traffic from their peers (other BitTorrent clients) and thus reducing the detection success rates.

Luckily, we can leverage flow data to detect BitTorrent traffic. By design, a BitTorrent client makes a lot of connections with peers in a short time interval, peers are located all around the globe and many connections fail because peers are offline. In short, we can detect encrypted BitTorrent communication with flow data as it is based on client behavior rather than the contents of the communication. Learn more about sources of flow data in our previous post Where Do The Flows Come From?.

In summary, with encryption on the rise it no longer makes sense to monitor your network with packet capture solutions. With flows it doesn't matter whether the content of communication is encrypted or not. We leverage information and metrics from lower network layers, which are not encrypted. Thanks to this approach, our network and security monitoring solution works even on encrypted traffic. 

Thank you to Roman Luks from Flowmon for the article.