How to Monitor a Network in Real-Time

Check out Shodan Monitor for an easy way to setup network monitoring

This guide will cover how to keep track of changes to a public network's IP range using the command-line interface and API. To do so we will be using the network alerts feature of the API.

Introducing Network Alerts

A network alert is a real-time feed of data that is being collected by Shodan for a network range. Think of it as a private firehose where the only banners you see are from your monitored IPs. With network alerts you can immediately respond to new discoveries - no searching is required.

In this guide, we're going to:

Setup a network alert for the IP range 198.20.0.0/16
Subscribe and store all the banners collected
Write a script to email us whenever an industrial control system shows up

Requirements

To follow along with this guide you must have the Shodan Python library installed. Most Unix-based operating systems already come with Python installed so all you need to type to install the Shodan library is:

$ sudo easy_install shodan

And make sure the command-line interface is initialized with your API key:

$ shodan init APIKEY

The Shodan API key can be obtained from the Shodan Account page.

Creating the Network Alert

First, lets create the network alert for the IP range we want to receive notifications for:

$ shodan alert create "Production Network" 198.20.0.0/16

And now lets confirm that the alert was properly created by listing all the alerts on your account:

$ shodan alert list

Subscribing to the Private Firehose

There are 2 ways to subscribe to alerts

Provide a list of alert IDs
Subscribe to all alerts

This is useful if you have different workflows and requirements. For now, lets simply store banners for all networks and store them a directory called ~/shodan-data/:

$ shodan stream --alert=all --datadir=~/shodan-data/

Lets break down the parameters:

--alert: this parameter accepts either an alert ID or the special keyword all which subscribes to all alerts.
--datadir: give this parameter the directory name where results should be stored. It automatically rotates files every day and names them YYYY-MM-DD.json.gz

At this point, we've successfully created our own, private firehose and are storing all new events in a directory. Saving the results is good practice and makes it possible to generate daily/ weekly/ monthly reports. However, for most situations you also want to immediately respond to the events in the firehose.

Monitoring for High Risk Events

The final step is to have a Python script that listens for high risk services - things that should never be publicly accessible on the network. A few examples of high risk services are: industrial control systems (ICS), unauthenticated databases or Telnet. Depending on your network you may also want to be aware of services such as Tor or BitTorrent (DHT).

For now, the goal is to monitor the network for industrial control systems and send out an email immediately. To achieve this we're going to write a custom Python script. We're going to start with boilerplate code which is used in most Shodan scripts:

#!/usr/bin/env python

from shodan import Shodan
from shodan.helpers import get_ip
from shodan.cli.helpers import get_api_key

# Setup the Shodan API connection
api = Shodan(get_api_key())

The get_api_key() method grabs the API key from the CLI so your scripts don't need to store the Shodan API key. That makes them easier to share and less dependent on static API keys. The only requirement is that you've initialized the local environment using:

shodan init YOUR_API_KEY

Moving on, lets subscribe to all alerts and use the tags property to find out whether a service belongs to an industrial control system. If Shodan identifies an ICS banner then it adds an ics tag to the banner. We have to check for the existence of the tags property because not all banners have tags"

# Subscribe to results for all networks:
for banner in api.stream.alert():
    # Check whether the banner is from an ICS service
    if 'tags' in banner and 'ics' in banner['tags']:
        ip = get_ip(banner) # Easy way to grab either IPv4 or IPv6 address
        send_mail('ICS Alert: {}'.format(ip), """
ICS Service information:

Port: {port}
Data: {data}

""".format(banner))

And finally, a simple Python method to send out email using a local mail server:

def send_mail(subject, content):
    """Send an email using a local mail server."""
    from smtplib import SMTP
    server = SMTP()
    server.connect()
    server.sendmail(EMAIL_FROM, EMAIL_TO, 'Subject: {}\n\n{}'.format(subject, content))
    server.quit()

Putting everything together we now have a script that subscribes to all of our network alerts, checks whether the banner belongs to an industrial control system and if it does then it immediately sends out an email to a designated address:

#!/usr/bin/env python

from shodan import Shodan
from shodan.helpers import get_ip
from shodan.cli.helpers import get_api_key


# Configuration
EMAIL_TO = 'alerts@myemail.com'
EMAIL_FROM  = 'ics-alerts'


# Setup the Shodan API connection
api = Shodan(get_api_key())

# Subscribe to results for all networks:
for banner in api.stream.alert():
    # Check whether the banner is from an ICS service
    if 'tags' in banner and 'ics' in banner['tags']:
        send_mail()


def send_mail(subject, content):
    """Send an email using a local mail server."""
    from smtplib import SMTP
    server = SMTP()
    server.connect()
    server.sendmail(EMAIL_FROM, EMAIL_TO, 'Subject: {}\n\n{}'.format(subject, content))
    server.quit()