Shodan crawls the entire Internet at least once a week, but if you want to request Shodan to scan a network immediately you can do so using the on-demand scanning capabilities of the API. A few common reasons to launch a scan are:
In this article we will be discussing the various options available at Shodan for performing external network scans.
Unlike scanning via a tool such as Nmap, the scanning with Shodan is done asynchronously. This means that after you submit a request to Shodan you don't get back the results immediately. It is up to the developer to decide how the results of the scan should be gathered: by looking up the IP information, searching Shodan or subscribing to the real-time stream.
The number of IPs that you're able to scan is determined by the number of scan credits available on your API plan. To see how many scan credits your account has access to either visit your Developer Dashboard or use the Shodan CLI:
$ shodan info
Query credits available: 100000
Scan credits available: 100000
For each IP that you wish to scan via Shodan you will use 1 scan credit. Effectively, this is the conversion rate:
1 IP requires 1 scan credit
There are 2 ways to request a scan:
The Shodan CLI is the easiest way to get started with the on-demand scanning capabilities. It lets you automate the task of submitting scan requests to Shodan without needing to write any code.
The Shodan API provides more flexibility but requires you to create your own scripts.
The command to launch a scan via the CLI is:
$ shodan scan submit <network or ip address>
To get the list of available options for the command simply add a -h to the command:
$ shodan scan submit -h
Usage: shodan scan submit [OPTIONS] <ip address>
Scan an IP/ netblock using Shodan.
Options:
--wait INTEGER How long to wait for results to come back. If this is set
to "0" or below return immediately.
--filename TEXT Save the results in the given file.
--force
--verbose
-h, --help Show this message and exit.
For example, to request a scan of the IP 198.20.69.74 you would simply enter:
$ shodan scan submit 198.20.69.74
You can also ask Shodan to scan a network range by providing an address in CIDR notation. For example, the following would request a scan for devices 198.20.69.0 through 198.20.69.255:
$ shodan scan submit 198.20.69.0/24
By default, the shodan scan submit command will output the results to the screen but doesn't otherwise save them. If you want to save the results you need to provide the --filename parameter:
$ shodan scan submit --filename scan-results.json.gz 198.20.69.74
The file generated by the scan command is the standard Shodan data file format which you get when downloading data from the website/ API. This means you can analyze the scan results the same way you'd analyze any other data collected from Shodan.
As mentioned earlier, all network scanning is performed asynchronously which means you don't know exactly when all the results have been collected by the crawlers. The Shodan CLI has a default timeout of 30 seconds which should be enough but depending on your network size you may want to wait longer. Sometimes you also just want to kick off a scan request but don't need to collect the results immediately. For those cases, you can submit a scan request using a wait time of zero:
$ shodan scan submit --wait 0 198.20.69.74
That will submit the request and then immediately exit.
If an IP or network was recently crawled by Shodan then you won't be able to scan it again within the next 24 hours - unless you have an Enterprise subscription to Shodan. With an Enterprise subscription you can use the --force option to force the Shodan crawlers to re-check an IP/ network:
$ shodan scan submit --force 198.20.69.74
We will be using the Python library for Shodan but there are API bindings available in most programming languages - simply pick the language you're most comfortable in.
Lets get started by looking at how to submit a scan request in Python:
from shodan import Shodan
# Setup the Shodan API object
api = Shodan(API_KEY)
# Submit a scan request for 1 IP and 1 network range
scan = api.scan(['198.20.49.30', '198.20.74.0/24'])
After submitting a scan request the API will return the following object:
{
'id': 'R2XRT5HH6X67PFAB',
'count': 257,
'credits_left': 4139
}
Right off the bat there are a few new things to take note of. Firstly, you can submit multiple networks/ IPs using a single scan request. This means that if you're trying to scan many networks you can request all of them at once using the API. Secondly, every scan request has a unique ID associated with it which can be used to grab the scan results.
The scanning API also allows you to submit hostnames:
scan = api.scan('google.com')
And you can combine them:
scan = api.scan(['microsoft.com', '198.20.69.0/24'])
By default, the Shodan crawlers will check all the standard ports it normally crawls. However, you can also provide a specific list of ports/ protocols that you'd like the crawlers to use. For example, the following is a request to scan the IP 198.20.69.74 for Modbus on port 503 and SSH on port 2900:
scan = api.scan({
'198.20.69.74': [
(503, 'modbus'),
(2900, 'ssh'),
]
})
A list of available protocols/ modules that can be used for checking ports is available via the API at /shodan/protocols. Or you can see the list using the CLI:
$ shodan scan protocols
Now that you know how to launch a scan lets take a look at how to retrieve the results. With the scan ID you have the choice of either streaming the results in real-time or bulk downloading them later.
The easiest way to get the results of the scan is by using the scan search filter. For example, the following search query would return the results for the above scan (assuming you had launched the scan):
scan:R2XRT5HH6X67PFAB
You simply provide the scan ID and the Shodan API will return the results. The following shodan command would download the data and store the results in a file called scan-results.json.gz:
$ shodan download --limit -1 scan-results.json.gz scan:R2XRT5HH6X67PFAB
And here's how to iterate over the scan results in Python:
# Use the search_cursor() method to automatically page through all results
for banner in api.search_cursor('scan:{}'.format(scan['id'])):
# Perform special actions depending on the types of banners you see in the results
# We're just printing out the entire banner for the purpose of this article.
print(banner)
If you want to see the results from the scan instantly as it's being done then you have the option of subscribing to a private firehose for the scan. This can also be done either via the command-line or the API but requires some additional setup. The workflow is as follows:
The 3rd part of the workflow has already been discussed so we'll quickly show how to achieve the other steps using the CLI. First, lets setup the network alert which is a private firehose containing the data for a network range:
$ shodan alert create "Scan results" 198.20.69.0/24
Successfully created network alert!
Alert ID: 4I2SWAZ6WV2CISKR
Once that's created we can go ahead and subscribe to it using the shodan stream command:
$ shodan stream --alerts=4I2SWAZ6WV2CISKR --datadir=scan-results/
In the above example we are providing the alert ID that was given to us in the first step. And the banners flowing through the private firehose will be stored in the scan-results directory. After we launch the command it will keep listening for results until we cancel it. Now that we've created a network alert that will give us all the information about a network range we are ready to submit our scan. In a separate terminal, enter the usual command for submitting a scan:
$ shodan scan submit --wait 0 198.20.69.0/24
Note that this time we're telling the CLI to immediately exit as the other terminal is already capturing any responses.
As you might have guessed, the real-time workflow is largely aimed at organizations that are permanently monitoring their external network via Shodan's private firehose streams. Most companies have an on-going subscription to their private firehose so they can get notified in real-time if something unexpected pops-up on their network (ex. a BitTorrent server).
Important: this feature is only available to our Enterprise Data License customers.
The Shodan API also allows you to request a scan of the entire Internet - you simply specify the port and protocol/ module. For example, the following command launches a scan for web servers that are running on port 9302:
$ shodan scan internet 9302 http
The results of the scan can be consumed either via the website or the Shodan Firehose. By default, the above command will wait 30 seconds and once results start coming in it will store it in a file called port-protocol.json.gz. For example, the above command would generate a file called 9302-http.json.gz. If you want more control over how the results are collected you can either use the Shodan API directly or you can check out the shodan stream command. Below is the command you would run to stream the results from the above scan and save the data:
$ mkdir scan-results
$ shodan stream --datadir scan-results/ --ports 9302
We first create a directory called scan-results which is where we're going to store the data. And then we stream data from the firehose, explicitly asking to only get results on port 9302.
Here is a video covering a few other options available when streaming from the Shodan Firehose: