This guide will cover how to use your Shodan API plan to download data instead of using the website.
If you have an API plan then you get a certain number of query credits that you can spend each month. For people with the Shodan Membership that means you get 100 query credits per month while for the API plans it can range from 10,000 up to unlimited.
1 query credit = 100 results
Every query credit gets you up to 100 results, which means that you can download at least 10,000 results every month - regardless of the type of search you're performing.
If you have an Enterprise Data subscription then you can download the entire Shodan database. For more information on our enterprise offerings please contact sales@shodan.io
To follow along with this guide you must have the Shodan Python library installed. Most Unix-based operating systems already come with Python installed so all you need to type to install the Shodan library is:
$ sudo easy_install shodan
If you're on Ubuntu and don't yet have easy_install available please run the following command first:
$ sudo apt-get install python-setuptools
After you've installed the library you can check that it's working simply by typing shodan into the terminal/ command prompt:
$ shodan
Usage: shodan [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help Show this message and exit.
Commands:
alert Manage the network alerts for your account
convert Convert the given input data file into a...
count Returns the number of results for a search
download Download search results and save them in a...
honeyscore Check whether the IP is a honeypot or not.
host View all available information for an IP...
info Shows general information about your account
init Initialize the Shodan command-line
myip Print your external IP address
parse Extract information out of compressed JSON...
scan Scan an IP/ netblock using Shodan.
search Search the Shodan database
stats Provide summary information about a search...
stream Stream data in real-time.
Finally, initialize the tool using your API key which you can get from your account page:
$ shodan init YOUR_API_KEY
The Shodan CLI provides access to most functions of the API in a user-friendly interface. It also includes a command to easily download data using the query credits from your API. Here's a quick video that shows how it works in action:
The basics of it are:
$ shodan download --limit <number of results> <filename> <search query>
For example, this is the command to download 500 results for the search query "product:mongodb" which returns Internet-facing MongoDB services:
$ shodan download --limit 500 mongodb-results product:mongodb
The results of the above command will be saved in a file called mongodb-results.json.gz. At this point, you can easily convert the file into CSV, KML or simply output a list of IP:port pairs by using the shodan parse command:
$ shodan parse --fields ip_str,port --separator , mongodb.json.gz
The CLI should work for most purposes but sometimes you want to perform custom transformations on the banners as you're downloading them. Or you don't want to store the information in a local file. In those cases, you can use a convenient helper method provided by the Python library for Shodan called search_cursor() to iterate over the results:
from shodan import Shodan
from shodan.cli.helpers import get_api_key
api = Shodan(get_api_key())
limit = 500
counter = 0
for banner in api.search_cursor('product:mongodb'):
# Perform some custom manipulations or stream the results to a database
# For this example, I'll just print out the "data" property
print(banner['data'])
# Keep track of how many results have been downloaded so we don't use up all our query credits
counter += 1
if counter >= limit:
break
We've discussed how to download data with the CLI, how to extract properties out of it, how to convert it to other file formats and how to process search results directly in Python. For a complete example that uses these skills check out how to track hacked websites.