How to Download Data with the API
This guide will cover how to use your Shodan API plan to download data instead of using the website. There are a few benefits in using the API, the most notable one being that downloading via the API uses query credits which are renewed at the start of every month whereas the website consumes export credits which are single-use.
How much data can I download?
If you have an API plan then you get a certain number of query credits that you can spend each month. For people with the Shodan Membership that means you get 100 query credits per month while for the API plans it can range from 10,000 up to unlimited.
1 query credit = 100 results
Every query credit gets you up to 100 results, which means that you can download at least 10,000 results every month - regardless of the type of search you're performing.
If you have an Enterprise Data subscription then you can download the entire Shodan database. For more information on our enterprise offerings please contact email@example.com
To follow along with this guide you must have the Shodan Python library installed. Most Unix-based operating systems already come with Python installed so all you need to type to install the Shodan library is:
$ sudo easy_install shodan
If you're on Ubuntu and don't yet have easy_install available please run the following command first:
$ sudo apt-get install python-setuptools
After you've installed the library you can check that it's working simply by typing shodan into the terminal/ command prompt:
$ shodan Usage: shodan [OPTIONS] COMMAND [ARGS]... Options: -h, --help Show this message and exit. Commands: alert Manage the network alerts for your account convert Convert the given input data file into a... count Returns the number of results for a search download Download search results and save them in a... honeyscore Check whether the IP is a honeypot or not. host View all available information for an IP... info Shows general information about your account init Initialize the Shodan command-line myip Print your external IP address parse Extract information out of compressed JSON... scan Scan an IP/ netblock using Shodan. search Search the Shodan database stats Provide summary information about a search... stream Stream data in real-time.
$ shodan init YOUR_API_KEY
Using the Command-Line Interface
The Shodan CLI provides access to most functions of the API in a user-friendly interface. It also includes a command to easily download data using the query credits from your API. Here's a quick video that shows how it works in action:
The basics of it are:
shodan download --limit <number of results> <filename> <search query>
For example, this is the command to download 500 results for the search query "product:mongodb" which returns Internet-facing MongoDB services:
shodan download --limit 500 mongodb-results product:mongodb
The results of the above command will be saved in a file called mongodb-results.json.gz. At this point, you can easily convert the file into CSV, KML or simply output a list of IP:port pairs by using the shodan parse command:
shodan parse --fields ip_str,port --separator , mongodb.json.gz
Programming with the Shodan API
The CLI should work for most purposes but sometimes you want to perform custom transformations on the banners as you're downloading them. Or you don't want to store the information in a local file. In those cases, you can use a convenient helper method provided by the Python library for Shodan called search_cursor() to iterate over the results:
from shodan import Shodan from shodan.cli.helpers import get_api_key api = Shodan(get_api_key()) limit = 500 counter = 0 for banner in api.search_cursor('product:mongodb'): # Perform some custom manipulations or stream the results to a database # For this example, I'll just print out the "data" property print(banner['data']) # Keep track of how many results have been downloaded so we don't use up all our query credits counter += 1 if counter >= limit: break
We've discussed how to download data with the CLI, how to extract properties out of it, how to convert it to other file formats and how to process search results directly in Python. For a complete example that uses these skills check out how to track hacked websites.