How to Download Data with the API

This guide will cover how to use your Shodan API plan to download data instead of using the website. There are a few benefits in using the API, the most notable one being that downloading via the API uses query credits which are renewed at the start of every month whereas the website consumes export credits which are single-use.

How much data can I download?

If you have an API plan then you get a certain number of query credits that you can spend each month. For people with the Shodan Membership that means you get 100 query credits per month while for the API plans it can range from 10,000 up to unlimited.

1 query credit = 100 results

Every query credit gets you up to 100 results, which means that you can download at least 10,000 results every month - regardless of the type of search you're performing.

If you have an Enterprise Data subscription then you can download the entire Shodan database. For more information on our enterprise offerings please contact sales@shodan.io

Requirements

To follow along with this guide you must have the Shodan Python library installed. Most Unix-based operating systems already come with Python installed so all you need to type to install the Shodan library is:

$ sudo easy_install shodan

If you're on Ubuntu and don't yet have easy_install available please run the following command first:

$ sudo apt-get install python-setuptools

After you've installed the library you can check that it's working simply by typing shodan into the terminal/ command prompt:

$ shodan
Usage: shodan [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

Commands:
  alert       Manage the network alerts for your account
  convert     Convert the given input data file into a...
  count       Returns the number of results for a search
  download    Download search results and save them in a...
  honeyscore  Check whether the IP is a honeypot or not.
  host        View all available information for an IP...
  info        Shows general information about your account
  init        Initialize the Shodan command-line
  myip        Print your external IP address
  parse       Extract information out of compressed JSON...
  scan        Scan an IP/ netblock using Shodan.
  search      Search the Shodan database
  stats       Provide summary information about a search...
  stream      Stream data in real-time.

Finally, initialize the tool using your API key which you can get either from the front page of Shodan or on your account page:

$ shodan init YOUR_API_KEY

Using the Command-Line Interface

The Shodan CLI provides access to most functions of the API in a user-friendly interface. It also includes a command to easily download data using the query credits from your API. Here's a quick video that shows how it works in action:

The basics of it are:

shodan download --limit <number of results> <filename> <search query>

For example, this is the command to download 500 results for the search query "product:mongodb" which returns Internet-facing MongoDB services:

shodan download --limit 500 mongodb-results product:mongodb

The results of the above command will be saved in a file called mongodb-results.json.gz. At this point, you can easily convert the file into CSV, KML or simply output a list of IP:port pairs by using the shodan parse command:

shodan parse --fields ip_str,port --separator , mongodb.json.gz

Programming with the Shodan API

The CLI should work for most purposes but sometimes you want to perform custom transformations on the banners as you're downloading them. Or you don't want to store the information in a local file. In those cases, you can use a convenient helper method provided by the Python library for Shodan called search_cursor() to iterate over the results:

from shodan import Shodan
from shodan.cli.helpers import get_api_key

api = Shodan(get_api_key())

limit = 500
counter = 0
for banner in api.search_cursor('product:mongodb'):
    # Perform some custom manipulations or stream the results to a database
    # For this example, I'll just print out the "data" property
    print(banner['data'])

    # Keep track of how many results have been downloaded so we don't use up all our query credits
    counter += 1
    if counter >= limit:
        break

Conclusion

We've discussed how to download data with the CLI, how to extract properties out of it, how to convert it to other file formats and how to process search results directly in Python. For a complete example that uses these skills check out how to track hacked websites.

Still need help? Get in touch!
Last updated on 28th May 2018