If you have an API plan then you get a certain number of query credits that you can spend each month. For people with the Shodan Membership that means you get 100 query credits per month while for the API plans it can range from 10,000 up to unlimited.

1 query credit = 100 results

Every query credit gets you up to 100 results, which means that you can download at least 10,000 results every month - regardless of the type of search you're performing.

Using the Command-Line Tool

The Shodan CLI provides a command to easily download data using the query credits from your API. Here's a quick video that shows how it works in action:

The basics of it are:

shodan download --limit <number of results> <filename> <search query>

For example, this is the command to download 500 results for the search query "product:mongodb" which returns Internet-facing MongoDB services:

shodan download --limit 500 mongodb-results product:mongodb

The results of the above command will be saved in a file called mongodb-results.json.gz. At this point, you can easily convert the file into CSV, KML or simply output a list of IP:port pairs by using the shodan parse command:

shodan parse --fields ip_str,port --separator , mongodb.json.gz

Programming with the Shodan API

The CLI should work for most people but sometimes you want to perform custom transformations on the banners as you're downloading them. Or you don't want to store the information in a local file. In those cases, you can use a convenient helper method provided by the Python library for Shodan called search_cursor() to iterate over the results:

import shodan

api = shodan.Shodan('Your API key')

limit = 500
counter = 0
for banner in api.search_cursor('product:mongodb'):
    # Perform some custom manipulations or stream the results to a database
    # For this example, I'll just print out the "data" property
    print(banner['data'])

    # Keep track of how many results have been downloaded so we don't use up all our query credits
    counter += 1
    if counter >= limit:
        break

Do you want to keep an eye on the latest results coming into Shodan? Want to create your own custom data feeds? Or want to grab a few thousand random web servers for your research? Using the Streaming API from Shodan you can directly subscribe to the raw data feed from the crawlers! The feed streams between 400-500 banners every second and depending on your API plan you have access to all or a fraction of it. And to get started with the stream you don't need any programming knowledge, just install the Shodan command-line tool and you're good to go. I've created a video that highlights some of the basic usage using the shodan command. Note that unless you use --limit or hit CTRL + C the stream will continue going forever:

shodan stream

At the heart is the stream command that when run by itself will simply stream all data you have access to and print it to your terminal. It won't store the data anywhere or perform any operations on it. Use this command if you'd like to explore random IPs on the Internet.

--ports

Often you're only interested in a certain type of service, and for those instances you can narrow down the stream using the --ports option. You can provide one port:

shodan stream --ports 23

Or many ports:

shodan stream --ports 23,1023

And without any other arguments it will once again just print the results to the terminal.

--datadir

Most of the time you also want to store the results so you're not throwing away information. To do so, simply create a directory and supply the --datadir option to the streaming command. This will result in the shodan tool storing the results from the stream in a file in the data folder, where the file name is the current date in YYYY-MM-DD.json.gz format:

mkdir shodan-data
shodan stream --datadir shodan-data

This is useful so you can keep the streaming command running and every day a new file will automatically be created for you. And then you can use the shodan parse command to extract the information you care about.

--limit

Sometimes you want to get a random sample of results. Lets say you'd like to see how many of the most recent 10,000 results are Nginx vs Apache vs Lighttpd etc. You can take periodic samplings to see how those trends change over time using your own computer. To get 10,000 web server results use the --limit option to make the stream command exit after it has received the provided number of results:

shodan stream --limit 10000 --ports 80

The above command would filter the stream for web servers running on port 80 (--port 80) and it would exit after 10,000 results were received (--limit).

I use the real-time stream for a lot of my own research and I hope you'll find it useful as well! If you have any thoughts, questions or suggestions please let me know @achillean