Do you want to keep an eye on the latest results coming into Shodan? Want to create your own custom data feeds? Or want to grab a few thousand random web servers for your research? Using the Streaming API from Shodan you can directly subscribe to the raw data feed from the crawlers! The feed streams between 400-500 banners every second and depending on your API plan you have access to all or a fraction of it. And to get started with the stream you don't need any programming knowledge, just install the Shodan command-line tool and you're good to go. I've created a video that highlights some of the basic usage using the shodan command. Note that unless you use --limit or hit CTRL + C the stream will continue going forever:
shodan stream
At the heart is the stream command that when run by itself will simply stream all data you have access to and print it to your terminal. It won't store the data anywhere or perform any operations on it. Use this command if you'd like to explore random IPs on the Internet.
--ports
Often you're only interested in a certain type of service, and for those instances you can narrow down the stream using the --ports option. You can provide one port:
shodan stream --ports 23
Or many ports:
shodan stream --ports 23,1023
And without any other arguments it will once again just print the results to the terminal.
--datadir
Most of the time you also want to store the results so you're not throwing away information. To do so, simply create a directory and supply the --datadir option to the streaming command. This will result in the shodan tool storing the results from the stream in a file in the data folder, where the file name is the current date in YYYY-MM-DD.json.gz format:
mkdir shodan-data
shodan stream --datadir shodan-data
This is useful so you can keep the streaming command running and every day a new file will automatically be created for you. And then you can use the shodan parse command to extract the information you care about.
--limit
Sometimes you want to get a random sample of results. Lets say you'd like to see how many of the most recent 10,000 results are Nginx vs Apache vs Lighttpd etc. You can take periodic samplings to see how those trends change over time using your own computer. To get 10,000 web server results use the --limit option to make the stream command exit after it has received the provided number of results:
shodan stream --limit 10000 --ports 80
The above command would filter the stream for web servers running on port 80 (--port 80) and it would exit after 10,000 results were received (--limit).
I use the real-time stream for a lot of my own research and I hope you'll find it useful as well! If you have any thoughts, questions or suggestions please let me know @achillean