Introducing Data Feeds for Search Results

It's now possible for enterprise customers to subscribe to a data feed of search results. Instead of running a search query every day to ask for new results you can stay connected to a new API endpoint and Shodan will send you any banners that meet the search criteria. Read on to learn more about streaming queries.

There are 2 parts of the Shodan API: REST and Streaming.

• The REST API lets you do IP lookups, run search queries and setup network monitoring among other things. Most of Shodan's capabilities are accessible via the REST API. You send a request to the REST API and you get a response.
• The Streaming API (aka Firehose) is much smaller and provides methods to get real-time streams of data. You connect to a streaming method and then you get a never-ending stream of data until you disconnect from the Streaming API.

Shodan actually uses the Streaming API to build the REST API and send out notifications in Monitor. Here is an overview of our architecture:

The Shodan crawlers collect data, publish it to the Streaming API, the REST API consumes the Streaming API to store/ index the data and the websites use both APIs to access the data. Shodan is an API-first company and every website is built on-top of the same public API that you have access to.

We've now added the ability to create custom data feeds from search queries using the new /shodan/custom method of the Streaming API:

There is a difference though between the search query syntax of the REST API/ website and of this new custom Streaming API method: streaming queries are case-sensitive. Otherwise you should be able to take your existing search query, plug it into the Streaming API and get a real-time data feed.

The Shodan CLI has been updated with a new --custom-filters option to run streaming queries. Here are a few examples:

• Compromised services in the US:

shodan stream --custom-filters "tag:compromised country:US"  

• Elastic or Kubernetes services deployed on Google Cloud:

shodan stream --custom-filters 'org:"Google LLC" product:Elastic,Kubernetes'  

• Industrial control systems in Germany, Switzerland or France:

shodan stream --custom-filters "tag:ics country:CH,DE,FR"  

Streaming queries are an efficient way to keep track of new results and lets you focus on the data that is relevant for your use case.