For fun I decided to see whether I can figure out how many Minecraft players are online at the moment. And it turns out that it's fairly straight-forward so here's how I did it.

As of now June 1st 2017 at 18:55 there are 96,418 players online on public servers.

To get started I downloaded the latest list of Minecraft servers from Shodan:

shodan download --limit -1 minecraft-servers product:minecraft port:25565

Now the next task is to parse that list of servers and request the number of players that are currently online. To speed things up the plan is to asynchronously perform the requests to the Minecraft servers using the gevent library in Python. It lets you write code that looks synchronous but actually runs asynchronously which means you can perform many connections in parallel. This is the usual template I use when grabbing a bunch of data using gevent:

#!/usr/bin/env python
#
# Shodan Async Workers

## Configuration
NUM_WORKERS = 100


# Make the stdlib async. This is where the gevent magic happens
import gevent.monkey
gevent.monkey.patch_all(subprocess=True, sys=True)


from gevent.pool import Pool
from shodan.helpers import iterate_files
from socket import setdefaulttimeout, socket, AF_INET, SOCK_STREAM

setdefaulttimeout(2.0)

def worker(banner):
    # Here's where you do the network stuff
    # Example:
    # con = socket(AF_INET, SOCK_STREAM)
    # con.connect((banner['ip_str']
    # con.send('hello world\n')
    # data = con.recv(5120)
    return True

def main(files):
    pool = Pool(NUM_WORKERS)

    # Loop through the banners in the file(s) and launch a worker
    # for each banner. When the pool is full it will cause the loop to
    # block until a worker finishes and opens up a spot in the pool.
    for banner in iterate_files(files):
        pool.spawn(worker, banner)

    # Wait for the workers to finish up
    pool.join()

    return True


if __name__ == '__main__':
    import sys
    sys.exit(main(sys.argv[1:])

If you're working with Shodan data files I recommend checking out the shodan.helpers.iterate_files() method since it'll make it easy for you to access the banners. You can give it either a single file:

for banner in iterate_files('minecraft-data.json.gz'):
    ...

Or you can provide it a list of files:

for banner in iterate_files(['minecraft-2017-04.json.gz', minecraft-2017-05.json.gz']):
    ...

To get the player count I added a method in the worker() that looks up the Minecraft info based on their current protocol and kicked it off:

$ python global-player-count.py minecraft-data.json.gz
96418

And that's how I'm now keeping track of how many players are at any moment online on Minecraft!

Note that this method only looks at Minecraft servers running on the default port (25565) and that are publicly-accessible on the Internet.

There's been much focus on MongoDB, Elastic and Redis in terms of data exposure on the Internet due to their general popularity in the developer community. However, in terms of data volume it turns out that HDFS is the real juggernaut. To give you a better idea here's a quick comparison between MongoDB and HDFS:

Even though there are more MongoDB databases connected to the Internet without authentication in terms of data exposure it is dwarfed by HDFS clusters (25 TB vs 5 PB). Where are all these instances located?

Most of the HDFS NameNodes are located in the US (1,900) and China (1,426). And nearly all of the HDFS instances are hosted on the cloud with Amazon leading the charge (1,059) followed by Alibaba (507).

The ransomware attacks on databases that were widely publicized earlier in the year are still happening. And they're impacting both MongoDB and HDFS deployments. For HDFS, Shodan has discovered roughly 207 clusters that have a message warning of the public exposure. And a quick glance at search results in Shodan reveals that most of the public MongoDB instances seem to be compromised. I've previously written on the reason behind these exposures but note that both products nowadays have extensive documentation on secure deployment.

Technical Details

If you'd like to replicate the above findings or perform your own investigations into data exposure, this is how I measured the above.

1. Download data using the Shodan command-line interface:

   shodan download --limit -1 hdfs-servers product:namenode
   

2. Write a Python script to measure the amount of exposed data (hdfs-exposure.py):

   from shodan.helpers import iterate_files, humanize_bytes
   from sys import argv, exit
   
   
   if len(argv) <=1 :
       print('Usage: {} <file1.json.gz> ...'.format(argv[0]))
       exit(1)
   
   
   datasize = 0
   clusters = {}
   
   
   # Loop over all the banners in the provided files
   for banner in iterate_files(argv[1:]):
       try:
           # Grab the HDFS information that Shodan gathers
           info = banner['opts']['hdfs-namenode']
           cid = info['ClusterId']
           # Skip clusters we've already counted
           if cid in clusters:
               continue
           datasize += info['Used']
           clusters[cid] = True
       except:
           pass
   
   
   print(humanize_bytes(datasize))
   

3. Run the Python script to get the amount of data exposed:

   $ python hdfs-exposure.py hdfs-data.json.gz
   5.0 PB