Ready To Boost Your Security Intelligence Collection Process?

It's time to take control of your threat landscape by increasing visibility and minimizing your attack surface. The chunks of untouched data floating around your network can be collected and correlated to establish intelligence and reduce uncertainty in your environment. Hurricane Labs has created a new tool called "Machinae," which is beneficial to your security intelligence collection process.

MACHINAE

Machinae is a tool that has been created to collect intelligence, about various security-related pieces of data, from public sites and feeds, including: IP addresses, domain names, URLs, email addresses, file hashes and SSL fingerprints. This tool was created to improve upon Automater, another excellent tool for collecting information. Read on, and/or check out more detailed (and continuously updated) information on the tool here: Machinae Security Intelligence Collector.

ENHANCEMENTS

Machinae provides enhancements to the four following areas:

  1. Codebase - Bring Automater to python3 compatibility, while making the code more pythonic
  2. Configuration - Use a more human readable configuration format (YAML)
  3. Inputs - Support JSON parsing out-of-the-box, without the need to write regular expressions, but still support regex scraping when needed
  4. Outputs - Support additional output types, including JSON, while making extraneous output optional

INSTALLATION

Before installing Machinae these are the things you should know:

  • Machinae can be installed using pip3: pip3 install machinae
  • Or, if you're feeling adventurous, can be installed directly from github: pip3 install git+https://github.com/HurricaneLabs/machinae.git
  • You will need to have the dependencies that are required on your system for compiling Python modules (on Debian based systems,python3-dev), as well as the libyaml development package (on Debian based systems, libyaml-dev).
  • You'll also want to grab the latest configuration file and place it in /etc/machinae.yml.

CONFIGURATION

Machinae supports a simple configuration merging system that will allow you to make adjustments to the configuration without modifying the machinae.yml we provide you, making configuration updates a snap.

This is done by finding a system-wide default configuration (default /etc/machinae.yml), merging that into a system-wide local configuration (/etc/machinae.local.yml) and finally a per-user local configuration (~/.machinae.yml). The system-wide configuration can also be located in the current working directory, that can be set using the MACHINAE_CONFIG environment variable, or of course by using the -c or --config command line options. Configuration merging can be disabled by passing the --nomerge option, which will cause Machinae to only load the default system-wide configuration (or the one passed on the command line).

EXAMPLE

Let's look at an example. Say you'd like to enable the Fortinet Category site, which is disabled by default. You could modify /etc/machinae.yml, but these changes would be overwritten by an update.

Instead, you can put in either /etc/machinae.local.yml or ~/.machinae.yml:

fortinet_classify:
    default: true

Or, conversely, to disable a site, such as Virus Total pDNS:

vt_ip:
    default: false
vt_domain:
    default: false

USAGE

Machinae usage is very similar to Automater:

usage: machinae [-h] [-c CONFIG] [-d DELAY] [-f FILE] [--nomerge] [-o {D,J,N}]
                [-O {ipv4,ipv6,fqdn,email,sslfp,hash,url}] [-q] [-s SITES]
                targets [targets ...]

  • See above for details on the -c/--config and --nomerge options.
  • Machinae supports a -d/--delay option, like Automater. However, Machinae uses 0 by default.
  • Machinae output is controlled by two arguments: 1.) -o controls the output format, and can be followed by a single character to indicated the desired type of output: N is the default output ("Normal"); D is the default output, but dot characters are replaced; J is JSON output 2.) -f /--file specifies the file where output should be written. The default is "-" for stdout.
  • Machinae will attempt to auto-detect the type of target passed in (Machinae refers to targets as "observables" and the type as "otype"). This detection can be overridden with the -O /--otype option.
  • The choices are listed in the usage: Pass a comma separated list of sites to run (use the top level key from the configuration). Pass the special keyword all to run through all services including those marked as "default: false"Note that in both cases, otype validation is still applied.
  • Lastly, a list of targets should be passed. All arguments other than the options listed above will be interpreted as targets.

OUT-OF-THE-BOX DATA SOURCES

Machinae comes with out-of-the-box support for the following data sources:

  • IPVoid
  • URLVoid
  • URL Unshortener (http://www.toolsvoid.com/unshorten-url)
  • Malc0de
  • SANS
  • Telize GeoIP
  • Fortinet Category
  • VirusTotal pDNS (via web scrape - commented out)
  • VirusTotal pDNS (via JSON API)
  • VirusTotal URL Report (via JSON API)
  • VirusTotal File Report (via JSON API)
  • Reputation Authority
  • ThreatExpert
  • VxVault
  • ProjectHoneypot
  • McAfee Threat Intelligence
  • StopForumSpam
  • Cymru MHR
  • ICSI Certificate Notary
  • TotalHash (disabled by default)
  • DomainTools Parsed Whois (Requires API key)
  • DomainTools Reverse Whois (Requires API key)
  • DomainTools Reputation (Requires API key)
  • IP WHOIS (Using RIR REST interfaces)
  • Additional data sources on the way

DISABLED BY DEFAULT

The following sites are disabled by default:

  • Fortinet Category (fortinet_classify)
  • TotalHash (totalhash_ip)
  • DomainTools Parsed Whois (domaintools_parsed_whois)
  • DomainTools Reverse Whois (domaintools_reverse_whois)
  • DomainTools Reputation (domaintools_reputation)

OUTPUT FORMATS

Machinae comes with a limited set of output formats: normal, normal with dot escaping, and JSON. We plan to add additional output formats in the future.

KNOWN ISSUES

  • Some ISP's on IPvoid contain double-encoded HTML entities, which are not double-decoded.

OVERALL

Overall, Machinae is a tool aimed at making the analysis of security-related pieces of data easier for you. The collection and evaluation of this type of intelligence can strengthen your strategy and security infrastructure as a whole.