Wednesday, March 26, 2014

Extending Burp Proxy With Extensions

By Chris Bush.

The world of information security is awash with tools to help security practitioners do their jobs more easily, accurately and productively. Regardless of whether you are responsible for doing PCI audits, network vulnerability assessments, enterprise risk assessments, social engineering, or what have you, there’s a tool for that. Usually there are several. Some are good, some not so much. One of the reasons a tool may or may not ultimately be useful is the ability for its functionality to be customized or extended to meet the needs of the practitioner using it. This is never truer than in application security, where every application the security tester confronts is different from the last. Bespoke applications demand bespoke security testing, and this requires that the tools used by the application security professional be not only robust and feature rich, but customizable in a way that allows them to be rapidly extended to fit to the needs of the job at hand.

Pound for pound (or maybe dollar for dollar), the Burp Suite is one of the best tools an application security professional can have in their tool kit. It has capabilities and features on par with, or exceeding, those of big-name commercial application scanners costing tens of thousands of dollars more, all in a single UI where all of the tools integrate and work together seamlessly. Often overlooked is the fact that Burp includes an extensibility framework that allows you to extend Burp’s functionality in a number of useful ways, through loading 3rd party extensions, or writing your own.

An Overview of Extending Burp

The Burp extensibility framework provides the ability to easily extend Burp’s functionality in many useful ways, including:

  • Analyzing and modifying HTTP requests/responses
  • Customizing the placement of attack insertion points within scanned requests
  • Implementing custom scan checks
  • Implementing custom session handling
  • Creating and issuing HTTP requests
  • Controlling and initiating actions within Burp, such as initiating scans or spidering
  • Customizing the Burp UI with custom tabs and context menus
  • Much, much more

There are a growing number of 3rd party extensions available that you can download and use. The BApp Store was recently created, providing access to a number of useful extensions that you can download and add to Burp. Beginning with Burp Suite version 1.6 beta, released along with the BApp Store on March 4, 2014, access to the BApp Store is also provided directly from within Burp’s UI.

Additionally, there are a number of examples available in the Portswigger blog that provide an excellent starting point for writing your own extension, depending on what you are trying to accomplish. Go to the Burp Extender page to see an overview of some of these examples, including a links to the full blog posts and downloadable code. And of course, you can always turn to the Burp Extension User Forum for help with writing your own extensions, and more examples contributed by the user community.

In the rest of this article, we’ll provide a quick overview of the Burp Extender tool, which you will use to load extensions and configure Burp to run those extensions. Then we’ll dive right into writing our own custom extension, and create an extension that performs a couple of custom passive scans.

Burp Extender Tool

First, let’s take a look at the Burp Extender Tool. When you select the Extender tab in Burp Suite, there are four sub-tabs that provide access to the functionality and configuration options of the Extender.


The Extensions tab (shown below) allows you to load and manage the extensions you are using in Burp. From this you can add and remove your extensions, as well as manage the order in which extensions and their resources are invoked. The panel at the bottom provides details on a selected extension, as well as tabs to display any output written by the extension, as well as any error messages produced by the extension.

BApp Store

The BApp Store tab, new with version 1.6 beta of Burp Suite, provides direct access to downloadable extensions from the Portswigger BApp Store.


The APIs tab essentially just provides a convenient reference to the Burp Extensibility API. From here, you can also download the Java interface files, for inclusion in your Java project, as well as download the Javadocs as a set of HTML files that you can access locally for reference.


Finally, the Options tab is where you will configure things like the location of different environments required to run your extensions, depending on whether the extension is written in Java, Python, or Ruby. To run extensions written in Python requires the use of Jython, a Python interpreter that is written in Java. Similarly, to run extensions written in Ruby requires the use of JRuby, a Ruby interpreter written in Java. The Options tab allows you to specify the locations of the Jython JAR file or JRuby JAR file respectively. Download the most recent versions of these and configure Burp Extender to point to them if you will be writing extensions in Python or Ruby.

Loading and managing extensions and configuring the runtime options needed is very straightforward and simple. Refer to the Burp Extender Help page online for additional information.

Writing Your Own Extensions

You can write your own extensions in Burp using the Burp Extensibility API. The API consists of a number of Java interfaces that you will provide implementations of, depending upon what you are trying to accomplish. It is beyond the scope of this article to cover the entire API in detail. Refer to the Burp Extender Javadoc page online for a complete description. Instead, we’ll cover a few key interfaces that are used by all extensions, some that are of practical use to nearly all extensions, as well as some that will be useful in understanding the example extension that will be presented later in this article. As indicated previously, Burp extensions can be written in Java, Python, or Ruby. Choose the language you are most familiar with. We’ll use Python here, for its familiarity and the ease of development as an interpreted language. While code examples provided in this article will be in Python, they should be easily read and understood by anyone with a programming background in Java or another high-level language.


The IBurpExtender class is the foundation of every extension you will write. A Burp extension must provide an implementation of IBurpExtender that is declared public, and implements a single method, called registerExtenderCallbacks. This method is invoked when the extension is loaded into Burp, and is passed an instance of the IBurpExtenderCallbacks class. This class provides a number of useful methods that can be invoked by your extension to perform a variety of actions. At its very simplest, an extension in Burp starts out like this:

 from burp import IBurpExtender
class BurpExtender(IBurpExtender):
    def registerExtenderCallbacks(self, callbacks):
        # put your extension code here


As indicated above, an instance of this class is passed to the registerExtenderCallbacks method in your IBurpExtender implementation when the extension is loaded in Burp. Through this callbacks object, you have access to a wide variety of useful methods that will help you create your extension. I’ll point out just a few, as these will be of importance as we build out our example custom scanner extension to follow.

  • getHelpers – Obtain an instance of the IExtensionHelpers class, which provides numerous useful "helper" methods that can be used to add functionality to an extension.
  • setExtensionName – Sets the name of the extension as it will appear in the Extensions tab in Burp Suite.
  • registerScannerCheck – Used to register an extension as a custom Scanner check.
  • applyMarkers – Used to apply markers, or highlights, to areas of a request or response. For instance, this may be used to mark a vulnerable parameter in a request, or an area of the response that indicates a vulnerability, such as a reflected XSS payload.

This is just a small taste. There are many other methods available to you in an instance of IBurpExtenderCallbacks. Consult the Burp Extender Javadoc page online for complete details.


The IExtensionHelpers class provides access to another large set of methods that you will undoubtedly find useful. It will be the rare extension that doesn’t get an instance of this class, which is obtained using a call to the getHelpers() method of IBurpExtenderCallbacks (see above). Just a few examples of the methods provided by this class are the following:

  • analyzeRequest – Used to analyze an HTTP request, and obtain various details, such as a list of parameters , headers, and more.
  • analyzeResponse – Used to analyze an HTTP response, and obtain various details, such as a list of cookies, headers, the response code, and more.
  • urlEncode – Perform URL encoding on a piece of data.
  • urlDecode – Perform URL decoding on a piece of data.
  • indexOf – Searches a piece of data for a specified pattern. This is very useful for search a request or a response for a specific value, such as PII, or examining the response for a parameter value that was in the corresponding request.
  • bytesToString – Converts data from an array of bytes to a String object. Many of the methods in the Burp API operate on an array of bytes, so this comes in quite handy.


Extensions implement this interface when they are going to be used to perform custom scan checks. Your extension must call the registerScannerCheck method of the IBurpExtenderCallbacks class to tell Burp that it is implementing this interface. Burp will then know to use your extension when performing active or passive scans on a base request/response pair, as well as to report any issues (see IScanIssue below) identified by your custom scan checks. The following three methods may be implemented by an IScannerCheck class:

  • consolidateDuplicateIssues – This method is invoked when the custom Scanner check has reported multiple issues for the same URL. You use this to tell Burp whether to keep the existing issue, or replace with the new issue, based on whatever criteria you decide.
  • doActiveScan – This method is invoked for each insertion point that is actively scanned by Burp. An implementation of this will then construct a new request, based on the base request passed, and insert a test payload into the specified insertion point. It will then issue that new request, and examine the response for an indication that the inserted payload reveals a vulnerability.
  • doPassiveScan – This method is invoked for each base request/response pair that Burp passively scans. An implementation of this will typically examine the base request and/or response for patterns of interest. No new requests should be generated from a passive scan.

Both the doActiveScan and doPassiveScan methods must return a list of IScanIssue objects, which Burp will then automatically include in the Scanner issues report.


The IScanIssue class provides a representation of a Scanner issue. An extension may retrieve current issues (IScanIssue objects) from the Scanner tool by registering an IScannerListener callback, or by calling the getScanIssue method of the IBurpExtenderCallbacks class. Scanner issues can be added to Burp by implementing the IScanIssue class in your extension, and calling the addScanIssue method of the IBurpExtenderCallbacks class with specific instances. Additionally, Scanner issues can be added via a custom scan check, by creating a list of instances of IScanIssue that is returned by either the doPassiveScan or doActiveScan methods of an IScannerCheck implementation.

Implementing the IScanIssue interface involves implementing a constructor method to set the details of the Scanner issue, as well as a number of getter methods to retrieve those details. We won’t go into details of the various methods here, as they will often be as simple as setting a class variable with a value passed to the constructor, and implementing a getter method that returns this value.

A Custom Passive Scanner

To conclude our discussion, we will present an example extension that implements a custom scanner, which will perform two different passive scan checks:

  • Reflection Checks – Using the values of the parameters in the base request that is being passively scanned, this check searches the corresponding response for those same values, providing a candidate point for further testing for reflected XSS vulnerabilities.
  • Regular Expression Match – Can be used to examine the base response of a passive scan request, looking for any string that matches a particular regular expression. In the context of this example extension, this check is used to do a customized search of application responses using a regular expression designed to match potentially sensitive personally identifiable information (PII) unique to a specific, non-US, country.

The full source code for this example extension can be downloaded from our GitHub page. This extension is written in Python, so to try it out you will first need to download the latest Jython library from The Jython Project, and configure the Burp Extender to use it. Then add the extension, and try it out.

The source code is extensively documented with comments. With the information provided above, as well as the Burp API Javadocs and the comments in the code, it should be easy to grasp what’s going on in the code. In the remainder of this article, I’ll go into a little detail for a few key sections of the code that may be particularly interesting or require some further context for understanding.

Earlier, we showed the simplest example of a Burp extension that does nothing. Recall that at a minimum an extension must implement the IBurpExtender interface, which has one method – registerExtenderCallbacks. Let’s take a quick look at the implementation of our registerExtenderCallbacks method.

 Line 15-26
def registerExtenderCallbacks(self, callbacks):
      # Put the callbacks parameter into a class variable so we have class-level scope
      self._callbacks = callbacks

      # Set the name of our extension, which will appear in the Extender tool when loaded
      self._callbacks.setExtensionName("Custom Passive Scanner")
      # Register our extension as a custom scanner check, so Burp will use this extension
      # to perform active or passive scanning and report on scan issues returned

The registerExtenderCallbacks method is passed an instance of the IBurpExtenderCallbacks class. On line 17 above, we are simply storing this callbacks object in a class variable, so that it has class-level scope, allowing any other methods within our BurpExtender class to access it. On line 20, we use one of the methods of the callbacks object, setExtensionName, to set the name of our extension. This is how the extension will be identified in the Extender tool when it is loaded. Finally, on line 24, we call the registerScannerCheck method of the callbacks object. This tells Burp that our extension implements a custom scanner check, and Burp will now call the doActiveScan and doPassiveScan methods of our extension whenever it is performing an active or passive scan, respectively. In our extension, we have only implemented doPassiveScan.

Our implementation of doPassiveScan makes use of a custom class that we have created, called CustomScans, which is not an implementation of anything in the Burp API.

 Line 47
self._CustomScans = CustomScans(baseRequestResponse, self._callbacks)

As we see above, within doPassiveScan, an instance of this class is created, passing the base request/response pair, as well as our instance of IBurpExtenderCallbacks that was created as a class variable in the registerExtenderCallbacks earlier. The purpose of the CustomScans class is to implement one or more methods that we can call that perform unique scan checks against the base request/response pair being passively scanned. In this extension, we’ve implemented two methods in CustomScans, called findReflections and findRegEx, whose purpose was described above.

Next, the extension’s implementation of doPassiveScan calls the findReflections method of CustomScans. This method will examine the base request/response pair, passed previously to the constructor for CustomScans, and identify any request parameters whose value appears in the corresponding response.

 Line 51-62
issuename = "Possible Reflected XSS"
issuelevel = "Information"
issuedetail = """The value of the $param$ request parameter appears
        in the corresponding response.  This indicates that there is a
        potential for reflected cross-site scripting (XSS), and this URL
        should be tested for XSS vulnerabilities using active scans and
        thorough manual testing and verification. """

tmp_issues = self._CustomScans.findReflections(issuename, issuelevel, issuedetail)
# Add the issues (if any) from findReflections to the list of issues to be returned
scan_issues = scan_issues + tmp_issues

Three arguments passed to findReflections provide information used to construct any new scan issues, including the issue name, level (or severity), and issue details. Note that the argument representing the issue details contains HTML tags. Burp will interpret these tags and render the issue details in its UI accordingly. Finally, the findReflections method returns a list of scan issues, in tmp_issues, which is then appended to the list of issues, scan_issues, which will ultimately be returned to Burp from doPassiveScan.

Following the above code, lines 69-81 follow a similar patter, calling CustomScans.findRegEx, and appending any resulting issues to the scan_issues list. Lines 85-88 then returns scan_issues if it is not empty, else it returns None (think null). Burp will then include the returned issues, if any, in the Scanner issues report.

The findReflections and findRegEx methods of CustomScans should be fairly straightforward to understand, and each follows a very similar flow. Lines 127-136 of findReflections, and lines 160-169 of findRegEx in particular follow a very similar pattern, which we’ll explain below.

 Line 127-136 (findReflections)
offset[0] = start
offset[1] = start + len(paramVal)
# Create a ScanIssue object and append it to our list of issues, marking
# the reflected parameter value in the response.
        [self._callbacks.applyMarkers(self._requestResponse, None, offsets)],
        issuename, issuelevel, issuedetail.replace("$param$", paramName)))

The first three lines set up an array that is used to store offsets used to apply a marker to a region of the response, in this case to highlight the reflected parameter value. The first array element contains the start position of the identified value, and the second element contains its end position. This array of two values is then appended to a list, called offsets, which will be passed to the applyMarkers method of IBurpExtenderCallbacks when the new scan issue is created. The applyMarkers method expects a List of arrays in its last two arguments, each array containing the start and end values of regions to be marked in the request and response respectively.

The last four lines above create an instance of our ScanIssue class, which is our extension’s implementation of the IScanIssue interface, by calling its constructor using a number of arguments. We then append that new instance of ScanIssue to a list, called scan_issues, which will be returned back to our caller, doPassiveScan. In the call to the constructor for our Scan, we call the applyMarkers method of IBurpExtenderCallbacks, passing the base request/response pair, and offsets for applying markers to the request (None in this case), and to the response, using the list, offsets, described above. The last three arguments to the ScanIssue constructor provide the issue name, issue level (severity), and issue detail information that was passed as arguments to findReflections. Here, we are replacing a token in the literal string passed in the issuedetail argument with the actual parameter value that was reflected in the response. This adds useful detail to the new scan issue for the tester, and also makes it so Burp will identify the issue as a unique instance when it calls our extension’s consolidateDuplicateIssues method.

The findRegEx method in CustomScans follows a similar pattern to that described above for findReflections. It makes use of Python’s regular expression operations to search the response, but otherwise uses the same techniques to create new scan issues as in findReflections.

One part of findReflections that is perhaps non-intuitive when examining the code is the following:

 Line 122
if len(paramVal) > 3:

Here we are checking the length of the variable paramVal, which contains the value of the current parameter we are checking for reflection of. In order to prevent a lot of noise from coincidental matches, we are simply checking that the parameter’s value is greater than three characters long. This is a fairly simplistic approach, and you are free to try any heuristic you can think of here. Regardless of what you try, since this is a passive scan, eliminating potentially coincidental matches may also eliminate true reflection of parameter values that may in fact be vulnerable to XSS. Remember, the point of this passive XSS scan is only to identify candidate points for further examination and testing, not to actually identify XSS vulnerabilities. Caveat emptor.

Lastly, there are some additional classes from the Burp Extensibility API that are being used in the example extension that have not been covered here. These classes are not explicitly implemented in the extension, but are used implicitly, typically as return values from other methods in the Burp API. They are mentioned briefly below, but you are encouraged to examine these in more detail in the Burp Extender Javadoc page online.

  • IHttpRequestResponse – Representation of an HTTP message
  • IHttpService – Representation of an HTTP service, to which requests can be sent
  • IRequesetInfo – Representation of an HTTP request
  • IParameter – Representation of an HTTP request parameter


As you can see, while it does involve some programming, creating Burp extensions is really quite straightforward, and should be no problem for anyone with a reasonable programming or scripting background. In the above example, we have the basics of a fairly useful custom scan check extension that performs two different passive scan checks, in around 160 lines of code (excluding comments).

Try out the example above, visit the new BApp Store, study the Burp Extensibility APIs, and you’re sure to come up with ideas for your own extensions that will help you do your job more easily, accurately, and productively, and get better results by customizing Burp to meet your particular needs. Best of luck.

Tuesday, March 18, 2014

Combatting AppScan's "Scan out of session"

By Kunal Garg.

Web application scanners may be full of repetition and obvious vulnerabilities but they do have their place in a web application penetration test. While they should never be used as the sole way to identify vulnerabilities, they can provide baseline and act as another available tool to achieving maximum results. All web application scanners are different and some require finer tuning then others. One common issue we see with IBM's AppScan is the "Scan out of session" error. This blog post aims to give advice around setting up the scan and working around the issue.

When running post authentication scans “In-session” detection is important concept to maximize scan coverage. Anytime the scan will go out of session, notification “scan out of session”will be displayed to user and scan will be suspended.

With the in session management we basically select a unique pattern on an in-session page, which Appscan continually polls to find out if scan is in session or not. This pattern needs to be unique and should be available on post authentication pages. It can be any text such as “welcome userabc” displayed after a specific user logs on or it can be a logout button (if present on all the pages).

Recording the login

First step in configuring the in-session pattern is to record the login using Appscan macro. Once the login is recorded, tick the checkbox “I want to configure In-session detection” and click on next.

Notice all the URL’s recorded in login macro previously appear here, and select the page which is post authentication and contains our unique identifier. In our case, test application after login routes to “main.apsx”, this page therefore is selected as In-session page (Right click and set as In-session).

Now, it’s time to select the In-session pattern this can be selected using the “Select in session pattern” button.

Usually, Appscan will select the session identifier on its own. But it is always advisable to look into the pattern and change the pattern if it’s not unique. In my personal experience with the automatically selected identifiers scans tend to run out of session.

Session pattern can be either selected from the page or its response body in the appscan browser window as shown below.

Session pattern is marked as “signoff”.

If the scan goes out of session there are certain points which we need to consider:

  1. Session cookies are not properly tracked. If the session cookies are not getting tracked it can be marked for tracking from “Login management”.

  2. Check if the application is still accessible.
  3. Check if the user account is not locked out.

Note: While run In-session scans make sure that, login and logout pages are out of scope, and please take due care while running automated scans and configuring them.

Tuesday, March 11, 2014

Identifying Malware Traffic with Bro and the Collective Intelligence Framework (CIF)

By Ismael Valenzuela.

In this post we will walk through some of the most effective techniques used to filter suspicious connections and investigate network data for traces of malware using Bro, some quick and dirty scripting and other free available tools like CIF.

This post doesn’t pretend to be a comprehensive introduction to Bro (check the references section at then end of the post for that) but rather a quick reference with tips and hints on how to spot malware traffic using Bro logs and other open source tools and resources.

All the pcap files used throughout this post can be obtained from GitHub. Some of them have been obtained from the large dataset of pcaps available at contagiodump.

Finally, if you are new to Bro I suggest that you start by downloading the latest version of Security Onion , a must-have Linux distribution for packet ninjas. Since version 12.04.4 Security Onion comes with the new Bro 2.2 installed by default so all you need to do is to open the terminal, grab the samples and maybe some coffee… (There is never enough coffee!).

Traffic Analysis with Bro

We will start replaying our first sample through Bro with:
 $ bro –r sample1.pcap local 

This command tells Bro to read and process sample1.pcap, pretty much like tcpdump or any other pcap tool does. By adding the keyword “local” at the end of the command, we ask Bro to load the ‘local’ script file, which in SecurityOnion is located in /opt/bro/share/bro/site/local.bro.

When the command is completed, Bro will generate a number of logs in the current working directory. These logs are highly structured, plain text ASCII and therefore Unix friendly, meaning that you can use your command line kung-fu with awk, grep, sort, uniq, head, tail and all the other usual suspects.

To see the summary of connections for sample1.pcap we can have a quick look at conn.log:

 $ cat conn.log

The figure above shows an excerpt of the output of this command. Notice how the output of Bro logs is structured in columns, each of them representing different fields. These fields are shown in the 7th line of the output header, starting with "ts" (timestamp in seconds since epoch) and "uid" (a unique identifier of the connection that is used to correlate information across Bro logs). Refer to the Bro documentation to learn more about the rest of the fields.

 #separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open 2014-03-07-13-51-01
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytesresp_pkts resp_ip_bytes tunnel_parents
#types time string addr port addr port enum string interval count count string bool count string count count count count table[string]

We can observe a number of connections to port 80 (tcp) and port 53 (udp). Conn.log also reports the result of these connections under the field conn_state. Let’s have a closer look at that using bro-cut an awk-based field extractor for Bro logs.

 $ cat conn.log | bro-cut id.orig_h, id.orig_p, id.resp_h, id.resp_p, proto, conn_state
… 49508 80 tcp REJ 49510 80 tcp REJ 57852 53 udp SF 49509 80 tcp REJ 57399 53 udp SF 49510 80 tcp REJ 57456 53 udp SF 49511 80 tcp S0 62602 53 udp SF 54957 53 udp SF 49511 80 tcp SH 49512 80 tcp S0 64623 53 udp SF 53702 53 udp SF 49512 80 tcp SH 49513 80 tcp S0 52164 53 udp SF 49513 80 tcp SH 49516 80 tcp S0 54832 53 udp SF 49516 80 tcp SH 49517 80 tcp S0 64102 53 udp SF 51110 53 udp SF 49517 80 tcp SH 49518 80 tcp S0 55957 53 udp SF 49519 80 tcp S0 58988 53 udp SF 49518 80 tcp SH

In this case, we can observe that some of the connections attempted on port 80 were rejected (REJ), while others never had a reply (S0) or left the connection half-open (SH, which means a SYN-ACK from the responder was never seen). The reason for this behavior is that sample1.pcap was obtained from one of my sandboxes where is a Virtual Machine running Remnux with fakedns and netcat listening on port 80 instead of a full web server.

Since we know that there is some http traffic going on here, let’s have a look at another log generated by Bro: http.log

 $ cat http.log | bro-cut id.orig_h, id.orig_p, id.resp_h, id.resp_p, host, uri, referrer 49493 80 / - 49495 80 / - 49511 80 / - 49512 80 / - 49513 80 / - 49516 80 / - 49517 80 / - 49518 80 / - 49519 80 / -

Anything weird here? Definitely! The host field of the http.log shows entries that don’t seem to correspond with normal browsing.

A closer look at the dns.log produced by Bro will confirm this:

 $ cat dns.log | bro-cut query | sort –u
(output truncated)

Looking at the length of the domains requested we could observe a pattern. First of all we will cut out the TLDs (com, info, net…) and then calculate the length of each of the strings.

 $ cat dns.log | bro-cut query | sort -u | cut -d . -f1 > domains-withoutTLD
 $ for i in `cat domains-withoutTLD`; do echo "${#i}"; done | sort –u


So all these strings are within a close range of 34 to 43 characters long. Casualty? Not really, a variant of the ZeuS botnet, the so-called ZeuS Gameover, is known for implementing P2P and Domain Generation Algorithm (DGA) communications to determine the current Command and Control (C&C) domain. When these bots can’t communicate with its botnet via P2P, DGA is used. The domain names generated by ZeuS Gameover consist of a string with a length of 32 to 48 chars and one of the following TLDs: ru, com, biz, net or org. The list contains over 1000 domains and changes every 7 days, based on the current date.

A regular expression like this can be used to search for ZeuS domains:


ZeuS Gameover has been reported as one of the most active banking Trojan in 2013, along with Citadel, another well-known piece of malware that has targeted a large number of financial organizations with focus on Europe and the Middle East. maintains a list of 1000 valid domains for ZeuS Gameover and updates it every week. A simple bash script can compare a list of domains obtained from dns.log to the list published by

 $ cat dns.log | bro-cut query | sort -u | > domains

$ for i in `cat domains`; do grep $i ZeusGameover_Domains; done

SSL Traffic and Notice.log

Malware authors are making increased use of SSL traffic to mask communications with C&C servers, data exfiltration and other malicious actions. Since decrypting SSL communications is not feasible in most of the scenarios, malware analysts must employ other techniques to spot badness in encrypted sessions. TLS or SSL handshake failures, suspicious, invalid or weird certificates can be indicators of such badness in your network traffic and the good news is that Bro, by default, does some of that analysis already for you, suggesting potentially interesting network activity for you to investigate.

To demonstrate how Bro can help with finding those indicators, we’ll look at sample2.pcap

 $ bro -r sample2.pcap local

See that a notice.log file has been created in the working directory, along with http.log, ssl.log and others.

Let’s have a look at the contents of notice.log:

 $ cat notice.log | bro-cut msg, sub

SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
SSL certificate validation failed with (unable to get local issuer certificate)
(output truncated)

Hmmm… that looks really suspicious again!

Let’s have a look at the contents of the ssl.log now:

 $ cat ssl.log | bro-cut server_name, subject, issuer_subject

Again, parsing these logs with bro-cut and other command line tools to generate a list of suspicious domains is straightforward. That list can be compared to a list of well-known malicious domains, or used with various domain reputation services. We will talk more about how to leverage threat intelligence feeds with Bro later in this post.

Let’s carry on with our analysis. A closer look at the http.log reveals some potentially interesting User Agents under the user_agent field:

 $ cat http.log | bro-cut user_agent | sort –u

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)

cgminer 2.7.5

Can you see that cgminer user agent? It is a well known fact that malware can use unusual, weird or unique user agents as part of the headers of the HTTP requests. A good study on that was written by Robert Vandenbrink.

In this case the user agent indicates that we’re looking at a bot whose purpose is to deliver bitcoin mining traffic. For more information about this particular bot check Liam Randall’s solutions and scripts on his GitHub

The new file analysis framework

The file analysis framework is a new feature introduced with Bro 2.2 that provides plenty of new functionalities to network analysts. One of the most powerful features is the ability to extract files from network streams based on multiple criteria: geo spatial (i.e. per country of origin), signature based, destination based, etc.

Files can be extracted from various protocols including FTP, HTTP, SMTP and IRC. Others like Bit torrent and SMB will be added in the near future.

Thanks to the powerful Bro language, the new file analysis framework can be combined with actions to do awesome stuff like look up in a malware hash registry, upload to virustotal, to a cuckoo sandbox or even tweet the results of your analysis!

To demonstrate some of its capabilities we’ll analyze sample3.pcap. As usual we start replaying the capture with Bro:

 $ bro -r sample3.pcap local

You should have a new log: files.log. Let’s have a look at its contents:

 $ cat files.log | bro-cut fuid, mime_type, filename, total_bytes, md5

FC7cMq18xeqtT9IGD3 application/zip - 31044 0cbc25ade65bcd7a28dd8ac62ea20186

We have a unique entry. We don’t have a filename but Bro has recorded the MIME type and even computed the MD5 hash for us!

Can we extract that file? Of course we can! Open your text editor of choice and save these lines as extract-all.bro
 event file_new(f: fa_file)
                Files::add_analyzer(f, Files::ANALYZER_EXTRACT);

Congratulations! You’ve written your first Bro script. Next, run the capture against Bro again, this time replacing the ‘local’ script with the new one you just created. You might need to run this as root:

 $ bro -r sample3.pcap extract-all.bro

This command will create a new directory extract_files where all files extracted will be located:

 $ ls extract_files


Let’s confirm what kind of file we’re looking at:

 $ file extract-HTTP-FC7cMq18xeqtT9IGD3 

extract-HTTP-FC7cMq18xeqtT9IGD3: Zip archive data, at least v2.0 to extract

$ xxd extract-HTTP-FC7cMq18xeqtT9IGD3 | head -10

0000000: 504b 0304 1400 0808 0800 208f 1c41 0000  PK........ ..A..
0000010: 0000 0000 0000 0000 0000 0d00 0000 6234  ..............b4
0000020: 612f 6234 612e 636c 6173 73c5 7979 5c9b  a/b4a.class.yy\.
0000030: 5b76 d8b9 9240 427c 8010 1606 db18 63fb  [v...@B|......c.
0000040: 6110 606c 24b0 0783 0149 0801 daf7 0d09  a.`l$....I......
0000050: edfb 2eb4 22e4 7979 33c9 bc74 3259 babd  ....".yy3..t2Y..
0000060: d725 93ce 6bac a493 f4bd e729 76e3 cc8c  .%..k......)v...
0000070: d32d 69d2 25d3 769a a64d 9ba6 49da a4cd  .-i.%.v..M..I...
0000080: d2c9 d2e9 b44d 9c73 0578 c36f dee4 af9a  .....M.s.x.o....
0000090: 9fbe 7bbe 7bcf 3dfb 39f7 9ecf 3fff a73f  ..{.{.=.9...?..?

$ xxd extract-HTTP-FC7cMq18xeqtT9IGD3 | tail -10

00078b0: db66 0000 6234 612f 6234 642e 636c 6173  .f..b4a/b4d.clas
00078c0: 7350 4b01 0214 0014 0008 0808 0020 8f1c  sPK.......... ..
00078d0: 4167 fdc8 0309 0700 00a7 0f00 000d 0000  Ag..............
00078e0: 0000 0000 0000 0000 0000 0034 7000 0062  ...........4p..b
00078f0: 3461 2f62 3465 2e63 6c61 7373 504b 0102  4a/b4e.classPK..
0007900: 0a00 0a00 0008 0000 208f 1c41 0000 0000  ........ ..A....
0007910: 0000 0000 0000 0000 0400 0000 0000 0000  ................
0007920: 0000 0000 0000 7877 0000 6234 612f 504b  ......xw..b4a/PK
0007930: 0506 0000 0000 0700 0700 9401 0000 9a77  ...............w
0007940: 0000 0000                                ....

While the first bytes in the file header (also known as magic numbers) suggest a ZIP file, the content of the file indicates the presence of Java class files. We can easily confirm that by executing:

 $ jar xf extract-HTTP-FC7cMq18xeqtT9IGD3

Which extracts the Java classes to the b4d directory.

We’ll leave the analysis of the Java classes for now, but can you identify if this is a malicious file with the information we have at this moment? Well, let’s see what others know about this file. Remember the MD5 hash included in the files.log? A quick search in Virustotal reveals that we’re looking at a Java 0-day that was included in the Blackhole Exploit Kit (CVE-2012-4681).

As you can see, the possibilities of using the new file analysis framework are endless. Add a bit of knowledge of the Bro programming language, some python scripting goodness and a few APIs to malware analysis services and you have an awesome cocktail!

Bro, Threat Intelligence and CIF

Threat Intelligence is the new holy grail of security. Finding relevant and up-to-date information on malicious threats is key for all the phases of the security lifecycle, from prevention, to detection, incident response, containment and forensic analysis. The most common types of threat intelligence required by analysts are IP addresses, domains, urls and file hashes that have been observed in relation to malicious activity.

Many organizations provide data feeds that are freely available and that can be used with the new Bro’s Intel Framework to log hits seen in network streams, like those from ZeuS and SpyEye Tracker, Malware Domains, Spamhaus, Shadowserver, Dragon Research Group, and others.

While you could download these data feeds on a regular basis, maintaining an updated repository that is actually usable by your tools can be a daunting task, especially given the number of sources and disparity of formats used. This is where the Collective Intelligence Framework (CIF) comes to the rescue.

CIF is now on version 1 (stable) and allows you to parse, normalize, store, process, query, share and produce data sets of threat intelligence.

Having installed a few CIF servers I can tell you it’s somewhat complex (maybe not complex but rather tedious), so I will refer you to the official documentation if you want to set up your own instance (see the References below). For the rest of this section I will assume that you have access to a running instance of CIF.

To enable the Bro Intel Framework and allow the integration of CIF feeds, add these three lines to your local.bro file (in Security Onion that’s in /opt/bro/share/bro/site/local.bro):

 @load frameworks/intel/seen
@load frameworks/intel/do_notice
@load policy/integration/collective-intel

CIF is used mainly in two ways: either to query for data stored about an IP address, a domain or a url, or to produce feeds based on the stored data sets. The data feeds available in version 1 can be seen here:

In our example, we’ll generate a list of domains related to malware with a confidence level of 75 or greater. To make sure the output is formatted for Bro append “-p bro

 $ cif -q domain/malware -c 75 –p bro >

Note that this command won’t work if you don’t have CIF installed. If you don’t have access to a CIF server you can grab a copy of a file formatted for Bro here (note that this will be outdated by the time you download it so use it for testing purposes only).

The figure below shows the contents of the file generated in CIF’s native format (without using the BRO plugin).

In order to import the new data feed we just generated we need to configure Bro’s Input Framework. To do so, add the following lines to your local.bro file:

 redef Intel::read_files += {

Where /opt/bro/feeds/ is where you have placed the file generated by CIF. You can add as many files as you want. For more information about different methods to refer to these .intel files check

Now the Input Framework will read the information from our text-based file and will send it to the Intel Framework for processing.

To demonstrate the combined usage of Bro and CIF I have created sample4.pcap, a simple capture that contains a DNS query to a malicious domain ( Let’s replay this capture with Bro after making all the changes described above:

 $ bro -r sample4.pcap local

See how a new file, intel.log has been created:

 $ cat intel.log 

#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path intel
#open 2014-03-07-21-28-09
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where sources
#types time string addr port addr port string string string string enum enum table[string]
1394223877.224159 C7J79H2v6YLWMaJEk6 54212 53 - - - Intel::DOMAIN DNS::IN_REQUEST CIF - need-to-know
#close 2014-03-07-21-28-10

Since was included in the feed generated by CIF and imported into Bro, now we can identify any attempt of connection to this malicious domain.


Security analysts will never have enough tools or resources to fight malware. Bro and CIF are two of those invaluable resources that every malware analyst should be aware of.

As their creators state, Bro is much more than an IDS. Bro is a full-featured network analysis framework created with a powerful tool, the Bro Programming Language.

If you want to know more about Bro, CIF, Malware Analysis or Network Forensics check the References section.

About the author

Ismael Valenzuela (GCFA, GREM, GCIA, GCIH, GPEN, GWAPT, GCWN, GCUX, GSNA, CISSP, CISM, 27001 Lead Auditor & ITIL Certified) works as a Principal Architect at McAfee Foundstone Services EMEA. Find him on twitter at @aboutsecurity or at


Pcap samples used in this post:
Catching “bayas” on the Wire: Practical. Kung-Fu to detect Malware Traffic. SANS EU Forensic Summit:
Liam Randall’s samples, exercises and scripts:
Toolsmith: Collective Intelligence Framework:
The Bro Network Security Monitor:
Malware dumps and pcaps:
Collective Intelligence Framework:
Security Onion: