• Bug fixes (tokenization and mapping updates)
  
• Global Search error handling, keyword highlighting
  
• Stacking on URL Domain and DNS, fixed stacking registry
  
• Reindex data utility added (see wiki article for usage)
  
• Upgrade feature added, you can now update the sourcecode from yum without downloading a new iso (see wiki article for usage)
  
• Rotate /media folder to remove old collections after 1 day (or > 2GB foldersize)
  
• Added w32system (system info)
  
• Removed static mapping in postController for hostname
  
• Fixed issue with building audit aggs where default_field was not being passed to ES.
  

1. Single view endpoint forensics (multiple audit types).
   
2. Global search.
   
3. Timelining.
   
4. Stacking.
   
5. Tagging.
   
6. Interactive process tree view.
   
7. Multiple file upload & Named investigations.
   

nightHawk ISO
To make it straight forward for users of nightHawk, we built an ISO with everything setup ready to go. That means you get the following;

1. Latest nightHawk source.
   
2. CentOS 7 Minimal with core libs needed to operate nightHawk.
   
3. Nginx and UWSGI setup in reverse proxy (socketed and optimized), SSL enabled.
   
4. Latest Elasticsearch/Kibana (Kibana is exposed and useable if desired).
   
5. Sysctrl for all core services.
   
6. Logging (rotated) for all core services.
   
7. Configurable system settings, a list of these can be found in the /opt/nighthawk/etc/nightHawk.json file.
   

Starting the system :
Before building your VM with the supplied ISO, take into consideration the following;

1. CPU/RAM.

Pending : Setup the Elastic service to be dual nodes with 1/4 of the allocated system memory per node. This means if you give it 2GB RAM, each ES node will be 512mb and the system will remain with 1GB to operate.

If you want to set this any different way, ssh into the box and configure your desired way.

1. HDD.

A minimum of 20GB should be considered. An audit file can be large and therefore its advised you allocate a lot of storage to handle ingesting many collections.
Pending : User based storage setup for large scale instances. If you desire to setup extra partitions, you can do this yourself, a few changes can be made to point the ES data storage to your new partition.

Installation :
Download ISO: nightHawk v1.0.2
Configure the hardware, mount the ISO into the VM, start the installtion script.

Once complete, in your browser (Chrome/FireFox), goto; https://192.168.42.173 .

If you need to access Kibana, goto; https://192.168.42.173:8443 .

If you need to SSH into the box, the login details are; admin/nightHawk .

If you want to change the IP address (reflected application wide); /opt/nighthawk/bin/nighthawkctl set-ip <new_ipaddress>

Redline Audit Collection Script can be found in the root of this repo. Use this when using the standalone redline collector as this will return the documents you need to populate nightHawk correctly.

Uploading:

IMPORTANT : Creating audit zip file to upload (Redline stand alone collector):
step_1: Navigate to Sessions\AnalysisSessionX\Audits<ComputerName> where X is analysis number which is 1 for most cases.
step_2: Create zip of folder containing audit files i.e. 20160708085733
step_3: Upload 20160708085733.zip


IMPORTANT : Use exisiting HX audit file (HX collector): FireEye HX audits are an extension ending in .mans. The audit from HX differs from the Redline collector because the .mans that it returns is actually a zip file. This means it can be uploaded directly unlike the Redline audit which you need to follow the instructions above.
Navigate to the "Upload" icon on the nav bar, select an audit .zip (or multiple), a case name (otherwise the system will supply you with one) and submit. If you have used our Redline audit script to build your collection, follow the "Redline Collector" instructions just above.

Once processed, the endpoint will appear in the "Current Investigations" tree node. Under the endpoint you will be presented with all audit types available for that endpoint. The upload feature of this web app spawns pOpen subprocesss that calls the GO application to parse the redline audit and push data into Elasticsearch. There are 2 options for uploading, one is sequential, the other is concurrent.
Please Note: Concurrent uploads are limited to 5 at a time and can be resource intensive, if you have an underpowered machine then restrict usage of this feature to 2-3.

Tagging:
You can click on any row in any table (in response view) to tag that data. Once tagged you can view the comments in the comments view.

Elasticsearch:
There are custom mappings (supplied in git root) and advisory comments on the following;

1. Parent/Child relationships:
   Documents are indexed via the GO app as parent/child relation. This was chosen because it is able to give relatively logical path to view documents, ie. the parent is the endpoint name and the children are audit types. Performing aggregations on a parent/child relational document at scale seems to make sense as well. The stacking framework relies on building parents in an array to then get all child document aggregations for certain audit types.
   
   
2. Sharding:
   Elasticsearch setups require tuning and proper design recognition. Sharding is important to understand because of the way we are linking parent/child documents. The child is ALWAYS routed to the parent, it cannot exist on its own. This means consideration must be given to how many shards are resident on the index. From what we understand, it may be wise to choose a setup that encorporates many nodes with single shards. To gain performance out of this kind of setup we are working on shard routed searches.
   We are currently working on designing the best possible configuration for fast searching.
   
   
3. Scaling:
   This application is designed to scale immensely. From inital design concept, we were able to run it smoothly on a single cpu 2gb ubuntu VM with 3 ES nodes (Macbook Pro), with about 4million+ documents (or 50 endpoints ingested). If going into production, running a setup with 64/128GB RAM and SAS storage, you would be able to maintain a lightning fast response time on document retrival whilst having many analysts working on the application at once.
   

Considerations:

1. DataTables mixed processing:
   There are several audit types ingested that are much to large to return all documents to the table. For example, URL history and Registry may return 15k doc's back to the DOM, rendering this would put strain on the client browser. To combat this, we are using ServerSide processing to page through results of certain audit types. This means you can also search over documents in audit type using Elasticsearch in the backend.
   
2. Tagging:
   Currently we can tag documents and view those comments. We can update them or change them. The analyst is able to give context such as Date/Analyst Name/Comment to the document.
   

Dependencies (all preinstalled):
elasticsearch-dsl.py django 1.8 python requests

To Do:
Process Handles (in progress).
Time selection sliders for time based generators (in progress).
Context menu for Current/Previous investigations.
Tagging context. The tagging system will integrate in a websocket loop for live comments across analyst panes (in progress).
Application context.
Ability to move endpoints between either context.
Potentially redesign node tree to be investigation date driven.
Selective stacking, currently the root node selector is enabled.
Shard routing searches.
Redline Audit script template.
More extensive integration with AngularJS (in progress).
Responsive design. (in progress).
Administrative control page for configuration of core settings (in progress).

Screenshots: