TLDR:

• Packed malware machine learning classifier can only previously identify 10 packers
• Solution was a customized version of model ensembling, which is to train multiple models and resolve their results
• It works with a slight caveat of more extended training and processing, which I could happily live with

I recently presented VGL4NT, my tool that uses machine learning to classify packed malware, at the Blackhat Middle East and Africa meetup. During my talk, I candidly shared one of the tool's limitations which is it can only identify 10 packers because of my hardware constraints. If I want it to be able to identify more, I need to get more GPU (which will be costly) or keep my money and come up with a clever solution. Well, this post is about the latter.

A Simple Solution

The solution I came up with isn't exactly original. It's based on Task Decomposition, which involves training separate models for different categories and combining their predictions. This way, I could double the classification capacity without requiring additional hardware resources.

This was implemented by creating multiple machine learning models, each specializing in recognizing a subset of packers. The real challenge, however, lies in combining the predictions from these models to form a unified output.

Here's how the process works:

The packed malware file is fed into Model 1, which outputs probabilities for Packer 1, Packer 2, and Others. For example, it might produce:

• Packer 1: 10%
• Packer 2: 20%
• Others: 70%

The same file is then fed into Model 2, which outputs probabilities for Packer 3, Packer 4, and Others. For instance:

• Packer 3: 60%
• Packer 4: 30%
• Others: 10%

I then take the 'Others' category with the lowest probability. For our example, the final 'Others' probability would be 10% from Model 2.

The final probabilities are:

• Packer 1: 10%
• Packer 2: 20%
• Packer 3: 60%
• Packer 4: 30%
• Others: 10%

Packer 3 has the highest probability in this example, and the file is classified as such.

This simple combination approach ensures I maintain a suitable probability distribution while leveraging each model's strengths. The beauty of this method is not only its efficiency but also its scalability. I can introduce more models, each specializing in different packers, to further increase the classification capabilities.

Now you might wonder why I'd even write about this if the solution is this simple. The funny thing is I've explored multiple approaches to unifying the output. Before this, I fully implemented a complicated approach, only to later realize while writing this blog post that a much simpler approach works well enough for the tool's purpose.

Downsides

I am conscious that this may or may not be the most effective method to tackle this problem. But what is essential is that the current computation is simple and can maintain the appropriate prediction distribution based on the relative percentages. In essence, the category with the highest confidence score will always come out on top in the final output, primarily what users of my tool are interested in.

Aside from this, I am concerned that increasing the number of categories also increases training and prediction time. I'm not too worried about the increase in training time because this happens behind the scenes and remains unseen to users of my tool. I'm slightly concerned about the longer prediction time, as all models need to process each submission to the tool. And as I plan to incorporate more packer tool categories, the prediction time will definitely rise.

These downsides are not too much of a problem, however. They can easily be fixed if I find they are not meeting the tool's goals. For now, these will do.

Conclusion

I am genuinely happy with my progress with the VGL4NT Malware Packer Classifier. There are other topics I want to tackle, but I'll save those for future blog posts.

In the meantime, I invite you to check out the tool and see the changes yourself. Visit the VGL4NT website to get started. And for a more detailed walkthrough, you can also watch this YouTube video I created

The recent rise in popularity of AI reignited my interest in machine learning. It inspired me to dive deeper into understanding how it can be applied to malware analysis and, more importantly, how to better detect malware packers, as almost every malware nowadays uses them.

My research and experiments eventually led me to make a web app, which I call the VGL4NT Malware Packer Classifier (https://packers.vgl4nt.com/).).

(For those curious, V.G.L.4.N.T. is a play on "Vigilant" and stands for "Visual Guided Learning 4 Neutralizing Threats")

Current State of Packer Detection

Traditional packer detection approaches like DiE (Detect it Easy) and Yara rules depend on known signatures and patterns to identify packers. These tools scrutinize a file for specific indicators, like unique sequences of bytes or strings. While effective in many cases, they have drawbacks, like when a packer is modified or if the sequence of bytes or strings are altered.

By using machine learning, the VGL4NT Malware Packer Classifier can be able to take into account minute differences and still be able to detect the packer used.

How it works

• The uploaded executable file's bytes are converted into grayscale values, creating an image..
• The grayscale image is then fed into an image machine-learning model I trained from scratch.
• It returns a list of percentages on how similar it is to other Packers.

The approach above is nothing new and is based on this academic paper. The difference is that the paper has a tool that classifies malware families, while mine classifies the packers used.

Most of the magic happens in the model itself. I've trained it on several packed malware samples and measured its accuracy using multiple iterations. The latest version of this model has a 94% accuracy, which is calculated by comparing the model's predictions to the actual packer labels in a dataset that the model hasn't seen before (the test dataset).

Current limitations

The app works for the most part, but it has its limitations. For example, users can only upload executable files (EXE, Bin, ELF, DLLs, etc) with a maximum size limit of 10MB.

Furthermore, due to costs of GPU resources during training, only the following packer tools can be classified:

• aspack
• alienyze
• amber
• mew
• mpress
• nspack
• pecompact
• petite
• themida
• upx
• others (Everything else)

The list of packer tools above was chosen based on available real-world malware samples that I have encountered or studied.

Future Plans and Updates

If this project gains enough interest, then I plan to add more improvements, such as:

• Increase GPU resources to increase the model's capacity to classify more categories
• Improvements in the training method by handpicking the most important parts of the executable and then feeding that to the model
• Offer an API for integration with existing tools and processes.

Of course, this project would improve a lot with the community's help. I encourage users to provide feedback, report issues, or request new features. Feel free to throw your thoughts to me through my email, karlo@accidentalrebel.com, or Twitter at @accidentalrebel.

I came across the Blue-Jupyter project on Github while researching Jupyter notebooks. This short demo video got me excited, so I cloned the project and added some improvements that automate many things when I am looking for malware to investigate.

What are Jupyter Notebooks?

For readers who may be unfamiliar, Jupyter Notebooks are a web-based tool that allows users to create and share documents that contain live code, equations, visualizations, and narrative text. They are a popular tool among data scientists and researchers but have also adapted for use in other fields, such as cybersecurity.

My Additions to the Blue-Jupyter

Many of the changes I've made are focused on automating the process of quickly looking for interesting new samples to investigate.

One addition to the notebook is the automated downloading of samples from Malware Bazaar. This can download a maximum of 100 samples continuously. Additional information is listed to highlight some interesting points about the sample, like the malware signature. It also can skip samples that have already been downloaded to save bandwidth.

The second significant addition is the automated generation of Capa results for each downloaded sample. This makes it easy to see which malware has a particular capability so I can quickly see which ones are interesting enough to investigate further.

I also added minor improvements like error handling, additional logging for troubleshooting, and some cleanup code just in case I want to start fresh.

Check it out

If you are interested in checking it out, you can view my fork of the repository here. I did not request for a pull request on the original branch because I've changed a lot of things that the original owner might not prefer to have. Of course, I encourage everyone to fork what I made and make it their own. That's the beauty of Jupyter notebooks, anyway.