vdisec/cve-2019-19871-auditguideAudit Guide for the Citrix ADC Vulnerability CVE-2019-19871. Collected from multiple sources and threat assessments. Will be updated as new methods come up.
There is now a tool from FireEye that will help scan these items below. The key to this is that you need to have enough logs to go back to 1-9-2020 to have a chance to see what was done beyond the exploit was ran. If it finds .XML payload files then you need to use the information below to make a decision on what action to take. https://www.fireeye.com/blog/products-and-services/2020/01/fireeye-and-citrix-tool-scans-for-iocs-related-to-vulnerability.html
https://github.com/citrix/ioc-scanner-CVE-2019-19781 https://github.com/fireeye/ioc-scanner-CVE-2019-19781/
There is not an easy way right now to prove what someone has done. With 4x public exploits they each leave a different artifact\signature which is helpful for detection but is not 100%. The key to remember these are the Public exploits the were private until the 10th and that also doesn’t mean there still are others out there in the wild that people not sharing for their own gains. Exploits are editable in most cases where someone can change the name of the file that will be dropped, the name of the user account, the name of the process, the path of the query and many other options which means the possibilities increase exponentially that someone may have exploited the system. There are also advanced attackers and basic attackers too, one will clean up after themselves and come up with very clever ways to disappear into the system to hide from detection.
If you run Nessus, then you can use this .YAR file below to do a privileged scan to look for the common detection methods. https://github.com/Neo23x0/signature-base/blob/master/yara/exploit_shitrix.yar
Great Links about this Auditing Process. https://nerdscaler.com/2020/01/13/citrix-adc-cve-2019-19781-exploited-what-now/amp/ http://deyda.net/index.php/en/2020/01/15/checklist-for-citrix-adc-cve-2019-19781/
Disclaimer just like stated before this will not detect every exploit, but it may help detect some anomalies if the attacker didn’t modify the public exploits and/or didn’t clean up after themselves. Most attackers will use the default exploit, and these are some of the documented artifacts that could be left behind. This list of commands was grabbed from many sources and will be very dynamic as other variants, workarounds and new developments come up. We can count on this changing constantly as more infections happen and further forensics is completed with a larger sample set.
Look at this first for things you didn’t do. If you don’t normally do a lot on your ADC these should be very quiet along with, they may have entries from weeks, months or years ago the last time you were on. There are more precise queries just below. Also understand this is a cat and mouse game even in this blog. As we disclose what we have seen and how to spot how to find the attackers they are using this information against us by changing their tactics to avoid detection.
Your mileage will always vary on what you need to do based on your threat landscape. Here are some of my thoughts I have been telling customers that have found evidence of an exploit executed.
What compliance bodies cover your business? Finance\Banking, SOX, PCI, HIPPA, State/Local and Government Laws.
If you are under one of these frameworks, then you need to follow those procedures for those compliance bodies. There are also ethical considerations based on what certifications and professional groups you are part of that have provisions for incident response and disclosure.
Here are links to these two well know Incident Response processes guides https://security.berkeley.edu/incident-response-planning-guideline https://csrc.nist.gov/publications/detail/sp/800-61/rev-2/final
One thing we know so far is that around the 1-10-20 the first public exploit was released and there are some reports of infections on the 9th as that is right when it came out. In most cases the risk is much lower if you patched your system before 2020 versus later this month.
December 17th-December 31st Lowest Risk of Exploitation January 1-8th Lower Risk of Exploitation January 9th-13th – Higher Risk of Exploitation January 14th- Present – Highest Risk of Exploitation
This should come into your process for your next steps.
I have found traces of an exploitation what now? This still depends, one thing most Citrix ADC deployments don’t get configured is good SNMP and SYSLOG logging and they may not have a good way to search, filter or alert if there are artifacts found. If you have absolute logging and you are confident, they didn’t do anything then you may be able to move on with your life. Then if you did find something and you were able to confidently remove their remote access then you could move on.
But most will find they will find some trails and they may not be able to connect the dots on what was done and where they may have gone to and it could be easier after detection to just reset the devices.
My Next Advice will be changing over the next 2 weeks.
There is no right and perfect answer I can give that will fit everyone’s situation these are my thoughts as of now on 1-19-20 and they may change after this as I learn more about next steps and as things are released on the defense and or attack side related to this vulnerability. There is no right answer, IT security is the land like most lands that is ruled by “it depends”. I highly suggest if you find anything else other than these files only in those 3 directories, I would consider the box compromised and go down the more cautious path. In some of these I’m suggesting taking the more cautious path especially when there is no logging to confirm what they did or didn’t do. You need to work with your team to decide the best course of action based on your situation because this is a team sport. There can always be a better way to fix things like this but based on the evidence you have on the device and around the device (1st Tier Targets) then you may be ok by reducing your risk and going from there.
I hope people can remove infection and have good enough logging to feel confident they are no longer in and be able to resume your normal activities without some of these steps.
There are two main things you need to make a decision on if there are any traces of exploitation.
I recommend changing your passwords for all your local accounts on the box if there is any inkling of a successful exploit. Go ahead and change it because in many deployments it may have never been changed after it was originally deployed 4-7 years ago. If you see signs of command line access and/or tampering you can most likely count on the attacker being able to crack the password on Pre 11.0 firmware’s it was AES256 and in later builds it is using AES512 which can also be susceptible with cracking too. Make sure it is bound to LDAP securely also and you have alerts setup for NSROOT logins.
If you have saw some levels of exploitation, I would also make sure and change any service account that was defined within the Citrix ADC configuration. The most common is the LDAP\Kerberos bind account. Someone getting an exploit to run on a Citrix ADC doesn’t mean they are Domain Admins, but it may not take too long depending on your controls and logging. This is a very easy change that if tested can be seamless to users.
Depending on what you found with your audit will help figure this one out too. If you had good logging and you can see access requested to this file, then you must rekey. If you don’t have good logging, then you should also rekey. If you have a wildcard certificate, then that is also another big problem too and the more sites it is bound to the more your risk and exposure will be. The worse thing that can happen is that you assume it is ok and someone is standing up phishing site with your certificate that all your training will not prevent the clicks. This can lead to much bigger problems if someone has access to your certificates and I would suggest proceeding with caution and doing a rekey. This could be an ok time based on the expiration date of the current certificate. I have seen some swap to another certificate registrar in this process to just to change things up, but they had logs that the file was accessed and downloaded along with other advanced techniques were detected.
Last but not least some credits for some of the people that have been working on this issue since it first arrived. There are so many more people that are not on this list because they behind the scenes that I have not seen too.
