Worse than looking for a needle in a haystack is having to do so in the dark without your glasses. In our second installment of the series, we look at the role of endpoint visibility in threat hunting.
Missed our first post where we explain purpose and goals of threat hunting in detail? Please visit this link.
You Can’t Find What You Can’t See
Worse than looking for a needle in a haystack is having to do so in the dark without your glasses.
Superior endpoint visibility is the key to detection, alert triage, incident investigation, response, remediation and effective threat hunting. With acute vision and clarity, concessions do not have to be made by the security team and threat hunting efforts can enhance the security posture of the enterprise without limitation. A 2017 SANS report indicated that almost all enterprises surveyed improved the speed and accuracy of their response as a result of threat hunting and were able to reduce dwell time significantly.
Leading cybersecurity vendors deploy agents on endpoints to be their cyber eyes. But just as some people have 20/20 vision, others need spectacles and the blind cannot see at all, some agents can clearly discern objects, less capable agents can see them only in fuzzy outline, while others miss them altogether.
Two important factors contribute to the level of visibility of the endpoint agent:
- Kernel-mode vs. User-mode execution
- Process-level vs. Thread-level visibility
Kernel and User Mode
In every endpoint’s operating system (Windows, Linux, MacOS, etc.), two parallel worlds exist when it comes to process execution: kernel-mode and user-mode.
As the name indicates, user applications run in user-mode where their processes are restricted to their own address space. They cannot affect other user applications nor shared computer resources. To gain access to computer hardware and shared services (e.g., write to a file on the drive, send a message via an Ethernet port, get the system date and time), processes running in user-mode issue a system call to the operating system that executes in kernel-mode. Upon making this system call, the user-mode process is put on hold while the operating system executes the action and, upon completion, re-starts the user process where it left off, adding new information (successful write, error on the send, here is the system date and time).
Core processes (process scheduler, device driver, memory manager, etc.) that deal with the computer hardware and shared services (and undertake a lot of other tasks) run in kernel-mode, endowing these processes with a higher privilege and wider field of vision than their user-mode counterparts.
Kernel-mode processes can pre-empt user-mode processes but not vice versa. In addition, certain kernel-mode processes can view and access the entire virtual memory of the endpoint including the memory belonging to any user-process while user-mode processes cannot view memory beyond their own.
Kernel Agent Reporting for Duty
Should an endpoint agent run in user-mode or kernel-mode? Are the benefits of kernel-mode execution worth the risk?
Agent stability is an important consideration. If an agent operating in user-mode malfunctions and crashes, the endpoint continues executing instructions of other user and kernel processes without error. However, when an agent in kernel-mode malfunctions, its privileged access also means it has the capability to bring down the entire endpoint and force a reboot, known as a Blue Screen Of Death (BSOD) for Windows systems. A poorly architected kernel-mode process also has a greater propensity to monopolize hardware resources, causing a cascade of poor performance for all other processes.
The other consideration is visibility. While a kernel-mode agent can access all, an agent that runs in user-mode is governed by its reliance on system calls to access hardware or memory, creating two issues of concern:
- malware also running in user-mode that executes before discovery can bypass and disarm the agents’ security actions; and
- malware running in kernel-mode is able to peer into user memory as well as monitor, modify, and control any agent request to the kernel, effectively allowing it to obscure information and evade countermeasures.
When it comes to cybersecurity, enterprises absolutely must have a kernel-mode agent for visibility and other purposes (e.g., anti-tampering). Though, creating a stable and efficient kernel-mode agent is not a trivial matter and cybersecurity vendors with user-mode agents cannot simply re-architect them to take advantage of kernel-mode benefits without significant overhaul. This is where experience of the vendor is key, superior software developers are able to deliver stable and efficient kernel-mode agents, eliminating the low-risk benefit of user-mode agents.
But User-Mode, Too, Please
Not all agent functions are necessary for spotting malware on endpoints. Data aggregation and communication with backend servers, for example, can be undertaken in user-mode so as not to tie up computer resources. Having these tasks operate in user-mode rather than kernel-mod also greatly lowers the risk of affecting system and user productivity.
Cybersecurity teams should make sure that the agents they deploy are capable of both kernel-mode and user-mode operation, maximizing security as well as efficiency.
Sighting Threats that Hang by a Thread
There is another important criterion that determines the acute vision of the endpoint agent: process-level vs. thread-level.
Cybersecurity agents that run on endpoints are designed to carefully watch processes. However, just as looking out over the ocean with binoculars will expose the pelicans, waves and ships, but hide the fish, seaweed and submarines, process-level observation does not provide deep-enough visibility to spot certain threats.
In recent sophisticated cyberattacks, exploits surreptitiously run code within a user process by injecting the process with malicious threads. For example, an attack might target an instance of svchost.exe. (Windows systems typically have multiple instances of svchost.exe running for a variety of services; these run at process-level.) The attack “influences” an instance of svchost.exe to load a malicious library (DLL) as a thread belonging to its own process. Observing at the process level, the cybersecurity agent will attribute tasks performed—even malicious ones—to the whitelisted svchost.exe. It won’t discern the threat lurking “under the radar” at the thread-level. The thread-level threat is then free to wreak havoc undetected and appearing as normal behavior.
Avant-garde threats like EternalBlue and DoublePulsar exploit this thread-level blindspot. More such attacks are being developed all the time. Cybersecurity teams without thread-level visibility are as blind to the danger as a gazelle at a croc-filled watering hole.
I Can See Clearly Now
Effective cybersecurity requires that analysts go on the offensive, attacking cyber threats instead of merely reacting to alerts. Today, enterprises require more than 101 days to detect breaches. Threat hunting boosts the speed and accuracy of detection and response and slashes dwell time.
To maximize their effectiveness, threat hunters must have the lenses to see clearly. Endpoint agents that operate in both kernel-mode and user-mode, and that observe thread-level activity, bestow reliable 20/20 vision on the blind. They are the microscopes, binoculars and periscopes that provide threat hunters with the visibility to find advanced threats and eradicate them before they can cause damage.