Here is a picture from "Securi-Tay 2020: Offensive Tradecraft - Defence Evasion" by - Paul Laîné which briefly sums up the technologies used by AVs to detect malware
We as attackers have to find ways to bypass and hide these detections.
I would also like to add a couple more things
  • Sandboxing: Trigger payload in a controlled environment to track heuristic behaviors and flag known malware behavior
  • Data Mining: Uses a known data set like virustotal to make algorithms to detect AV or just to build signatures to flag for
  • In memory scanning: periodically scans or scans executable regions for malware in real time to detect malware
  • Library load events
  • Kernel callbacks
  • EtwTi Kernel-mode event provider