Tuesday, October 13, 2026

YARA in Cyber Threat Analyzer: from rule file to remediation

 

A script lands in a folder on a PC. CTA scans it against a rule you wrote, raises an alert, and an analyst quarantines the file before anyone runs it. This post follows that path from start to finish, then covers what changes when your rule set grows from a handful of rules to hundreds of thousands.

YARA matches on what a file contains, not on its name, its hash or who signed it. In Cyber Threat Analyzer (CTA) that closes a gap the other file checks leave open: a hash lookup finds nothing for a binary nobody has seen, and a signature check passes anything validly signed.

You get a worked custom rule, the exact steps to load it, and a checklist for running a large rule set. A ready-to-use copy of the rule file, custom_cta_rules.yar, goes with this post.

Everything here comes from reading the Cyber-Threat-Analyzer(190) package (src/analysis/yara_scanner.py, docs/yara.md, the response code and the GUI). I could not run CTA or yara-python in my sandbox, so the only numbers I measured myself are in the scale section, and they are labelled.

What YARA does inside CTA

CTA compiles every rule file in one folder, then scans files, memory dumps and live processes against the result. Matching runs inside the application, so it needs no API key and no network.

  • Rules folder. yara_rules/ next to main.py (setting yara.rules_dir). CTA reads every .yar and .yara file under it, subfolders included. A rule placed directly in the folder is universal. A rule under yara_rules/clients/<client id>/ runs only on PCs assigned to that client.
  • Background scan. With yara.enabled on and File Integrity Monitoring (FIM) watching a path, each new or modified file with a listed extension is queued to a worker thread. The default extensions are .exe .dll .sys .scr .msi .ps1 .bat .cmd .vbs .js .jar .dmp .bin .tmp. Files over 200 MB are skipped, browser and temp folders are excluded, the queue holds 500 files before it drops new ones with a warning, and the same file is not alerted twice within 60 seconds.
  • On-demand scan. The YARA Scan tab scans any file, memory dump or running process by PID, with no extension filter. Create Alert for This Finding saves a reviewed match as an alert.
  • Severity. The alert takes the highest meta: severity among the matched rules (low, medium, high or critical), and defaults to high when a rule sets none. Risk scores are 30, 55, 75 and 95. File matches raise YARA-FILE-MATCH (event 90008) and process-memory matches raise YARA-MEMORY-MATCH (event 90009).
  • In the box. example_rules.yar holds five demonstration rules: an EICAR test string, an encoded-PowerShell pattern, a base64-smuggled PE header, a double-extension filename and some credential-dumping strings. The project's own notes say these prove the pipeline works and are not a real detection ruleset.
  • Endpoints. The agent reuses the same scanner for removable-media scans, reading a rules folder on the endpoint itself.

Write your own rule

A good CTA rule has narrow strings, a size limit and a severity you chose on purpose. Here are two rules built for the project. The first flags an internal confidentiality marker, which also makes a safe, harmless test. The second flags a hidden PowerShell downloader.

rule CTA_Custom_Internal_Marker : custom dlp
{
    meta:
        id = "CUSTOM-0001"
        name = "Internal document marker found in a scanned file"
        author = "CyberForge SOC"
        version = "1.0"
        description = "File carries the internal CONFIDENTIAL marker. Check whether it left an approved location."
        severity = "medium"
        confidence = "medium"
        false_positive_notes = "Expected on approved file servers. Most useful on removable media and download folders."
        reference = "https://intranet.example/policies/data-handling"
    strings:
        $m1 = "CYBERFORGE-CONFIDENTIAL" ascii wide nocase
        $m2 = "INTERNAL USE ONLY - CYBERFORGE" ascii wide nocase
    condition:
        filesize < 20MB and any of them
}

rule CTA_Custom_Hidden_PowerShell_Downloader : custom script
{
    meta:
        id = "CUSTOM-0002"
        name = "Hidden PowerShell that downloads from a URL"
        author = "CyberForge SOC"
        version = "1.0"
        description = "Script runs PowerShell with a hidden window and downloads content from a URL, a common stager pattern."
        severity = "high"
        confidence = "medium"
        mitre = "T1059.001"
        false_positive_notes = "Some software installers and admin scripts do this. Review the URL before acting."
        reference = "https://attack.mitre.org/techniques/T1059/001/"
    strings:
        $hid1 = "-WindowStyle Hidden" nocase
        $hid2 = "-w hidden" nocase
        $dl1  = "DownloadString" nocase
        $dl2  = "DownloadFile" nocase
        $dl3  = "Invoke-WebRequest" nocase
        $url  = /https?:\/\/[a-z0-9\.\-]{4,60}\// nocase
    condition:
        filesize < 1MB and any of ($hid*) and any of ($dl*) and $url
}

Why it is written this way.

  • severity is always set. CTA reads it to set the alert severity. Leave it out and the alert defaults to high, which is wrong for a low-confidence rule.
  • description shows up in the alert. The alert text lists each matched rule with its tags, its string-match count and this description, so write it for the analyst reading it at 2 a.m.
  • The other meta keys feed the Detection Content inventory. CTA reads id, name, version, author, mitre (or mitre_technique), confidence, false_positive_notes and reference. It only parses quoted string values, so write numbers and flags as quoted text too.
  • filesize caps the cost and the noise. A rule that cannot match a 500 MB file should say so, and YARA then skips it quickly.
  • The second rule needs three things at once: a hidden-window flag, a download call and a URL. Any one alone matches too many legitimate scripts.

I ran CTA's own metadata reader over this file and it parsed both rules, with the right ids, severities and the MITRE technique. I could not compile it with yara-python here, so CTA's startup log is the real test. A syntax error shows up there as a compile error.

Put the rule in the project folder

CTA has no upload screen for YARA rules yet, so rules go in by copying files into the folder and restarting. The project's own folder notes say the same.

  1. Save the rule as a .yar or .yara file, such as custom_cta_rules.yar.
  2. Copy it into the rules folder, which sits next to main.py (inside Windows-Threat-Analyzer(178) in your package). Put it directly in yara_rules\ to run it everywhere, or in yara_rules\clients\<client id>\ to run it only for that client.
  3. Install the dependency if you have not: pip install yara-python.
  4. Restart CTA, started from the folder that contains main.py.
  5. Open the YARA Scan tab. Its status line reports how many rules loaded, or why loading failed. The log at logs/wta.log also records how many rules were compiled from how many files.
  6. Test with the harmless file below, scanned from the YARA Scan tab.

To make the rule universal, run this from the folder that contains main.py:

copy custom_cta_rules.yar yara_rules\

To limit it to one client instead:

mkdir yara_rules\clients\acme
copy custom_cta_rules.yar yara_rules\clients\acme\

Save this as marker_test.txt for the test:

This is a harmless test file.
CYBERFORGE-CONFIDENTIAL

A .txt file is not in the background-scan extension list, so scan it on demand. To exercise the background path, save it as marker_test.bin inside a folder FIM is watching.

Things that go wrong.

  • One bad rule file stops the whole load. CTA compiles all files in a single step. A syntax error anywhere makes startup report a compile error and turns YARA off for that session. A later reload that fails keeps the previously loaded rules.
  • The client id must match. A PC loads the universal rules plus the folder for the client it is assigned to on the This PC / My Session screen, and skips every other client's folder.
  • Reload has no button. The scanner can recompile without a restart, but nothing in the interface calls it. The per-client notes say to restart, so restart.
  • The versioned content API does not turn on YARA rules. The Detection Content repository accepts a yara content type for inventory and version history. In the code I read, publishing only wires Sigma rules into the engine, so the folder stays the source of truth for YARA matching.

Follow one match from alert to remediation

A YARA hit only matters when an analyst acts on a specific file or process, so the cycle runs from rule to alert to response and back to rule tuning. The scenario: a script named update.ps1 lands in a watched folder on a PC.

[embed: node/d526f917-ef20]

A match is not the end of the cycle: what the analyst learns during triage goes back into the rule. Here is each step in detail.

  1. The rule loads. At startup CTA compiles CTA_Custom_Hidden_PowerShell_Downloader together with every other rule.
  2. A file lands. update.ps1 is created in a folder that FIM watches. .ps1 is on the extension list and the folder is not excluded, so FIM queues it.
  3. The worker scans. It checks the file is under 200 MB, runs the rules and records one match.
  4. An event and an alert are stored. The event goes on channel YARA. The alert is YARA-FILE-MATCH, named after the matched rules, severity HIGH, risk score 75. Its text lists each rule with tags, string-match count, description and the file path. A critical match adds an instruction to isolate the host.
  5. The analyst triages. Read the script, check the URL, and compare against the rule's false_positive_notes. If the script may already be running, scan the suspect process by PID from the YARA Scan tab.
  6. The analyst remediates with the response actions on the alert (below).
  7. The analyst closes it out. Acknowledge the alert and record a disposition; the alert record carries both fields.
  8. The rule gets tuned. If it was a false positive, tighten the condition, require another string, lower the severity, or add a path exclusion, then copy the file back and restart.

The remediation actions.

  • Quarantine File moves the file into a quarantine folder as <timestamp>__update.ps1.quarantined and removes its execute permission. It keeps the file for inspection and never deletes it.
  • Terminate Process stops a process by PID after a confirmation prompt.
  • Isolate blocks all inbound and outbound network traffic on the machine. Reach for it on critical matches, or when the script has already run.
  • Remote machines. When the alert maps to a registered endpoint agent, the same buttons send the command to that agent, and the confirmation names the remote host instead of "this machine".

One gap to know about. The Quarantine and Terminate dialogs pre-fill their choices by looking for FIM-style File : lines and Sysmon fields in the alert. A YARA alert states its target as "matched file <path>" or "(PID n)", so expect an empty suggestion list. Both dialogs have a manual field: paste the full path or type the PID from the alert text.

Scale to hundreds of thousands of rules

CTA can load a very large rule folder, but it was designed around a curated set, and a few behaviours change at scale. What follows separates what the code does, what I measured, and what I did not.

What I measured. At startup CTA reads every rule file twice, once to count rules and once to build the Detection Content inventory. I ran those two real functions over synthetic rule sets split into 1,000-rule files. This is only that bookkeeping, not YARA itself.

Rules

Files

Size on disk

Startup bookkeeping

Peak memory

10,000

10

4 MB

0.4 s

about 41 MB

100,000

100

37 MB

4.0 s

about 129 MB

300,000

300

111 MB

12.8 s

about 390 MB

What I did not measure. YARA's own compile time, its memory use and its scan time per file. yara-python is not available in my sandbox. Treat those as unknown until you test on your hardware.

What the code does that matters at scale.

  • One compile for everything. Every file is compiled together, each in its own namespace, so two feeds can reuse a rule name without clashing. The cost is that a single bad file fails the whole load.
  • One scan worker. Background scans run on a single thread against the full rule set. The queue holds 500 files, then drops new ones with a warning. A slow rule set therefore means dropped scans, not blocked file monitoring, though the worker still uses CPU while it runs.
  • Every rule becomes an inventory row. The Detection Content tab lists each rule, so 300,000 rules means 300,000 rows.
  • Missing severity means HIGH. Community feeds often omit it. With a large feed, that default can bury your real high-severity alerts.

How to run a large set safely.

  1. Curate before you scale. Start from the maintained feeds the project suggests, such as Yara-Rules and Neo23x0's signature-base, and load only the families you can act on. More rules mean more alerts to triage, not just more coverage.
  2. Validate every file on its own before deployment. Compile each file separately in a script or CI job and reject the ones that fail. This keeps one broken file from switching YARA off on every PC.
  3. Set a severity on every rule. Add it in bulk to a feed with a script before it reaches the folder.
  4. Split rules by client. Use yara_rules/clients/<client id>/ so each PC loads the universal set plus only its own client's rules.
  5. Narrow what gets scanned. Trim yara.extensions, add exclude_paths, and lower max_file_size_mb so the single worker keeps up. Watch logs/wta.log for the queue-full warning.
  6. Write cheap rules. Anchor with a size limit or a header check, avoid very short strings, and keep regex strings few. These are standard YARA practices and they decide scan speed more than rule count does.
  7. Roll out in stages. Go from 1,000 to 10,000 to 100,000 rules, and at each step record startup time, memory and queue warnings on your slowest PC.

What is not in the project today. The scanner does not skip a bad file, cache compiled rules between restarts, or push rules to endpoints. I found no code that copies YARA rules to agents; the agent's removable-media scan reads a local folder. Those would be sensible additions if your rule set keeps growing.

Before you go live

Work through this list the first time you put a rule set on a production PC.

  • yara-python is installed on every PC that scans.
  • Each rule file compiles on its own.
  • Every rule sets severity and description.
  • Files are in yara_rules\ or the right yara_rules\clients\<client id>\ folder.
  • CTA was restarted from the folder with main.py, and the YARA Scan status line shows the rule count you expect.
  • A harmless test file produced the expected match on demand.
  • For background scans, yara.enabled is true and FIM watches at least one path.
  • Extensions, excluded paths and the size limit have been reviewed.
  • Analysts know to paste the path or PID by hand for Quarantine and Terminate.

Limits to remember.

  • Background scans only see new or modified files in FIM watch paths. Files already on disk need an on-demand scan.
  • Process-memory scanning is Windows-only and needs CTA to run with enough privilege to read the process, typically as Administrator.
  • Agent telemetry arriving at the server is not YARA-scanned. The server code notes that hash, signature and YARA enrichment stay on the desktop.
  • A match shows content, not intent. Quarantine keeps the file and is easy to undo. Host isolation blocks all traffic and is not, so confirm before using it on an unfamiliar file.
  • Community rules carry licences. The project's notes say to review the terms before you redistribute any rule feed.

A natural next step is an upload screen or an API route for rules, so analysts do not need file access to the PC. CTA's web console already talks to the same server API and could host it, but that would need new server routes and checks first.

No comments:

Post a Comment

Hands-On with CTA ASM/EASM: Practical Techniques, Tips and Tricks for Effective Attack Surface Management

  A practical field guide for SOC analysts, security engineers, vulnerability managers, penetration testers, and cybersecurity consultants I...