Building Sift: A Faster, More Flexible Alternative to Snaffler

TL;DR

Sift is a modern alternative to Snaffler, which is a tool loved by many in the security community for years but has fallen behind its cutting edge origins in a story common in the open-source community.

Journey

For a time, the team here at Stratus maintained an internal fork of Snaffler and made pull requests where possible with additional features, including an upgrade to allow it to run under modern frameworks which significantly increased performance and allowed natively compiled binaries. For those .NET wizards reading this, it was conditional compilation to allow Framework builds for use over a C2 since a native compilation causes issues both loading as a .NET binary and size issues running in most C2s.

After a while, our internal version stacked up more and more features which weren’t included in the original and of course, we want to respect l0ss by not actively forking his repo. The work maintaining this also accumulated and eventually it was easier to create a whole new project than to continue patching the existing codebase, hence “Sift” was born.

Since we were making a brand new tool, we wanted to deal with the architectural challenges that come with something like this. Most importantly, the pain points many of us have dealt with during penetration tests such as bad computers provided by clients capping out under the load from the scan, waiting a LONG time for them to finish, crashed scans, missing rules for modern secrets and much more.

Since the tool supported so much more and especially due to the cloud scanning, our outcomes on internal network penetration tests became significantly better. This was great and all, especially making our testing much easier to do and finding a lot of issues missed by historical tests (especially by competitors!), but it also caused a few arguments internally when the discussion came up of open-sourcing. We could help a lot of businesses around the world find a lot of severe data security issues in their company, but we also give away the tooling we spent time and resources developing so there were a few arguments between sales and tech about the pros and cons, ultimately coming to an agreement that it should be released for all as I hope we can do for all our tools.

Features

Now back to the tech! Since we now had an idea of what we wanted in the tool, the features developed over time as the need arose in real engagements. The original version was a plain share scanner like Snaffler, but we quickly added the extra features. Below we have outlined our top features that don’t exist in the public Snaffler but we added to Sift, enabling our scanner to work so much more effectively.

Native Binaries

Many people assume .NET programs only run under .NET Framework or .NET Core but it has come a long way in the last decade. One of the more modern features is NativeAoT, which is just a fancy way of compiling the binaries to a single machine-code binary instead of a IL managed one like .NET usually does. Now to run the tool, it has no dependencies and can run on Windows, Mac or Linux without issue. In the Sift repo you can find automatically generated releases compiled this way for each major platform on ARM and x86-64 machines.

False Positives

If you ever ran Snaffler on a large network, you will be more than familiar with the effort that goes into reviewing the results due to the extensive false positives. We can’t prevent false positives in many cases, but we made sure to review every rule in Sift to make sure it has at least as much coverage while tightening a lot of them to prevent common false positives. On top of just improving the rules a little, we implemented a new rule format to make them clearer, easier to edit and added 2 major features to validate beyond the initial match. The first of which we call “validators” and the latter is some AI magic.

Validators are effectively code defined to do additional analysis on results beyond what a simple JSON/TOML rule can. For example, if you were looking for credit card numbers you would find a LOT of vulnerabilities so we made a Luhn algorithm validator which is a standard followed by credit card numbers beyond the normal regex. For you fellow nerds, it is as simple as creating a class like this:

public class LuhnValidator : BaseValidator
{
    public override string Name => ClassifierValidatorCatalog.Luhn;

    public override ValidationResult Validate(ValidationContext context)
    {
        string candidate = context.Candidate;

        // 1. FAST FAIL: Basic cleanup check
        if (string.IsNullOrWhiteSpace(candidate)) 
            return new ValidationResult { IsValid = false, Reason = "Empty" };

        // 2. MATH CHECK (Optimized)
        // We perform the math first because it is CPU-cheap compared to string searching.
        if (!PassesLuhnMath(candidate))
        {
            return new ValidationResult { IsValid = false, Reason = "Checksum Mismatch" };
        }

        // 3. CONTEXT CHECK (The "Smart" Layer)
        // If the math passes, we now check if it's likely a False Positive.
        var contextResult = CheckCommonContextClues(context);
        if (contextResult != null) return contextResult;

        // If we passed math and found no "test" indicators, it's a High Confidence finding.
        return new ValidationResult { IsValid = true, Confidence = 1.0 };
    }

    /// <summary>
    /// Zero-allocation Luhn implementation.
    /// Avoids LINQ to reduce Garbage Collection pressure during massive scans.
    /// </summary>
    private bool PassesLuhnMath(string candidate)
    {
        int sum = 0;
        bool doubleDigit = false;
        bool hasDigits = false;

        // Iterate backwards through the string manually
        for (int i = candidate.Length - 1; i >= 0; i--)
        {
            char c = candidate[i];
            
            // Skip non-digits (dashes/spaces) without creating new strings
            if (c < '0' || c > '9') continue;

            hasDigits = true;
            int digit = c - '0';

            if (doubleDigit)
            {
                digit *= 2;
                if (digit > 9) digit -= 9;
            }

            sum += digit;
            doubleDigit = !doubleDigit;
        }

        return hasDigits && (sum % 10) == 0;
    }
}

Finally, after the rules and validators are done you’ll still end up with a stack of false positives on any decent sized environment, so we added a magic AI feature to allow review of the found credentials automatically. It only supports local LLMs for privacy since the info found is very sensitive, but it can either filter in real-time or by being fed a previous output. The AI will be provided each result one by one and will decide whether or not to keep it. In our experience, even a relatively small model like Gemma4:31b will be able to give reliable results here since it is provided only a small packet of information at a time and makes decisions one by one.

Once the results have gone through all these steps, the accuracy is close to (if not) 100% every time. AI is coming for our jobs!

Connectors

After we made the results more reliable, we needed to be able to scan more things. This is a topic of contention among penetration testers, but we believe an internal network pentest should simulate the entire estate beyond just on-prem and include (within reason) cloud services used by employees like Teams, Slack, SharePoint and Confluence. Our opinion is that an internal pentest is a simulation of a threat actor gaining access to the internal network via an employees account, not a strict review of the internal network itself or we would just request an admin on day 1 and go to town.

With this in mind, we needed to support the places often overlooked. SharePoint has become part of many offensive security playbooks since it’s poorly monitored compared to on-prem shares. This is partly due to the lack of detection rules people often forget to place in SharePoint but more importantly, it’s because everyone is doing scans with Snaffler and there’s no tooling to support it in this way… until now!

Over several internal pentest engagements, we developed support for Microsoft/Sharepoint which naturally includes Teams and OneDrive along with support for Atlassian (Jira+Confluence) and Slack so all the most common places people dump secrets are covered. As alluded to earlier, this has made a huge difference to our tests since the industry standard software didn’t support any of this and ad-hoc searches the more advanced testers were doing just didn’t measure up to the comprehensiveness of this. If you have tried manually doing “SharePoint dorks” you know exactly what I mean.

Although we support these as the current connectors, the code has a simple interface which makes creating a new connector quick and easy. For now, Sift supports:

• Local Drives
• AD Enumerated Shares
• Network Shares (including subnet scanning!)
• Microsoft (SharePoint/Teams/OneDrive)
• Atlassian (Jira/Confluence)
• Slack

It’s amazing how many people are storing credentials in Jira tickets and comments or pasting their password to tech support in Slack…

Resume

We all know Windows likes to randomly restart or go to sleep, even if you told it not to. It likes to punish us for hacking it, so instead of fighting it we made the tool resilient to the gremlins running around inside Windows. To do this, we added support for checkpointing scans as they go. It’s a little complicated but effectively it will write a temp file with your current scans checkpoint periodically and as long as you restart the scan with –resume and provide the same flags, it will start from where it left off.

Windows restarts for updates? Resume! Gremlins turned off the power? Resume! Pick up where you left off and never cry in the bathroom over a lost scan again.

Depth

It’s 2026 and files are bigger and more complex than ever, so while developing our inspection functionality we decided to push it further after some experimentation to look deeper in files than ever before. This effectively breaks down to a few point: scanning chunk limits, tail scanning and file support.

The first thing we did was stream files instead of opening the whole thing, this along gave us a huge improvement to our results. If a file is 200MB, we will still scan the first 1MB by streaming just that chunk of the file and the last 1MB while Snaffler would simply skip the file entirely (or scan the whole thing under 1MB), helping us find those juicy secrets sitting at the top or bottom of large scripts or logs. In our experiments we found that it is by far most likely for secrets to be at the start or end of files so we decided that was the best choice. We also increased the inspection to read the entire file up to 10MB to allow full inspection for a larger range of files.

On top of the improved detection logic, we also added support for a few extra handy dandy file types to improve it further. First, binary files (docx, pdf, etc) are supported but much like Snaffler (which required a separate manual build) it requires opt-in since it’s much slower than plaintext scanning, although the implementation is significantly faster so it’s often a good idea to include unless you’re working across a huge estate. Additionally, we added zip support which means all those sneaky backup.zip files will now be parsed and run through the tool. Since many legacy tools and most DLP software have issues with zips, it is an often overlooked location with good stuff to investigate. Finally, before you sneaky hackers decide to place a zip bomb on a drive for us, we thought of that and added mitigations to prevent triggering it. Bad hacker.

Performance

All our changes mean wonderful things for detection accuracy and in some cases, speed. The updated rules mean faster regexin’ but the extra rules mean we do even more work than Snaffler did so additional improvements had to be made (or because we’re obsessive, you can choose!).

The whole design was around optimizing memory, CPU, disk and network impact so modern .NET paradigms like Channels and asynchronous programming are used among other things but that’s a conversation for another day. People assume .NET being a high level language means it has to be slow, but with a little magic you can optimize it with modern code to avoid things like allocating to the heap in hot paths, so a huge amount of memory and CPU usage can be saved. We used many programming tricks, but one of the easiest and most impactful ones wasn’t even on purpose, it was just using the most recent version of the language. In our previous work we identified you could significantly drop usage in Snaffler just by compiling it in .NET 10 and in Sift, we have dropped it even more with this extra effort.

We also redesigned the rules, there was a fair bit of discussion around keeping compatibility with Snaffler style TOML but ultimately a more familiar JSON was chosen. There are performance specific improvements here, but you can read the rules section below for more info.

To quantify performance, we ran 3 comparable scans against synthetic file repositories using the most recent Snaffler and Sift. Snaffler was patched to report completion immediately instead of waiting for its once-per-minute check-in, keeping the comparison fair:

Scenario	Snaffler	Sift	Improvement
250,000 small files	25.48 s	10.61 s	58.4% shorter
5.5 GiB content throughput	6.32 s	0.69 s	89.1% shorter
Deep and wide tree	2.37 s	1.12 s	52.7% shorter

Each test was run three times. The aggregate averages are shown below:

Metric	Snaffler	Sift	Improvement
Duration	34.18 s	12.42 s	63.7% quicker
Total CPU time	276.56 s	62.11 s	77.5% less CPU time
Average CPU load	25.3%	15.6%	38.3% lower
Average memory	337 MiB	92 MiB	73% lower
Memory-time	11.24 GiB·s	1.11 GiB·s	90.1% lower
Peak memory	429.5 MiB	102.2 MiB	76.2% lower

As you can see in our benchmark results, despite the extra capabilities Sift comes in way lower in all items and in real-world scans we get results in a fraction of the time without crippling our machine.

Our final performance feature is important too. Speed is great, but I know we have personally slowed down client machines, networks, or shared devices by capping out the resources so we have included custom thread counts, disk rate-limits, etc. so you can cap the amount of resources used at any time.

Much More!

If you made it this far, you are probably bored of hearing about all the extra features and engineering going on. So here’s a rapid fire list of the other important things:

• Kerberos support!
• Explicit credential support! (No more runas)
• Safety features! (Avoid syncing an entire OneDrive estate accidentally or infinite looping)
• Cross-platform! (Windows+Mac+Linux)
• Custom DNS servers!

Rules

Rules! Hackers love them, right? We at least love making our own, so we reimagined a new rule format from scratch to enable flexible and clear use of all the features in Sift.

The default ruleset can be found in the repo here and they are bundled with the binary by default so you don’t have to think too hard but if you would like to customize, you can provide your own like this:

.\sift.exe local –path C:\Review –rules .\my-rules

Complicated! Now, if you wanted to make a rule you have 2 types to worry about, ignore rules and normal rules.

Ignore Rules

An ignore rule simply allows users to ignore files or folders and lives in the IgnoreRules folder. An example is this rule:

[
{ “Pattern”: “IPC$”, “MatchTarget”: “ShareName”, “Description”: “Administrative Share”, “IsEnabled”: true },
{ “Pattern”: “PRINT$”, “MatchTarget”: “ShareName”, “Description”: “Administrative Share”, “IsEnabled”: true }
]

It prevents trying to enumerate these common, useless shares. Others avoid generally wasteful directories, etc and use a minimal version of the format used for normal rules, except they’re just a plain array of matches.

Sifting Rules

Sifting rules are our plain rules, which exist in a folder named Rules because we’re creative over here. They are a little more complicated, for example this one for detecting OpenVPN configs:

{
“Name”: “OpenVPN Configuration”,
“Description”: “Detects OpenVPN config files containing embedded private keys”,
“Severity”: “High”,
“Matches”: [
{
“Target”: “Content”,
“Keywords”: [
“<key>”,
“—–BEGIN PRIVATE KEY—–“
],
“Patterns”: [
“<key>\\s*—–BEGIN.*?PRIVATE KEY—–“
],
“ExtensionProfile”: “SourceAndConfig”,
“IncludedExtensions”: [
“.ovpn”
]
}
],
“EnableLlmValidation”: false
}

Forgive the blogs lack of indentation, the rule defines a name and severity defined to inform the output in the terminal, a description purely for documentation, an array of Matches which define what content to match and some extra validation fields although only name and matches are required fields. Here’s all options for reference:

Field	Required	Purpose
Name	Yes	Stable rule and finding name.
Matches	Yes	One or more metadata or content match blocks.
FindingName	No	Different name for the reported finding. The default is Name.
Description	No	Short explanation of what the rule detects.
Severity	No	Info, Informational, Low, Medium, High, or Critical. The default is Medium.
Enabled	No	Enables the rule. The default is true.
Validator	No	Built-in validation step used after a pattern matches.
EntropyThreshold	No	Minimum Shannon entropy for the matched value. 0, the default, disables this check.
EnableLlmValidation	No	Allows the match to be checked when the scan uses --llm-validate. The default is true, but no LLM is called unless that command option is supplied.
MinMatchCount	No	Number of matches required on one item before a finding is kept. The default is 1.
IncludePaths	No	Allowlist of full-path wildcard expressions.
ExcludePaths	No	Blocklist of full-path wildcard expressions.
StopOnMatch	No	Stops the current matching pass after this rule reports a match. The default is false.
ReportFinding	No	Set to false for a parent rule that only gates its SubRules. The default is true.
SubRules	No	Second-stage rules evaluated after their parent matches.

Each item in Matches supports:

Field	Required	Purpose
Patterns	Yes	One or more literal values or .NET regular expressions.
Target	No	The part of the item to inspect. The default is Content. See the table below.
IsLiteral	No	Treats patterns as plain text. The default is false, which uses regex for content patterns.
CaseSensitive	No	Makes patterns and keywords case-sensitive. The default is false.
Keywords	No	Fast prefilter for content rules. At least one keyword must be present before the regex runs. The regex must still match.
ExtensionProfile	For content¹	Named extension allowlist. SourceAndConfig is currently provided.
IncludedExtensions	For content¹	Additional extensions such as .txt, .config, or .ps1.

¹ A top-level content match must declare ExtensionProfile, IncludedExtensions, or both. A content SubRule inherits the scope established by its parent.

The most important new concept here is the keywords, which enable a huge speedup in scanning. The keywords are effectively plaintext strings which must have at least one match before a pattern gets evaluated, if you’ve ever done a performance evaluation of a tool like this you will know a huge majority of the resources are used in regexes. Sifts architecture allows the resource usage to be reduced dramatically as seen in our benchmarks so less data goes through your machine and the hold up generally becomes a disk or networks speed. In the case of our OpenVPN rule, the keyword defines <key> and —–BEGIN PRIVATE KEY—– so if these strings don’t exist, the scan moves on without touching that costly regex resulting in exponentially less regexes being evaluated.

The other new concept here is the extension profile and excluded extensions. Included extensions provides an allowlist of extensions to check for a rule just like Snaffler. We also added extension profiles, they’ll be added to a third folder in an update, but for now it’s just a shorthand way of saying a rule should include a pre-defined list of extensions in its allowlist. For posterity, here is the code for it:

[SourceAndConfig] = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
{
string.Empty,
“.ascx”, “.ashx”, “.asmx”, “.asp”, “.aspx”,
“.bash_history”, “.bashrc”, “.bat”, “.bicep”, “.bicepparam”, “.c”, “.cc”, “.cfm”, “.cjs”, “.cmd”,
“.cnf”, “.conf”, “.config”, “.cpp”, “.credentials”, “.cs”, “.cshtml”, “.csv”, “.cue”, “.cxx”,
“.dart”, “.dist”, “.do”, “.dockerfile”, “.dockerignore”, “.docx”,
“.editorconfig”, “.env”, “.es”, “.es6”, “.exports”, “.extra”,
“.fdb”, “.fs”, “.fsx”, “.functions”,
“.gemrc”, “.git-credentials”, “.gitconfig”, “.go”, “.gql”, “.gradle”, “.graphql”, “.groovy”,
“.h”, “.har”, “.hcl”, “.hpp”, “.hta”, “.http”,
“.inc”, “.inf”, “.ini”, “.irb_history”,
“.ipynb”, “.java”, “.js”, “.json”, “.jsonl”, “.jsp”, “.jsx”,
“.key”, “.kt”, “.kts”, “.log”, “.ls”, “.lua”,
“.markdown”, “.md”, “.mdc”, “.mjs”,
“.ndjson”, “.netrc”, “.nix”, “.npmrc”, “.nuget”, “.pdf”, “.pem”, “.php”, “.php3”, “.php5”, “.php7”,
“.phtml”, “.pl”, “.profile”, “.prompt”, “.prompty”, “.properties”, “.ps1”, “.psd1”, “.psm1”,
“.pub”, “.py”, “.pypirc”,
“.proto”, “.r”, “.rb”, “.rc”, “.rego”, “.rest”, “.rs”,
“.scala”, “.service”, “.sh”, “.sh_history”, “.sql”, “.sqlite”, “.sqlite3”, “.svelte”, “.swift”,
“.targets”, “.templ”, “.tf”, “.tfplan”, “.tfrc”, “.tfstate”, “.tfvars”, “.toml”, “.ts”, “.tsv”, “.tsx”, “.txt”,
“.vb”, “.vbe”, “.vbs”, “.vue”, “.wsc”, “.wsf”,
“.xlsx”, “.xml”, “.yaml”, “.yml”, “.zsh_history”, “.zshrc”
}

Snaffler rules were prone to misisng common extensions since they needed to be repeated every rule, so we made an abstraction to allow repitition.

Conclusion

This tool has done wonderful things for us and our clients, so please let us know how it works for you! We’re happy to hear from anyone who loves the tool, has feedback or even just needs a hand setting it up. If it’s too hard to use, it’s an issue that needs fixing, not an issue with you.