In order to accept requests from Nightfall, a Webhook server must use a signing key to verify requests.

To access or generate your Webhook signing key, start by logging in to the Nightfall dashboard.

Select the Developer Platform > Manage API Keys using the navigation bar on the left side of the page. You will see the Webhook signing section:

Unlike the API Key, it is possible to reveal the signature via the "eye" icon furtherest to the left of the three icons displayed.

You may copy the current value to your clipboard with the "copy" icon in the center of the three icons displayed.

You may also regenerate the key with the circular arrow icon furthest to the right.

Use this value as shown in the code examples that are used in the following sections.

The Nightfall API supports the ability to send asynchronous notifications when findings are detected as part of a scan request.

The supported destinations for these notifications include external platforms, such as Slack, email, or url to a SIEM log collector as well as to a webhook server.

Nightfall issues notifications under the following scenarios:

• to notify a client about the results of a . File scans themselves are always performed asynchronously because of complexity relating to text extraction and data volume.

• to notify a client about results from a text scan request. Although results are already delivered synchronously in the response object, clients may configure the request to forward results to other platforms such a webhook, SIEM endpoint, or email through a

To create a webhook you will need to and then set up a

For more information on how webhooks and asynchronous notifications are used please see our guides on:

Policies allow customers to create templates for their most common workflows by unifying a set of Detection Rules with the actions to be taken when those rules are triggered, including:

• automated actions such as redaction of findings

• alerting through webhooks

Once defined, a Policy may be used in requests to the Nightfall API, such as calls to scan file uploads, though automated redactions are not available for uploaded files at this time.

To create a policy:

1. Log in to Nightfall.

2. Click Policies under the Configuration section.

3. Click + New Policy.

4. Select Developer APIs.

5. Select the Detection Rules to be included in the policy and click Next.

6. Configure a notification channel. Click to learn more about alert channel configuration. If you wish to setup a webhook alert channel, click for details.

1. Click Next.

2. (Optional) Enable the Redact Message toggle switch. This is an automated action that is triggered when sensitive data is found. The action automatically redacts sensitive data.

3. Click Next.

4. Enter a name for the policy.

5. (Optional) Enter a Description for the policy and click Next.

6. Verify the configurations and click Submit.

Configuring Webhook Alerts

To configure Webhook as an alert channel:

1. Enable the Webhook Alert notification channel.

2. In the Configure Webhook URL field, enter the URL of the Webhook to which you wish to send notifications.

3. (Optional) Click Add Headers to add header key value pairs.

4. Click Validate.

5. Once validated, you can click Next to proceed.

Similarly, you can also use HTTP alerts channel.

The API expects an API Key to be passed via the Authorization: Bearer <key> HTTP header.

To create and manage API keys:

1. Log in to Nightfall.

2. Click Overview under the DEVELOPER APIS section.

3. Click Create key.

The Generate API Key window is displayed.

1. Enter a name for the API key and click Create.

The API key is generated and displayed. Click the copy button to copy the API key and store it in a. secure location. Once you click the Got it button, you cannot retrieve the API key again.

🚧Be Sure to Record the API Key's Value

For security reasons, after closing the window, you will not be able to recover the key's value.

Once you close the window, the Your API Keys section will display your newly generated key, with the majority of the Key redacted.

You can click the Create Key button to create new keys (assuming your license allows you to generate additional keys). You can click the Delete icon against an API key to delete the key.

The Nightfall API uses API keys to authenticate requests. You can create and view your API keys in the Nightfall app on the page.

Your API keys carry many privileges, so be sure to keep them secure. Do not share your secret API keys in publicly accessible areas such as GitHub, client-side code, or anywhere else that would compromise their secrecy. If you believe one of your API Keys has been compromised, you should delete it through the Dashboard.

All API requests must be made over HTTPS.

Calls made over plain HTTP will fail.

API requests without authentication will fail.

Protected health information (PHI), also referred to as personal health information, describes a patient's medical history — including ailments, various treatments, and outcomes. PHI may include:

• demographic information

• test and laboratory results

• mental health conditions

• insurance information

The Health Insurance Portability and Accountability Act (HIPAA) of 1996 is the primary law that oversees the use of, access to, and disclosure of PHI in the United States. HIPAA lists 18 different personal information identifiers (PII) that, when paired with health information, become PHI. In order to more accurately detect potential PHI, Nightfall has introduced specific new detectors that allow for specialized combinations.

These HIPAA PII and PHI-specific detectors intelligently aggregate Nightfall's built-in detector to ensure compliance with governing law. For example, finding a patient's name in a document or message is not considered HIPAA PII as it does not uniquely identify an individual, many people can share the same name. However, the information would be considered HIPAA PII if the patient's name and address were in the same message.

Specific PHI and HIPAA PII can be detected with greater confidence, especially as they relate to specific medical codes or terms in association with specific logical combinations of other PII. For instance when the patient's name and date of birth or a person's name and street address or any of a set of particular PII (phone number email, SSN, etc) it would be considered HIPAA PII.

If the combined detectors all match with a confidence of "Very Likely" it would match our "HIPAA PII Very Likely" Detection Rule. Otherwise if these detectors match with a confidence of "Likely" it would match our "HIPAA PII Likely" Detection Rule.

Alternatively when any of the above PII options are found in conjunction with a specific set of medical related codes or terms (IDC Codes, FDA Drug Names or Codes, Procedures), that finding could be flagged as PHI.

When all the detectors within these PHI Detection Rules make findings that have a confidence of "Very Likely," that would match our "PHI Very Likely" Detection Rule, while if some are all are met with a confidence of "Likely" that would match our "PHI Likely" Detection Rule.

Our PHI Detectors may be used just like other Detectors with or .

Before you use the scan endpoint, there are a number of actions to do within the Nightfall dashboard to get your environment set up properly.

• See to see how to create the necessary Authentication token for making API calls.

• See for how to define your own custom logic for detecting sensitive data

• See for how to aggregate Detectors for use in the scan endpoint

• See for how to set up common workflows that combine your Detection Rules with remediation actions such as alerting.

Welcome to the amazing world of the Nightfall Developer APIs (formerly known as Firewall for AI). Here you can find all the information about Nightfall's APIs, and SDKs, and also usage examples of these APIs and SDKs.

The following sample datasets can be used to test Nightfall's advanced AI-based detection capabilities.

This data has been fully de-identified and can be used to test any data loss prevention (DLP) platform.

Welcome to Nightfall's Firewall for AI Developers Scan and Workflow APIs documentation. This documentation helps developers leverage Nightfall AI's industry-leading detection engine to identify and protect sensitive customer and corporate data anywhere. It prevents unauthorized access and data breaches and allows you to focus on innovation.

Scan APIs

Scan prompts, text, documents, spreadsheets, logs, zips, JSON, images, etc., for PII, PHI, PCI, banking information, API keys, passwords, and network information with the highest accuracy and lightning-fast response times. Redact sensitive findings with customizable formatting.

Workflow APIs

Leverage the full potential of the Nightfall console application through our Workflow APIs. Customize your SIEM workflows and reporting, take actions, update support tickets, alert users, search violations, annotate findings, create reports, and more.

Key Features

• AI-Powered Identification: Utilize advanced AI models to detect and prevent security threats in real-time.

• Comprehensive Sensitive Data Detection: Identify PII, PHI, PCI, banking information, API keys, passwords, and network information across various formats including text, documents, spreadsheets, logs, zips, and images.

• Customizable Redaction: Tailor data protection to your needs with fully customizable redaction for each sensitive entity type.

• Flexible Detectors: Leverage Nightfall’s comprehensive list of machine learning-based detectors, customize them, or create your own with specialized logic.

• High Accuracy and Performance: Achieve precision and recall rates of 95% or higher, handle over 1K requests per second, and experience latency of less than 100 ms.

• Seamless Integration: Easily integrate with your existing AI development and data engineering tools for smooth and efficient operation.

Customizable and Built-in Machine Learning-based Detectors

You can leverage Nightfall’s machine learning-based detectors or create your own detectors with customized logic to scan third-party apps, internal services, and data silos to identify instances of potentially sensitive types of data such as:

• Personally Identifiable Information (PII) including Social Security Numbers, passport numbers, email addresses, or date of birth

• Protected Health Information (PHI) such as insurance claim numbers or ICD10 codes

• Financial information like credit card numbers or bank routing numbers

• Secrets such as API and cryptographic Keys, database connection strings, passwords, etc.

• Network information such as IP Address or MAC Address

A Flexible Data Security Solution

Key features of Nightfall’s detection engine include:

• Defining minimum confidence thresholds and minimum finding counts on detectors to reduce the chance of false positives.

• Specifying context rules and exclusion rules on detectors to fine-tune their accuracy to better suit your use cases.

• Choosing which detectors are triggered for each policy.

Using the API

The Nightfall API consumes arbitrary data as input either as strings or as files and allows you to use any combination of detectors to return a collection of “findings" objects.

The detectors may be referenced in an API call or defined as part of the payload to an API call.

The findings display the relevant detector, the likelihood of a match, and the location within the given data where the matched token occurred (not only in terms bytes — there is support for tabular and JSON data as well).

You can take protective action on sensitive text by redacting, substituting, or encrypting it with the API. You may also set up webhooks to receive asynchronous notifications when findings are detected.

The Nightfall API is RESTful and uses JSON for its payloads. Our API is designed to have predictable, resource-oriented URLs for each endpoint and uses HTTP response codes to indicate any API errors.

You may test out the API through the interactive reference documentation.

Where to Go From Here

The following guide will walk you through getting started and describe the API functionality in more detail. If you want to execute an API call immediately, see our Quickstart guide to see how to obtain an API Key and make a simple scan request.

After that, you can learn about Nightfall with our Key Concepts section, which will also help you get set up with Nightfall.

If you’re looking for more ideas about best to leverage Nightfall’s functionality, see our Use Cases guide.

We have created numerous tutorials and example implementations that demonstrate how to implement DLP for a variety of platforms (including OpenAI, LangChang, Amazon, Datadog, and Elasticsearch) and handle various scenarios (such as detecting sensitive data in GenAI prompts or detecting PII on your machine in real-time).

We also have several language-specific SDKs to get you up and running in Java, Python, Go, Node.js, and Ruby.

You can also quickly test out Nightfall detectors or your custom Detection Rules in the Nightfall Playground. Please also consult our Detector Glossary to see the variety of built-in detectors that Nightfall offers.

The Firewall for AI Overview page allows you to create API keys and manage Detectors and Detection Rules through a straightforward user interface. Log in here to access the Dashboard, or sign up to create a free account.

For frequently asked questions, feedback, and other help, please contact Nightfall support at [email protected]. We also host Nightfall Developer Office Hours on Wednesdays at 12pm PT to help answer questions, talk through any ideas, and chat about data security. We would love to see you there!

The scan endpoint allows you to apply Policies and Detection Rules to a list of text strings provided as a payload.

curl --request POST \
     --url https://api.nightfall.ai/v3/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer  NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                         {
                              "minNumFindings": 1,
                              "minConfidence": "LIKELY",
                              "displayName": "US Social Security Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER"
                         }
                    ],
                    "name": "My Match Rule",
                    "logicalOp": "ANY"
               }
          ]
     },
     "payload": [
          "The customer social security number is 458-02-6124",
          "No PII in this string"
     ]
}

You may use Pre-Configured Detection Rules or Create Inline Detection Rules

Text scanning supports the use of Exclusion Rules, Context Rules, and Redaction as well as other Scanning Features.

For scanning files, see Scanning Files.

Note that you must generate an API key to send requests to the Nightfall API.

Nightfall offers many useful features beyond its detectors, including:

The ability to use Context Rules and Exclusion Rules to narrow the scope of matches.

The ability to create Redactions in a way that is highly configurable so that sensitive data is appropriately obfuscated.

The ability to create Policies that determine how leaks of sensitive information should be mitigated (i.e. through alerts sent to email or Slack).

Nightfall’s file scan API allows a user to upload a file in chunks, then to scan it with Detection Rules once the upload is complete.

The scan will then be processed asynchronously before sending the results to the webhook URL that is provided along with your Detection Rules.

The following sequence diagram illustrates the full process for scanning a binary file with Nightfall.

For a detailed walkthrough of the API calls necessary to upload and scan a file and full script that shows the entire process, see Uploading and Scanning Files.

Prerequisites

In order to utilize the File Scanning API you need the following:

• An active API Key authorized for file scanning passed via the header Authorization: Bearer <key> — see Authentication and Security

• A Nightfall Detection Policy associated with a webhook URL

• A web server configured to listen for file scanning results (detailed information to follow)

File scanning also support Nightfall's functionality for Using Exclusion Rules and Using Context Rules as part of your scan requests.

This section describes the terms you will need to know when using the API.

Detectors

Detectors provide the logic to find potentially sensitive pieces of data.

When this logic detects such data, the Detector is considered "triggered."

Nightfall's has numerous pre-built Detectors that are trained via machine learning. Detectors may also be defined with regular expressions or dictionaries. Their accuracy may be further refined with exclusion rules and context rules. Whether a Detector is triggered may be controlled by a minimum confidence threshold per Detector and minimum number of findings per Detector as set on a Detection Rule.

The built-in set of Detectors cover a number of different categories of data, including:

• Standard PII (e.g. social security number, driver's license number, ID card image)

• PCI (Credit Card Number, credit card image)

• Healthcare (e.g. PHI, US Medicare Beneficiary Number)

• Finance - Banking (e.g. SWIFT code, IBAN code, US bank routing number)

• Network (e.g. an IP Address)

The full set is enumerated in the .

Custom Detectors

Nightfall also supports and word lists for any custom detectors that you may want to implement.

Over time, we've aggregated the following , which you're welcome to select from to save you some time. Please note that a regular expression is an established yet limited method that searches for pre-defined patterns, so your mileage may vary.

You can test regular expressions .

You can input custom detectors in two ways: directly in the Nightfall by navigating to Detectors → New Detector → Regular expression, or

Exclusion Rules

An exclusion rule is a regular expression or word list that will be used once a Detector is triggered by its primary expression or word list to eliminate false positives.

For instance, you may have a Detector designed to detect phone numbers. However, you may have a particular set of phone numbers that you use for testing purposes that are known not to be valid (e.g. they start with the prefix 555) and this should be ignored. Adding an exclusion rule would allow you to prevent those matches from being returned by the API.

See:

Context Rules

Context Rules are additional matching expressions for a Detector that may be used to adjust the confidence score of a match.

You may provide a regular expression and the number of leading or trailing characters within which a match of that expression must occur in order to adjust the confidence level to a particular level.

For instance, if you found a sequence that appeared to be a social security number based on its length or formatting, you might boost the confidence score if it was preceded by the text like “SSN” or “Social Security Number.”

Returning Surrounding Context

You may request that a sequence of bytes of a given length be provided from before and after the text that triggers a Detection Rule.

This information can help you better understand whether or not something is an actual violation by observing the circumstances within which the detected text was found.

You are limited to a maximum of 40 bytes of this context text preceding and trailing the match for a total of 80 bytes overall.

See:

Detection Rules

Detection Rules are aggregations of Detectors that are assigned a minimum confidence level. The identifiers of Detection Rules are used as a parameter to the API.

You may create Detection Rules as described in the section and use their identifier as part of API calls to scan content.

Alternatively you may specify Detection Rules in each API call, as described in the scan method documentation below.

A Detection Rule is composed of a list of Detectors with which you wish to scan each request payload, where any or all Detectors may be satisfied in order to trigger the rule. You can add up to 50 total Detectors with a limit of 30 regular expression type custom detectors.

Additionally, each Detector in the Detection Rule is assigned a “minimum confidence” level (see and a minimum number of findings to determine if the Detection Rule should be considered triggered.

Confidence Levels

Detection results will be returned with one of the following confidence values.

In practice, the API will only return detections assigned a POSSIBLE or higher confidence level.

• VERY_LIKELY (recommended)

• LIKELY

• POSSIBLE

• UNLIKELY

• VERY_UNLIKELY

Learn more about what different confidence levels mean and how to choose the right minimum confidence level for your detection rule .

Policies

Policies allow you to create templates for the most common workflows by unifying a set of Detection Rules with the actions to be taken when those rules are triggered, including:

• automated actions such as redaction of findings

• alerting through webhooks

Once defined, a Policy may be used in requests to the Nightfall API, such as calls to scan file uploads, though automated redactions are not available for uploaded files at this time.

Using regex to identify long patterns in images can be challenging because OCR systems. In such cases, even Nightfall may not achieve 100% character-by-character accuracy. To improve results, you must introduce higher levels of flexibility into your regex patterns to accommodate common OCR inconsistencies. Here are some typical OCR challenges to keep in mind:

• Spell-check noise: Spell-checking tools can add artifacts like red underlines, which may interfere with text recognition.

• Character ambiguity:

  • The digit 0 may be misinterpreted as the letter O (or vice versa), depending on the font.

  • The character l (lowercase L) may be read as the digit 1.

  • The letter B may appear as the digit 8.

• Underscore handling: An underscore (_) is sometimes interpreted as a space, particularly when spell-check artifacts are present.

• Line wrapping: OCR may introduce unexpected newlines when text wraps across multiple lines.

• Periods and punctuation: Spell-check artifacts or font issues may result in extraneous periods (.) or other punctuation being added to the output. En dash (–) and hyphens (-) may be interchanged.

For reference, OCR tools like Tesseract typically achieve 85-98% character accuracy for similar input, and our system operates within a similar range. Given this, tuning your regex to be more forgiving (e.g., allowing for optional characters or slight variations) can significantly improve detection rates.

Example Regex (original and loosened)

original: ATATT3xFfGF0[A-Za-z0-9=_\-]*[=A-Za-z0-9]{9}

loosened: ATATT[A-Za-z0-9_\-– @.\n=]*[A-Za-z0-9_\- @.\n]{7,11}

• shortened the literal match prefix

• excluded the the literal zero (0) from the prefix

• added period (.) and newline () chars

• relaxed the char length

As part of submitting a file scan request, the request payload must contain a reference to a webhook server URL defined as part of a policy defined inline.

When Nightfall prepares a file scan operation, it will issue a challenge to the webhook server to verify its legitimacy.

After the file scan has been processed asynchronously, the results will be delivered to the webhook.

Webhook Payload and Findings for File Scans

For a file scan, your webhook will receive a request body that will be a JSON payload containing:

• the upload UUID (uploadID)

• a boolean indicating whether or not any data in the file matched the provided detection rules (findingsPresent)

• a pre-signed S3 URL where the caller may fetch the findings for the scan (findingsURL). if there are no findings in the file, this field will be empty.

• the date until which the findingsURL is valid (validUntil) formatted to RFC 3339. Results are valid for 24 hours after scan completion. The time will be in UTC.

• the value you supplied for requestMetadata. Callers may opt to use this to help identify their input file upon receiving a webhook response. Maximum length 10 KB.

Below is an example of a payload sent to the webhook URL.

{
    "findingsURL": "https://files.nightfall.ai/asdfasdf-asdf-asdf-asdf-asdfasdfasdf.json?Expires=1635135397&Signature=asdfasdfQ2qTmPFnS9uD5I3QGEqHY2KlsYv4S-WOeEEROj~~x6W2slP2GvPPgPlYs~lwdr-mtJjVFu4LtyDhdfYezC7B0ysfJytyMIyAFriVMqOGsRJXqoQfsg8Ckd2b6kRcyDZXJE25cW8zBS08lyVwMBCsGS0BKSin8uSuD7pQu3QAubT7p~MPkfc6PSXYIJREBr3q4-8c7UnrYOAiXfSW1AmFE47rr3Wxh2TpU3E-Fxu-6e3DKN4q6meACdgZb2KHZo3e-NK7ug9f8sxBp1YT0n5oiVuW4KXguIyXWN~aKEHMa6DzZ4cUJ61LmnMzGndc2sVKhii39FHwTsYog__&Key-Pair-Id=asdfOPZ1EKX0YC",
    "validUntil": "2021-10-25T04:16:37.734633129Z",
    "uploadID": "152848af-2ac9-4e0a-8563-2b82343d964a",
    "findingsPresent": true,
    "requestMetadata": "",
    "errors": []
}

If you follow the URL (before it expires) it will return a JSON representation of the findings similar to those returned by the Scan Plain Text endpoint.

In this example, we have uploaded a zip file with a python script (upload.py) and a README.md file. A Detector in our DetectionRule checks for the presence of the string http://localhost

{
   "findings":[
      {
         "path":"fileupload/upload.py",
         "detector":{
            "id":"58861dee-b213-4dbc-97fa-a148acb8bd1a",
            "name":"localhost url"
         },
         "finding":"http://localhost",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":105,
               "end":121
            },
            "codepointRange":{
               "start":105,
               "end":121
            },
            "lineRange":{
               "start":7,
               "end":7
            }
         },
         "beforeContext":"PLOAD_URL = getenv(\"FILE_UPLOAD_HOST\", \"",
         "afterContext":":8080/v3\")\nNF_API_KEY = getenv(\"NF_API_K",
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      },
      {
         "path":"fileupload/README.md",
         "detector":{
            "id":"58861dee-b213-4dbc-97fa-a148acb8bd1a",
            "name":"localhost url"
         },
         "finding":"http://localhost",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":570,
               "end":586
            },
            "codepointRange":{
               "start":570,
               "end":586
            },
            "lineRange":{
               "start":22,
               "end":22
            }
         },
         "beforeContext":"t the script will send the requests to `",
         "afterContext":":8080`, but this can be overridden using",
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      },
      {
         "path":"fileupload/README.md",
         "detector":{
            "id":"58861dee-b213-4dbc-97fa-a148acb8bd1a",
            "name":"localhost url"
         },
         "finding":"http://localhost",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":965,
               "end":981
            },
            "codepointRange":{
               "start":965,
               "end":981
            },
            "lineRange":{
               "start":26,
               "end":26
            }
         },
         "beforeContext":"ice deployment you want to connect to | ",
         "afterContext":":8080 |\n| `NF_API_KEY`      | the API Ke",
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      }
   ]
}

An Exclusion Rule allows you to refine a Detector to make sure false positives are not surfaced by Nightfall.

For instance you may want to detect whether credit card numbers are being shared inappropriately in your organization. However, there may be cases where members of your QA are sharing test credit card numbers, which should not be considered a violation and should be ignored by Nightfall.

In the following example, we define a Detector with a regular expression to match credit cards.

We then add an exclusion for some known test credit cards.

curl --location --request POST 'https://api.nightfall.ai/v3/scan' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--header 'Content-Type: application/json' \
--data-raw '{
    "policy": {
        "detectionRules": [
            {
                "detectors": [
                    {
                        "regex": {
                            "pattern": "(?:(4[0-9]{12}(?:[0-9]{3})?)|(5[1-5][0-9]{14})|(6(?:011|5[0-9]{2})[0-9]{12})|(3[47][0-9]{13})|(3(?:0[0-5]|[68][0-9])[0-9]{11})|((?:2131|1800|35[0-9]{3})[0-9]{11}))",
                            "isCaseSensitive": false
                        },
                        "exclusionRules": [
                            {
                                "wordList": {
                                    "values": [
                                        "4111111111111111",
                                        "5105105105105100"
                                    ]
                                },
                                "exclusionType": "WORD_LIST",
                                "matchType": "FULL"
                            }
                        ],
                        "minNumFindings": 1,
                        "minConfidence": "POSSIBLE",
                        "displayName": "Credit Card Reg Ex",
                        "detectorType": "REGEX"
                    }
                ],
                "name": "Credit Card Detection Rule",
                "logicalOp": "ALL"
            }
        ]
    },
    "payload": [
        "5105105105105100",
        "4111111111111111",
        "4012888888881881"
    ]
}'

As the resulting payload shows, only the 3rd provided Credit Card number matches because the first two items in the payload are included in our ExclusionRules word list.

{
   "findings":[
      [
         
      ],
      [
         
      ],
      [
         {
            "finding":"4012888888881881",
            "detector":{
               "name":"Credit Card Reg Ex",
               "uuid":"93024e88-e6de-4c84-8295-75157cdd1b52"
            },
            "confidence":"LIKELY",
            "location":{
               "byteRange":{
                  "start":0,
                  "end":16
               },
               "codepointRange":{
                  "start":0,
                  "end":16
               },
               "rowRange":null,
               "columnRange":null,
               "commitHash":""
            },
            "matchedDetectionRuleUUIDs":[
               
            ],
            "matchedDetectionRules":[
               "Credit Card Detection Rule"
            ]
         }
      ]
   ],
   "redactedPayload":[
      "",
      "",
      ""
   ]
}

Nightfall provides you the flexibility to easily integrate into applications using programming languages. The supported languages are as follows.

• Python

• Ruby

• Java

This section consists of use case tutorials for various scenarios of Firewall for AI. The tutorials explained in this section are as follows.

• LangChain Prompt Sanitization

• Deploy a File Scanner for Sensitive Data in 40 Lines of Code

• Redacting Sensitive Data in 4 Lines of Code

• Detecting Sensitive Data in SMS Automations

• Building Endpoint DLP to Detect PII on Your Machine in Real-Time

You can start scanning for sensitive data in just a few minutes. Our developer-friendly API and comprehensive documentation make it easy to integrate Firewall for AI into your application. Follow our Quickstart guide at this link for step-by-step instructions on setting up the API, configuring detectors, and making your first API call.

Absolutely! In addition to the pre-built detectors, Firewall for AI allows you to create custom detectors tailored to your specific requirements. You can either fine-tune one of our pre-configured detection rules or build your own detector from scratch using our intuitive API. Nightfall supports many traditional detector types such as regular expressions, exact data matching, and word list/dictionaries. Check out our dedicated guide on creating custom detectors for more information.

Firewall for AI offers a rich set of pre-built detectors that can identify many different types of sensitive data, including personally identifiable information (PII), payment card industry data (PCI), protected health information (PHI), secrets, and credentials. These detectors are powered by advanced machine learning models and can be easily integrated into your application with just a few lines of code. Refer to our detector glossary at docs.nightfall.ai/docs/detector-glossary for a complete list of available detectors.

At Nightfall, data security and privacy are our top priorities. We have implemented stringent security measures to protect your sensitive data at every stage of the scanning process. All data transmitted to our API is encrypted in transit using industry-standard protocols. We adhere to best practices for secure coding, undergo regular security audits, and maintain compliance with relevant security standards. Visit our security and compliance page at nightfall.ai/security for more details on our commitment to data protection.

Nightfall supports Detectors that will scan for file names, file types, and file finger prints.

Detecting File Names

In addition to scanning the content of files, you may configure the Detectors to scan file names as well.

This is done through the “scope” attribute of a Detector.

The scope attribute allows you to scan either within file contents, the file name, or both the file contents and file name.

File extensions can be scanned for by creating a Regular Expression type custom Detector with a scope to scan only file names ("File") or both the content and file name ("ContentAndFile"), as shown in the example request below.

curl --request POST \
     --url https://api.nightfall.ai/v3/upload/<fileid>/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer  NF-<yourNightfallKey> \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                         {
                              "regex": {
                                   "pattern": "*\.txt",
                                   "isCaseSensitive": false
                              },
                              "detectorType": "REGEX",
                              "scope": "ContentAndFile"
                         }
                    ],
                    "name": "File Name Detector",
                    "logicalOp": "ANY"
               }
          ]
     }
}

In addition to scanning based on file name, you may also use a File Type Detector which allows you to scan for files based on their mime-type.

Note that confidence sensitivity does not apply to file names. Sensitive findings will always be reported on.

Detecting File Types

Nightfall’s File Type detection allows you to implement compliance policies that detect and alert you when particular file types that are not allowed in a given location are discovered.

This functionality is implemented by creating a specific Detector called a “File Type Detector”

To create a File Type Detector, select “Detectors” from the left hand navigation and click the button labeled “+New Detector” in the upper right hand corner. From there a drop down list of Detector types will be displayed which will include the “File Type” Detector type.

You will then select one or more file types for which to scan by selecting from a list of mime-types

You can either scroll through the list of mime-types in the select box or you may type in a portion of the mime-type and the contents of the select box will be filtered to match your input.

Nightfall supports detection for a wide variety of mime-types. See the Internet Assigned Numbers Authority’s (IANA) website for a definitive list of mime-types. Note however that Nightfall does not support the detection of audio and video related mime-types.

Detection of file types is done based on the file contents, not its extension. However, you can create Detectors that scan file names by setting the scope attribute.

File Type Detectors vary from other Nightfall Detectors in that the attributes of scope and confidence are not relevant to File Type Detectors

Once you have added all the mime-types you wish to scan for, save your new Detector. You may then add your new Detector to Detection Rules and Policies.

Detecting Files Through Fingerprinting

Nightfall allows you to discover the location of specific files that you have deemed sensitive and want to avoid sharing.

This discovery is done through document fingerprinting. Fingerprinting is the process of algorithmically creating a unique identifier for a file by mapping the data of the document to a signature that can be recalled quickly. This allows the file to be identified in a manner akin to how human fingerprints uniquely identify individual people.

This functionality is achieved in Nightfall by creating a specific Detector type called a File Fingerprint Detector.

The Fingerprint Detector allows you to create a fingerprint for one more files (a sort “handful” of fingerprints, if you would).

To create a Fingerprint Detector, select “Detectors” from the left hand navigation and click the button labeled “+New Detector” in the upper right hand corner. From there a drop down list of Detector types will be displayed which will include the “Fingerprint” Detector type.

When you create a File Fingerprint Detector you can upload up to 50 files that need to be fingerprinted. The file size limit is 25MB.

Once the fingerprint is generated, the actual content of the file is discarded so no sensitive content is stored on Nightfall’s system.

These Detectors may only be created through the console.

You may then treat the Fingerprint detector like any other Detector and incorporate it into a Detection Rule using its unique Detector identifier.

You may incorporate these Detectors into Policies that will alert you whenever files that match the fingerprint are detected.

APIs to monitor and manager integrations

You can use the posture management APIs to search posture events, fetch posture events and also event details. Additionally, you can also view details of the user (actor) whose actions triggered an event, and details of the asset that triggered an event.

Firewall for AI DLP APIs enables developers to write custom code to sanitize data anywhere–RAG data sets, analytics data stores, data pipelines, and unsupported SaaS applications.

Nightfall has the ability to send alerts when a violation is detected.

Policies for alerting may be configured through the Nightfall app user interface or they may be set up . Policies that are configured under Developer Platform > Overview > Policies may be used in the API by referencing their Policy UUID.

The way that an alert notification presents itself depends on the platform in question.

For example, notifications sent to Slack will appear as formatted messages sent by the Nightfall Alerts Bot. Other destinations such as email, SIEM url, and webhooks, will present the information as JSON objects.

In the case of webhooks, detailed information about the finding will be sent. For other destinations, sensitive information is redacted.

Supported Alert Platforms

Slack

In order to use asynchronous notifications with Slack, you must install the Nightfall Alerts plugin from the Slack Marketplace.

See our end user documentation on installing for more details.

Once you have authenticated Nightfall to your Slack workspace, you can provide any public channel name (e.g. #general) as part of a request to the Nightfall API.

To send notifications to a private channel, a member of the channel should invite the Nightfall bot to the specific private channel and allow channel access to the bot.

Follow the steps below to invite Nightfall Alerts bot to a private channel:

1. Go to the Slack channel in question

2. Type /invite @Nightfall Alerts as a message

3. Press 'Enter' (you should see a message that Nightfall Alerts has now joined the channel)

If any findings are detected as part of that request, then the Nightfall Alerts bot will send a message to the channel you configured. Conversely, if there are no findings in the request payload, then Nightfall will not send an alert message.

Teams

Documentation TBD

Email

Email is unauthenticated, so you can get started using Nightfall to send email alerts without any initial setup work.

Nightfall will send an email to the provided address only if findings were detected as part of the request. The findings themselves will be attached in a JSON file.

SIEM

You may send your alerts to a designated url, such as an endpoint hosted by SIEM software for log collection.

In addition to the url, you may provide headers, either for security or logging purposes.

See in our end user guide or our for more details.

Webhook

You may use a webhook server to programmatically handle a finding, allowing you to create your own custom workflows with your own or 3rd party systems.

Nightfall will always send an alert to the client's webhook server if it is provided as part of an API request, even if the scan request yielded no findings.

See for more details.

Alert Schemas

The request body sent by Nightfall is JSON, and uses the schemas in the section documented below.

File Scans

Since file scans can produce a large number of results, findings are not transmitted directly in the notification that Nightfall sends. The notification object looks like the following:

The requestMetadata field contains arbitrary contents provided by the client at request time, and can be used by the client to correlate this response to the original request.

The value of the findingsURL field is a pre-signed URL, which means anyone with the link can download the file. Therefore, this URL itself should be treated as sensitive and must not be leaked. The object stored at this URL is a JSON file containing a single key findings containing a list of all data detected from the request. The schema for the finding object inside the list is shared between the text-based and file-based API endpoints.

Text Scans

The payload that is forwarded on behalf of text scanning requests is identical to the response body that is synchronously returned to the client. Refer to the for more details on this payload.

Learn how to set up a server to handle results of file scans and alerts sent based on policy alert configurations.

Webhook Challenges

Nightfall will send a POST request with a JSON payload with a single field challenge containing randomly-generated bytes when it sends a message to a user-provided webhook address. This is to ensure that the caller owns the server.

In order to authenticate your webhook server to Nightfall, you must reply with (1) a 200 HTTP Status Code, and (2) a plaintext request body containing only the value of the challenge key.

If Nightfall receives the expected value back, then the file scan operation will proceed; otherwise it will be aborted.

When a server responds successfully to a challenge request, the validity of that URL will be cached for up to 24 hours, after which it will need to be validated again.

If the webhook cannot be reached, you will receive an error with the code "40012" and the description "Webhook URL validation failed" when you initiate the scan.

If the webhook challenge fails, you will receive an error with the code "42201" and the description "Webhook returned incorrect challenge response" when you initiate the scan.

Webhook Signature Verification

When a customer signs up for the developer platform, Nightfall automatically generates a unique for them.

This secret is used to sign requests to the customer's configured webhook URL.

If you has any concerns that their signing secret may have leaked, you can request rotation at any time by reaching out to Nightfall Customer Success.

For security purposes, the webhook includes a signature header containing an HMAC-SHA256 digital signature that customers may use to authenticate the client.

In order to authenticate requests to the webhook URL, customers may use the following algorithm:

1. Check for the presence of the headers X-Nightfall-Signature and X-Nightfall-Timestamp. If these headers are not both present, discard the request.

2. Read the entire request body into a string body.

3. Verify that the value in the X-Nightfall-Timestamp header (the POSIX time in seconds) occurred recently. This is to protect against replay attacks, so a threshold on the order of magnitude of minutes should be reasonable. If a request occurred too far in the past, it should be discarded.

4. Concatenate the timestamp and body with a colon delimiter, i.e. timestamp:body.

5. Compute the HMAC SHA-256 hash of the payload from the previous step, using your unique signing secret as the key. Encode this computed value in hex.

6. Compare the value of the X-Nightfall-Signature header to the value computed in the previous step. If the values match, authentication is successful, and processing should proceed. Otherwise, the request must be discarded.

The snippet below shows how you might implement this authentication validation in Python:

Example Webhook Server

An example implementation of a simple webhook server is below.

You can test your webhook with a tool such as which allows you expose a web server running on your local machine to the internet.

In the above example, the webhook server is running on port 8075. To route ngrok requests to this server, once you run the python script (having installed the necessary dependencies such getenv and Flask), you would run ngrok as follow:

./ngrok http 8075

See the section on for details about the json payloads for the different messages sent to webhook servers.

Spreadsheets and Tabular Data

File scans of Microsoft Office, Apache parquet, csv, and tab separated files will provide additional properties to locate findings within the document beyond the standard byteRange, codepointRange, and lineRange properties.

Findings will contain a columnRange and a rowRange that will allow you to identify the specific row and column within the tabular data wherein the finding is present.

This functionality is applicable to the following mime types:

• text/csv

• text/tab-separated-values

• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

• application/vnd.ms-excel

data files are also accepted.

Below is a sample match of a spreadsheet containing dummy PII where a SSN was detected in the 2nd column and 55th row.

Git Repositories

Nightfall provides special handling for archives of GitHub repositories.

Nightfall will scan the repository history to discover findings in particular checkin, returning the hash for the checkin.

In order to scan the repository, you will need to create a clone, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

This creates a clone of the Nightfall go SDK.

You will then need to create an archive that can be uploaded using Nightfall's file scanning sequence.

zip -r directory.zip directory

Note that in order to work, the hidden directory .github must be included in the archive.

When you initiate the with this file, you will receive scan results that contain the commitHash property filled in.

Using the Nightfall go SDK archive created above, a simple example would be to scan for URLs (i.e. strings starting with http:// or https://), which will send results such as the following:

To retrieve the specific checkout, you will need to clone the repository, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

You can then checkout the specific commit using the commit hash returned by Nightfall.

Note that you are in a when working with this sort of check out of a repository.

You can use the surrounding context of a match to help determine how likely it is that your potential match should actually be considered as a match by adjusting its confidence rating.

You can also tell the Detection Rule to return a portion of the surrounding context for manual review.

In the following example, in addition to providing a regular expression to match Social Security Numbers, we also look to see if someone has written the text “SSN” before and after the match, which might be a label indicating it is indeed a social security number. In which case, we change our confidence score to “VERY_LIKELY.” We then provide two possible matches in our payload, the first of which contains the string “SSN”.

In the results, you can see the confidence for the first finding in the payload has been set to VERY_LIKELY while the second item is only LIKELY.

While using Nightfall's Scan API, you may encounter some of the common errors outlined below. Try following the provided troubleshooting steps.

If problems persist, please for further assistance.

HTTP Error Codes

The following error codes are returned as part of a standard HTTP response.

The Nightfall Developer Playground () is a sample app that you may use to test out API functionality before writing any code.

Our playground environment allows you to:

• Test Detectors and Detection Rules. Here are some .

• Generate sample data for DLP testing.

• Explore a sample app built on our APIs

We offer a free tier that allows you to sign up and start using Firewall for AI with zero upfront costs or commitments. This tier provides a generous data scanning capacity and access to all the core features.

We offer enterprise pricing plans for advanced requirements such as higher data volumes, custom rate limits, and dedicated support.

Contact our team at [email protected] or via the contact form on our website to discuss your specific needs and get a tailored pricing quote.

You can use the exfiltration APIs to search exfiltration events, fetch exfiltration events and also event details. Additionally, you can also view details of the user (actor) whose actions triggered an event, and details of the asset that triggered an event.

Don't hesitate to get in touch with us directly via email at or through the c on our website.

We host on Wednesdays at 12 pm PT to help answer questions, talk through any ideas, and chat about data security. We would love to see you there!

Leverage our software development kits (SDKs) to enable easier, faster, and more stable engagement with the Nightfall APIs. Nightfall has a growing library of language specific SDKs including for:

If there is a language-specific SDK that you would find valuable but is not here, please don't hesitate to reach out to .

Yes, you can test out the detection engine, including 70+ pre-built detectors without writing any code or having to sign up in our .

The Document will guide you in making your first API request.

This page will get you up and running with the Nightfall API so you can start scanning for sensitive data.

Obtain an API key

The Nightfall API requires a valid API key to authenticate your API requests.

You can create API keys in the Dashboard.

Learn more about Authentication and Security.

Make an API Scan Request

Below is an example request to the scan endpoint.

To run this example yourself, replace the API key (NF-rEpLaCe...) with the one you created in the dashboard or set it as the environment variable NIGHTFALL_API_KEY as necessary.

The cURL example may be run from the command line without any additional installation. To run the Python example, you will need to download the corresponding SDK.

curl --request POST \
     --url https://api.nightfall.ai/v3/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer  NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                         {
                              "minNumFindings": 1,
                              "minConfidence": "VERY_LIKELY",
                              "displayName": "US Social Security Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER"
                         },
                         {
                              "redactionConfig": {
                                   "maskConfig": {
                                        "charsToIgnore": [
                                             "-"
                                        ],
                                        "maskingChar": "X",
                                        "maskRightToLeft":true,
                                        "numCharsToLeaveUnMasked":4
                                   }
                              },
                              "minNumFindings": 1,
                              "minConfidence": "VERY_LIKELY",
                              "displayName": "Credit Card Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "CREDIT_CARD_NUMBER"
                         }
                    ],
                    "name": "My Match Rule",
                    "logicalOp": "ANY"
               }
          ]
     },
     "payload": [
          "The customer social security number is 458-02-6124",
          "No PII in this string",
          "My credit card number is 5310-2768-6832-9293"
     ]
}
'

// By default, the client reads your API key from the environment variable NIGHTFALL_API_KEY
const nfClient = new Nightfall();

const payload = [
          "The customer social security number is 458-02-6124",
          "No PII in this string",
          "My credit card number is 5310-2768-6832-9293"
     ];
     

const policy = {
		 "detectionRules": [
               {
                    "detectors": [
                         {
                              "minNumFindings": 1,
                              "minConfidence": "LIKELY",
                              "displayName": "US Social Security Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER"
                         },
                         {
                              "redactionConfig": {
                                   "maskConfig": {
                                        "charsToIgnore": [
                                             "-"
                                        ],
                                        "maskingChar": "#"
                                   }
                              },
                              "minNumFindings": 1,
                              "minConfidence": "LIKELY",
                              "displayName": "Credit Card Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "CREDIT_CARD_NUMBER"
                         }
                    ],
                    "name": "My Match Rule",
                    "logicalOp": "ANY"
               }
          ]
     };
     
const response = await nfClient.scanText(payload, policy);

if (response.isError) {
  console.log(response.getError());
} else {
  response.data.findings.forEach((finding) => {
    if (finding.length > 0) {
      finding.forEach((result) => {
        console.log(`Finding: ${result.finding}, Confidence: ${result.confidence}`);
      });
    }
  });
}// Some code

>>> from nightfall import Confidence, DetectionRule, Detector, Nightfall

>>> # By default, the client reads the API key from the environment variable NIGHTFALL_API_KEY
>>> nightfall = Nightfall()

>>> # A rule contains a set of detectors to scan with
>>> cc = Detector(min_confidence=Confidence.LIKELY, nightfall_detector="CREDIT_CARD_NUMBER")
>>> ssn = Detector(min_confidence=Confidence.POSSIBLE, nightfall_detector="US_SOCIAL_SECURITY_NUMBER")
>>> detection_rule = DetectionRule([cc, ssn])
>>> payload = ["hello world", "my SSN is 678-99-8212", "4242-4242-4242-4242"]
>>> findings, _ = nightfall.scan_text( payload, detection_rules=[detection_rule])

The Policy (policy) you define indicates what to scan for in your payload with a logical grouped (ANY or ALL) set of Detection Rules (detectionRules).

Detection Rules can be defined two ways:

• inline as code, as shown above

• in the Nightall app, which you will then reference by UUID.

Learn more about setting up Nightfall in the Nightfall app to create your own Detectors, Detection Rules, and Policies. See Using Pre-Configured Detection Rules for an example as to how to execute queries using an existing Detection Rules UUID.

In the example above, two of Nightfall's native Detectors are being used: US_SOCIAL_SECURITY_NUMBER and CREDIT_CARD_NUMBER.

You can find a full list of native Detectors in the Detector Glossary.

If you don't want to create your Detectors, Detection Rules, and Policies in the Nightfall app, but would prefer to do it in code, it is possible to define Detectors inline with your own regular expressions or word list as well as extend our native Detectors with exclusion and context rules.

When defining a Detection Rule, you configure the minimum confidence level (minConfidence) and minimum number of times the match must be found (minNumFindings) for the rule to be triggered.

Another feature Nightfall offers is the ability to redact sensitive findings. Detectors may be configured (via redactionConfig) to replace the text that triggered them with a variety of customizable masks, including an encrypted version of the text.

In the payload body, you can see that we are submitting a list of three different strings to scan (payload). The first will trigger the U.S. Social Security Detector. The last will trigger the credit card Detector. The middle example will trigger neither.

Example Nightfall API Scan Response

The Nightfall API returns a response with an array (findings) with a length that corresponds to the length of the payload array. In this example, only the first and last items in the request payload triggered the Detectors, so the second element of the array is empty.

In the first element of the array, you can see details about which Detection Rule was triggered and the data that was found (finding). The response also provides a confidence level (confidence), as well as the location within the original text where the data was found either in terms of bytes (byteRange) or characters (codepointRange).

{
  "findings": [
    [
      {
        "finding": "458-02-6124",
        "redactedFinding": "XXX-XXXX-XXXX-9293",
        "detector": {
          "name": "US Social Security Number",
          "uuid": "e30d9a87-f6c7-46b9-a8f4-16547901e069"
        },
        "confidence": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "start": 39,
            "end": 50
          },
          "codepointRange": {
            "start": 39,
            "end": 50
          },
          "rowRange": null,
          "columnRange": null,
          "commitHash": ""
        },
        "matchedDetectionRuleUUIDs": [],
        "matchedDetectionRules": [
          "My Match Rule"
        ]
      }
    ],
    [],
    [
      {
        "finding": "5310-2768-6832-9293",
        "redactedFinding": "XXXX-XXXX-XXXX-9293",
        "detector": {
          "name": "Credit Card Number",
          "uuid": "74c1815e-c0c3-4df5-8b1e-6cf98864a454"
        },
        "confidence": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "start": 25,
            "end": 44
          },
          "codepointRange": {
            "start": 25,
            "end": 44
          },
          "rowRange": null,
          "columnRange": null,
          "commitHash": ""
        },
        "redactedLocation": {
          "byteRange": {
            "start": 25,
            "end": 44
          },
          "codepointRange": {
            "start": 25,
            "end": 44
          },
          "rowRange": null,
          "columnRange": null,
          "commitHash": ""
        },
        "matchedDetectionRuleUUIDs": [],
        "matchedDetectionRules": [
          "My Match Rule"
        ]
      }
    ]
  ],
  "redactedPayload": [
    "",
    "",
    "My credit card number is XXXX-XXXX-XXXX-9293"
  ]
}

Congratulations! You have successfully completed the Nightfall Quickstart.

You can modify the Detectors or payload in the example request to get more practice with the Nightfall API.

Leaked secrets, such as credentials needed to authenticate and authorize a cloud provider’s API request, expose company software, services, infrastructure, and data to hackers.

Nightfall has developed technology to detect secrets and label findings to speed SecOPs workflows from being clogged and eliminate false positive alerts.

Overall Coverage

Nightfall uses machine learning models trained on a large (millions of lines of code) diverse dataset (including all programming languages and application types) to ensure best-in-class secret detection accuracy and coverage.

Explicit Labeling and Endpoint Validation for Popular Services

For a growing set of the most popular services, Nightfall will:

• label detected secrets by vendor and service type (returned the kind field of the response)

• label detected secrets as active risks by validating supported credential types with their associated service endpoints (returned as the status of the service)

Our current solution supports the following vendors covering a diverse set of use cases, including cloud storage/infrastructure, communication, social networks, software development, banking, observability, and payment processing.

This list is not static and will continue to grow as we add support for detecting API keys from additional services. If you want to detect API keys from a service not listed below, please contact us.

Key Detection Example

Below is an example of how an AWS Key would be shown in a finding.

{
  "finding": "zImaKNJJ8u/seIbm1UszokVz3SSARukJs6cghEBXD",
  "detector": {
    "name": "API key",
    "uuid": "0e95732f-bc5c-448f-9d15-bd1417177360"
  },
  "confidence": "VERY_LIKELY",
  ...
  "findingMetadata": {
    "apiKeyMetadata": {
      "status": "ACTIVE",
      "kind": "AWS",
      "description": "Access Key ID: AKIA52FSMBPZS1JIDTPX"
    }
  }
}

The following values are returned for the status field:

• ACTIVE

• EXPIRED

• UNVERIFIED

This value will be based on what information is returned by the corresponding service when attempting the validate the key. If no data is returned fro the service, it will be considered UNVERIFIED.

To use this functionality, you use our existing built-in API_KEY detector to scan a data source such as Git Repository. Below is an example using a detection rule defined in line for a text scan.

curl --request POST \
     --url https://api.nightfall.ai/v3/scan \
     --header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '{
       "policy": {
            "detectionRules": [
                 {
                      "detectors": [
                           {
                                "detectorType": "NIGHTFALL_DETECTOR",
                                "nightfallDetector": "API_KEY",
                                "minNumFindings": 1,
                                "minConfidence": "LIKELY",
                                "displayName": "API Key"
                           }
                      ],
                      "name": "My Match Rule",
                      "logicalOp": "ANY"
                 }
            ]
       },
       "payload": [
            "Is this an active nightfall key? NF-OZ6F9fzF2z5mRxMrUdfL8FddFS51kPzE"
       ]
     }'

To prevent misuse and ensure the stability of our platform, we enforce a rate limit on an API Key and endpoint basis, similar to the way many other APIs enforce rate limits.

When operating under our Free plan, accounts and their corresponding API Keys have a rate limit of 5 requests per second on average, with support for bursts of 15 requests per second. If you upgrade to a paid plan – the Enterprise plan – this rate increases to a limit of 10 requests per second on average and bursts of 50 requests per second.

The Nightfall API follows standard practices and conventions to signal when these rate limits have been exceeded.

Successful requests return a header X-Rate-Limit-Remaining with the integer number of requests remaining before errors will be returned to the client.

When your application exceeds the rate limit for a given API endpoint, the Nightfall API will return an HTTP response code of 429 "Too Many Requests.” If your use case requires increased rate limiting, please reach out to [email protected].

Additionally, these unsuccessful requests return the number of seconds to wait before retrying the request in a Retry-After Header.

Request Rate Limiting

Your Request Rate Limiting throttles how frequently you can make requests to the API. You can monitor your rate limit usage via the `X-Rate-Limit-Remaining` header, which tells you how many remaining requests you can make within the next second before being throttled.

Quotas

Your Quota limits how many requests you can make within a given period. Your current remaining quota and the end of your current quota period are denoted by the following response headers.

For the free plan, we allow 5 requests per second and 10000 requests in a day.

In two ways:

• Nightfall’s out of the box detectors can be modified with context rules and exclusion rules.

• Nightfall also supports inputting custom regular expressions or word lists (i.e. dictionaries) as detectors in the RE2 standard as documented here.

Firewall for AI provides a flexible and extensible API that allows you to scan a wide variety of data types, including plain text, structured and unstructured files, and even images. Our API can handle data in various formats such as JSON, XML, CSV, and more. Visit our detector glossary at docs.nightfall.ai/docs/detector-glossary to explore the comprehensive list of supported data types and file formats

Firewall for AI is a powerful API that acts as a middleware layer or client wrapper to protect your AI models from consuming sensitive data. By integrating Firewall for AI into your application via API calls, you can proactively prevent data leaks and maintain compliance without disrupting your existing workflows or model updates.

To prevent misuse and ensure the stability of our platform, we enforce a rate limit on an API Key and endpoint basis, similar to the way many other APIs enforce rate limits.

When operating under our Free plan, accounts and their corresponding API Keys have a rate limit of 5 requests per second on average, with support for bursts of 15 requests per second. If you upgrade to a paid plan – the Enterprise plan – this rate increases to a limit of 10 requests per second on average and bursts of 50 requests per second.

The Nightfall API follows standard practices and conventions to signal when these rate limits have been exceeded.

Successful requests return a header X-Rate-Limit-Remaining with the integer number of requests remaining before errors will be returned to the client.

When your application exceeds the rate limit for a given API endpoint, the Nightfall API will return an HTTP response code of 429 "Too Many Requests.” If your use case requires increased rate limiting, please reach out to [email protected].

Additionally, these unsuccessful requests return the number of seconds to wait before retrying the request in a Retry-After Header.

Request Rate Limiting

Your Request Rate Limiting throttles how frequently you can make requests to the API. You can monitor your rate limit usage via the `X-Rate-Limit-Remaining` header, which tells you how many remaining requests you can make within the next second before being throttled.

Quotas

Your Quota limits how many bytes of data you're permitted to scan within a given period. Your current remaining quota and the end of your current quota period are denoted by the following response headers.

The native SaaS app APIs can be utilized by customers using Nightfall’s SaaS apps, supported natively, to fetch violations, search violations by app meta-data attributes, and fetch findings within violations. These DLP APIs do not provide access to violations for apps scanned via the developer platform. These APIs require you to create an API key as outlined in the Getting Started with the Developer Platform section. However, to use these APIs, you need not create any detectors, detection rules, and policies in the developer platform.

If you are using Nightfall SaaS apps, you can use APIs to fetch violations, search through the violations, and fetch specific findings within the Violations. To scan data in any custom apps or cloud infrastructure services like AWS S3, you must use the APIs in the DLP APIs - Firewall for AI Platform section.

Policies allow customers to create templates for their most common workflows such as sending alerts when detection rules are triggered.

These policies may be through the dashboard or may be defined programmatically.

When defining an a Policy inline, in addition to specifying the Detection Rules (either by referencing the UUID of an existing Detection Rule or), you must define an alertConfig which will determine where findings are sent.

The alertConfig can be either:

• an email address

• a Slack channel

• a webhook url

• a url to a SIEM host as well authentication and other headers

Below is a simple example of a payload with a policy that will send alerts to an email address that you would use with our endpoint for .

You will receive the following response:

Note that you may also use a pre-defined policy defined under Developer Platform > Overview > Policies by copying the Policy UUID and sending a request as shown below.

The following payload will be sent to the given email address with the subject "🚨 Findings Detected by Nightfall! 🚨" as an attachment with the name nightfall-findings.json:

This attachment has the same content as the response payload to the initial request.

Note that the sender address will be [email protected]

This email address will not respond to messages sent to it.

Using Webhooks with Policies

Policies also allow you to send findings to a callback designated URL using the url property of the alertConfig object.

This mechanism allows you to programmatically consume findings and the data sent will contain sensitive information as well as additional metadata like the location of the findings in the payload. For this reason the URL must be an HTTPS URL and the service backing it be implemented to properly respond with your and act as a

Below is what Webhook URL should like in your policy's alertConfig in a payload sent to our endpoint used for scanning plain text.

Using Slack Channels With Policies

Another option supported by Policies is sending finding data to a designated Slack channel.

This feature requires that you have configured the .

Below is a sample payload for scanning plain text.

Below is an example as to how the violation will appear in Slack.

See the section on Slack in the overview on for more details.

Sending Alerts to SIEMs and other HTTP Event Collectors

SIEM (pronounced “sim”) is a combination of security information management (SIM) and security event management systems. SIEM technology collects event log data for analysis in order to provide visibility into network activity.

It is possible to send findings from a policy to a SIEM service such as LogRhythm, SumoLogic, or Splunk using the siem alertConfig.

This configuration will require a URL to a collector that uses an HTTPS endpoint.

Note that the URL for the siem alertConfig must:

• use the HTTPS scheme

• be able to accept requests made with the POST verb

• respond with a 200 status code upon receipt of the event

See the documentation for your SIEM service for how to set up this URL.

Unlike the url alertConfig option, the siem alertConfig does not require that the endpoint for the service implement a custom challenge response. Events sent to the siem alertConfig endpoint contain a subset of what is sent to the url alertConfig. Furthermore the findings are sent in a redacted form similar to Slack or email alerts.

In addition to the URL, you may provide headers such as those that are used for authorization.

The headers in the SIEM alertConfig are divided into sensitiveHeaders and plainTextHeaders header mappings.

The sensitiveHeaders field is specifically for header values like authentication. Nightfall ensures that these header values are always hidden in our service. They are never logged or saved in analytic events.

You can use plainTextHeaders for all other type of information you would like passed along with Nightfall alerts to you HTTP endpoint. Nightfall assumes that the values stored plainTextHeaders do not contain any sensitive information so we do not take any action to hide or protect these values.

Below is an example of a payload using a siem alertConfig.

Other Policy Features

Using Redaction Within a Policy

A policy may be configured with default redaction rules as a defaultRedactionConfig that will affect the content of the redactedPayload field of the content that is sent to the alert locations specified in the policy alertConfig. Note that this redaction does not affect the findings themselves.

These redaction rules will be applied to Detection Rules that do not have a specified redaction configuration.

The redactionConfig specified must be one and only one of the four available redaction types:

• maskConfig

• infoTypeSubstitutionConfig

• substitutionConfig

• cryptoConfig

For more information on Redactions see:

Below is a simple example of a payload for using a policy set up to use a defaultRedactionConfig

Using Context Bytes Within a Policy

In additional to a defaultRedactionConfig it is possible to set the number of bytes to include as before and after a given finding as the contextBytes. This context can provide meaning to how the finding appears within the text to allow human readers to better understand the meaning of the finding. The maximum value for contextBytes is 40.

Schedule a Demo

You can schedule a demo or a meeting with our sales/solutions engineering team directly via Calendly . If you don't see a suitable time, please email us at .

Email Us

For support inquiries, please email us at .

For sales inquiries, please email us at .

In this example, we'll walk through making a request to the scan endpoint.

The endpoint inspects the data you provide via the request body and reports any detected occurrences of the sensitive data types you are searching for.

Please refer to the API reference of the scan endpoint for more detailed information on the request and response schemas.

In this sample request, we provide two main fields:

1. a policy and its detection rules that we want to use when scanning the text payload

2. a list of text strings to scan

In the example below we will use a Detection Rule that has been configured in the user interface by supplying its UUID.

The aggregate length of all strings in payload list must not exceed 500 KB, and the number of items in the payload may not exceed 50,000.

curl --request POST \
     --url https://api.nightfall.ai/v3/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRuleUUIDs": [
               "950833c9-8608-4c66-8a3a-0734eac11157"
          ]
     },
     "payload": [
          "4916-6734-7572-5015 is my credit card number",
          "This string does not have any sensitive data",
          "my api key is yr+ZWwIZp6ifFgaHV8410b2BxbRt5QiAj1EZx1qj and my 💳 credit card number 💰 is 30204861594838"
     ]
}
'

Alternatively you may define your policy in code by using a built in Nightfall detector from the Detector Glossary as follows:

curl --request POST \
     --url https://api.nightfall.ai/v3/scan \
     --header 'accept: application/json' \
     --header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'content-type: application/json' \
     --data '
{
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                         {
                              "nightfallDetector": "CREDIT_CARD_NUMBER",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "minConfidence": "POSSIBLE",
                              "minNumFindings": 1
                         }
                    ],
                    "logicalOp": "ALL"
               }
          ]
     },
     "payload": [
          "4916-6734-7572-5015 is my credit card number",
          "This string does not have any sensitive data",
          "my api key is yr+ZWwIZp6ifFgaHV8410b2BxbRt5QiAj1EZx1qj and my 💳 credit card number 💰 is 30204861594838"
     ]
}
'

See Creating an Inline Detection Rule for more information about how policies and detection rules may be defined through code.

Executing the curl request will yield a response as follows.

{
    "findings": [
        [
            {
                "finding": "4916-6734-7572-5015",
                "detector": {
                    "name": "Credit card number",
                    "uuid": "74c1815e-c0c3-4df5-8b1e-6cf98864a454"
                },
                "confidence": "VERY_LIKELY",
                "location": {
                    "byteRange": {
                        "start": 0,
                        "end": 19
                    },
                    "codepointRange": {
                        "start": 0,
                        "end": 19
                    }
                },
                "matchedDetectionRuleUUIDs": [
                    "950833c9-8608-4c66-8a3a-0734eac11157"
                ],
                "matchedDetectionRules": []
            }
        ],
        [],
        [
            {
                "finding": "30204861594838",
                "detector": {
                    "name": "Phone number",
                    "uuid": "d08edfc4-b5e2-420a-a5fe-3693fb6276c4"
                },
                "confidence": "LIKELY",
                "location": {
                    "byteRange": {
                        "start": 94,
                        "end": 108
                    },
                    "codepointRange": {
                        "start": 88,
                        "end": 102
                    }
                },
                "matchedDetectionRuleUUIDs": [
                    "950833c9-8608-4c66-8a3a-0734eac11157"
                ],
                "matchedDetectionRules": []
            },
            {
                "finding": "30204861594838",
                "detector": {
                    "name": "Credit card number",
                    "uuid": "74c1815e-c0c3-4df5-8b1e-6cf98864a454"
                },
                "confidence": "LIKELY",
                "location": {
                    "byteRange": {
                        "start": 94,
                        "end": 108
                    },
                    "codepointRange": {
                        "start": 88,
                        "end": 102
                    }
                },
                "matchedDetectionRuleUUIDs": [
                    "950833c9-8608-4c66-8a3a-0734eac11157"
                ],
                "matchedDetectionRules": []
            }
        ]
    ]
}
 
      "location": {
        "byteRange": {
          "start": 94,
          "end": 108
        },
        "codepointRange": {
          "start": 88,
          "end": 102
        }
      },
      "matchedDetectionRuleUUIDs": [
        "950833c9-8608-4c66-8a3a-0734eac11157"
      ],
      "matchedDetectionRules": []
    },
    {
      "finding": "30204861594838",
      "detector": {
        "name": "Credit card number",
        "uuid": "74c1815e-c0c3-4df5-8b1e-6cf98864a454"
      },
      "confidence": "LIKELY",
      "location": {
        "byteRange": {
          "start": 94,
          "end": 108
        },
        "codepointRange": {
          "start": 88,
          "end": 102
        }
      },
      "matchedDetectionRuleUUIDs": [
        "950833c9-8608-4c66-8a3a-0734eac11157"
      ],
      "matchedDetectionRules": []
    }
  ]
]

The API call returns a list, where the item at each index is a sublist of matches for the provided detector types.

The indices of the response list correspond directly to the indices of the list provided in the request payload.

In this example, the first item in the response list contains a finding because one credit card number was detected in the first string we provided. The second item in the response list is an empty list because there is no sensitive data in the second input string we provided. The third item in the returned list contains multiple findings as a result of multiple Detectors within the Detection Rule being triggered.

You can read further about the fields in the response object in the Nightfall APIs.

Nightfall's upload process is built to accommodate files of any size. Once files are uploaded, they may be scanned with Detection Rules and Policies to detect potential violations.

Many users will find it more convenient to use our our native language SDKs to complete the upload process.

Uploading files using Client SDK libraries requires fewer steps as all the required API operations are wrapped in a single function call. Furthermore these SDKs handle all the programmatic logic necessary to send files in smaller chunks to Nightfall.

For users that are looking to understand the entire upload process end-to-end, that is also outlined in this document. We will walk you through the order of operations necessary to upload the file.

Using Nightfall's SDKs to Upload Files

Rather than implementing the full sequence of API calls for the upload functionality yourself, the Nightfall’s native language SDKs provide a single method that wraps the steps required to upload your file.

Below is an example of uploading a file from our Python SDK and our Node SDK.

>>> from nightfall import Confidence, DetectionRule, Detector, Nightfall, EmailAlert, AlertConfig
>>> import os

>>> # use your API Key here
>>> nightfall = Nightfall("NF-y0uRaPiK3yG03sH3r3")

>>> # A rule contains a set of detectors to scan with
>>> cc = Detector(min_confidence=Confidence.LIKELY, nightfall_detector="CREDIT_CARD_NUMBER")
>>> ssn = Detector(min_confidence=Confidence.POSSIBLE, nightfall_detector="US_SOCIAL_SECURITY_NUMBER")
>>> detection_rule = DetectionRule([cc, ssn])
>>> # The scanning is done asynchronously, so provide a valid email address as the simplest way of getting results
>>> alertconfig = alert_config=AlertConfig(email=EmailAlert("[email protected]"))
    

>>> # Upload the file and start the scan.
>>> id, message = nightfall.scan_file( "./README.md", detection_rules=[detection_rule], alert_config=alertconfig)
>>> print("started scan", id, message)

//this script assumes the node sdk has been installed locally with `npm install` and `npm run build`
import { Nightfall } from "./nightfall-nodejs-sdk/dist/nightfall.js";
import { Detector } from "./nightfall-nodejs-sdk/dist/types/detectors.js";


// By default, the client reads your API key from the environment variable NIGHTFALL_API_KEY
const uploadit = async() => {
    var data = null;
    
    const nfClient = new Nightfall();
    	
    try{
   
		const response = await nfClient.scanFile('./README.md', {
		  detectionRules: [
			{
			  name: 'Secrets Scanner',
			  logicalOp: 'ANY',
			  detectors: [
				{
				  minNumFindings: 1,
				  minConfidence: Detector.Confidence.Possible,
				  displayName: 'Credit Card Number',
				  detectorType: Detector.Type.Nightfall,
				  nightfallDetector: 'CREDIT_CARD_NUMBER',
				},
			  ],
			},
		  ],
		  alertConfig: {
				email: {
						address: "[email protected]"
					}
		   }
		});

		if (response.isError) {
		  data = response.getError();
		}
		else{ 
			data = (response.data.id);
		}
	 
    }
	catch(e){
		console.log(e);
	}


	return data;

}

uploadit().then(data => console.log(data));

To run the node sample script you must compile it as TypesScript. Save it as a .ts file and run

tsc <yourfilename>.ts -lib ES2015,DOM

You can then run the resulting JavaScript file:

NIGHTFALL_API_KEY=<YourApiKey> node yourscriptname.js

Note that these examples use an email address to receive the results for simplicity.

You may also want to use a webhook. See Webhooks and Asynchronous Notifications for additional information on how to set up Webhook server to receive these results.

The Upload Process

The upload process consists of 3 stages:

• Initializing

• Uploading

• Completing

Once the upload is complete, you may initiate the file scan.

After we discuss each API call in the sequence, you will find a script that walks through the full sequence at the end of this guide.

Initializing Phase

POST /v3/upload

The first step in the process of scanning a binary file is to initiate an upload in order to get a fileId through the Initiate a File Upload endpoint.

As part of the initialization you must provide the total byte size of the file being uploaded.

You may also provide the mime-type, otherwise the system will attempt to determine it once the upload is complete.

curl --location --request POST 'https://api.nightfall.ai/v3/upload' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '{
    "fileSizeBytes": 73891,
    "mimeType" : "image/png"
}'

The id of the returned JSON object will be used as the fileId in subsequent requests.

The chunkSize is the maximum number of bytes to upload during the uploading phase.

{
    "id": "f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d",
    "fileSizeBytes": 73891,
    "chunkSize": 10485760,
    "mimeType": "image/png"
}

Uploading Phase

PATCH /v3/upload/<uploadUUID>

Use the Upload a Chunk of a File endpoint to upload the file contents in chunks.

The size of these chunks are determined by the chunkSize value returned by POST /upload endpoint used in the previous step.

Below is a simple example where the file is less than the chunkSize so may safely be uploaded with one call to the upload endpoint.

curl --location --request PATCH 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d' \
--header 'X-Upload-Offset: 0' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-binary '@/Users/myname/Documents/work/Nightfall/Nightfall Upload Sequence.png'

If your file's size exceeds the chunkSize, to upload the complete file you will need to send iterative requests as you read portions of the file's contents. This means you will send multiple requests to the upload endpoint as shown above. As you do so, you will be updating the value of the X-Upload-Offset header based on the portion of the file being sent.

Each request should send a chunk of the file exactly chunkSize bytes long except for the final uploaded chunk. The final uploaded chunk is allowed to contain fewer bytes as the remainder of the file may be less than the chunkSize returned by the initialization step.

The request body should be the contents of the chunk being uploaded.

The value of the X-UPLOAD-OFFSET header should be the byte offset specifying where to insert the data into the file as an integer. This byte offset is zero-indexed.

Successful calls to this endpoint return an empty response with an HTTP status code of 204

See the full example script below for an illustration as to how this upload process can be done programmatically.

Completion Phase

POST /v3/upload/<uploadUUID>/finish

Once all chunks are uploaded, mark the upload as completed using the Complete a File Upload endpoint.

curl --location --request POST 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/finish' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer  NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '""'

When an upload completes successfully, the returned payload will indicate the mimeType the system determined to file to be if it was not provided during upload initialization.

{
    "id": "152848af-2ac9-4e0a-8563-2b82343d964a",
    "fileSizeBytes": 2349,
    "chunkSize": 10485760,
    "mimeType": "application/zip"
}

Once a file has been marked as completed, you may initiate a scan of the uploaded file.

Scanning Uploaded Files

After an upload is finalized, it can be scanned against a Detection Policy. A Detection Policy represents a pairing of:

• a webhook URL

• a set of detection rules to scan data against

The scanning process is asynchronous, with results being delivered to the webhook URL configured on the detection policy. See Webhooks and Asynchronous Notifications for more information about creating a Webhook server.

Exactly one policy should be provided in the request body, which includes a webhookURL to which the callback will be made once the file scan has been completed (this must be an HTTPS URL) as well as a Detection Rule as either an a list of UUIDs or as a rule that has been defined in-line.

You may also supply a value to the requestMetadata field to help identify the input file upon receiving a response to your webhook. This field has a maximum length 10 KB.

curl --request POST \
     --url https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRuleUUIDs": [
               "950833c9-8608-4c66-8a3a-0734eac11157"
          ],
          "webhookURL": "https://mycompany.org/webhookservice"
     },
     "requestMetadata": "your file metadata"
}
'

Full Upload Process Example Script

Below is a sample Python script that handles the complete sequence of API calls to upload a file using a path specified as an argument.

from os import getenv, path

import fire
import requests


BASE_UPLOAD_URL = getenv("FILE_UPLOAD_HOST", "http://api.nightfall.ai/v3")
NF_API_KEY = getenv("NF_API_KEY")


def upload(filepath, mimetype, policy_uuid):
    """Upload the given file using the provided MIMEType and PolicyUUID.

    Arguments:
        file_path -- an absolute or relative path to the file that will be
            uploaded to the API.
        mimetype -- (optional) The mimetype of the file being uploaded.
        policy_uuid -- The UUID corresponding to an existing policy. This
            policy must be active and have a webhook URL associated with it.
    """
    default_headers = {
        "Authorization": F"Bearer {NF_API_KEY}",
    }

    # =*=*=*=*=* Initiate Upload =*=*=*=*=*=*
    file_size = path.getsize(filepath)
    upload_request_body = {"fileSizeBytes": file_size, "mimeType": mimetype}
    r = requests.post(F"{BASE_UPLOAD_URL}/upload",
                      headers=default_headers,
                      json=upload_request_body)
    upload = r.json()
    if not r.ok:
        raise Exception(F"Unexpected error initializing upload - {upload}")

    # =*=*=*=*=*=* Upload Chunks =*=*=*=*=*=*
    chunk_size = upload["chunkSize"]
    i = 0
    with open(filepath, "rb") as file:
        while file.tell() < file_size:
            upload_chunk_headers = {
                **default_headers,
                "X-UPLOAD-OFFSET": str(file.tell())
            }
            r = requests.patch(F"{BASE_UPLOAD_URL}/upload/{upload['id']}",
                               headers=upload_chunk_headers,
                               data=file.read(chunk_size))
            if not r.ok:
                raise Exception(F"Unexpected error uploading chunk - {r.text}")
            i += 1

    # =*=*=*=*=*=* Finish Upload =*=*=*=*=*=*
    r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/finish",
                      headers=default_headers)
    if not r.ok:
        raise Exception(F"Unexpected error finalizing upload - {r.text}")

    # =*=*=*=*=* Scan Uploaded File =*=*=*=*=*
    r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/scan",
                      json={"policyUUID": policy_uuid},
                      headers=default_headers)
    if not r.ok:
        raise Exception(F"Unexpected error initiating scan - {r.text}")

    print("Scan Initiated Successfully - await response on configured webhook")
    quota_remaining = r.headers.get('X-Quota-Remaining')
    if quota_remaining is not None and int(quota_remaining) <= 0:
        print(F"Scan quota exhausted - Quota will reset on {r.headers['X-Quota-Period-End']}")


if __name__ == "__main__":
    fire.Fire(upload)

The Firewall for AI Platform differs from other solutions like Google DLP and Amazon Macie, as well as open source solutions like truffleHog, on a number of dimensions summarised below.

Accuracy

• While solutions like Google DLP have a broad set of detectors, many of them are rules or regex based, which means many of the detectors are not usable in practice. Likewise, detection has been found to be inconsistent in some cases, perhaps due to internal A/B testing.

• Because of the limitations of regex-based rules, instead of leveraging machine learning based detectors, OSS detection solutions tend to have a much higher rate of false positives compared to Nightfall.

• Detector configurability and ability to provide metrics at the token level makes Nightfall accurate and actionable to engineering & security teams.

Convenience

• Want to leave the last 4 digits of a credit card number visible, securely encrypt emails, and completely remove SSNs from your data? The Nightfall platform allows you to redact/replace, substitute, and/or encrypt sensitive data findings in the same API call as your inspection request.

Ease of use

• All inspection configuration in Google DLP is done as code, which makes it challenging to easily update, visualize, and modify detection rules and configuration. Nightfall allows for configuration as code, as well as the Nightfall Dashboard for creating and updating detection rules, which makes it easier to collaborate.

• OSS secret detection tools tend to rely heavily on manual creation of regex-based detection compared to an ability to programmatically scan text and file inputs using 150+ detectors in Nightfall – e.g. truffleHog only enables you to scan for secrets like passwords and private keys whereas Nightfall scans for not only secrets and credentials, but also allows you to use our vast detector library to scan for PII, PCI, and PHI.

File parsing

• To parse files with Google DLP and Macie, each requires that they be in their respective cloud storage (Google Cloud Storage or S3, respectively). With the Nightfall Developer Platform, we take care of storage requirements for you. Uploaded assets are stored encrypted at rest with minimal access permissions, and are automatically deleted after 24 hours.

• Amazon’s file parsers are limited to around 20 file types. Most notably, Macie does not support images. Text extraction via machine-learning based OCR for images is a core component of Nightfall’s file scanning endpoint.

• Open source secrets detection solutions are limited in their detection capabilities. Namely, these projects do not support scanning binary files. Nightfall supports binary files and the ability to scan diff files.

Platform agnostic

• Each cloud provider's DLP products are geared towards protecting their own cloud services. For example, Google DLP’s native integrations are limited to Google Cloud offerings such as BigQuery. Similarly, Macie is primarily designed around scanning AWS S3 buckets. The interface is largely geared towards exploring sensitive data across S3 buckets. To scan content outside of S3, Amazon’s recommendation is to move or replicate the data into S3 to scan, which is impractical.

• OSS solutions are primarily designed around git repositories.

• Nightfall has native integrations with many cloud applications like Slack, Atlassian, GitHub, Google Drive, as well a broad set of tutorials and open source code so you can build integrations into any data silo with ease. For example, this includes services like Snowflake, Airtable, and more.

Support and documentation

• Google DLP and Macie are loosely supported products and with many cloud offerings, support is hard to come by. Nightfall is laser-focused on best-of-breed content inspection and we are ready to address your questions and use cases.

• Nightfall also has extensive documentation including SDKs for multiple languages including Python, Java, NodeJS, and Go - with more under consistent development.

Cost and scale

• Costs can balloon quickly with commercial services. They also have rate limits that don’t suit high data volumes.

• Open source solutions have high hidden costs in the form of TCO, maintenance, and opportunity cost.

• Nightfall offers a custom enterprise tier that can help you scale pricing based on your anticipated usage as well as custom rate limits.

Say you have a number of files containing customer or patient data and you are not sure which of them are ok to share in a less secure manner. By leveraging Nightfall’s API you can easily verify whether a file contains sensitive PII, PHI, or PCI.

To make a request to the Nightfall API you will need:

• A Nightfall API key

• A list of data types you wish to scan for

• Data to scan. Note that the API interprets data as plaintext, so you may pass it in any structured or unstructured format.

You can read more about or about our in the linked reference guides.

To run the following API call, we will be using Python's standard json, os, and requests libraries.

First we define the endpoint we want to reach with our API call.

Next we define the headers of our API request. In this example, we have our API key set via an environment variable called "NIGHTFALL_API_KEY". Your API key should never be hard-coded directly into your script.

Next we define the detectors with which we wish to scan our data. The detectors must be formatted as a list of key-value pairs of format {‘name’:’DETECTOR_NAME’}.

Next, we build the request body, which contains the detectors from above, as well as the raw data that you wish to scan. In this example, we will read it from a file called sample_data.csv.

Here we assume that the file is under the 500 KB payload limit of the Scan API. If your file is larger than the limit, consider breaking it down into smaller pieces across multiple API requests.

Now we are ready to call the Nightfall API to check if there is any sensitive data in our file. If there are no sensitive findings in our file, the response will be "[[]]".

This guide describes how to use Nightfall with the Python programming language.

The example below will demonstrate how to use Nightfall’s text scanning functionality to verify whether a string contains sensitive PII using the Nightfall Python SDK.

To request the Nightfall API you will need:

• A Nightfall API key

• An existing Nightfall Detection Rule

• Data to scan. Note that the API interprets data as plaintext, so you may pass it in any structured or unstructured format.

You can read more about obtaining a Nightfall API key or about our available data detectors in the linked reference guides.

In this tutorial, we will be downloading, setting up, and using the Python SDK provided by Nightfall.

We recommend you first set up a virtual environment. You can learn more about that here.

You can download the Nightfall SDK from PyPi like this:

We will be using the built-in os library to help run this sample API script. This will be used to help extract the API Key from the OS as an environment variable.

import os

from nightfall import Confidence, DetectionRule, Detector, LogicalOp, Nightfall

Next, we extract our API Key, and abstract a nightfall class from the SDK, for it. In this example, we have our API key set via an environment variable called NIGHTFALL_API_KEY. Your API key should never be hard-coded directly into your script.

nightfall = Nightfall(os.environ['NIGHTFALL_API_KEY'])

Next we define the Detection Rule with which we wish to scan our data. The Detection Rule can be pre-made in the Nightfall web app and referenced by UUID.

detection_rule_uuid = os.environ.get('DETECTION_RULE_UUID')

In this example, we will use some example data in the payload List.

🚧Payload Limit

Payloads must be under 500 KB when using the Scan API. If your file is larger than the limit, consider using the file api, which is also available via the Python SDK.

We will ignore the second parameter as we do not have redaction configured for this request.

With the Nightfall API, you can redact and mask your findings. You can add a Redaction Config, as part of your Detection Rule. For more information on how to use redaction, and its specific options, please refer to the guide here.

payload = [
    "The customer social security number is 458-02-6124",
    "No PII in this string",
    "My credit card number is 4916-6734-7572-5015"
]

result, _ = nightfall.scan_text(
        payload,
        detection_rule_uuids=[detection_rule_uuid]
    )

payload = [
    "The customer social security number is 458-02-6124",
    "No PII in this string",
    "My credit card number is 4916-6734-7572-5015"
]

result, _ = nightfall.scan_text(
    payload,
    detection_rules=[
        DetectionRule(
            name="Sample_Detection_Rule",
            logical_op=LogicalOp.ANY,
            detectors=[
                Detector(
                    min_confidence=Confidence.VERY_LIKELY,
                    min_num_findings=1,
                    display_name="Credit Card",
                    nightfall_detector="CREDIT_CARD_NUMBER",
                ),
                Detector(
                    min_confidence=Confidence.VERY_LIKELY,
                    min_num_findings=1,
                    display_name="Social",
                    nightfall_detector="US_SOCIAL_SECURITY_NUMBER",
                )
            ]
        )
    ]
)

Reviewing Results

Now we are ready to review the results from the Nightfall SDK to check if there is any sensitive data in our file. Since the results will be in a dataclass, we can use the built-in __repr__ functions to format the results in a user-friendly and readable manner.

All data and sample findings shown below are validated, non-sensitive, examples of sample data.

If there are no sensitive findings in our payload, the response will be as shown in the 'empty response' pane below:

[
    [Finding(finding='458-02-6124', redacted_finding=None, before_context=None, after_context=None, detector_name='US social security number (SSN)', detector_uuid='e30d9a87-f6c7-46b9-a8f4-16547901e069', confidence=<Confidence.VERY_LIKELY: 'VERY_LIKELY'>, byte_range=Range(start=39, end=50), codepoint_range=Range(start=39, end=50), matched_detection_rule_uuids=['c67e3dd7-560e-438f-8c72-6ec54979396f'], matched_detection_rules=[])],
    [],
    [Finding(finding='4916-6734-7572-5015', redacted_finding=None, before_context=None, after_context=None, detector_name='Credit card number', detector_uuid='74c1815e-c0c3-4df5-8b1e-6cf98864a454', confidence=<Confidence.VERY_LIKELY: 'VERY_LIKELY'>, byte_range=Range(start=25, end=44), codepoint_range=Range(start=25, end=44), matched_detection_rule_uuids=['c67e3dd7-560e-438f-8c72-6ec54979396f'], matched_detection_rules=[])]
]

[
    [Finding(finding='458-02-6124', redacted_finding=None, before_context=None, after_context=None, detector_name='Social', detector_uuid='e30d9a87-f6c7-46b9-a8f4-16547901e069', confidence=<Confidence.VERY_LIKELY: 'VERY_LIKELY'>, byte_range=Range(start=39, end=50), codepoint_range=Range(start=39, end=50), matched_detection_rule_uuids=[], matched_detection_rules=['Sample_Detection_Rule'])],
    [],
    [Finding(finding='4916-6734-7572-5015', redacted_finding=None, before_context=None, after_context=None, detector_name='Credit Card', detector_uuid='74c1815e-c0c3-4df5-8b1e-6cf98864a454', confidence=<Confidence.VERY_LIKELY: 'VERY_LIKELY'>, byte_range=Range(start=25, end=44), codepoint_range=Range(start=25, end=44), matched_detection_rule_uuids=[], matched_detection_rules=['Sample_Detection_Rule'])],
]

[[], [], []]

And that's it 🎉

You are now ready to use the Python SDK for other scenarios.

This guide describes how to use Nightfall with the Java programming language.

The example below will demonstrate how to use Nightfall’s text scanning functionality to verify whether a string contains sensitive PII using the Nightfall Java SDK.

In this tutorial, we will be downloading, setting up, and using the Java SDK provided by Nightfall.

To make a request to the Nightfall API you will need:

• A Nightfall API key

• Plaintext data to scan.

You can read more about obtaining API key or about our available data detectors from the linked reference guides.

You can add the Nightfall package to your project by adding a dependency to your pom.xml:

<!--pom.xml-->

<?xml version="1.0" encoding="UTF-8"?>
 <project xmlns="http://maven.apache.org/POM/4.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>

     <groupId>com.foo</groupId>
     <artifactId>my-artifact</artifactId>
     <version>1.0.0</version>

     <name>${project.groupId}:${project.artifactId}</name>
     <packaging>jar</packaging>

     <dependencies>
         <dependency>
             <groupId>ai.nightfall</groupId>
             <artifactId>scan-api</artifactId>
             <version>1.0.1</version>
         </dependency>
     </dependencies>
 </project>

First add the required imports to the top of the file.

These are the objects we will use from the Nightfall SDK, as well as some collection classes for data handling.

//List of imports

import ai.nightfall.scan.NightfallClient;
 import ai.nightfall.scan.model.Confidence;
 import ai.nightfall.scan.model.DetectionRule;
 import ai.nightfall.scan.model.Detector;
 import ai.nightfall.scan.model.LogicalOp;
 import ai.nightfall.scan.model.NightfallAPIException;
 import ai.nightfall.scan.model.ScanTextConfig;
 import ai.nightfall.scan.model.ScanTextRequest;
 import ai.nightfall.scan.model.ScanTextResponse;

 import java.util.Arrays;
 import java.util.List;

We can then declare some data to scan in a List:

//Sample Payload

List<String> payload = Arrays.asList(
  "hello", 
  "world", 
  "my data is 4242-4242-4242-4242 but shhhh 🙊 ", 
  "my ssn is 678-99-8212"
);

Create a ScanTextRequest to scan the payload with. First create a new instance of the credit card detector, and set to trigger if there are any findings that are confidence LIKELY or above.

Add a second detector, looking for social security numbers. Set it to be triggered if there is at least a possible finding.

Combine these detectors into a detection rule, which will return findings if either of these detectors are triggered.

Finally, combine the payload and configuration together as a new ScanTextRequest, and return it.

//Build the Scan Request

public static ScanTextRequest buildScanTextRequest() {
  	// Define some detectors to use to scan your data
  	Detector creditCard = new Detector("CREDIT_CARD_NUMBER");
  	creditCard.setMinConfidence(Confidence.LIKELY);
  	creditCard.setMinNumFindings(1);

    Detector ssn = new Detector("US_SOCIAL_SECURITY_NUMBER");
    ssn.setMinConfidence(Confidence.POSSIBLE);
    ssn.setMinNumFindings(1);
    DetectionRule rule = new DetectionRule(Arrays.asList(creditCard, ssn), LogicalOp.ANY);
    ScanTextConfig config = ScanTextConfig.fromDetectionRules(Arrays.asList(rule), 20);

    return new ScanTextRequest(payload, config);
}

Use the ScanTextRequest instance with a NightfallClient to send your request to Nightfall.

The resulting ScanTextResponse may be used to print out the results:

//Run the Scan Request

public class Runner {
     public static void main(String[] args) {
         try (NightfallClient c = NightfallClient.Builder.defaultClient()) {
             try {
                 ScanTextResponse response = c.scanText(buildScanTextRequest());
                 System.out.println("response: " + response.getFindings());
             } catch (NightfallAPIException e) {
                 // not a checked exception, just for illustrative purposes
                 System.out.println("got error: " + e);
             }
         }
     }
 }

And that's it 🎉

You are now ready to use the Java SDK for other scenarios.

The file scan API has first-class support for text extraction and scanning on all MIME types enumerated below.

Certain file types receive special handling, such as and , that results in more precise information about the location of findings within the source file.

Accepted Text and Derivatives

• application/json

• application/x-ndjson

• application/x-php

• text/calendar

• text/css

• text/csv (treated as and may be )

• text/html

• text/javascript

• text/plain

• text/tab-separated-values (treated as )

• text/tsv (treated as )

• text/x-php

Accepted Office Formats

• application/pdf

• application/vnd.openxmlformats-officedocument.presentationml.presentation

• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet (treated as )

• application/vnd.openxmlformats-officedocument.wordprocessingml.document

• application/vnd.ms-excel (treated as )

Accepted Archive and Compressed File Types

• application/bzip2

• application/ear

• application/gzip

• application/jar

• application/java-archive

• application/tar+gzip

• application/vnd.android.package-archive

• application/war

• application/x-bzip2

• application/x-gzip

• application/x-rar-compressed

• application/x-tar

• application/x-webarchive

• application/x-zip-compressed

• application/x-zip

• application/zip

Accepted Image File Types

• image/apng

• image/avif

• image/gif

• image/jpeg

• image/jpg

• image/png

• image/svg+xml

• image/tiff

• image/webp

Rejected MIME Types

The file scan API explicitly rejects requests with MIME types that are not conducive to extracting or scanning text. Sample rejected MIME types include:

• application/photoshop

• audio/midi

• audio/wav

• video/mp4

• video/quicktime

Spreadsheets and Tabular Data

File scans of Microsoft Office, Apache parquet, csv, and tab separated files will provide additional properties to locate findings within the document beyond the standard byteRange, codepointRange, and lineRange properties.

Findings will contain a columnRange and a rowRange that will allow you to identify the specific row and column within the tabular data wherein the finding is present.

This functionality is applicable to the following mime types:

• text/csv

• text/tab-separated-values

• text/tsv

• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

• application/vnd.ms-excel

data files are also accepted.

Below is a sample match of a spreadsheet containing dummy PII where a SSN was detected in the 2nd column and 55th row.

Redacting CSV Files

Findings within csv files may be redacted.

To enable redaction in files, set the enableFileRedaction flag of your policy to "true"

The csv file will be redacted based on the configuration of the defaultRedactionConfig of the policy

Below is an example curl request for a csv file that has already been .

When results are sent to the location specified in the alertConfig (in this case an email address) a redactedFile property will be set with a fileURL in addition the findingsURL

This redacted file will be a modified version of the original csv file.

Below is an example of a redacted csv file.

Git Repositories

Nightfall provides special handling for archives of Git repositories.

Nightfall will scan the repository history to discover findings in particular checkin, returning the hash for the checkin.

In order to scan the repository, you will need to create a clone, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

This creates a clone of the Nightfall go SDK.

You will then need to create an archive that can be uploaded using Nightfall's file scanning sequence.

zip -r directory.zip directory

Note that in order to work, the hidden directory .github must be included in the archive.

When you initiate the with this file, you will receive scan results that contain the commitHash property filled in.

Using the Nightfall go SDK archive created above, a simple example would be to scan for URLs (i.e. strings starting with http:// or https://), which will send results such as the following:

Large repositories result in a large volume of data sent at once. We are working on changes to allow these and other large surges of data to be processed in a more controlled manner, and will increase the limit or remove it altogether once those changes are complete.

To retrieve the specific checkout, you will need to clone the repository, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

You can then checkout the specific commit using the commit hash returned by Nightfall.

Note that you are in a when workin with this sort of check out of a repository.

File Scanning Limitations

• CSV Files: Only the first 250,000 rows will be scanned.

• Spreadsheet Files: Up to 100,000 rows per sheet will be scanned, with a maximum of 1 million rows across all tabs in multi-sheet spreadsheets.

• PDF Files: Scanning is limited to the first 100 pages, including a maximum of 50 images within those pages.

• Images: Images smaller than 5KB or larger than 50MB will be excluded from scanning.

• Archive Files: A maximum of 1,000 files will be extracted and scanned. Files larger than 100MB requiring extraction will not be scanned.