The Document will guide you in making your first API request.

This page will get you up and running with the Nightfall API so you can start scanning for sensitive data.

Obtain an API key

The Nightfall API requires a valid API key to authenticate your API requests.

Make an API Scan Request

Below is an example request to the scan endpoint.

To run this example yourself, replace the API key (NF-rEpLaCe...) with the one you created in the dashboard or set it as the environment variable NIGHTFALL_API_KEY as necessary.

curl --request POST \
     --url https://api.nightfall.ai/v3/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer  NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                         {
                              "minNumFindings": 1,
                              "minConfidence": "VERY_LIKELY",
                              "displayName": "US Social Security Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER"
                         },
                         {
                              "redactionConfig": {
                                   "maskConfig": {
                                        "charsToIgnore": [
                                             "-"
                                        ],
                                        "maskingChar": "X",
                                        "maskRightToLeft":true,
                                        "numCharsToLeaveUnMasked":4
                                   }
                              },
                              "minNumFindings": 1,
                              "minConfidence": "VERY_LIKELY",
                              "displayName": "Credit Card Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "CREDIT_CARD_NUMBER"
                         }
                    ],
                    "name": "My Match Rule",
                    "logicalOp": "ANY"
               }
          ]
     },
     "payload": [
          "The customer social security number is 458-02-6124",
          "No PII in this string",
          "My credit card number is 5310-2768-6832-9293"
     ]
}
'

// By default, the client reads your API key from the environment variable NIGHTFALL_API_KEY
const nfClient = new Nightfall();

const payload = [
          "The customer social security number is 458-02-6124",
          "No PII in this string",
          "My credit card number is 5310-2768-6832-9293"
     ];
     

const policy = {
		 "detectionRules": [
               {
                    "detectors": [
                         {
                              "minNumFindings": 1,
                              "minConfidence": "LIKELY",
                              "displayName": "US Social Security Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER"
                         },
                         {
                              "redactionConfig": {
                                   "maskConfig": {
                                        "charsToIgnore": [
                                             "-"
                                        ],
                                        "maskingChar": "#"
                                   }
                              },
                              "minNumFindings": 1,
                              "minConfidence": "LIKELY",
                              "displayName": "Credit Card Number",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "nightfallDetector": "CREDIT_CARD_NUMBER"
                         }
                    ],
                    "name": "My Match Rule",
                    "logicalOp": "ANY"
               }
          ]
     };
     
const response = await nfClient.scanText(payload, policy);

if (response.isError) {
  console.log(response.getError());
} else {
  response.data.findings.forEach((finding) => {
    if (finding.length > 0) {
      finding.forEach((result) => {
        console.log(`Finding: ${result.finding}, Confidence: ${result.confidence}`);
      });
    }
  });
}// Some code

>>> from nightfall import Confidence, DetectionRule, Detector, Nightfall

>>> # By default, the client reads the API key from the environment variable NIGHTFALL_API_KEY
>>> nightfall = Nightfall()

>>> # A rule contains a set of detectors to scan with
>>> cc = Detector(min_confidence=Confidence.LIKELY, nightfall_detector="CREDIT_CARD_NUMBER")
>>> ssn = Detector(min_confidence=Confidence.POSSIBLE, nightfall_detector="US_SOCIAL_SECURITY_NUMBER")
>>> detection_rule = DetectionRule([cc, ssn])
>>> payload = ["hello world", "my SSN is 678-99-8212", "4242-4242-4242-4242"]
>>> findings, _ = nightfall.scan_text( payload, detection_rules=[detection_rule])

The Policy (policy) you define indicates what to scan for in your payload with a logical grouped (ANY or ALL) set of Detection Rules (detectionRules).

Detection Rules can be defined two ways:

In the example above, two of Nightfall's native Detectors are being used: US_SOCIAL_SECURITY_NUMBER and CREDIT_CARD_NUMBER.

In the payload body, you can see that we are submitting a list of three different strings to scan (payload). The first will trigger the U.S. Social Security Detector. The last will trigger the credit card Detector. The middle example will trigger neither.

Example Nightfall API Scan Response

The Nightfall API returns a response with an array (findings) with a length that corresponds to the length of the payload array. In this example, only the first and last items in the request payload triggered the Detectors, so the second element of the array is empty.

In the first element of the array, you can see details about which Detection Rule was triggered and the data that was found (finding). The response also provides a confidence level (confidence), as well as the location within the original text where the data was found either in terms of bytes (byteRange) or characters (codepointRange).

{
  "findings": [
    [
      {
        "finding": "458-02-6124",
        "redactedFinding": "XXX-XXXX-XXXX-9293",
        "detector": {
          "name": "US Social Security Number",
          "uuid": "e30d9a87-f6c7-46b9-a8f4-16547901e069"
        },
        "confidence": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "start": 39,
            "end": 50
          },
          "codepointRange": {
            "start": 39,
            "end": 50
          },
          "rowRange": null,
          "columnRange": null,
          "commitHash": ""
        },
        "matchedDetectionRuleUUIDs": [],
        "matchedDetectionRules": [
          "My Match Rule"
        ]
      }
    ],
    [],
    [
      {
        "finding": "5310-2768-6832-9293",
        "redactedFinding": "XXXX-XXXX-XXXX-9293",
        "detector": {
          "name": "Credit Card Number",
          "uuid": "74c1815e-c0c3-4df5-8b1e-6cf98864a454"
        },
        "confidence": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "start": 25,
            "end": 44
          },
          "codepointRange": {
            "start": 25,
            "end": 44
          },
          "rowRange": null,
          "columnRange": null,
          "commitHash": ""
        },
        "redactedLocation": {
          "byteRange": {
            "start": 25,
            "end": 44
          },
          "codepointRange": {
            "start": 25,
            "end": 44
          },
          "rowRange": null,
          "columnRange": null,
          "commitHash": ""
        },
        "matchedDetectionRuleUUIDs": [],
        "matchedDetectionRules": [
          "My Match Rule"
        ]
      }
    ]
  ],
  "redactedPayload": [
    "",
    "",
    "My credit card number is XXXX-XXXX-XXXX-9293"
  ]
}

Congratulations! You have successfully completed the Nightfall Quickstart.

You can modify the Detectors or payload in the example request to get more practice with the Nightfall API.

Welcome to the amazing world of the Nightfall Firewall for AI (formerly known as Nightfall Developer Platform). Here you can find all the information about Nightfall's APIs, and SDKs, and also usage examples of these APIs and SDKs.

There are many use cases for a high accuracy data classification and protection system like Nightfall. Here are some of the most popular to spark your imagination.

Protect sensitive data from transferring to downstream 3rd party services like LLM APIs.

Motivation

• Third-party APIs provide services that greatly augment the capabilities of your applications.

  • For example, GenAI LLMs can automatically generate content. These LLMs can be accessed via APIs, such as OpenAI or Anthropic APIs.

  • Another example are telecom/communications APIs like SendGrid and Twilio that provide communications infrastructure.

• The challenge is that these services may unnecessarily receive sensitive or confidential information from your application that is calling these APIs, which can pose data privacy risks because customer data is being shared outside the intended scope. For example, LLMs can handle very large inputs, or prompts, and these prompts may contain sensitive customer information.

Benefits

• By filtering out customer data from API inputs, you will be able to leverage cutting-edge third-party services and APIs without introducing data privacy risks by oversharing sensitive or confidential information.

Sanitize user input to prevent unnecessary collection or proliferation of sensitive customer data.

Motivation

• Applications collect and store sensitive information from consumers. Users may “overshare” or incorrectly input information, leading to sensitive data ending up in places it is not expected, or internal services may proliferate or handle this data in unexpected ways.

  • Fintech applications that intake, store, and generate files with PII like W-2s and paystubs.

  • Healthcare applications that handle protected health information or SSNs.

• Marketplaces and social media applications allow for user generated content that may contain sensitive or illicit information, such as profanity, toxicity.

• Support channels receive any inbound information from consumers, and can include highly sensitive information or over-sharing that is then exposed to support agents.

• This data can come in a variety of unstructured formats - whether that be screenshots, images, documents, plaintext, compressed folders or archives, so to inspect this content requires high quality text extraction.

Benefits

• Reduce the possibility of users inputting sensitive data that should not be collected or retained within your application or service by scanning data upon submission. Warn or prevent users from inputting sensitive data into form fields or file uploads.

• Diminish collection of sensitive data types that could result in regulatory fines or brand damage, if leaked or breached.

• Limit exposure of sensitive data to internal personnel like support agents that could lead to accidental misuse or intentional theft.

Audit and remove sensitive data in data silos and processing workflows for compliance.

Motivation

• Compliance regimes like FedRAMP, PCI, and HIPAA may require that sensitive data is not proliferating into unsanctioned data silos, like project management systems, data warehouses, and logging infrastructure.

• Many different development teams may be writing data into these internal services like logging and data warehousing, so it is challenging to enforce data sanitization on data ingress.

• CDP tools like Segment and Fivetran can further proliferate sensitive data into a broader set of data silos than its original location.

• Data analytics and data science teams may replicate and transform data, leading to further copies and versions across internal systems.

• Edge cases, unexpected errors, and stack traces can lead to sensitive data landing or replicating in application logs.

Benefits

• Identify and remove sensitive data from places that it shouldn’t be.

• Monitor data at rest in data silos instead of at points of ingress/egress that would be hard to monitor or track.

• Scan extremely high volumes of unstructured data at scale.

• Build workflows to delete data, redact data, or alert the right teams when sensitive data is found where it shouldn’t be.

Build data classification and DLP features directly into your SaaS application.

Motivation

• Data classification and DLP capabilities are increasingly expected by regulated institutions such as big banks.

• Building data classification and DLP from scratch is complex and has high opportunity costs in moving developers away from working on the core product offering. Building a half-baked solution erodes customer trust, especially when there is already a high degree of skepticism around the quality of traditional DLP solutions.

• SaaS and security vendors can deliver additional customer value and drive additional revenue through premium enterprise feature tiers that include security features like DLP, SAML SSO, audit logging, and more.

Benefits

• Reduce time-to-market by leveraging out of the box components.

• Reduce the overhead of an in-house data classification service that requires text extraction services, detector research and tuning, machine learning model development and deployment, maintenance & support.

• Deliver best in class accuracy, reducing the risk of alert fatigue or missing sensitive data that erodes customer trust.

Centralize detection logic, custom detectors & regexes all in one place instead of embedded directly in code, and reduce the number of regexes required.

Motivation

• Detecting a single type of sensitive data well (e.g. a credit card number) can be complex - requiring research and maintenance as the detector evolves over time. This becomes especially challenging for esoteric detectors, for example those that are region or industry-specific.

• Managing regexes and input validation is complex and evolving. For example, a regex embedded in code to validate a Google Docs link may need to be updated over time as the format for Google Docs links changes, false positives are identified and accounted for, any performance implications are observed.

• Many data types cannot be detected accurately with a regex because they require a certain level of validation, are heavily context dependent, or are highly variable or entropic in nature leading to a regex being overly sensitive or overly specific.

Benefits

• Leverage out of the box detectors so no engineering time is spent on research, training, tuning detectors. No need to reinvent the wheel. These detectors span the categories of PII, PCI, PHI, credentials & secrets, ID numbers, and more.

• Reduce time spent finding, tuning, and sharing regular expressions.

• Build upon out of the box detectors with custom logic, instead of having to start from scratch with a regex or custom validation logic.

Improve accuracy of existing content inspection systems.

Motivations

• Existing content inspection systems may yield a high degree of false positives (i.e. noise), leading to alert fatigue and significant time wasted on inaccurate alerts.

• On the contrary, existing solutions may also be very limited in detection scope, leading to a high degree of false negatives (i.e. misses), putting the business at risk when sensitive data is missed.

Benefits

• Replace existing, brittle solutions with a highly accurate content inspection system.

• Reduce engineering time spent analyzing false positives and attempting to tune them out.

Sanitize inputs to labeled data used to train machine learning models.

Motivation

• In training complex learning models, data scientists must compile and use large corpuses of data to improve the accuracy of the trained model. Unknowingly leveraging sensitive data in this effort can lead to violations of compliance regimes like HIPAA, GDPR, or PCI.

• Models that focus on health, finance, public sector applications are particularly at risk for ingesting sensitive data that may violate industry specific compliance mandates.

• Labeled data is often ingested from unregulated sources like customer communications, emails, public repos, and more. Inspecting all of these input sources manually is untenable.

• Additionally, the data being leveraged may be in a variety of unstructured formats like screenshots, images, documents, plaintext, compressed folders or archives – to inspect this content requires high quality text extraction.

Benefits

• Ensure the hygiene of the labeled data you are using to train your machine learning models

• Diminish collection of sensitive data types that could result in regulatory fines or brand damage, if leaked or breached.

Example use cases by team and industry.

• Healthcare: Detect PHI to ensure HIPAA compliance in your apps

• Financial services: Secure PII and PCI like bank account numbers, payment card details, and social security numbers

• E-commerce: Prevent costly data breaches of PII and PCI that can damage brand reputation

• Education: Protect student and faculty privacy within applications

• Customer support: Redact sensitive data in customer support system, shielding agents from information they shouldn’t see

• IT Operations: Search for API keys, credentials, and secrets across internal and external data silos

• Product: Create custom solutions for data classification, DLP, content moderation and more within your applications

• Compliance: Address PCI-DSS, HIPAA, FedRAMP, GDPR, CCPA, GLBA, FERPA, PHIPA, and more

• People & Community: Content moderation to detect profanity, toxicity

• Gaming: Detecting profanity, toxicity, or even personal or financial information being shared in community chat rooms

This section describes the terms you will need to know when using the API.

Detectors

Detectors provide the logic to find potentially sensitive pieces of data.

When this logic detects such data, the Detector is considered "triggered."

Nightfall's has numerous pre-built Detectors that are trained via machine learning. Detectors may also be defined with regular expressions or dictionaries. Their accuracy may be further refined with exclusion rules and context rules. Whether a Detector is triggered may be controlled by a minimum confidence threshold per Detector and minimum number of findings per Detector as set on a Detection Rule.

The built-in set of Detectors cover a number of different categories of data, including:

• Standard PII (e.g. social security number, driver's license number, ID card image)

• PCI (Credit Card Number, credit card image)

• Healthcare (e.g. PHI, US Medicare Beneficiary Number)

• Finance - Banking (e.g. SWIFT code, IBAN code, US bank routing number)

• Network (e.g. an IP Address)

Custom Detectors

Exclusion Rules

An exclusion rule is a regular expression or word list that will be used once a Detector is triggered by its primary expression or word list to eliminate false positives.

For instance, you may have a Detector designed to detect phone numbers. However, you may have a particular set of phone numbers that you use for testing purposes that are known not to be valid (e.g. they start with the prefix 555) and this should be ignored. Adding an exclusion rule would allow you to prevent those matches from being returned by the API.

Context Rules

Context Rules are additional matching expressions for a Detector that may be used to adjust the confidence score of a match.

You may provide a regular expression and the number of leading or trailing characters within which a match of that expression must occur in order to adjust the confidence level to a particular level.

For instance, if you found a sequence that appeared to be a social security number based on its length or formatting, you might boost the confidence score if it was preceded by the text like “SSN” or “Social Security Number.”

Returning Surrounding Context

You may request that a sequence of bytes of a given length be provided from before and after the text that triggers a Detection Rule.

This information can help you better understand whether or not something is an actual violation by observing the circumstances within which the detected text was found.

You are limited to a maximum of 40 bytes of this context text preceding and trailing the match for a total of 80 bytes overall.

Detection Rules

Detection Rules are aggregations of Detectors that are assigned a minimum confidence level. The identifiers of Detection Rules are used as a parameter to the API.

A Detection Rule is composed of a list of Detectors with which you wish to scan each request payload, where any or all Detectors may be satisfied in order to trigger the rule. You can add up to 50 total Detectors with a limit of 30 regular expression type custom detectors.

Confidence Levels

Detection results will be returned with one of the following confidence values.

In practice, the API will only return detections assigned a POSSIBLE or higher confidence level.

• VERY_LIKELY (recommended)

• LIKELY

• POSSIBLE

• UNLIKELY

• VERY_UNLIKELY

Policies

Policies allow you to create templates for the most common workflows by unifying a set of Detection Rules with the actions to be taken when those rules are triggered, including:

• automated actions such as redaction of findings

• alerting through webhooks

Once defined, a Policy may be used in requests to the Nightfall API, such as calls to scan file uploads, though automated redactions are not available for uploaded files at this time.

Your API keys carry many privileges, so be sure to keep them secure. Do not share your secret API keys in publicly accessible areas such as GitHub, client-side code, or anywhere else that would compromise their secrecy. If you believe one of your API Keys has been compromised, you should delete it through the Dashboard.

All API requests must be made over HTTPS.

Calls made over plain HTTP will fail.

API requests without authentication will fail.

Welcome to Nightfall's Firewall for AI Developers Scan and Workflow APIs documentation. This documentation helps developers leverage Nightfall AI's industry-leading detection engine to identify and protect sensitive customer and corporate data anywhere. It prevents unauthorized access and data breaches and allows you to focus on innovation.

Scan APIs

Scan prompts, text, documents, spreadsheets, logs, zips, JSON, images, etc., for PII, PHI, PCI, banking information, API keys, passwords, and network information with the highest accuracy and lightning-fast response times. Redact sensitive findings with customizable formatting.

Workflow APIs

Leverage the full potential of the Nightfall console application through our Workflow APIs. Customize your SIEM workflows and reporting, take actions, update support tickets, alert users, search violations, annotate findings, create reports, and more.

Key Features

• AI-Powered Identification: Utilize advanced AI models to detect and prevent security threats in real-time.

• Comprehensive Sensitive Data Detection: Identify PII, PHI, PCI, banking information, API keys, passwords, and network information across various formats including text, documents, spreadsheets, logs, zips, and images.

• Customizable Redaction: Tailor data protection to your needs with fully customizable redaction for each sensitive entity type.

• Flexible Detectors: Leverage Nightfall’s comprehensive list of machine learning-based detectors, customize them, or create your own with specialized logic.

• High Accuracy and Performance: Achieve precision and recall rates of 95% or higher, handle over 1K requests per second, and experience latency of less than 100 ms.

• Seamless Integration: Easily integrate with your existing AI development and data engineering tools for smooth and efficient operation.

Customizable and Built-in Machine Learning-based Detectors

You can leverage Nightfall’s machine learning-based detectors or create your own detectors with customized logic to scan third-party apps, internal services, and data silos to identify instances of potentially sensitive types of data such as:

• Personally Identifiable Information (PII) including Social Security Numbers, passport numbers, email addresses, or date of birth

• Protected Health Information (PHI) such as insurance claim numbers or ICD10 codes

• Financial information like credit card numbers or bank routing numbers

• Secrets such as API and cryptographic Keys, database connection strings, passwords, etc.

• Network information such as IP Address or MAC Address

A Flexible Data Security Solution

Key features of Nightfall’s detection engine include:

• Defining minimum confidence thresholds and minimum finding counts on detectors to reduce the chance of false positives.

• Choosing which detectors are triggered for each policy.

Using the API

The findings display the relevant detector, the likelihood of a match, and the location within the given data where the matched token occurred (not only in terms bytes — there is support for tabular and JSON data as well).

Where to Go From Here

After that, you can learn about Nightfall with our Key Concepts section, which will also help you get set up with Nightfall.

You can define Detection Rules “inline” in the body of each request to the scan endpoint. See the example in the walk through of the scan endpoint Creating an Inline Detection Rule.

You may add up to 50 detectors to your detection rule.

To create a Detection Rule in the Nightfall UI, Select "Detection Rules" from the left hand navigation.

Click the + New Detection Rule button in the upper right hand corner.

First, enter a name for your Detection Rule as well as an optional description.

Then click the + Detectors button to add Detectors to your Detection Rule.

In this example we have selected the US drivers license and Canada Government ID detectors.

Click the Add button in the lower right hand corner at the end of the detector list when you are done adding detectors.

Now that your Detectors are set, choose a minimum confidence level and a minimum # of findings for each detector.

If these minimums for a Detector are not met, the Detection Rule will not be triggered.

Save your Detection Rule in the lower left hand corner once you are done.

Once the Detection Rule is saved, it is available for use in requests to the Nightfall API to scan your data for sensitive information. Pass in the UUID of the Detection Rule as the detectionRuleUUIDs field of your requests to the the scan endpoints.

The UUID may be obtained by clicking the "copy" icon, the left most icon in the set of icons that appear next to the Detection Rules name when your cursor highlights a Detection Rule in the list of Detection Rules.

See Using Pre-Configured Detection Rules for an example of using a Detection Rule UUID.

The way that an alert notification presents itself depends on the platform in question.

For example, notifications sent to Slack will appear as formatted messages sent by the Nightfall Alerts Bot. Other destinations such as email, SIEM url, and webhooks, will present the information as JSON objects.

In the case of webhooks, detailed information about the finding will be sent. For other destinations, sensitive information is redacted.

Supported Alert Platforms

Slack

In order to use asynchronous notifications with Slack, you must install the Nightfall Alerts plugin from the Slack Marketplace.

Once you have authenticated Nightfall to your Slack workspace, you can provide any public channel name (e.g. #general) as part of a request to the Nightfall API.

To send notifications to a private channel, a member of the channel should invite the Nightfall bot to the specific private channel and allow channel access to the bot.

Follow the steps below to invite Nightfall Alerts bot to a private channel:

1. Go to the Slack channel in question

2. Type /invite @Nightfall Alerts as a message

3. Press 'Enter' (you should see a message that Nightfall Alerts has now joined the channel)

If any findings are detected as part of that request, then the Nightfall Alerts bot will send a message to the channel you configured. Conversely, if there are no findings in the request payload, then Nightfall will not send an alert message.

Teams

Documentation TBD

Email

Email is unauthenticated, so you can get started using Nightfall to send email alerts without any initial setup work.

Nightfall will send an email to the provided address only if findings were detected as part of the request. The findings themselves will be attached in a JSON file.

SIEM

You may send your alerts to a designated url, such as an endpoint hosted by SIEM software for log collection.

In addition to the url, you may provide headers, either for security or logging purposes.

Webhook

You may use a webhook server to programmatically handle a finding, allowing you to create your own custom workflows with your own or 3rd party systems.

Nightfall will always send an alert to the client's webhook server if it is provided as part of an API request, even if the scan request yielded no findings.

Alert Schemas

The request body sent by Nightfall is JSON, and uses the schemas in the section documented below.

File Scans

Since file scans can produce a large number of results, findings are not transmitted directly in the notification that Nightfall sends. The notification object looks like the following:

{
  "errors": [],
  "findingsPresent": true,
  "findingsURL": "https://files.nightfall.ai/877442c5-1573-4637-a223-595bf620e3e5.json?Expires=1645722381&Signature=C-kQbtonFAPXfooGcm0dYgbsn9jfGu~vGSv5yK5j1z2f7aAhk0WuaL4bISUwx5MZkQmPVFgeyMwemvEoI8aI11lPA-ORsX5LtRdGJBOma4sPVl~9f9qBPKE2VSrdGDmT4EpBLc8ewUtKrLm2xE-0BzW~5PdLSvZ~NQxtB7OMBaYm7h~y2NSUZfpqzdzENyKhyHx5QxH2PJvxeN5IvMXqNUrKyZsxviSYY6kDNAiGExS-u6PmKKS1GhXOaFLdJSRjgtFhUxDLyWl~xTYR-lJol5UTgtcuYU8AaJ3xVTF1-1JYRlioRlaf9shAvme4djFyg8k~zOB8bYgzBeaRqSjeWA__&Key-Pair-Id=K3RYMP51FKX5HX",
  "requestMetadata": "some data",
  "uploadID": "877442c5-1573-4637-a223-595bf620e3e5",
  "validUntil": "2022-02-24T17:06:21.412377682Z"
}

The requestMetadata field contains arbitrary contents provided by the client at request time, and can be used by the client to correlate this response to the original request.

The value of the findingsURL field is a pre-signed URL, which means anyone with the link can download the file. Therefore, this URL itself should be treated as sensitive and must not be leaked. The object stored at this URL is a JSON file containing a single key findings containing a list of all data detected from the request. The schema for the finding object inside the list is shared between the text-based and file-based API endpoints.

{
  "findings": [
    {
      "detector": {
        "id": "74d1315e-c0c3-4ef5-8b1e-6cf98664a854"
      },
      "finding": "4242-4242-4242-4242",
      "confidence": "VERY_LIKELY",
      "location": {
        "byteRange": {
          "start": 146,
          "end": 165
        },
        "codepointRange": {
          "start": 146,
          "end": 165
        },
        "lineRange": {
          "start": 3,
          "end": 3
        },
        "rowRange": null,
        "columnRange": null,
        "commitHash": ""
      },
      "beforeContext": "nd HIPAA Defined PII\nHIPAA HIPAA hooray\n",
      "afterContext": " is my credit card number\n\n",
      "matchedDetectionRuleUUIDs": ["7bd6166a-b9af-4069-847d-487a88788122"],
      "matchedDetectionRules": []
    } 
  ]    
}

Text Scans

{
  "findings": [
    [
      {
        "finding": "4242-4242-4242-4242",
        "beforeContext": "hello world cc ",
        "detector": {
          "name": "Credit card number",
          "uuid": "74c1815e-c0c3-4df5-8b1e-6cf98864a454"
        },
        "confidence": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "start": 15,
            "end": 34
          },
          "codepointRange": {
            "start": 15,
            "end": 34
          },
          "rowRange": null,
          "columnRange": null,
          "commitHash": ""
        },
        "matchedDetectionRuleUUIDs": [
          "42efe36c-6479-412a-9049-fd8cdf895ced"
        ],
        "matchedDetectionRules": []
      }
    ]
  ],
  "redactedPayload": [""]
}

Before you use the scan endpoint, there are a number of actions to do within the Nightfall dashboard to get your environment set up properly.

Policies allow customers to create templates for their most common workflows by unifying a set of Detection Rules with the actions to be taken when those rules are triggered, including:

• automated actions such as redaction of findings

• alerting through webhooks

Once defined, a Policy may be used in requests to the Nightfall API, such as calls to scan file uploads, though automated redactions are not available for uploaded files at this time.

To create a policy:

1. Log in to Nightfall.

2. Click Overview under the Firewall for AI section.

1. Click Create Policy.

The policy creation page is displayed as follows.

1. Enter a name for the policy.

2. (Optional) Enter a Description for the policy.

3. Click + Detection rule to add a Detector rule to the policy.

4. Select the check box of the Detector rules that you wish you add to the Policy.

1. Select the Redact Violations check box to mask sensitive information found in your transmitted data.

2. Select one of the alerting method available.

   • Click + HTTP Alerts to configure a website as alert notification channel.

   • Click + Email to notify recipients through an Email.

   • Click + Slack to select a Slack channel to which the alerts must be sent.

3. Click Save Policy.

Configuring Webhook Alerts

When you click + Application Webhook, the following window is displayed.

If you have custom headers you would like to add to requests sent to the Webhook URL, you can do this from the overlay that appears when you click the "+ Webhook" button on the policy creation and edit page. These headers may be used for the purpose of authentication as well as integrating with Security Incidents and Event Management (SIEMs) or similar tools that aggregate content through HTTP event collection.

Click the "Add Header" button to add your custom headers.

Once your header key and value is entered you may obfuscate it by clicking on the "lock" icon next to the value field for the header. Click the "Save" button to persist your changes to the headers.

When you have completed configuring your Webhook URL and Headers, click the "Save" button.

🚧Limits On Webhook Headers

It is currently not possible to configure headers for webhooks programmatically when defining policies through the API.

After you click the "Save Policy" button, your policy should be immediately available for use. You can refer to the API Docs for the comprehensive list of endpoints that support policy UUIDs.

The API expects an API Key to be passed via the Authorization: Bearer <key> HTTP header.

To create and manage API keys:

1. Log in to Nightfall.

2. Click Overview under the Firewall for AI section.

1. Click Create key.

The Generate API Key window is displayed.

1. Enter a name for the API key and click Create.

The API key is generated and displayed (blurred in the following image). Click the copy button to copy the API key and store it in a. secure location. Once you click the Got it button, you cannot retrieve the API key again.

🚧Be Sure to Record the API Key's Value

For security reasons, after closing the window, you will not be able to recover the key's value.

Once you close the window, the My API Keys page will display your newly generated key, with the majority of the Key redacted.

You can return to the Overview page at any time to create new keys (assuming your license allows you to generate additional keys) or delete old keys.

The file scan API has first-class support for text extraction and scanning on all MIME types enumerated below.

Accepted Text and Derivatives

• application/json

• application/x-ndjson

• application/x-php

• text/calendar

• text/css

• text/html

• text/javascript

• text/plain

• text/x-php

Accepted Office Formats

• application/pdf

• application/vnd.openxmlformats-officedocument.presentationml.presentation

• application/vnd.openxmlformats-officedocument.wordprocessingml.document

Accepted Archive and Compressed File Types

• application/bzip2

• application/ear

• application/gzip

• application/jar

• application/java-archive

• application/tar+gzip

• application/vnd.android.package-archive

• application/war

• application/x-bzip2

• application/x-gzip

• application/x-rar-compressed

• application/x-tar

• application/x-webarchive

• application/x-zip-compressed

• application/x-zip

• application/zip

Accepted Image File Types

• image/apng

• image/avif

• image/gif

• image/jpeg

• image/jpg

• image/png

• image/svg+xml

• image/tiff

• image/webp

Rejected MIME Types

The file scan API explicitly rejects requests with MIME types that are not conducive to extracting or scanning text. Sample rejected MIME types include:

• application/photoshop

• audio/midi

• audio/wav

• video/mp4

• video/quicktime

Spreadsheets and Tabular Data

File scans of Microsoft Office, Apache parquet, csv, and tab separated files will provide additional properties to locate findings within the document beyond the standard byteRange, codepointRange, and lineRange properties.

Findings will contain a columnRange and a rowRange that will allow you to identify the specific row and column within the tabular data wherein the finding is present.

This functionality is applicable to the following mime types:

• text/csv

• text/tab-separated-values

• text/tsv

• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

• application/vnd.ms-excel

Below is a sample match of a spreadsheet containing dummy PII where a SSN was detected in the 2nd column and 55th row.

{
   "findings":[
      {
         "path":"Sheet1 (5)",
         "detector":{
            "id":"e30d9a87-f6c7-46b9-a8f4-16547901e069",
            "name":"US social security number (SSN)",
            "version":1
         },
         "finding":"624-84-9182",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":2505,
               "end":2516
            },
            "codepointRange":{
               "start":2452,
               "end":2463
            },
            "lineRange":{
               "start":55,
               "end":55
            },
            "rowRange":{
               "start":55,
               "end":55
            },
            "columnRange":{
               "start":2,
               "end":2
            },
            "commitHash":""
         },
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      },
...

Redacting CSV Files

Findings within csv files may be redacted.

To enable redaction in files, set the enableFileRedaction flag of your policy to "true"

The csv file will be redacted based on the configuration of the defaultRedactionConfig of the policy

curl --request POST \
     --url https://api.nightfall.ai/v3/upload/02a0c5e1-c950-4e28-a988-f6fffefc4205/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer NF-<Your API Key>' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRuleUUIDs": [
               "950833c9-8608-4c66-8a3a-0734eac11157"
          ],
          "alertConfig": {
               "email": {
                    "address": "<your email addres>"
               }
          },
          "defaultRedactionConfig": {
               "maskConfig": {
                    "charsToIgnore": [
                         "-",
                         "@"
                    ],
                    "maskingChar": "*"
               }
          },
          "enableFileRedaction": true

     },
     "requestMetadata": "csv redaction test"
}
'

When results are sent to the location specified in the alertConfig (in this case an email address) a redactedFile property will be set with a fileURL in addition the findingsURL

{
   "errors":null,
   "findingsPresent":true,
   "findingsURL":"https://files.nightfall.ai/asdfc5e1-c950-4e28-a988-f6fffefc4205.json?Expires=1655324479&Signature=zjo1nT-PECHC-fiTvAgdA8aDnceoY~6iGfzOBCcBjscKqOHnIar8hoH4gGufffiulBw5BpfJuvWwBW~lXO~ZNhN139LDwoTsfLJswJiQCB2Hj-Az0Em6go~1j8WBqCS8G0Gk17M-zcPedHGX3z~1pw8nm5sh6Pa-jJwfw9NIEiqmBb3Vdcj3J-~Wzag~ENV4499rnG299ee-ig5Ms1oVlzycb4YxzgTMrTL5Q07ozNenwFZcGDNQre1inLXmV-m8teLX-K3boklenp9KXiNDDV0wi74ADN-QfIR1q1oU7mEI1f3aVC3kju0QRErp2lsfs08EtZKLE3C4N17jDJdYcw__&Key-Pair-Id=K24YOPZ1EKX0YC",
   "redactedFile":{
      "fileURL":"https://files.nightfall.ai/asdfc5e1-c950-4e28-a988-f6fffefc4205-redacted.csv?Expires=1655324479&Signature=Hx8kRh88maLeStysy3fsLbFVG9VELEtfemtQe2lWUnFjAMd9HqlEksTmirqAWFWV4zPVUB73izlMj5cSer8v2N5ZCcnD3dz~nnwR4P5LewGJ2CQzGnDnXgh70HW5qp04gnUD-pYWp~bGPVspkJKCkl1zH-EoGonvcNVq3SNsVzOlsVIjep7Y7otQKEEyAZ7JmHiVfuBxrvn8pleuC5lEJ3f9miPyoRqH9DyPlNTJTIuijqe9q32Qcui2RsDR6IT-foFX52dy6rRa01ZV0gZMDWJokMlCr8Iu5An~qnhxC49bqTtI82oz9FcBaP-Yea8cq1TiAfGxX7CJ0~JeTLvr6g__&Key-Pair-Id=K24YOPZ1EKX0YC",
      "validUntil":"2022-06-15T20:21:19.750990823Z"
   },
   "requestMetadata":"csv redaction test",
   "uploadID":"02a0c5e1-c950-4e28-a988-f6fffefc4205",
   "validUntil":"2022-06-15T20:21:19.723045787Z"
}

This redacted file will be a modified version of the original csv file.

Below is an example of a redacted csv file.

name,email,phone,alphanumeric
Ulric Burton,*****@*************,*-***-***-****,TEL82EBM1GQ
Wade Jones,******************@***********,(********-****,VVF64PJV2EF
Molly Mccullough,*****************@**********,(********-****,OHO41SFZ2BR
Raja Riggs,************@**********,(********-****,UVD51JTE5NZ
Colin Carter,**********************@*********,(********-****,LNI34LLC5WV// Some code

Git Repositories

Nightfall provides special handling for archives of Git repositories.

Nightfall will scan the repository history to discover findings in particular checkin, returning the hash for the checkin.

In order to scan the repository, you will need to create a clone, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

This creates a clone of the Nightfall go SDK.

You will then need to create an archive that can be uploaded using Nightfall's file scanning sequence.

zip -r directory.zip directory

Note that in order to work, the hidden directory .github must be included in the archive.

Using the Nightfall go SDK archive created above, a simple example would be to scan for URLs (i.e. strings starting with http:// or https://), which will send results such as the following:

{
   "findings":[
      {
         "path":"f607a067..53e59684/nightfall.go",
         "detector":{
            "id":"6123060e-2d9f-4f35-a7a1-743379ea5616",
            "name":"URL"
         },
         "finding":"https://api.nightfall.ai/\"",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":142,
               "end":168
            },
            "codepointRange":{
               "start":142,
               "end":168
            },
            "lineRange":{
               "start":16,
               "end":16
            },
            "rowRange":{
               "start":0,
               "end":0
            },
            "columnRange":{
               "start":0,
               "end":0
            },
            "commitHash":"53e59684d9778ceb0f0ed6a4b949c464c24d35ce"
         },
         "beforeContext":"tp\"\n\t\"os\"\n\t\"time\"\n)\n\nconst (\n\tAPIURL = \"",
         "afterContext":"\n\n\tDefaultFileUploadConcurrency = 1\n\tDef",
         "matchedDetectionRuleUUIDs":[
            "cda0367f-aa75-4d6a-904f-0311209b3383"
         ],
         "matchedDetectionRules":[
            
         ]
      },
 ...

Large repositories result in a large volume of data sent at once. We are working on changes to allow these and other large surges of data to be processed in a more controlled manner, and will increase the limit or remove it altogether once those changes are complete.

To retrieve the specific checkout, you will need to clone the repository, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

You can then checkout the specific commit using the commit hash returned by Nightfall.

cd nightfall-go-sdk
git checkout 53e59684d9778ceb0f0ed6a4b949c464c24d35ce

File Scanning Limitations

• CSV Files: Only the first 250,000 rows will be scanned.

• Spreadsheet Files: Up to 100,000 rows per sheet will be scanned, with a maximum of 1 million rows across all tabs in multi-sheet spreadsheets.

• PDF Files: Scanning is limited to the first 100 pages, including a maximum of 50 images within those pages.

• Images: Images smaller than 5KB or larger than 50MB will be excluded from scanning.

• Archive Files: A maximum of 1,000 files will be extracted and scanned. Files larger than 100MB requiring extraction will not be scanned.

After the file scan has been processed asynchronously, the results will be delivered to the webhook.

Webhook Payload and Findings for File Scans

For a file scan, your webhook will receive a request body that will be a JSON payload containing:

• the upload UUID (uploadID)

• a boolean indicating whether or not any data in the file matched the provided detection rules (findingsPresent)

• a pre-signed S3 URL where the caller may fetch the findings for the scan (findingsURL). if there are no findings in the file, this field will be empty.

• the value you supplied for requestMetadata. Callers may opt to use this to help identify their input file upon receiving a webhook response. Maximum length 10 KB.

Below is an example of a payload sent to the webhook URL.

{
    "findingsURL": "https://files.nightfall.ai/asdfasdf-asdf-asdf-asdf-asdfasdfasdf.json?Expires=1635135397&Signature=asdfasdfQ2qTmPFnS9uD5I3QGEqHY2KlsYv4S-WOeEEROj~~x6W2slP2GvPPgPlYs~lwdr-mtJjVFu4LtyDhdfYezC7B0ysfJytyMIyAFriVMqOGsRJXqoQfsg8Ckd2b6kRcyDZXJE25cW8zBS08lyVwMBCsGS0BKSin8uSuD7pQu3QAubT7p~MPkfc6PSXYIJREBr3q4-8c7UnrYOAiXfSW1AmFE47rr3Wxh2TpU3E-Fxu-6e3DKN4q6meACdgZb2KHZo3e-NK7ug9f8sxBp1YT0n5oiVuW4KXguIyXWN~aKEHMa6DzZ4cUJ61LmnMzGndc2sVKhii39FHwTsYog__&Key-Pair-Id=asdfOPZ1EKX0YC",
    "validUntil": "2021-10-25T04:16:37.734633129Z",
    "uploadID": "152848af-2ac9-4e0a-8563-2b82343d964a",
    "findingsPresent": true,
    "requestMetadata": "",
    "errors": []
}

In this example, we have uploaded a zip file with a python script (upload.py) and a README.md file. A Detector in our DetectionRule checks for the presence of the string http://localhost

{
   "findings":[
      {
         "path":"fileupload/upload.py",
         "detector":{
            "id":"58861dee-b213-4dbc-97fa-a148acb8bd1a",
            "name":"localhost url"
         },
         "finding":"http://localhost",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":105,
               "end":121
            },
            "codepointRange":{
               "start":105,
               "end":121
            },
            "lineRange":{
               "start":7,
               "end":7
            }
         },
         "beforeContext":"PLOAD_URL = getenv(\"FILE_UPLOAD_HOST\", \"",
         "afterContext":":8080/v3\")\nNF_API_KEY = getenv(\"NF_API_K",
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      },
      {
         "path":"fileupload/README.md",
         "detector":{
            "id":"58861dee-b213-4dbc-97fa-a148acb8bd1a",
            "name":"localhost url"
         },
         "finding":"http://localhost",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":570,
               "end":586
            },
            "codepointRange":{
               "start":570,
               "end":586
            },
            "lineRange":{
               "start":22,
               "end":22
            }
         },
         "beforeContext":"t the script will send the requests to `",
         "afterContext":":8080`, but this can be overridden using",
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      },
      {
         "path":"fileupload/README.md",
         "detector":{
            "id":"58861dee-b213-4dbc-97fa-a148acb8bd1a",
            "name":"localhost url"
         },
         "finding":"http://localhost",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":965,
               "end":981
            },
            "codepointRange":{
               "start":965,
               "end":981
            },
            "lineRange":{
               "start":26,
               "end":26
            }
         },
         "beforeContext":"ice deployment you want to connect to | ",
         "afterContext":":8080 |\n| `NF_API_KEY`      | the API Ke",
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      }
   ]
}

To create a Detector, select "Detectors" from the left-hand navigation and click the + New Detector button

Custom detectors can add context and exclusion rules on top of pre-built Nightfall detectors, or can be built off your own custom regular expressions.

Be aware that you may not have two detectors based on the same Nightfall data type within the same detection rule.

A full glossary of Nightfall's prebuilt detectors can be found in the Detector Glossary

Updated 2 months ago

Uploading files using Client SDK libraries requires fewer steps as all the required API operations are wrapped in a single function call. Furthermore these SDKs handle all the programmatic logic necessary to send files in smaller chunks to Nightfall.

Using Nightfall's SDKs to Upload Files

>>> from nightfall import Confidence, DetectionRule, Detector, Nightfall, EmailAlert, AlertConfig
>>> import os

>>> # use your API Key here
>>> nightfall = Nightfall("NF-y0uRaPiK3yG03sH3r3")

>>> # A rule contains a set of detectors to scan with
>>> cc = Detector(min_confidence=Confidence.LIKELY, nightfall_detector="CREDIT_CARD_NUMBER")
>>> ssn = Detector(min_confidence=Confidence.POSSIBLE, nightfall_detector="US_SOCIAL_SECURITY_NUMBER")
>>> detection_rule = DetectionRule([cc, ssn])
>>> # The scanning is done asynchronously, so provide a valid email address as the simplest way of getting results
>>> alertconfig = alert_config=AlertConfig(email=EmailAlert("whatever@example.com"))
    

>>> # Upload the file and start the scan.
>>> id, message = nightfall.scan_file( "./README.md", detection_rules=[detection_rule], alert_config=alertconfig)
>>> print("started scan", id, message)

//this script assumes the node sdk has been installed locally with `npm install` and `npm run build`
import { Nightfall } from "./nightfall-nodejs-sdk/dist/nightfall.js";
import { Detector } from "./nightfall-nodejs-sdk/dist/types/detectors.js";


// By default, the client reads your API key from the environment variable NIGHTFALL_API_KEY
const uploadit = async() => {
    var data = null;
    
    const nfClient = new Nightfall();
    	
    try{
   
		const response = await nfClient.scanFile('./README.md', {
		  detectionRules: [
			{
			  name: 'Secrets Scanner',
			  logicalOp: 'ANY',
			  detectors: [
				{
				  minNumFindings: 1,
				  minConfidence: Detector.Confidence.Possible,
				  displayName: 'Credit Card Number',
				  detectorType: Detector.Type.Nightfall,
				  nightfallDetector: 'CREDIT_CARD_NUMBER',
				},
			  ],
			},
		  ],
		  alertConfig: {
				email: {
						address: "whatever@example.com"
					}
		   }
		});

		if (response.isError) {
		  data = response.getError();
		}
		else{ 
			data = (response.data.id);
		}
	 
    }
	catch(e){
		console.log(e);
	}


	return data;

}

uploadit().then(data => console.log(data));

To run the node sample script you must compile it as TypesScript. Save it as a .ts file and run

tsc <yourfilename>.ts -lib ES2015,DOM

You can then run the resulting JavaScript file:

NIGHTFALL_API_KEY=<YourApiKey> node yourscriptname.js

Note that these examples use an email address to receive the results for simplicity.

The Upload Process

The upload process consists of 3 stages:

Initializing Phase

POST /v3/upload

As part of the initialization you must provide the total byte size of the file being uploaded.

You may also provide the mime-type, otherwise the system will attempt to determine it once the upload is complete.

curl --location --request POST 'https://api.nightfall.ai/v3/upload' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '{
    "fileSizeBytes": 73891,
    "mimeType" : "image/png"
}'

The id of the returned JSON object will be used as the fileId in subsequent requests.

The chunkSize is the maximum number of bytes to upload during the uploading phase.

{
    "id": "f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d",
    "fileSizeBytes": 73891,
    "chunkSize": 10485760,
    "mimeType": "image/png"
}

Uploading Phase

PATCH /v3/upload/<uploadUUID>

The size of these chunks are determined by the chunkSize value returned by POST /upload endpoint used in the previous step.

Below is a simple example where the file is less than the chunkSize so may safely be uploaded with one call to the upload endpoint.

curl --location --request PATCH 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d' \
--header 'X-Upload-Offset: 0' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-binary '@/Users/myname/Documents/work/Nightfall/Nightfall Upload Sequence.png'

If your file's size exceeds the chunkSize, to upload the complete file you will need to send iterative requests as you read portions of the file's contents. This means you will send multiple requests to the upload endpoint as shown above. As you do so, you will be updating the value of the X-Upload-Offset header based on the portion of the file being sent.

Each request should send a chunk of the file exactly chunkSize bytes long except for the final uploaded chunk. The final uploaded chunk is allowed to contain fewer bytes as the remainder of the file may be less than the chunkSize returned by the initialization step.

The request body should be the contents of the chunk being uploaded.

The value of the X-UPLOAD-OFFSET header should be the byte offset specifying where to insert the data into the file as an integer. This byte offset is zero-indexed.

Successful calls to this endpoint return an empty response with an HTTP status code of 204

Completion Phase

POST /v3/upload/<uploadUUID>/finish

curl --location --request POST 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/finish' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer  NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '""'

When an upload completes successfully, the returned payload will indicate the mimeType the system determined to file to be if it was not provided during upload initialization.

{
    "id": "152848af-2ac9-4e0a-8563-2b82343d964a",
    "fileSizeBytes": 2349,
    "chunkSize": 10485760,
    "mimeType": "application/zip"
}

Once a file has been marked as completed, you may initiate a scan of the uploaded file.

Scanning Uploaded Files

After an upload is finalized, it can be scanned against a Detection Policy. A Detection Policy represents a pairing of:

• a webhook URL

• a set of detection rules to scan data against

You may also supply a value to the requestMetadata field to help identify the input file upon receiving a response to your webhook. This field has a maximum length 10 KB.

curl --request POST \
     --url https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRuleUUIDs": [
               "950833c9-8608-4c66-8a3a-0734eac11157"
          ],
          "webhookURL": "https://mycompany.org/webhookservice"
     },
     "requestMetadata": "your file metadata"
}
'

Full Upload Process Example Script

Below is a sample Python script that handles the complete sequence of API calls to upload a file using a path specified as an argument.

from os import getenv, path

import fire
import requests


BASE_UPLOAD_URL = getenv("FILE_UPLOAD_HOST", "http://api.nightfall.ai/v3")
NF_API_KEY = getenv("NF_API_KEY")


def upload(filepath, mimetype, policy_uuid):
    """Upload the given file using the provided MIMEType and PolicyUUID.

    Arguments:
        file_path -- an absolute or relative path to the file that will be
            uploaded to the API.
        mimetype -- (optional) The mimetype of the file being uploaded.
        policy_uuid -- The UUID corresponding to an existing policy. This
            policy must be active and have a webhook URL associated with it.
    """
    default_headers = {
        "Authorization": F"Bearer {NF_API_KEY}",
    }

    # =*=*=*=*=* Initiate Upload =*=*=*=*=*=*
    file_size = path.getsize(filepath)
    upload_request_body = {"fileSizeBytes": file_size, "mimeType": mimetype}
    r = requests.post(F"{BASE_UPLOAD_URL}/upload",
                      headers=default_headers,
                      json=upload_request_body)
    upload = r.json()
    if not r.ok:
        raise Exception(F"Unexpected error initializing upload - {upload}")

    # =*=*=*=*=*=* Upload Chunks =*=*=*=*=*=*
    chunk_size = upload["chunkSize"]
    i = 0
    with open(filepath, "rb") as file:
        while file.tell() < file_size:
            upload_chunk_headers = {
                **default_headers,
                "X-UPLOAD-OFFSET": str(file.tell())
            }
            r = requests.patch(F"{BASE_UPLOAD_URL}/upload/{upload['id']}",
                               headers=upload_chunk_headers,
                               data=file.read(chunk_size))
            if not r.ok:
                raise Exception(F"Unexpected error uploading chunk - {r.text}")
            i += 1

    # =*=*=*=*=*=* Finish Upload =*=*=*=*=*=*
    r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/finish",
                      headers=default_headers)
    if not r.ok:
        raise Exception(F"Unexpected error finalizing upload - {r.text}")

    # =*=*=*=*=* Scan Uploaded File =*=*=*=*=*
    r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/scan",
                      json={"policyUUID": policy_uuid},
                      headers=default_headers)
    if not r.ok:
        raise Exception(F"Unexpected error initiating scan - {r.text}")

    print("Scan Initiated Successfully - await response on configured webhook")
    quota_remaining = r.headers.get('X-Quota-Remaining')
    if quota_remaining is not None and int(quota_remaining) <= 0:
        print(F"Scan quota exhausted - Quota will reset on {r.headers['X-Quota-Period-End']}")


if __name__ == "__main__":
    fire.Fire(upload)

Nightfall’s file scan API allows a user to upload a file in chunks, then to scan it with Detection Rules once the upload is complete.

The scan will then be processed asynchronously before sending the results to the webhook URL that is provided along with your Detection Rules.

The following sequence diagram illustrates the full process for scanning a binary file with Nightfall.

Prerequisites

In order to utilize the File Scanning API you need the following:

• A Nightfall Detection Policy associated with a webhook URL

• A web server configured to listen for file scanning results (detailed information to follow)

Spreadsheets and Tabular Data

File scans of Microsoft Office, Apache parquet, csv, and tab separated files will provide additional properties to locate findings within the document beyond the standard byteRange, codepointRange, and lineRange properties.

Findings will contain a columnRange and a rowRange that will allow you to identify the specific row and column within the tabular data wherein the finding is present.

This functionality is applicable to the following mime types:

• text/csv

• text/tab-separated-values

• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

• application/vnd.ms-excel

Below is a sample match of a spreadsheet containing dummy PII where a SSN was detected in the 2nd column and 55th row.

{
   "findings":[
      {
         "path":"Sheet1 (5)",
         "detector":{
            "id":"e30d9a87-f6c7-46b9-a8f4-16547901e069",
            "name":"US social security number (SSN)",
            "version":1
         },
         "finding":"624-84-9182",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":2505,
               "end":2516
            },
            "codepointRange":{
               "start":2452,
               "end":2463
            },
            "lineRange":{
               "start":55,
               "end":55
            },
            "rowRange":{
               "start":55,
               "end":55
            },
            "columnRange":{
               "start":2,
               "end":2
            },
            "commitHash":""
         },
         "matchedDetectionRuleUUIDs":[
            "950833c9-8608-4c66-8a3a-0734eac11157"
         ],
         "matchedDetectionRules":[
            
         ]
      },
...

Git Repositories

Nightfall provides special handling for archives of GitHub repositories.

Nightfall will scan the repository history to discover findings in particular checkin, returning the hash for the checkin.

In order to scan the repository, you will need to create a clone, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

This creates a clone of the Nightfall go SDK.

You will then need to create an archive that can be uploaded using Nightfall's file scanning sequence.

zip -r directory.zip directory

Note that in order to work, the hidden directory .github must be included in the archive.

Using the Nightfall go SDK archive created above, a simple example would be to scan for URLs (i.e. strings starting with http:// or https://), which will send results such as the following:

{
   "findings":[
      {
         "path":"f607a067..53e59684/nightfall.go",
         "detector":{
            "id":"6123060e-2d9f-4f35-a7a1-743379ea5616",
            "name":"URL"
         },
         "finding":"https://api.nightfall.ai/\"",
         "confidence":"LIKELY",
         "location":{
            "byteRange":{
               "start":142,
               "end":168
            },
            "codepointRange":{
               "start":142,
               "end":168
            },
            "lineRange":{
               "start":16,
               "end":16
            },
            "rowRange":{
               "start":0,
               "end":0
            },
            "columnRange":{
               "start":0,
               "end":0
            },
            "commitHash":"53e59684d9778ceb0f0ed6a4b949c464c24d35ce"
         },
         "beforeContext":"tp\"\n\t\"os\"\n\t\"time\"\n)\n\nconst (\n\tAPIURL = \"",
         "afterContext":"\n\n\tDefaultFileUploadConcurrency = 1\n\tDef",
         "matchedDetectionRuleUUIDs":[
            "cda0367f-aa75-4d6a-904f-0311209b3383"
         ],
         "matchedDetectionRules":[
            
         ]
      },
 ...

To retrieve the specific checkout, you will need to clone the repository, i.e.

git clone https://github.com/nightfallai/nightfall-go-sdk.git

You can then checkout the specific commit using the commit hash returned by Nightfall.

cd nightfall-go-sdk
git checkout 53e59684d9778ceb0f0ed6a4b949c464c24d35ce

The Nightfall API supports the ability to send asynchronous notifications when findings are detected as part of a scan request.

The supported destinations for these notifications include external platforms, such as Slack, email, or url to a SIEM log collector as well as to a webhook server.

Nightfall issues notifications under the following scenarios:

For more information on how webhooks and asynchronous notifications are used please see our guides on:

In order to accept requests from Nightfall, a Webhook server must use a signing key to verify requests.

Select the Developer Platform > Manage API Keys using the navigation bar on the left side of the page. You will see the Webhook signing section:

Unlike the API Key, it is possible to reveal the signature via the "eye" icon furtherest to the left of the three icons displayed.

You may copy the current value to your clipboard with the "copy" icon in the center of the three icons displayed.

You may also regenerate the key with the circular arrow icon furthest to the right.

Use this value as shown in the code examples that are used in the following sections.

Nightfall supports Detectors that will scan for file names, file types, and file finger prints.

Detecting File Names

In addition to scanning the content of files, you may configure the Detectors to scan file names as well.

This is done through the “scope” attribute of a Detector.

The scope attribute allows you to scan either within file contents, the file name, or both the file contents and file name.

File extensions can be scanned for by creating a Regular Expression type custom Detector with a scope to scan only file names ("File") or both the content and file name ("ContentAndFile"), as shown in the example request below.

Note that confidence sensitivity does not apply to file names. Sensitive findings will always be reported on.

Detecting File Types

Nightfall’s File Type detection allows you to implement compliance policies that detect and alert you when particular file types that are not allowed in a given location are discovered.

This functionality is implemented by creating a specific Detector called a “File Type Detector”

To create a File Type Detector, select “Detectors” from the left hand navigation and click the button labeled “+New Detector” in the upper right hand corner. From there a drop down list of Detector types will be displayed which will include the “File Type” Detector type.

You can either scroll through the list of mime-types in the select box or you may type in a portion of the mime-type and the contents of the select box will be filtered to match your input.

File Type Detectors vary from other Nightfall Detectors in that the attributes of scope and confidence are not relevant to File Type Detectors

Detecting Files Through Fingerprinting

Nightfall allows you to discover the location of specific files that you have deemed sensitive and want to avoid sharing.

This discovery is done through document fingerprinting. Fingerprinting is the process of algorithmically creating a unique identifier for a file by mapping the data of the document to a signature that can be recalled quickly. This allows the file to be identified in a manner akin to how human fingerprints uniquely identify individual people.

This functionality is achieved in Nightfall by creating a specific Detector type called a File Fingerprint Detector.

The Fingerprint Detector allows you to create a fingerprint for one more files (a sort “handful” of fingerprints, if you would).

To create a Fingerprint Detector, select “Detectors” from the left hand navigation and click the button labeled “+New Detector” in the upper right hand corner. From there a drop down list of Detector types will be displayed which will include the “Fingerprint” Detector type.

When you create a File Fingerprint Detector you can upload up to 50 files that need to be fingerprinted. The file size limit is 25MB.

Once the fingerprint is generated, the actual content of the file is discarded so no sensitive content is stored on Nightfall’s system.

These Detectors may only be created through the console.

Nightfall offers many useful features beyond its detectors, including:

You can use the surrounding context of a match to help determine how likely it is that your potential match should actually be considered as a match by adjusting its confidence rating.

You can also tell the Detection Rule to return a portion of the surrounding context for manual review.

In the following example, in addition to providing a regular expression to match Social Security Numbers, we also look to see if someone has written the text “SSN” before and after the match, which might be a label indicating it is indeed a social security number. In which case, we change our confidence score to “VERY_LIKELY.” We then provide two possible matches in our payload, the first of which contains the string “SSN”.

curl --location --request POST 'https://api.nightfall.ai/v3/scan' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--header 'Content-Type: application/json' \
--data-raw '{
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                         {
                              "regex": {
                                   "isCaseSensitive": false,
                                   "pattern": "\\d{3}-\\d{2}-\\d{4}"
                              },
                              "contextRules": [
                                   {
                                        "regex": {
                                             "pattern": "SSN",
                                             "isCaseSensitive": false
                                        },
                                        "proximity": {
                                             "windowBefore": 20,
                                             "windowAfter": 20
                                        },
                                        "confidenceAdjustment": {
                                             "fixedConfidence": "VERY_LIKELY"
                                        }
                                   }
                              ],
                              "minNumFindings": 1,
                              "minConfidence": "POSSIBLE",
                              "detectorType": "REGEX",
                              "displayName": "SSN Match Detector"
                         }
                    ],
                    "name": "SSN Match Detection Rule",
                    "logicalOp": "ALL"
               }
          ],
          "contextBytes": 20
     },
     "payload": [
          "My SSN is 555-55-5555",
          "Here it is : 555-55-5555"
     ]
}
'

In the results, you can see the confidence for the first finding in the payload has been set to VERY_LIKELY while the second item is only LIKELY.

{
   "findings":[
      [
         {
            "finding":"555-55-5555",
            "beforeContext":"My SSN is ",
            "detector":{
               "name":"SSN Match Detector",
               "uuid":"6131f41c-dbdd-47a9-8c6f-1819c9baf388"
            },
            "confidence":"VERY_LIKELY",
            "location":{
               "byteRange":{
                  "start":10,
                  "end":21
               },
               "codepointRange":{
                  "start":10,
                  "end":21
               },
               "rowRange":null,
               "columnRange":null,
               "commitHash":""
            },
            "matchedDetectionRuleUUIDs":[
               
            ],
            "matchedDetectionRules":[
               "SSN Match Detection Rule"
            ]
         }
      ],
      [
         {
            "finding":"555-55-5555",
            "beforeContext":"Here it is : ",
            "detector":{
               "name":"SSN Match Detector",
               "uuid":"6131f41c-dbdd-47a9-8c6f-1819c9baf388"
            },
            "confidence":"LIKELY",
            "location":{
               "byteRange":{
                  "start":13,
                  "end":24
               },
               "codepointRange":{
                  "start":13,
                  "end":24
               },
               "rowRange":null,
               "columnRange":null,
               "commitHash":""
            },
            "matchedDetectionRuleUUIDs":[
               
            ],
            "matchedDetectionRules":[
               "SSN Match Detection Rule"
            ]
         }
      ]
   ],
   "redactedPayload":[
      "",
      ""
   ]
}

Using regex to identify long patterns in images can be challenging because OCR systems. In such cases, even Nightfall may not achieve 100% character-by-character accuracy. To improve results, you must introduce higher levels of flexibility into your regex patterns to accommodate common OCR inconsistencies. Here are some typical OCR challenges to keep in mind:

• Spell-check noise: Spell-checking tools can add artifacts like red underlines, which may interfere with text recognition.

• Character ambiguity:

  • The digit 0 may be misinterpreted as the letter O (or vice versa), depending on the font.

  • The character l (lowercase L) may be read as the digit 1.

  • The letter B may appear as the digit 8.

• Underscore handling: An underscore (_) is sometimes interpreted as a space, particularly when spell-check artifacts are present.

• Line wrapping: OCR may introduce unexpected newlines when text wraps across multiple lines.

• Periods and punctuation: Spell-check artifacts or font issues may result in extraneous periods (.) or other punctuation being added to the output. En dash (–) and hyphens (-) may be interchanged.

For reference, OCR tools like Tesseract typically achieve 85-98% character accuracy for similar input, and our system operates within a similar range. Given this, tuning your regex to be more forgiving (e.g., allowing for optional characters or slight variations) can significantly improve detection rates.

Example Regex (original and loosened)

original: ATATT3xFfGF0[A-Za-z0-9=_\-]*[=A-Za-z0-9]{9}

loosened: ATATT[A-Za-z0-9_\-– @.\n=]*[A-Za-z0-9_\- @.\n]{7,11}

• shortened the literal match prefix

• excluded the the literal zero (0) from the prefix

• added period (.) and newline () chars

• relaxed the char length

You can use the posture management APIs to search posture events, fetch posture events and also event details. Additionally, you can also view details of the user (actor) whose actions triggered an event, and details of the asset that triggered an event.

You can use the exfiltration APIs to search exfiltration events, fetch exfiltration events and also event details. Additionally, you can also view details of the user (actor) whose actions triggered an event, and details of the asset that triggered an event.

Protected health information (PHI), also referred to as personal health information, describes a patient's medical history — including ailments, various treatments, and outcomes. PHI may include:

• demographic information

• test and laboratory results

• mental health conditions

• insurance information

The Health Insurance Portability and Accountability Act (HIPAA) of 1996 is the primary law that oversees the use of, access to, and disclosure of PHI in the United States. HIPAA lists 18 different personal information identifiers (PII) that, when paired with health information, become PHI. In order to more accurately detect potential PHI, Nightfall has introduced specific new detectors that allow for specialized combinations.

These HIPAA PII and PHI-specific detectors intelligently aggregate Nightfall's built-in detector to ensure compliance with governing law. For example, finding a patient's name in a document or message is not considered HIPAA PII as it does not uniquely identify an individual, many people can share the same name. However, the information would be considered HIPAA PII if the patient's name and address were in the same message.

Specific PHI and HIPAA PII can be detected with greater confidence, especially as they relate to specific medical codes or terms in association with specific logical combinations of other PII. For instance when the patient's name and date of birth or a person's name and street address or any of a set of particular PII (phone number email, SSN, etc) it would be considered HIPAA PII.

If the combined detectors all match with a confidence of "Very Likely" it would match our "HIPAA PII Very Likely" Detection Rule. Otherwise if these detectors match with a confidence of "Likely" it would match our "HIPAA PII Likely" Detection Rule.

Alternatively when any of the above PII options are found in conjunction with a specific set of medical related codes or terms (IDC Codes, FDA Drug Names or Codes, Procedures), that finding could be flagged as PHI.

When all the detectors within these PHI Detection Rules make findings that have a confidence of "Very Likely," that would match our "PHI Very Likely" Detection Rule, while if some are all are met with a confidence of "Likely" that would match our "PHI Likely" Detection Rule.

While using Nightfall's Scan API, you may encounter some of the common errors outlined below. Try following the provided troubleshooting steps.

HTTP Error Codes

The following error codes are returned as part of a standard HTTP response.

Policies allow customers to create templates for their most common workflows such as sending alerts when detection rules are triggered.

The alertConfig can be either:

• an email address

• a Slack channel

• a webhook url

• a url to a SIEM host as well authentication and other headers

Below is a simple example of a payload with a policy that will send alerts to an email address that you would use with our endpoint for .

{     
  "policy": {
    "detectionRules": [
               {
                    "detectors": [
                        {
                            "detectorType": "NIGHTFALL_DETECTOR",
                            "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER",
                            "minNumFindings": 1,
                            "minConfidence": "LIKELY",
                            "displayName": "US Social Security Number"
                        }
                    ],
                    "name": "SSN Match Detection Rule",
                    "logicalOp": "ALL"
               }
          ],
    "contextBytes": 5,
    "alertConfig": {
      "email": {
        "address": "youremail@nightfall.ai"
      }
    }
  },
  "payload": [
        "The customer's social security number is 555-55-5555",
        "No SSN in this string"
   ]
}

You will receive the following response:

{
    "findings": [
        [
            {
                "finding": "555-55-5555",
                "beforeContext": "r is ",
                "detector": {
                    "name": "US Social Security Number",
                    "uuid": "e30d9a87-f6c7-46b9-a8f4-16547901e069"
                },
                "confidence": "VERY_LIKELY",
                "location": {
                    "byteRange": {
                        "start": 41,
                        "end": 52
                    },
                    "codepointRange": {
                        "start": 41,
                        "end": 52
                    },
                    "rowRange": null,
                    "columnRange": null,
                    "commitHash": ""
                },
                "matchedDetectionRuleUUIDs": [],
                "matchedDetectionRules": [
                    "SSN Match Detection Rule"
                ]
            }
        ],
        []
    ],
    "redactedPayload": [
        "",
        ""
    ]
}

Note that you may also use a pre-defined policy defined under Developer Platform > Overview > Policies by copying the Policy UUID and sending a request as shown below.

curl --request POST \
     --url https://api.nightfall.ai/v3/scan \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <InsertYourApiKeyHere>' \
     --header 'content-type: application/json' \
     --data '
{
     "policyUUIDs": [
          "2b2ced32-80c3-4a89-8757-489743ec4640"
     ],
     "payload": [
          "My payload to scan"
     ]
}
'

The following payload will be sent to the given email address with the subject "🚨 Findings Detected by Nightfall! 🚨" as an attachment with the name nightfall-findings.json:

{
  "redactedPayload": [
    "", 
    ""
  ], 
  "findings": [
    [
      {
        "confidence": "LIKELY", 
        "matchedDetectionRules": [
          "SSN Match Detection Rule"
        ], 
        "matchedDetectionRuleUUIDs": [], 
        "location": {
          "codepointRange": {
            "start": 41, 
            "end": 52
          }, 
          "rowRange": null, 
          "byteRange": {
            "start": 41, 
            "end": 52
          }, 
          "columnRange": null, 
          "commitHash": ""
        }, 
        "finding": "555-55-5555", 
        "detector": {
          "name": "SSN Match Detector", 
          "uuid": "7270ccd5-07c5-44e5-b280-c768e0028963"
        }, 
        "beforeContext": "r is "
      }
    ], 
    []
  ]
}

This attachment has the same content as the response payload to the initial request.

Note that the sender address will be no-reply@nightfall.ai

This email address will not respond to messages sent to it.

Using Webhooks with Policies

Policies also allow you to send findings to a callback designated URL using the url property of the alertConfig object.

Below is what Webhook URL should like in your policy's alertConfig in a payload sent to our endpoint used for scanning plain text.

{     
  "policy": {
    "detectionRuleUUIDs": [
      "c8d43147-0a63-4c01-8a57-83d8108422f5"
    ],
    "alertConfig": {
        "url": {
            "address": "https://mywebhookurl.com"
        }
    }
  },
  "payload": [
        "The customer's social security number is 555-55-5555"
   ]
}

Using Slack Channels With Policies

Another option supported by Policies is sending finding data to a designated Slack channel.

Below is a sample payload for scanning plain text.

{     
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                        {
                            "detectorType": "NIGHTFALL_DETECTOR",
                            "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER",
                            "minNumFindings": 1,
                            "minConfidence": "LIKELY",
                            "displayName": "US Social Security Number"
                        }
                    ],
                    "name": "Simple SSN Match Detection Rule",
                    "logicalOp": "ALL"
               }
          ],
        "alertConfig": {
            "slack": {
                "target": "#securityalert"
            }
        }
    },
     "payload": [
          "The customer's social security number is 555-55-5555",
          "No SSN in this string"
     ]
}

Below is an example as to how the violation will appear in Slack.

Sending Alerts to SIEMs and other HTTP Event Collectors

SIEM (pronounced “sim”) is a combination of security information management (SIM) and security event management systems. SIEM technology collects event log data for analysis in order to provide visibility into network activity.

It is possible to send findings from a policy to a SIEM service such as LogRhythm, SumoLogic, or Splunk using the siem alertConfig.

This configuration will require a URL to a collector that uses an HTTPS endpoint.

Note that the URL for the siem alertConfig must:

• use the HTTPS scheme

• be able to accept requests made with the POST verb

• respond with a 200 status code upon receipt of the event

See the documentation for your SIEM service for how to set up this URL.

Unlike the url alertConfig option, the siem alertConfig does not require that the endpoint for the service implement a custom challenge response. Events sent to the siem alertConfig endpoint contain a subset of what is sent to the url alertConfig. Furthermore the findings are sent in a redacted form similar to Slack or email alerts.

In addition to the URL, you may provide headers such as those that are used for authorization.

The headers in the SIEM alertConfig are divided into sensitiveHeaders and plainTextHeaders header mappings.

The sensitiveHeaders field is specifically for header values like authentication. Nightfall ensures that these header values are always hidden in our service. They are never logged or saved in analytic events.

You can use plainTextHeaders for all other type of information you would like passed along with Nightfall alerts to you HTTP endpoint. Nightfall assumes that the values stored plainTextHeaders do not contain any sensitive information so we do not take any action to hide or protect these values.

Below is an example of a payload using a siem alertConfig.

{
     "policy": {
          "detectionRules": [
               {
                    "detectors": [
                         {
                              "nightfallDetector": "CREDIT_CARD_NUMBER",
                              "detectorType": "NIGHTFALL_DETECTOR",
                              "minConfidence": "POSSIBLE",
                              "minNumFindings": 1
                         }
                    ],
                    "logicalOp": "ALL"
               }
          ],
		"alertConfig": {
		   "email": {
			"address": "<your email>"
		   },
		   "siem": {
				"sensitiveHeaders": {
					 "Authorization": "Splunk <your token value>"
				},
				"address": "https://http-inputs-<yourhost>.splunkcloud.com:8088/services/collector/event"
		   }
		}
     },
     "payload": [
          "4916-6734-7572-5015 is my credit card number",
          "This string does not have any sensitive data",
          "my api key is yr+ZWwIZp6ifFgaHV8410b2BxbRt5QiAj1EZx1qj and my 💳 credit card number 💰 is 30204861594838"
     ]
}

Other Policy Features

Using Redaction Within a Policy

A policy may be configured with default redaction rules as a defaultRedactionConfig that will affect the content of the redactedPayload field of the content that is sent to the alert locations specified in the policy alertConfig. Note that this redaction does not affect the findings themselves.

These redaction rules will be applied to Detection Rules that do not have a specified redaction configuration.

The redactionConfig specified must be one and only one of the four available redaction types:

• maskConfig

• infoTypeSubstitutionConfig

• substitutionConfig

• cryptoConfig

{
  "policy": {
    "detectionRules": [
      {
        "detectors": [
          {
            "detectorType": "NIGHTFALL_DETECTOR",
            "nightfallDetector": "US_SOCIAL_SECURITY_NUMBER",
            "minNumFindings": 1,
            "minConfidence": "LIKELY",
            "displayName": "US Social Security Number"
          }
        ],
        "name": "Simple SSN Match Detection Rule",
        "logicalOp": "ALL"
      }
    ],
      "defaultRedactionConfig": {
        "maskConfig": {
          "charsToIgnore": [
            "-"
          ],   
            "maskingChar": "#",
              "numCharsToLeaveUnmasked": 4,
                "maskLeftToRight": true
        }
      },
        "contextBytes": 5,
          "alertConfig": {
            "email": {
              "address": "eric@nightfall.ai"
            }
          }
  },
    "payload": [
      "The customers social security number is 555-55-5555",
      "No SSN in this string"
    ]
}

Using Context Bytes Within a Policy

In additional to a defaultRedactionConfig it is possible to set the number of bytes to include as before and after a given finding as the contextBytes. This context can provide meaning to how the finding appears within the text to allow human readers to better understand the meaning of the finding. The maximum value for contextBytes is 40.

Firewall for AI DLP APIs enables developers to write custom code to sanitize data anywhere–RAG data sets, analytics data stores, data pipelines, and unsupported SaaS applications.

If you are using Nightfall SaaS apps, you can use APIs to fetch violations, search through the violations, and fetch specific findings within the Violations. To scan data in any custom apps or cloud infrastructure services like AWS S3, you must use the APIs in the DLP APIs - Firewall for AI Platform section.

To prevent misuse and ensure the stability of our platform, we enforce a rate limit on an API Key and endpoint basis, similar to the way many other APIs enforce rate limits.

When operating under our Free plan, accounts and their corresponding API Keys have a rate limit of 5 requests per second on average, with support for bursts of 15 requests per second. If you upgrade to a paid plan – the Enterprise plan – this rate increases to a limit of 10 requests per second on average and bursts of 50 requests per second.

The Nightfall API follows standard practices and conventions to signal when these rate limits have been exceeded.

Successful requests return a header X-Rate-Limit-Remaining with the integer number of requests remaining before errors will be returned to the client.

Request Rate Limiting

Your Request Rate Limiting throttles how frequently you can make requests to the API. You can monitor your rate limit usage via the `X-Rate-Limit-Remaining` header, which tells you how many remaining requests you can make within the next second before being throttled.

Quotas

Your Quota limits how many requests you can make within a given period. Your current remaining quota and the end of your current quota period are denoted by the following response headers.

For the free plan, we allow 5 requests per second and 10000 requests in a day.

An Exclusion Rule allows you to refine a Detector to make sure false positives are not surfaced by Nightfall.

For instance you may want to detect whether credit card numbers are being shared inappropriately in your organization. However, there may be cases where members of your QA are sharing test credit card numbers, which should not be considered a violation and should be ignored by Nightfall.

In the following example, we define a Detector with a regular expression to match credit cards.

We then add an exclusion for some known test credit cards.

As the resulting payload shows, only the 3rd provided Credit Card number matches because the first two items in the payload are included in our ExclusionRules word list.

Our playground environment allows you to:

• Generate sample data for DLP testing.

• Explore a sample app built on our APIs

This guide describes how to use Nightfall with the Ruby programming language.

The example below will demonstrate how to use Nightfall’s text scanning functionality to verify whether a string contains sensitive PII using the Nightfall Python SDK.

To follow along, you will need:

• A Nightfall API Key

• An existing Detection Rule

• Data to scan. Note that the API interprets data as plaintext, so you may pass it in any structured or unstructured format.

• A local Ruby 2.6 or greater environment.

Start by creating a new file called nightfall_demo.rb

Now we will walk through the code step by step. If you'd like to skip ahead you can see the complete code sample at the bottom of this page.

We will be using a few built-in Ruby libraries to run this sample API script.

First, we will load some environment variables that will be used to interact with the Nightfall API. NIGHTFALL_API_KEY should be your Nightfall API Key, and NIGHTFALL_DETECTION_RULE_UUID should be the UUID for your existing Nightfall condition set.

Next, we will construct our payload to scan as an array. You can replace this with any data you'd like, or read plaintext from a file.

Next, we build the HTTP request headers and body using the environment variables that we previously defined.

Next, we build the HTTP object and make a request to the Nightfall API.

Lastly, we make the API request and process the response from Nightfall. If there are sensitive findings in the response we pretty-print them to the console. If there are no findings, we print a message to the console. Otherwise, if there is a problem with the HTTP request we print the status code and message to the console.

Usage

Now we can run our script:

If there are sensitive findings based on your Nightfall detection rule, you should see output similar to this in your console, corresponding to each of the 3 items inputted to scan in the payload.

Complete Sample Code

For your convenience, the complete Ruby code sample is shown below.

This section consists of various documents that assist you in scanning various popular SaaS GenAI services and frameworks using Nightfall APIs.

• OpenAI Prompt Sanitization Tutorial

• Anthropic Prompt Sanitization Tutorial

• LangChain Prompt Sanitization Tutorial