Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Nightfall offers many useful features beyond its detectors, including:
The ability to use Context Rules and Exclusion Rules to narrow the scope of matches.
The ability to create Redactions in a way that is highly configurable so that sensitive data is appropriately obfuscated.
The ability to create Policies that determine how leaks of sensitive information should be mitigated (i.e. through alerts sent to email or Slack).
In this example, we'll walk through making a request to the scan endpoint.
The endpoint inspects the data you provide via the request body and reports any detected occurrences of the sensitive data types you are searching for.
Please refer to the API reference of the scan endpoint for more detailed information on the request and response schemas.
In this sample request, we provide two main fields:
a policy and its detection rules that we want to use when scanning the text payload
a list of text strings to scan
In the example below we will use a Detection Rule that has been configured in the user interface by supplying its UUID.
The aggregate length of all strings in payload list must not exceed 500 KB, and the number of items in the payload may not exceed 50,000.
Alternatively you may define your policy in code by using a built in Nightfall detector from the Detector Glossary as follows:
See Creating an Inline Detection Rule for more information about how policies and detection rules may be defined through code.
Executing the curl request will yield a response as follows.
The API call returns a list, where the item at each index is a sublist of matches for the provided detector types.
The indices of the response list correspond directly to the indices of the list provided in the request payload.
In this example, the first item in the response list contains a finding because one credit card number was detected in the first string we provided. The second item in the response list is an empty list because there is no sensitive data in the second input string we provided. The third item in the returned list contains multiple findings as a result of multiple Detectors within the Detection Rule being triggered.
You can read further about the fields in the response object in the Nightfall APIs.
You can use the surrounding context of a match to help determine how likely it is that your potential match should actually be considered as a match by adjusting its confidence rating.
You can also tell the Detection Rule to return a portion of the surrounding context for manual review.
In the following example, in addition to providing a regular expression to match Social Security Numbers, we also look to see if someone has written the text “SSN” before and after the match, which might be a label indicating it is indeed a social security number. In which case, we change our confidence score to “VERY_LIKELY.” We then provide two possible matches in our payload, the first of which contains the string “SSN”.
In the results, you can see the confidence for the first finding in the payload has been set to VERY_LIKELY while the second item is only LIKELY.
An Exclusion Rule allows you to refine a Detector to make sure false positives are not surfaced by Nightfall.
For instance you may want to detect whether credit card numbers are being shared inappropriately in your organization. However, there may be cases where members of your QA are sharing test credit card numbers, which should not be considered a violation and should be ignored by Nightfall.
In the following example, we define a Detector with a regular expression to match credit cards.
We then add an exclusion for some known test credit cards.
As the resulting payload shows, only the 3rd provided Credit Card number matches because the first two items in the payload are included in our ExclusionRules word list.
In addition to using pre-defined Detection Rules, you may define Detection Rules within the body of your scan method by either supplying:
the identifier of one of Nightfall's native detectors
the UUID of an a Detector defined through the UI
a Regular Expression
a Word List.
Out of the box, Nightfall comes with an extensive library of native detectors.
In the example below two of Nightfall's native Detectors (detectorType = "NIGHTFALL_DETECTOR") are being used:
US_SOCIAL_SECURITY_NUMBER
CREDIT_CARD_NUMBER.
When defining a Detection Rule in line, you configure the minimum confidence level (minConfidence) and minimum number of times the match must be found (minNumFindings) for the rule to be triggered. .
In the payload body, you can see that we are submitting a list of three different strings to scan (payload). The first will trigger the U.S. Social Security Detector. The last will trigger the credit card Detector. The middle example will trigger neither.
For more information on the parameters related to redaction, see Using Redaction.
Below is the response payload to the previous request.
The following example shows a Detection Rule composed of two Detectors defined using regular expressions – one for the format of an International Standard Recording Code (ISRC) and one for the format of an International Standard Musical Work Code (ISWC) – matching either of which will trigger the Detection Rule (by using the logicalOp “Any”).
We will provide a payload of two strings, one of which will match the ISRC and one of which will match the ISWC.
The returned response demonstrates how findings are returned, with a finding per payload entry and the Detection Rule and Detector that matched the payload, if any.
The byte range that triggered the match is also provided. In the case of the 2nd item in the payload, since the match occurred at the beginning of the string, it has a location where the byteRange start is 0. In the case of the 3rd payload entry the location offset is 31.
The following example shows how a word list may be used instead of a regular expression.
Below is the resulting payload with the findings detected in our different payload strings.
Note that since the isCaseSensitive flag is set to "false" for the detector, so the first string in our payload matches a word from our word list.
Also note that the confidence level for a word list match defaults to "LIKELY" so you should not set a minConfidence level higher than that if you want matches to result.
The Nightfall API is capable of returning a redacted version of your scanned text when a Detector is triggered.
This functionality allows you to hide potentially sensitive information while retaining the original context in which that information appeared.
In order to redact content, when you call the scan endpoint you must provide a RedactionConfig as part of the definition of your Detection Rule.
You may specify one of the following different methods to redact content:
apply masking (e.g. asterisks)
substitute a custom phrase
substitute the name of the Detector triggered (referred to as "InfoType substitution")
use encryption
A RedactionConfig is defined per Detector in a Detection Rule, allowing you to specify a different redaction method for each type of Detector in the rule.
By default, the redaction feature will return both the sensitive finding and the redacted version of that finding. You may set the removeFinding field to true if you want only the redacted version of the finding returned in the response.
Specifying a MaskConfig as part of your RedactionConfig substitutes a character for each character in the matched text. By default the masking character is an asterisk (*). You may specify an alternate character to use instead (maskingChar).
You may also choose to only mask a portion of the original text by specifying a number of characters to leave unmasked (numCharsToLeaveUnmasked). For instance, if you want to mask all but the last 4 digits of a credit card number, set this value to 4 so that the redacted finding would be rendered as ***************4242.
In the case where you want to leave characters unmasked at the front of the string you may use the maskLeftToRight flag. This flag determines if masking is applied left to right (*****/1984) instead of right to left (01/01*****). By default, this value is false.
Below is an example of how a RedactionConfig would be configured to redact the text that triggers a DATE_OF_BIRTH Detector such that the text 01/11/1995 becomes ??/??/??95
The SubstitutionConfig substitutes a sensitive finding with the value assigned to the property substitutionPhrase.
If no value is assigned to substitutionPhrase, the finding will be replaced with an empty string.
It is possible to replace a sensitive finding with the name of the NIGHTFALL_DETECTOR that triggered it by using an InfoTypeSubstitutionConfig.
If you use the built in credit card Detector, the string 4242-4242-4242-4242 will be redacted to [CREDIT_CARD_NUMBER]
This config is only valid for Detector's with a detectorType of NIGHTFALL_DETECTOR.
A CryptoConfig will encrypt a sensitive finding with a public key (provided as the publicKey property of the config) using RSA encryption.
Note that you are responsible for passing public keys for encryption and handling any decryption of the response payload. Nightfall will not store your keys.
Below is an example of a CryptoConfig being used to redact an EMAIL_ADDRESS detector.
The results of applying redactions are returned in the response payload for requests made to the scan endpoint as both part of an array named redactedPayload as well as additional properties of the finding object.
The original input payload with redactions made inline are returned as a list of strings under the redactedPayload property. Each item in the list of redacted payloads corresponds to the list of strings in the original input payload and, if a Detector was triggered, it will contain a redacted version of that corresponding string.
If an item in the input payload did not have any findings, the entry for that index will be an empty string ("").
The redactedPayload property is omitted if no RedactionConfig was provided.
Additionally, the fields redactedFinding and redactedLocation are added to the finding object when the redaction feature is invoked.
The redactedFinding field contains the redacted version of only the text of the finding without its surrounding context. This is useful when you are masking a portion of the text that triggered a Detector.
The redactedLocation property will be returned as part of the finding that corresponds to an item in the payload. This may be distinct from the location property that is returned for a finding by default.
In the unlikely case where there are findings that overlap, Nightfall will default to replacing the text of the overlapping findings with [REDACTED BY NIGHTFALL].
The following example shows how the redaction functionality may be invoked, with a variety of different redaction methods applied to the different Detectors being used.
You can see in the response how the RedactionConfig associated with the various Detectors affects the different findings.
Note that because the 2nd item the payload matches multiple detectors, the redacted text in the redactedPayload property becomes [REDACTED BY NIGHTFALL]
Policies allow customers to create templates for their most common workflows such as sending alerts when detection rules are triggered.
These policies may be created manually through the dashboard or may be defined programmatically.
When defining an a Policy inline, in addition to specifying the Detection Rules (either by referencing the UUID of an existing Detection Rule or defining a Detection Rule and its Detectors inline), you must define an alertConfig which will determine where findings are sent.
The alertConfig can be either:
an email address
a Slack channel
a webhook url
a url to a SIEM host as well authentication and other headers
Below is a simple example of a payload with a policy that will send alerts to an email address that you would use with our endpoint for Scan Plain Text.
You will receive the following response:
Note that you may also use a pre-defined policy defined under Developer Platform > Overview > Policies by copying the Policy UUID and sending a request as shown below.
The policy object supersedes the config object. The use of config objects will still continue to be supported, but its use should be considered deprecated. If you specify policy object you cannot also specify a config object.
Also note that previous iterations of the API allowed for a simple list of policyUUIDs to be specified instead of of a policy object. This has been preserved for backwards compatibility, but it is recommended you use the policy object as it has a richer set of features. You may not use both a policyUUIDs list and a policy object.
The following payload will be sent to the given email address with the subject "🚨 Findings Detected by Nightfall! 🚨" as an attachment with the name nightfall-findings.json:
This attachment has the same content as the response payload to the initial request.
Note that the sender address will be no-reply@nightfall.ai
This email address will not respond to messages sent to it.
Policies also allow you to send findings to a callback designated URL using the url property of the alertConfig object.
This mechanism allows you to programmatically consume findings and the data sent will contain sensitive information as well as additional metadata like the location of the findings in the payload. For this reason the URL must be an HTTPS URL and the service backing it be implemented to properly respond with your Webhook signing key and act as a Webhook Server.
Below is what Webhook URL should like in your policy's alertConfig in a payload sent to our endpoint used for scanning plain text.
Another option supported by Policies is sending finding data to a designated Slack channel.
This feature requires that you have configured the Nightfall Slack integration.
Below is a sample payload for scanning plain text.
Below is an example as to how the violation will appear in Slack.
See the section on Slack in the overview on Alerting for more details.
SIEM (pronounced “sim”) is a combination of security information management (SIM) and security event management systems. SIEM technology collects event log data for analysis in order to provide visibility into network activity.
It is possible to send findings from a policy to a SIEM service such as LogRhythm, SumoLogic, or Splunk using the siem alertConfig.
This configuration will require a URL to a collector that uses an HTTPS endpoint.
Note that the URL for the siem alertConfig must:
use the HTTPS scheme
be able to accept requests made with the POST verb
respond with a 200 status code upon receipt of the event
See the documentation for your SIEM service for how to set up this URL.
Unlike the url alertConfig option, the siem alertConfig does not require that the endpoint for the service implement a custom challenge response. Events sent to the siem alertConfig endpoint contain a subset of what is sent to the url alertConfig. Furthermore the findings are sent in a redacted form similar to Slack or email alerts.
In addition to the URL, you may provide headers such as those that are used for authorization.
The headers in the SIEM alertConfig are divided into sensitiveHeaders and plainTextHeaders header mappings.
The sensitiveHeaders field is specifically for header values like authentication. Nightfall ensures that these header values are always hidden in our service. They are never logged or saved in analytic events.
You can use plainTextHeaders for all other type of information you would like passed along with Nightfall alerts to you HTTP endpoint. Nightfall assumes that the values stored plainTextHeaders do not contain any sensitive information so we do not take any action to hide or protect these values.
Below is an example of a payload using a siem alertConfig.
A policy may be configured with default redaction rules as a defaultRedactionConfig that will affect the content of the redactedPayload field of the content that is sent to the alert locations specified in the policy alertConfig. Note that this redaction does not affect the findings themselves.
These redaction rules will be applied to Detection Rules that do not have a specified redaction configuration.
The redactionConfig specified must be one and only one of the four available redaction types:
maskConfig
infoTypeSubstitutionConfig
substitutionConfig
cryptoConfig
For more information on Redactions see: Using Redaction
Below is a simple example of a payload for scanning plain text using a policy set up to use a defaultRedactionConfig
In additional to a defaultRedactionConfig it is possible to set the number of bytes to include as before and after a given finding as the contextBytes. This context can provide meaning to how the finding appears within the text to allow human readers to better understand the meaning of the finding. The maximum value for contextBytes is 40.
Leaked secrets, such as credentials needed to authenticate and authorize a cloud provider’s API request, expose company software, services, infrastructure, and data to hackers.
Nightfall has developed technology to detect secrets and label findings to speed SecOPs workflows from being clogged and eliminate false positive alerts.
Nightfall uses machine learning models trained on a large (millions of lines of code) diverse dataset (including all programming languages and application types) to ensure best-in-class secret detection accuracy and coverage.
For a growing set of the most popular services, Nightfall will:
label detected secrets by vendor and service type (returned the kind field of the response)
label detected secrets as active risks by validating supported credential types with their associated service endpoints (returned as the status of the service)
Our current solution supports the following vendors covering a diverse set of use cases, including cloud storage/infrastructure, communication, social networks, software development, banking, observability, and payment processing.
This list is not static and will continue to grow as we add support for detecting API keys from additional services. If you want to detect API keys from a service not listed below, please contact us.
Below is an example of how an AWS Key would be shown in a finding.
The following values are returned for the status field:
ACTIVE
EXPIRED
UNVERIFIED
This value will be based on what information is returned by the corresponding service when attempting the validate the key. If no data is returned fro the service, it will be considered UNVERIFIED.
To use this functionality, you use our existing built-in API_KEY detector to scan a data source such as Git Repository. Below is an example using a detection rule defined in line for a text scan.
Protected health information (PHI), also referred to as personal health information, describes a patient's medical history — including ailments, various treatments, and outcomes. PHI may include:
demographic information
test and laboratory results
mental health conditions
insurance information
The Health Insurance Portability and Accountability Act (HIPAA) of 1996 is the primary law that oversees the use of, access to, and disclosure of PHI in the United States. HIPAA lists 18 different personal information identifiers (PII) that, when paired with health information, become PHI. In order to more accurately detect potential PHI, Nightfall has introduced specific new detectors that allow for specialized combinations.
These HIPAA PII and PHI-specific detectors intelligently aggregate Nightfall's built-in detector to ensure compliance with governing law. For example, finding a patient's name in a document or message is not considered HIPAA PII as it does not uniquely identify an individual, many people can share the same name. However, the information would be considered HIPAA PII if the patient's name and address were in the same message.
Specific PHI and HIPAA PII can be detected with greater confidence, especially as they relate to specific medical codes or terms in association with specific logical combinations of other PII. For instance when the patient's name and date of birth or a person's name and street address or any of a set of particular PII (phone number email, SSN, etc) it would be considered HIPAA PII.
If the combined detectors all match with a confidence of "Very Likely" it would match our "HIPAA PII Very Likely" Detection Rule. Otherwise if these detectors match with a confidence of "Likely" it would match our "HIPAA PII Likely" Detection Rule.
Alternatively when any of the above PII options are found in conjunction with a specific set of medical related codes or terms (IDC Codes, FDA Drug Names or Codes, Procedures), that finding could be flagged as PHI.
When all the detectors within these PHI Detection Rules make findings that have a confidence of "Very Likely," that would match our "PHI Very Likely" Detection Rule, while if some are all are met with a confidence of "Likely" that would match our "PHI Likely" Detection Rule.
Our PHI Detectors may be used just like other Detectors with Policies or Inline Detection Rules.
AWS
Azure
Confluence
Confluent
Datadog
ElasticSearch
GCP
Google API
GitHub
GitLab
JIRA
JWT
Nightfall
Notion
Okta
Paypal
Plaid
Postmark
Postman
RapidAPI
Salesforce
Sendgrid
Slack
Snyk
Splunk
Square
Stripe
Twilio
Zapier
