Learn how to create custom detectors in Nightfall.
Nightfall allows you to create custom detectors that match your organization's requirements. If none of the existing Nightfall detectors match your requirements, you can use this option to create a custom detector.
Nightfall provides you with four types of custom detectors.
Dictionary: This Detector allows you to upload a text file. The text file must have a specific token which is sensitive data for your organization. The Detector looks for this token in all the data and if a match is found, it flags it as sensitive data found.
File Type: This Detector allows you to select a MIME (Multipurpose Internet mail extension). (roughly speaking, every file on the Internet is recognized by its format which is called MIME). The Detector scans all your data to check if it has a file with the selected MIME type.
File Fingerprint: This Detector accepts a file as input. You can add all the content that you feel is sensitive to your organization. The detector uses this file as the base and scans your data to check if any matching content is present. If found, it flags it as sensitive data.
Regular Expression: This Detector allows you to define regular expressions matching the RE2 syntax. The Detector scans all your data to find any content that matches the regular expression. If found, it flags it as sensitive data.
Extend a Nightfall Detector: This Detector type allows you to customize an out-of-the-box Detector by defining Context Rules and Token rules.
Apart from the above custom Detectors, you can also use the Request a new detector option to request an entirely new custom Detector created by Nightfall.
To view only custom detectors created by you, click the CUSTOM filter in the left pane.
To learn more about how to create each of the custom detectors, you can refer to the following documents.
Learn how to create a Regular Expression detector in Nightfall.
A regular expression detector allows you to define a regular expression pattern. Once you define a pattern, Nightfall scans your data to check if there is anything that matches the given pattern. If a match is found, it is flagged as sensitive data. You can also refer to to generate regular expressions.
To use the Regular Expression detector effectively, Nightfall provides you with two special rules; Context Rules and Exclusion Rules. When Nightfall scans your data and finds some sensitive data, these rules are used to further scrutinize the sensitive data to be sure that the data is sensitive. Context rules and Exclusion rules are optional and you can define them only if you wish to have added filtration on sensitive data to be sure that data that is termed as sensitive, is sensitive.
Context rules are a set of hot and cold words that help Nightfall to effectively identify sensitive data.
Exclusion rules define specific scenarios in which data flagged by a detector should be excluded from the list of findings.
You can find these rules in the Regular Expression detector. You must click + Rule to create a Context rule or Exclusion rule.
Context rules help Nightfall identify sensitive data by providing data about the text surrounding sensitive data. You can define a regular expression pattern to match the data that usually surrounds sensitive data. You can then instruct Nightfall as to where this surrounding data can be generally found. It can be found before sensitive data, after sensitive data, or both before and after sensitive data. You can also define confidence level settings when a match is found.
For example, credit card data is sensitive data. If a user shares it accidentally online, then they generally draft the following phrases before disclosing the credit card number.
After disclosing the credit card information, users generally draft the following messages.
You can also choose to make your regular expression case-sensitive by selecting the Case sensitive check box.
The terms "Finding" and "Sensitive data" are synonyms to each other.
Windows Setting: Once you create the regular expression to define the possible phrases that can surround the sensitive data, you can use the Windows setting to instruct Nightfall as to where exactly this surrounding text can exist around the sensitive data (before, after). Nightfall provides you the following Windows settings.
Before the finding: You can use this setting to define the number of characters before the prospective sensitive data at which surrounding text can be found. For example, if you define this setting as 20 characters before the finding, and if Nightfall detects the prospective sensitive data on line 4 column 60, it looks at the data on line 4 and from column 40 to check if it matches the defined regular expression for Context rules.
After the finding: You can use this setting to define the number of characters after the prospective sensitive data at which surrounding text can be found. For example, if you define this setting as 20 characters after the finding, and if Nightfall detects the prospective sensitive data on line 4 column 60, it looks at the data on line 4 and from column 60 to check it that matches the defined regular expression for Context rules.
Before or after the finding: You can use this setting to define the number of characters before or after the prospective sensitive data at which surrounding text can be found. For example, if you define this setting as 20 characters before or after the finding, and if Nightfall detects the prospective sensitive data on line 4 column 60, it looks at the data on line 4 and from column 60 and also on data at line 4 from column 40 to check it any of the two match the defined regular expression for Context rules.
Confidence Change: If Nightfall finds any data to be sensitive and if this prospective sensitive data matches the Context rules as well, you can then define Confidence change settings to instruct Nightfall on what to label this prospective sensitive data. Nightfall provides you with four Confidence change settings
Exclude: If you set the Confidence Change setting to Exclude, Nightfall does not consider the prospective Finding to be actually sensitive (even though it matched both; sensitive data's regular expression and also the Context rules' regular expression and Windows setting) and excludes it from Findings. You can use this setting to eliminate false positive findings. For example, consider that one of your prospective customer is testing your API and you need to share an API key for this testing. You share a dummy API key and do not wish this API key to be flagged as sensitive data. You can ask your employees to draft a specific phrase before sharing the API key (something like, the dummy API key is ). You can then define a context rule for this phrase, set the Windows setting to define where exactly it can be found, and then set Exclude as the Confidence change setting, to exclude this dummy API key from being flagged as sensitive data.
Once you create the regular expression for Context rule, define the Windows setting and the Confidence Change settings, you must click Save to create the context rule.
Once you save the Context rule, you can see that Nightfall creates a sentence which defines the summary of your Context rule. You can choose to edit or delete the rule, if required.
Exclusion rules specify data that is not sensitive in nature (false positive findings). If you feel that Context rules cannot help you to stop false positive findings being logged, you can use Exclusion rules. You can also use Exclusion rules and also Context rules together to eliminate the possibility of getting false positive findings.
Nightfall provides you four methods by which you can define Exclusion rules.
For example, consider that one of your prospective customer is testing your API and you need to share an API key for this testing. You create a series of dummy API keys (say ABCD1234, ABCD2345, ABCD3124, ABCD5412, and so on). You can create a regular expression to match each of these dummy APIs and select the Full match option. Nightfall does not treat any of these as sensitive data and excludes them.
You can observe that all the API keys start with the term ABCD. In such cases, you can use the Partial match option and just define regular expression for any one of the API keys. This is helpful if you have hundreds of dummy APIs, which have some common data between them. You need not create a regular expression for each of them.
Since all the dummy APIs start with ABCD, if Nightfall detects any of these API keys, it straight away excludes them. In this case, you must ensure that none of the live APIs have the term ABCD in them.
If you are facing a hard time creating regular expressions for content that needs to be excluded, you can use the dictionary option. This option allows you to define phrases that must not be considered as sensitive data. You can use this option to directly define commonly used passwords, dummy API keys, and so on.
The Dictionary exclusion rule has three options.
Manual Entry: In this option you can directly enter the password or API key to be excluded and press enter. You can add multiple items (need not have any delimiter). You can choose to match the string partially or fully.
Existing Dictionary: In this option, you can choose an existing Dictionary detector as the exclusion token. When you select the Existing Dictionary option, a new drop-down menu is displayed. You can select any of the previously created dictionary detector.
The File Type exclusion rule allows you to exclude a specific file type. If Nightfall finds that the prospective sensitive data is part of one of the file types defined in this section, it excludes it.
File type provides you two options.
New File Type: This option allows you to define new file type(s) that must be excluded.
Existing Custom Detector: This option allows you to use a File Type Detector created previously as the exclusion token. When you select this option, a new drop-down menu is displayed which consists of the list of previously created File type detectors. You can select the required detector.
This option allows you to upload a file with all the data that you feel is not sensitive. Nightfall checks if any of the prospective findings match the data in this file. If a match is found, the data is not considered to be sensitive and excluded from Findings. You can also upload an existing File Fingerprint detector as the input token.
There are two options in this rule.
Upload New File: This option allows you to upload a new file.
Existing Custom Detector: This option allows you to use a previously created custom File Fingerprint detector.
You can execute the following steps to create a Regex detector.
Navigate to the Detectors section from the left pane.
Click + Custom Detector and select Regular Expression.
Enter a name for your custom Detector in the Name field.
(Optional) Enter a description for the Detector in the Description field.
(Optional) Select the Case sensitive check box if you wish to match case sensitive pattern.
(Optional) Click + Rule under the CONTEXT RULES section and define regular expression pattern for Context rule.
(Optional) Click + Rule under the EXCLUSION RULES section and define regular expression pattern for Exclusion rule.
Click ADD.
Context rules involve creating regular expressions to define these types of phrases that surround the sensitive data. You can use to generate regular expressions to define data that surrounds sensitive data. You can define the Context rule regular expression in the Pattern to match text box.
Possible: If you set the Confidence Change setting to Possible, Nightfall classifies the prospective finding as an actual sensitive data and this finding is logged on the page as a Possible finding (around 20-30% chance of data actually being sensitive in nature). For example, in the above scenario for the Exclude option, if you wish to get notifications even for dummy APIs, to ensure that your employees might have not shared an actual API key instead of a dummy key, you can use the Possible setting. This setting logs the finding on the Violations page as Possible finding. You can check the finding on the Violations page and if its dummy API key, and not a live API key, you can set it to Ignore the finding from the Violations page.
Likely: If you set the Confidence Change setting to Likely, Nightfall classifies the prospective finding as an actual sensitive data and this finding is logged on the page as a Likely finding (around 50-60% chance of data actually being sensitive in nature). For example, consider the scenario described above for the Exclude option. If you feel that sometimes your employees may share a dummy API key with prospective customers without using the mandatory phrase (the dummy API key is). In such cases, you they might have used some other phrases. You cannot really be sure if they transmitted dummy API keys or shared live API keys. In such cases, you can set the Confidence to be Likely (50-60% chance of data actually being sensitive). Nightfall logs it as a Finding in the Violations page with Likely confidence level. You can view the Violations page, verify this Finding which is tagged as Likely and take appropriate actions based on whether the data is actually sensitive or not.
Very Likely: If you set the Confidence Change setting to Very Likely, Nightfall classifies the prospective finding as an actual sensitive data and this finding is logged on the page as a Very Likely finding (around 80-90% chance of data actually being sensitive in nature). For example, consider the scenario described above for the Exclude option. Apart from the dummy API key there are lots of sensitive data in your organization. If an employee accidentally shares such information, they generally use phrases like our organization's API keys are, our API OAuth key is, the password is... and so on. You can create a regular expression for all such phrases, define the Window settings for them and then set the Confidence level as Very Likely. Nightfall logs such findings on the Violations page as Very Likely findings (80-90% chance of data actually being sensitive). You can navigate to the Violations page and take appropriate actions.
You can use Regex method and define a regular expression. If a prospective sensitive data matches the defined regex, it is immediately excluded from being considered as a Finding. Nightfall also allows you to match either the whole regular expression or partial regular expression. You can use to generate regular expressions.
Upload Dictionary: With this option, you can upload a new dictionary. The process is same as in case of dictionary detector. You can refer to the document for details.
Enter the Regular expression pattern in , in the Regex field.
Learn how to create a File Fingerprint detector in Nightfall.
A File Fingerprint detector allows you to upload a file. You must enter the exact data that is sensitive to your organization, in this file. Nightfall scans your data to check if there is an exact match between the contents of the uploaded file and your organization's data. If a match is found, it is flagged as sensitive data.
To create a File Fingerprint Detector:
Navigate to the Detectors section from the left pane.
Click + Custom Detector and select File Fingerprint.
Enter a name for your custom Detector in the Name field.
(Optional) Enter a description for the Detector in the Description field.
Upload or drag and drop the text file whose content needs to be matched.
Click Add.
Learn how you can extend the functionality of an existing Nightfall detector.
You can customize a Nightfall detector to meet your organization's requirements. Once you customize a Nightfall detector, it is considered to be a custom detector. The Nigytfall detector that you used and customized, continues to exist as is.
Nightfall provides you with two customization options. Context rules and Exclusion rules. You must use at least one of these two rules to customize a Nightfall detector. The procedure to use Context rules and Exclusion rules is the same as in the case of Regular Expression Detector. You can learn how to configure Context rules and Exclusion rules from the#understanding-context-rules-and-exclusion-rules document.
To extend a Nightfall Detector:
Navigate to the Detectors section from the left pane.
Click + Custom Detector and select Extend a Nightfall detector.
Select the Nightfall detector that you wish to customize.
Enter a name for the customized detector in the Name field.
(Optional) Enter a description for the detector in the Description field.
Click Next.
Click + Add Context Rule to configure the context rule.
Click Save to save the rule.
Click Next.
Click + Add Token Rule to configure token rules.
Click Save to save the rule.
Click Next.
It is mandatory to either define the Context rule or the Token rule.
Review the configurations and click Add.
Learn how to create a File Type detector in Nightfall.
A File Type detector allows you to select MIME (multi-purpose mail extension) types. The selected MIME types are considered as tokens and Nightfall scans your data to identify if there is anything that matches the MIME token. If a match is found, it is flagged as sensitive data.
To create a File Type Detector:
Navigate to the Detectors section from the left pane.
Click + Custom Detector and select File Type.
Enter a name for your custom Detector in the Name field.
(Optional) Enter a description for the Detector in the Description field.
Click Add File Type and select an MIME type.
(Optional) Repeat step 3 to add multiple MIME types to your rule.
(Optional) Click the delete icon to delete an MIME type, if you do not require it.
Click Add.
Learn how to create a Dictionary detector in Nightfall.
A dictionary detector can detect a specific piece of sensitive data (referred to as a token). The token can be any type of sensitive data like a password, API key, IP address, PII, and so on. You must add the token to a file and then upload the file while creating the dictionary. Nightfall scans your data to check if the token present in the uploaded file exists in your data. If a match is found, it fis flagged as sensitive data exposure.
To create a Dictionary Detector:
Navigate to the Detectors section from the left pane.
Click + Custom Detector and select Dictionary.
Enter a name for your custom Detector in the Name field.
(Optional) Enter a description for the Detector in the Description field.
Upload or drag and drop the text file which contains a token.
(Optional) Select the Match substrings check box to match the sub-strings of the text file.
Click Add.
