Nightfall Documentation
  • Data Detection and Response
  • Posture Management
  • Data Exfiltration Prevention
  • Data Encryption
  • Firewall for AI
  • Data Classification and Discovery
  • Welcome
  • Introduction to Firewall for AI
    • Overview
    • Quickstart
    • Use Cases
    • Authentication and Security
  • Key Concepts
    • Entities and Terms to Know
    • Setting Up Nightfall
      • Creating API Key
      • Creating Detectors
      • Creating Detection Rules
      • Creating Policies
    • Alerting
    • Scanning Text
    • Scanning Files
      • Supported File Types
      • File Scanning and Webhooks
      • Uploading and Scanning API Calls
      • Special File Types
      • Specialized File Detectors
      • Webhooks and Asynchronous Notifications
        • Accessing Your Webhook Signing Key
        • Creating a Webhook Server
    • Scanning Features
      • Using Pre-Configured Detection Rules
        • Scanning Images for patterns using Custom Regex Detectors
      • Creating an Inline Detection Rule
      • Using Exclusion Rules
      • Using Context Rules
      • Using Redaction
      • Using Policies to Send Alerts
      • Detecting Secrets
      • PHI Detection Rules
    • Detector Glossary
    • Test Datasets
    • Errors
    • Nightfall Playground
  • Nightfall APIs
    • DLP APIs - Firewall for AI Platform
      • Rate Limits for Firewall APIs
    • DLP APIs - Native SaaS Apps
      • Policy User Scope Update API
      • Rate Limits for Native SaaS app APIs
  • Exfiltration Prevention APIs
    • Default
    • Models
  • Posture Management APIs
    • Default
    • Models
  • Nightfall Software Development Kit (SDK)
    • Overview
    • Java SDK
    • Python SDK
    • Go SDK
    • Node.JS SDK
  • Language Specific Guides
    • Overview
    • Python
    • Ruby
    • Java
  • Tutorials
    • GenAI Protection
      • OpenAI Prompt Sanitization Tutorial
      • Anthropic Prompt Sanitization Tutorial
      • LangChain Prompt Sanitization Tutorial
    • SaaS Protection
      • HubSpot DLP Tutorial
      • Zendesk DLP Tutorial
    • Observability Protection
      • Datadog DLP Tutorial
      • New Relic DLP Tutorial
    • Datastore Protection
      • Airtable DLP Tutorial
      • Amazon Kinesis DLP Tutorial
      • Amazon RDS DLP Tutorial
      • Amazon RDS DLP Tutorial - Full Scan
      • Amazon S3 DLP Tutorial
      • Elasticsearch DLP Tutorial
      • Snowflake DLP Tutorial
  • Nightfall Use Cases
    • Overview
    • GenAI Content Filtering-How to prevent exposure of sensitive data
    • Redacting Sensitive Data in 4 Lines of Code
    • Detecting Sensitive Data in SMS Automations
    • Building Endpoint DLP to Detect PII on Your Machine in Real-Time
    • Deploy a File Scanner for Sensitive Data in 40 Lines of Code
    • Using Scan API (with Python)
  • FAQs
    • What Can I do with the Firewall for AI
    • How quickly can I get started with Firewall for AI?
    • What types of data can I scan with API?
    • What types of detectors are supported out of the box?
    • Can I customize or bring my own detectors?
    • What is the pricing model?
    • How do I know my data is secure?
    • How do I get in touch with you?
    • Can I test out the detection and my own detection rules before writing any code?
    • How does Nightfall support custom data types?
    • How does Nightfall's Firewall for AI differs from other solutions?
  • Nightfall Playground
  • Login to Nightfall
  • Contact Us
Powered by GitBook
On this page
  • Use Cases
  • Example 1
  • Example 2
  • Standard Pattern for Sending Transactional Email/SMS
  • Adding Sensitive Data Classification/Protection to the Pattern
  • Step 1. Setup
  • Step 2. Configure Detection
  • Step 3. Classify and/or Redact
  • Step 4. Handle Response and Send
  • Python Example

Was this helpful?

Export as PDF
  1. Nightfall Use Cases

Detecting Sensitive Data in SMS Automations

PreviousRedacting Sensitive Data in 4 Lines of CodeNextBuilding Endpoint DLP to Detect PII on Your Machine in Real-Time

Last updated 10 months ago

Was this helpful?

Transactional email and communication APIs like SendGrid, Twilio, SES, and Mailgun are critical components to modern applications. These services allow developers to easily incorporate end-user communication into their applications without the infrastructural overhead.

However, these services pose a new source of security risk as they can lead to accidental sharing of sensitive data if communications are sent to the wrong users or inadvertently contain sensitive data. Adding data loss prevention (DLP) into your business logic can provide the critical capability to classify & protect sensitive data before it is exposed, leaked, or stored.

Use Cases

The risk of exposing sensitive data is especially common in situations where these transactional communication services are handling user-generated content like messages between agents and users, or peer-to-peer. Here are a few examples:

Example 1

You're building a grocery delivery application like Instacart. The application allows Shoppers and Customers to send and receive text messages with each other, powered by Twilio. The Customer sends the Shopper a picture of their Driver's License since they won't be home for the delivery, even though their Driver's License needs to be verified in person. Now this image with sensitive PII is processed by Twilio, stored in your application's object store, and viewable by the Shopper and support agents.

Example 2

You're building an application for job seekers to connect with small business owners like restaurants that are hiring, and they can exchange messages over text, powered by Twilio. A malicious user signs up to impersonate a restaurant owner and uses this mechanism to collect PII from job seekers such as their SSN. Now this PII is transmitted by Twilio and is accessible by the attacker.

With the Nightfall API, you can scan transactional communications for sensitive data and remediate them accordingly. In this post, we’ll describe the pattern behind how to use Nightfall to scan for sensitive data in outgoing emails.

Standard Pattern for Sending Transactional Email/SMS

The typical pattern for using transactional communication services like SendGrid is as follows:

  1. Get an API key and set environment variables

  2. Initialize the SDK client (e.g. SendGrid Python client), or use the API directly to construct a request

  3. Construct your outgoing message, which includes information like the subject, body, recipients, etc.

  4. Use the SendGrid API or SDK client to send the message

Let's look at a simple example in Python. Note how easy it is to send sensitive data, in this case a credit card number.

import os
from twilio.rest import Client

account_sid = os.environ['TWILIO_ACCOUNT_SID']
auth_token = os.environ['TWILIO_AUTH_TOKEN']
client = Client(account_sid, auth_token)

message_body = "4916-6734-7572-5015 is my credit card number"

message = client.messages.create(
                     body=message_body,
                     from_='+15017122661',
                     to='+15558675310'
                 )

print(message.sid)

Adding Sensitive Data Classification/Protection to the Pattern

It is straightforward to update this pattern to use Nightfall to check for sensitive findings and ensure sensitive data isn’t sent out. Here’s how:

Step 1. Setup

Get API keys for both communication service (“CS”) and Nightfall (“NF”), and set environment variables. Learn more about creating a Nightfall API key here.

Step 2. Configure Detection

NF: Create a pre-configured detection rule in the Nightfall dashboard or inline detection rule with Nightfall API or SDK client.

📘Consider using Redaction

Note that if you specify a redaction config, you can automatically get de-identified data back, including a reconstructed, redacted copy of your original payload. Learn more about redaction here.

Step 3. Classify and/or Redact

NF: Send your outgoing message text (and any other metadata like the subject line, etc.) in a request payload to the Nightfall API text scan endpoint. For example, if you are interested in scanning the subject and body of an outgoing email, you can send these both in the input array payload: [ body, subject ]

# Nightfall API Response
 
{
  "findings": [
    [
      {
        "finding": "458-02-6124",
       "redactedFinding": "***-**-****",
        "detector": {
          "name": "US Social Security Number",
          "uuid": "e30d9a87-f6c7-46b9-a8f4-16547901e069"
        },
        "confidence": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "start": 39,
            "end": 50
          },
          "codepointRange": {
            "start": 39,
            "end": 50
          }
        },
        "redactedLocation": {
          "byteRange": {
            "start": 39,
            "end": 50
          },
          "codepointRange": {
            "start": 39,
            "end": 50
          }
        },
        "matchedDetectionRuleUUIDs": [],
        "matchedDetectionRules": [
          "My Match Rule"
        ]
      }
    ]
  ],
  "redactedPayload": [
    "Thanks for getting back to me. My SSN is ***-**-****."
  ]
}

Step 4. Handle Response and Send

  • Review the response to see if Nightfall has returned sensitive findings:

    • If there are sensitive findings:

      • You can choose to specify a redaction config in your request so that sensitive findings are redacted automatically

      • Without a redaction config, you can simply break out of the conditional statement, throw an exception, etc.

    • If no sensitive findings or you chose to redact findings with a redaction config:

      • Initialize the SDK client (e.g. SendGrid Python client), or use the API directly to construct a request

      • Construct your outgoing message, which includes information like the subject, body, recipients, etc.

        • If you specified a redaction config and want to replace raw sensitive findings with redacted ones, use the redacted payload that Nightfall returns to you

      • Use the SendGrid API or SDK client to send the sanitized message

Python Example

Let's take a look at what this would look like in a Python example using the Twilio and Nightfall Python SDKs in just 12 lines of code (with comments and formatting added for clarity):

import os
from twilio.rest import Client
from nightfall import Confidence, DetectionRule, Detector, RedactionConfig, MaskConfig, Nightfall

# Initialize the Twilio and Nightfall clients
account_sid = os.environ['TWILIO_ACCOUNT_SID']
auth_token = os.environ['TWILIO_AUTH_TOKEN']
client = Client(account_sid, auth_token)

nightfall = Nightfall() # By default Nightfall will read the NIGHTFALL_API_KEY environment variable

# The message you intend to send
payload = [ "4916-6734-7572-5015 is my credit card number" ]

# Send the message to Nightfall to scan it for sensitive data
# Nightfall returns the sensitive findings, and a copy of your input payload with sensitive data redacted
findings, redacted_payload = nightfall.scan_text(
				        payload,
				        [ DetectionRule([ # Define an inline detection rule that looks for Likely Credit Card Numbers
			        		Detector(
			        			min_confidence=Confidence.LIKELY,
		               			nightfall_detector="CREDIT_CARD_NUMBER",
		               			display_name="Credit Card Number",
			               		redaction_config=RedactionConfig(
									remove_finding=False, 
									substitution_phrase="[Redacted]")
			               	)])
				        ])

# If the message has sensitive data, use the redacted version, otherwise use the original message
if redacted_payload[0]:
	message_body = redacted_payload[0]
else:
	message_body = payload[0]

print(message_body)

# Send the sanitized message body via Twilio
message = client.messages.create(
                     body=message_body,
                     from_='+15017122661',
                     to='+15558675310'
                 )

print(message.sid)

You'll see that the message we originally intended to send had sensitive data: 4916-6734-7572-5015 is my credit card number

And the message we ultimately sent was redacted!

[Redacted] is my credit card number

Services like Twilio and SendGrid also support inbound communications from end-users, typically via webhook. Meaning inbound messages will be sent to a webhook handler that you specify. Hence, you would insert the above logic in your webhook handler upon receipt of an event payload.

Now that you understand the pattern, give it a shot!