# Uploading and Scanning API Calls

Nightfall's upload process is built to accommodate files of any size. Once files are uploaded, they may be scanned with[ Detection Rules](https://help.nightfall.ai/developer-api/key-concepts/scanning_features/pre_configured_detection_rules) and [Policies](https://help.nightfall.ai/developer-api/key-concepts/scanning_features/send_alerts_using_policies) to detect potential violations.

Many users will find it more convenient to use our our [native language SDKs](https://help.nightfall.ai/developer-api/nightfall_sdk/overview) to complete the upload process.

Uploading files using Client SDK libraries requires fewer steps as all the required API operations are wrapped in a single function call. Furthermore these SDKs handle all the programmatic logic necessary to send files in smaller chunks to Nightfall.

For users that are looking to understand the entire [upload process](https://help.nightfall.ai/developer-api/key-concepts/file_scan/scan_api_calls) end-to-end, that is also outlined in this document. We will walk you through the order of operations necessary to upload the file.

### Using Nightfall's SDKs to Upload Files

Rather than implementing the [full sequence of API calls](#the-upload-process) for the upload functionality yourself, the Nightfall’s [native language SDKs](https://help.nightfall.ai/developer-api/nightfall_sdk/overview) provide a single method that wraps the steps required to upload your file.

Below is an example of uploading a file from our [Python SDK](https://help.nightfall.ai/developer-api/nightfall_sdk/overview) and our [Node SDK](https://help.nightfall.ai/developer-api/nightfall_sdk/overview).

{% tabs %}
{% tab title="Python" %}

```python
>>> from nightfall import Confidence, DetectionRule, Detector, Nightfall, EmailAlert, AlertConfig
>>> import os

>>> # use your API Key here
>>> nightfall = Nightfall("NF-y0uRaPiK3yG03sH3r3")

>>> # A rule contains a set of detectors to scan with
>>> cc = Detector(min_confidence=Confidence.LIKELY, nightfall_detector="CREDIT_CARD_NUMBER")
>>> ssn = Detector(min_confidence=Confidence.POSSIBLE, nightfall_detector="US_SOCIAL_SECURITY_NUMBER")
>>> detection_rule = DetectionRule([cc, ssn])
>>> # The scanning is done asynchronously, so provide a valid email address as the simplest way of getting results
>>> alertconfig = alert_config=AlertConfig(email=EmailAlert("whatever@example.com"))
    

>>> # Upload the file and start the scan.
>>> id, message = nightfall.scan_file( "./README.md", detection_rules=[detection_rule], alert_config=alertconfig)
>>> print("started scan", id, message)
```

{% endtab %}

{% tab title="Javascript" %}

```javascript
//this script assumes the node sdk has been installed locally with `npm install` and `npm run build`
import { Nightfall } from "./nightfall-nodejs-sdk/dist/nightfall.js";
import { Detector } from "./nightfall-nodejs-sdk/dist/types/detectors.js";


// By default, the client reads your API key from the environment variable NIGHTFALL_API_KEY
const uploadit = async() => {
    var data = null;
    
    const nfClient = new Nightfall();
    	
    try{
   
		const response = await nfClient.scanFile('./README.md', {
		  detectionRules: [
			{
			  name: 'Secrets Scanner',
			  logicalOp: 'ANY',
			  detectors: [
				{
				  minNumFindings: 1,
				  minConfidence: Detector.Confidence.Possible,
				  displayName: 'Credit Card Number',
				  detectorType: Detector.Type.Nightfall,
				  nightfallDetector: 'CREDIT_CARD_NUMBER',
				},
			  ],
			},
		  ],
		  alertConfig: {
				email: {
						address: "whatever@example.com"
					}
		   }
		});

		if (response.isError) {
		  data = response.getError();
		}
		else{ 
			data = (response.data.id);
		}
	 
    }
	catch(e){
		console.log(e);
	}


	return data;

}

uploadit().then(data => console.log(data));
```

{% endtab %}
{% endtabs %}

To run the node sample script you must compile it as TypesScript. Save it as a .ts file and run

`tsc <yourfilename>.ts -lib ES2015,DOM`

You can then run the resulting JavaScript file:

`NIGHTFALL_API_KEY=<YourApiKey> node yourscriptname.js`

Note that these examples use an email address to receive the results for simplicity.

You may also want to use a webhook. See [Webhooks and Asynchronous Notifications](https://docs.nightfall.ai/docs/webhooks-and-asynchronous-notifications)[ ](https://help.nightfall.ai/developer-api/key-concepts/file_scan/webhooks_async_notifications)for additional information on how to set up Webhook server to receive these results.

### The Upload Process

The upload process consists of 3 stages:

* [Initializing](#initializing-phase)
* [Uploading](#uploading-phase)
* [Completing](#completion-phase)

Once the upload is complete, you may initiate the [file scan.](#scanning-uploaded-files)

After we discuss each API call in the sequence, you will find a script that walks through the [full sequence](#full-upload-process-example-script) at the end of this guide.

#### Initializing Phase

`POST /v3/upload`

The first step in the process of scanning a binary file is to initiate an upload in order to get a `fileId` through the Initiate a [File Upload endpoint](https://help.nightfall.ai/nightfall-firewall-for-ai/nightfall-apis/dlp-apis-firewall-for-ai-platform/initiate-file-upload).

As part of the initialization you must provide the total byte size of the file being uploaded.

You may also provide the mime-type, otherwise the system will attempt to determine it once the upload is complete.

```sh
curl --location --request POST 'https://api.nightfall.ai/v3/upload' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '{
    "fileSizeBytes": 73891,
    "mimeType" : "image/png"
}'
```

The `id` of the returned JSON object will be used as the `fileId` in subsequent requests.

The `chunkSize` is the maximum number of bytes to upload during the uploading phase.

```json
{
    "id": "f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d",
    "fileSizeBytes": 73891,
    "chunkSize": 10485760,
    "mimeType": "image/png"
}
```

#### Uploading Phase

```json
PATCH /v3/upload/<uploadUUID>
```

Use the [Upload a Chunk of a File](https://help.nightfall.ai/nightfall-firewall-for-ai/nightfall-apis/dlp-apis-firewall-for-ai-platform/upload-file-chunk) endpoint to upload the file contents in chunks.

The size of these chunks are determined by the `chunkSize` value returned by `POST /upload` endpoint used in the previous step.

Below is a simple example where the file is less than the `chunkSize` so may safely be uploaded with one call to the upload endpoint.

```sh
curl --location --request PATCH 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d' \
--header 'X-Upload-Offset: 0' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-binary '@/Users/myname/Documents/work/Nightfall/Nightfall Upload Sequence.png'
```

If your file's size exceeds the `chunkSize`, to upload the complete file you will need to send iterative requests as you read portions of the file's contents. This means you will send multiple requests to the `upload` endpoint as shown above. As you do so, you will be updating the value of the `X-Upload-Offset` header based on the portion of the file being sent.

Each request should send a chunk of the file exactly `chunkSize` bytes long except for the final uploaded chunk. The final uploaded chunk is allowed to contain fewer bytes as the remainder of the file may be less than the `chunkSize` returned by the initialization step.

The request body should be the contents of the chunk being uploaded.

The value of the `X-UPLOAD-OFFSET` header should be the byte offset specifying where to insert the data into the file as an integer. This byte offset is zero-indexed.

Successful calls to this endpoint return an empty response with an HTTP status code of `204`

See the[ full example script](#full-upload-process-example-script) below for an illustration as to how this upload process can be done programmatically.

#### Completion Phase

`POST /v3/upload/<uploadUUID>/finish`

Once all chunks are uploaded, mark the upload as completed using the [Complete a File Upload endpoint](https://help.nightfall.ai/nightfall-firewall-for-ai/nightfall-apis/dlp-apis-firewall-for-ai-platform/complete-file-upload).

```sh
curl --location --request POST 'https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/finish' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer  NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
--data-raw '""'
```

When an upload completes successfully, the returned payload will indicate the mimeType the system determined to file to be if it was not provided during upload initialization.

```json
{
    "id": "152848af-2ac9-4e0a-8563-2b82343d964a",
    "fileSizeBytes": 2349,
    "chunkSize": 10485760,
    "mimeType": "application/zip"
}
```

Once a file has been marked as completed, you may initiate a scan of the uploaded file.

### Scanning Uploaded Files

After an upload is finalized, it can be scanned against a Detection Policy. A Detection Policy represents a pairing of:

* a webhook URL
* a set of detection rules to scan data against

The scanning process is asynchronous, with results being delivered to the webhook URL configured on the detection policy. See [Webhooks and Asynchronous Notifications](https://help.nightfall.ai/developer-api/key-concepts/file_scan/webhooks_async_notifications) for more information about creating a Webhook server.

Exactly one `policy` should be provided in the request body, which includes a `webhookURL` to which the callback will be made once the file scan has been completed (this must be an HTTPS URL) as well as a Detection Rule as either an a [list of UUIDs](https://help.nightfall.ai/developer-api/key-concepts/scanning_features/pre_configured_detection_rules) or as a rule that has been [defined in-line](https://help.nightfall.ai/developer-api/key-concepts/scanning_features/inline_detection_rules).

You may also supply a value to the `requestMetadata` field to help identify the input file upon receiving a response to your webhook. This field has a maximum length 10 KB.

```sh
curl --request POST \
     --url https://api.nightfall.ai/v3/upload/f9dbdb15-c9fa-46ff-86ec-cd5c09aa550d/scan \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer NF-rEpLaCeM3w1ThYoUrNiGhTfAlLKeY123' \
     --header 'Content-Type: application/json' \
     --data '
{
     "policy": {
          "detectionRuleUUIDs": [
               "950833c9-8608-4c66-8a3a-0734eac11157"
          ],
          "webhookURL": "https://mycompany.org/webhookservice"
     },
     "requestMetadata": "your file metadata"
}
'
```

{% hint style="info" %}

### Webhook Verification

Nightfall will verify that the webhook URL is valid before launching its asynchronous scan by issuing a challenge.
{% endhint %}

### Full Upload Process Example Script

Below is a sample Python script that handles the complete sequence of API calls to upload a file using a path specified as an argument.

```python
from os import getenv, path

import fire
import requests


BASE_UPLOAD_URL = getenv("FILE_UPLOAD_HOST", "http://api.nightfall.ai/v3")
NF_API_KEY = getenv("NF_API_KEY")


def upload(filepath, mimetype, policy_uuid):
    """Upload the given file using the provided MIMEType and PolicyUUID.

    Arguments:
        file_path -- an absolute or relative path to the file that will be
            uploaded to the API.
        mimetype -- (optional) The mimetype of the file being uploaded.
        policy_uuid -- The UUID corresponding to an existing policy. This
            policy must be active and have a webhook URL associated with it.
    """
    default_headers = {
        "Authorization": F"Bearer {NF_API_KEY}",
    }

    # =*=*=*=*=* Initiate Upload =*=*=*=*=*=*
    file_size = path.getsize(filepath)
    upload_request_body = {"fileSizeBytes": file_size, "mimeType": mimetype}
    r = requests.post(F"{BASE_UPLOAD_URL}/upload",
                      headers=default_headers,
                      json=upload_request_body)
    upload = r.json()
    if not r.ok:
        raise Exception(F"Unexpected error initializing upload - {upload}")

    # =*=*=*=*=*=* Upload Chunks =*=*=*=*=*=*
    chunk_size = upload["chunkSize"]
    i = 0
    with open(filepath, "rb") as file:
        while file.tell() < file_size:
            upload_chunk_headers = {
                **default_headers,
                "X-UPLOAD-OFFSET": str(file.tell())
            }
            r = requests.patch(F"{BASE_UPLOAD_URL}/upload/{upload['id']}",
                               headers=upload_chunk_headers,
                               data=file.read(chunk_size))
            if not r.ok:
                raise Exception(F"Unexpected error uploading chunk - {r.text}")
            i += 1

    # =*=*=*=*=*=* Finish Upload =*=*=*=*=*=*
    r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/finish",
                      headers=default_headers)
    if not r.ok:
        raise Exception(F"Unexpected error finalizing upload - {r.text}")

    # =*=*=*=*=* Scan Uploaded File =*=*=*=*=*
    r = requests.post(F"{BASE_UPLOAD_URL}/upload/{upload['id']}/scan",
                      json={"policyUUID": policy_uuid},
                      headers=default_headers)
    if not r.ok:
        raise Exception(F"Unexpected error initiating scan - {r.text}")

    print("Scan Initiated Successfully - await response on configured webhook")
    quota_remaining = r.headers.get('X-Quota-Remaining')
    if quota_remaining is not None and int(quota_remaining) <= 0:
        print(F"Scan quota exhausted - Quota will reset on {r.headers['X-Quota-Period-End']}")


if __name__ == "__main__":
    fire.Fire(upload)
```