# Supported File Types The file scan API has first-class support for text extraction and scanning on all MIME types enumerated below. Certain file types receive special handling, such as [tabular data ](#spreadsheets-and-tabular-data)and [archives of Git repositories](#git-repositories), that results in more precise information about the location of findings within the source file. {% hint style="info" %} ### Handling of MIME Types Not Listed Files with a MIME type not listed below are processed using an unoptimized text extractor. As a result, the quality of the text extraction for unrecognized types may vary. {% endhint %} ### Accepted Text and Derivatives * application/json * application/x-ndjson * application/x-php * text/calendar * text/css * text/csv (treated as [tabular data](#spreadsheets-and-tabular-data) and may be[ redacted](#redacting-csv-files) ) * text/html * text/javascript * text/plain * text/tab-separated-values (treated as [tabular data](#spreadsheets-and-tabular-data)) * text/tsv (treated as [tabular data](#spreadsheets-and-tabular-data)) * text/x-php ### Accepted Office Formats * application/pdf * application/vnd.openxmlformats-officedocument.presentationml.presentation * application/vnd.openxmlformats-officedocument.spreadsheetml.sheet (treated as [tabular data](#spreadsheets-and-tabular-data)) * application/vnd.openxmlformats-officedocument.wordprocessingml.document * application/vnd.ms-excel (treated as [tabular data](#spreadsheets-and-tabular-data)) ### Accepted Archive and Compressed File Types * application/bzip2 * application/ear * application/gzip * application/jar * application/java-archive * application/tar+gzip * application/vnd.android.package-archive * application/war * application/x-bzip2 * application/x-gzip * application/x-rar-compressed * application/x-tar * application/x-webarchive * application/x-zip-compressed * application/x-zip * application/zip ### Accepted Image File Types * image/apng * image/avif * image/gif * image/jpeg * image/jpg * image/png * image/svg+xml * image/tiff * image/webp ### Rejected MIME Types The file scan API explicitly rejects requests with MIME types that are not conducive to extracting or scanning text. Sample rejected MIME types include: * application/photoshop * audio/midi * audio/wav * video/mp4 * video/quicktime ### Spreadsheets and Tabular Data File scans of Microsoft Office, Apache parquet, csv, and tab separated files will provide additional properties to locate findings within the document beyond the standard `byteRange`, `codepointRange`, and `lineRange` properties. Findings will contain a `columnRange` and a `rowRange` that will allow you to identify the specific row and column within the tabular data wherein the finding is present. This functionality is applicable to the following mime types: * text/csv * text/tab-separated-values * text/tsv * application/vnd.openxmlformats-officedocument.spreadsheetml.sheet * application/vnd.ms-excel [Apache parquet](https://parquet.apache.org/docs/) data files are also accepted. Below is a sample match of a spreadsheet containing dummy PII where a SSN was detected in the 2nd column and 55th row. {% code lineNumbers="true" %} ```json { "findings":[ { "path":"Sheet1 (5)", "detector":{ "id":"e30d9a87-f6c7-46b9-a8f4-16547901e069", "name":"US social security number (SSN)", "version":1 }, "finding":"624-84-9182", "confidence":"LIKELY", "location":{ "byteRange":{ "start":2505, "end":2516 }, "codepointRange":{ "start":2452, "end":2463 }, "lineRange":{ "start":55, "end":55 }, "rowRange":{ "start":55, "end":55 }, "columnRange":{ "start":2, "end":2 }, "commitHash":"" }, "matchedDetectionRuleUUIDs":[ "950833c9-8608-4c66-8a3a-0734eac11157" ], "matchedDetectionRules":[ ] }, ... ``` {% endcode %} ### Redacting CSV Files Findings within csv files may be redacted. To enable redaction in files, set the `enableFileRedaction` flag of your `policy` to "true" The csv file will be redacted based on the configuration of the `defaultRedactionConfig` of the `policy` Below is an example curl request for a csv file that has already been [uploaded](https://help.nightfall.ai/developer-api/key-concepts/file_scan/scan_api_calls) . ```shell curl --request POST \ --url https://api.nightfall.ai/v3/upload/02a0c5e1-c950-4e28-a988-f6fffefc4205/scan \ --header 'Accept: application/json' \ --header 'Authorization: Bearer NF-' \ --header 'Content-Type: application/json' \ --data ' { "policy": { "detectionRuleUUIDs": [ "950833c9-8608-4c66-8a3a-0734eac11157" ], "alertConfig": { "email": { "address": "" } }, "defaultRedactionConfig": { "maskConfig": { "charsToIgnore": [ "-", "@" ], "maskingChar": "*" } }, "enableFileRedaction": true }, "requestMetadata": "csv redaction test" } ' ``` When results are sent to the location specified in the `alertConfig` (in this case an email address) a `redactedFile` property will be set with a `fileURL` in addition the `findingsURL` ```shell { "errors":null, "findingsPresent":true, "findingsURL":"https://files.nightfall.ai/asdfc5e1-c950-4e28-a988-f6fffefc4205.json?Expires=1655324479&Signature=zjo1nT-PECHC-fiTvAgdA8aDnceoY~6iGfzOBCcBjscKqOHnIar8hoH4gGufffiulBw5BpfJuvWwBW~lXO~ZNhN139LDwoTsfLJswJiQCB2Hj-Az0Em6go~1j8WBqCS8G0Gk17M-zcPedHGX3z~1pw8nm5sh6Pa-jJwfw9NIEiqmBb3Vdcj3J-~Wzag~ENV4499rnG299ee-ig5Ms1oVlzycb4YxzgTMrTL5Q07ozNenwFZcGDNQre1inLXmV-m8teLX-K3boklenp9KXiNDDV0wi74ADN-QfIR1q1oU7mEI1f3aVC3kju0QRErp2lsfs08EtZKLE3C4N17jDJdYcw__&Key-Pair-Id=K24YOPZ1EKX0YC", "redactedFile":{ "fileURL":"https://files.nightfall.ai/asdfc5e1-c950-4e28-a988-f6fffefc4205-redacted.csv?Expires=1655324479&Signature=Hx8kRh88maLeStysy3fsLbFVG9VELEtfemtQe2lWUnFjAMd9HqlEksTmirqAWFWV4zPVUB73izlMj5cSer8v2N5ZCcnD3dz~nnwR4P5LewGJ2CQzGnDnXgh70HW5qp04gnUD-pYWp~bGPVspkJKCkl1zH-EoGonvcNVq3SNsVzOlsVIjep7Y7otQKEEyAZ7JmHiVfuBxrvn8pleuC5lEJ3f9miPyoRqH9DyPlNTJTIuijqe9q32Qcui2RsDR6IT-foFX52dy6rRa01ZV0gZMDWJokMlCr8Iu5An~qnhxC49bqTtI82oz9FcBaP-Yea8cq1TiAfGxX7CJ0~JeTLvr6g__&Key-Pair-Id=K24YOPZ1EKX0YC", "validUntil":"2022-06-15T20:21:19.750990823Z" }, "requestMetadata":"csv redaction test", "uploadID":"02a0c5e1-c950-4e28-a988-f6fffefc4205", "validUntil":"2022-06-15T20:21:19.723045787Z" } ``` This redacted file will be a modified version of the original csv file. Below is an example of a redacted csv file. ``` name,email,phone,alphanumeric Ulric Burton,*****@*************,*-***-***-****,TEL82EBM1GQ Wade Jones,******************@***********,(********-****,VVF64PJV2EF Molly Mccullough,*****************@**********,(********-****,OHO41SFZ2BR Raja Riggs,************@**********,(********-****,UVD51JTE5NZ Colin Carter,**********************@*********,(********-****,LNI34LLC5WV// Some code ``` ### Git Repositories Nightfall provides special handling for archives of Git repositories. Nightfall will scan the repository history to discover findings in particular checkin, returning the hash for the checkin. In order to scan the repository, you will need to create a clone, i.e. `git clone https://github.com/nightfallai/nightfall-go-sdk.git` This creates a clone of the Nightfall go SDK. You will then need to create an archive that can be uploaded using Nightfall's file scanning sequence. `zip -r directory.zip directory` Note that in order to work, the hidden directory `.github` must be included in the archive. When you initiate the [file upload sequence](https://help.nightfall.ai/developer-api/key-concepts/file_scan/scan_api_calls) with this file, you will receive scan results that contain the `commitHash` property filled in. Using the Nightfall go SDK archive created above, a simple example would be to scan for URLs (i.e. strings starting with `http://` or `https://`), which will send results such as the following: ```json { "findings":[ { "path":"f607a067..53e59684/nightfall.go", "detector":{ "id":"6123060e-2d9f-4f35-a7a1-743379ea5616", "name":"URL" }, "finding":"https://api.nightfall.ai/\"", "confidence":"LIKELY", "location":{ "byteRange":{ "start":142, "end":168 }, "codepointRange":{ "start":142, "end":168 }, "lineRange":{ "start":16, "end":16 }, "rowRange":{ "start":0, "end":0 }, "columnRange":{ "start":0, "end":0 }, "commitHash":"53e59684d9778ceb0f0ed6a4b949c464c24d35ce" }, "beforeContext":"tp\"\n\t\"os\"\n\t\"time\"\n)\n\nconst (\n\tAPIURL = \"", "afterContext":"\n\n\tDefaultFileUploadConcurrency = 1\n\tDef", "matchedDetectionRuleUUIDs":[ "cda0367f-aa75-4d6a-904f-0311209b3383" ], "matchedDetectionRules":[ ] }, ... ``` {% hint style="info" %} ### Support for Large Repositories Currently, processing is limited to repositories with a total number of commits lower than 5000. {% endhint %} Large repositories result in a large volume of data sent at once. We are working on changes to allow these and other large surges of data to be processed in a more controlled manner, and will increase the limit or remove it altogether once those changes are complete. {% hint style="info" %} ### Sensitive Data in GitHub Repositories If the finding in a GitHub repository is considered to be sensitive, it should be considered compromised and appropriate mitigation steps (i.e. secrets should be rotated). {% endhint %} To retrieve the specific checkout, you will need to clone the repository, i.e. `git clone https://github.com/nightfallai/nightfall-go-sdk.git` You can then checkout the specific commit using the commit hash returned by Nightfall. ``` cd nightfall-go-sdk git checkout 53e59684d9778ceb0f0ed6a4b949c464c24d35ce ``` Note that you are in a ['detached HEAD' state](https://www.git-tower.com/learn/git/faq/detached-head-when-checkout-commit) when workin with this sort of check out of a repository. ## File Scanning Limitations * CSV Files: Only the first 250,000 rows will be scanned. * Spreadsheet Files: Up to 100,000 rows per sheet will be scanned, with a maximum of 1 million rows across all tabs in multi-sheet spreadsheets. * PDF Files: Scanning is limited to the first 100 pages, including a maximum of 50 images within those pages. * Images: Images smaller than 5KB or larger than 50MB will be excluded from scanning. * Archive Files: A maximum of 1,000 files will be extracted and scanned. Files larger than 100MB requiring extraction will not be scanned.