droid-light

droid-light is a small Java library that identifies file formats from the "magic bytes" inside a file, using the signature file of the PRONOM project, the file-format registry of The National Archives (UK) that DROID is built on.

Give it a file or a stream and it tells you which format it is: the PRONOM identifier (PUID), the format name, the MIME type and the version.

// Create once - reading the signature file takes a second or two - and reuse it, also from many threads.
DroidSignatureVerifier verifier = new DroidSignatureVerifier(new File("DROID_SignatureFile_V125_experimental.xml"));

DetectionResult[] results = verifier.detect(new File("somefile.pdf"));   // or verifier.detect(inputStream)
if (results.length > 0) {
    System.out.println(results[0]);  // fmt/18  Acrobat PDF 1.4 - Portable Document Format  [application/pdf; version=1.4]
}

Why this library exists

DROID checks a signature that consists of several byte patterns separated by flexible gaps with nested loops that try every possible position of every pattern. For some files the number of combinations explodes, and DROID keeps running at 100% CPU for practically forever. This was reported to the DROID project: DROID hangs indefinitely (100% CPU) on WebP image. A tool that identifies files unattended, in large numbers, cannot afford a single file that never finishes.

droid-light reads the same signature file and applies the same matching rules for binary signatures, but it never tries all combinations: every pattern is searched for in a limited window, and the first fitting position is used for the next pattern in the chain. The work per file is therefore bounded, whatever the content.

The result

detect() returns up to 10 candidates, index 0 being the best match. Formats that DROID's priority rules prefer come first; an empty array means that no signature matched.

A DetectionResult holds the PUID (getCode()), the format name (getText()), the MIME type and the version, all taken from the signature file. There are three ways to get the MIME type:

Method	Returns
getPrimaryMimeTypeWithVersion()	e.g. "text/html; version=5". null when the signature file has no MIME type for the format, which is the case for about two thirds of the formats that have a signature (AVIF, for example).
getPrimaryMimeType()	the same without the version. When the signature file lists several MIME types, the first one.
getPrimaryMimeTypeWithVersionOrName()	never null: the MIME type with version when there is one, otherwise the format name, e.g. "AV1 Image File Format". Use it when a value is always needed, for example in a report or a search index. A returned name is not a valid MIME type.

Files, streams and large files

• detect(File) and detect(InputStream) give the same result, except for files between 40 and 100 MB that need a pattern in the middle: a file is read completely, a stream only at both ends. A stream is not closed. If it supports mark()/reset() it is reset to its start before reading; otherwise reading starts at its current position.
• Files up to 100 MB are read completely. Larger files, and streams longer than 40 MB, are examined through a window of 20 MB at the start and 20 MB at the end, because signatures look at the beginning and the end of a file. Memory use stays small and predictable, from a few bytes up to many gigabytes. A stream is still read to its end, because a stream cannot skip ahead.

Faster when you know the MIME type

If you already know the MIME type, for example from an HTTP Content-Type header, you can check only the signatures of the formats that declare that type (between 1 and about 50 signatures instead of all of them). Such a header can be wrong, so always fall back to the full scan:

DetectionResult[] r = new DetectionResult[0];
if (verifier.getSignatureCountForMimeType(contentType) > 0) {
    r = verifier.detectCommonMimeTypes(stream, contentType);   // an empty array if nothing matched
}
if (r.length == 0) {
    r = verifier.detect(stream);                               // the full scan
}

getSignatureCountForMimeType() returns 0 for unknown types and for types without any signature; skipping the shortcut then saves reading the stream twice.

The signature file

The signatures come from PRONOM. The project contains two signature files in src/main/resources, and both are packaged in the jar:

• DROID_SignatureFile_V125.xml is the official PRONOM signature file V125, released on 2 September 2026 and published by The National Archives under the Open Government Licence v3.0 (2,273 signatures). It is kept exactly as downloaded and is never modified, reformatted or renamed; it is there for comparison. V125 adds 14 formats: ForTheRecord TRM audio, FictionBook 2, Aseprite, FAB2 Book Container, Clock RINEX 2.0 and 3.0, LabVIEW Virtual Instrument, NumPy, TIGER/Line ASCII, TIGER/Line Shapefile, HALO Photonics Doppler LiDAR, Hugin project files, Address Event DATa and FASTQ.
• DROID_SignatureFile_V125_experimental.xml is the official file plus local additions, 16 signatures that PRONOM does not have, 2,289 in total. It is the default: the file the library is built and tested with, and the one used in the examples. The name says "experimental" because the file deviates from the official one, which this project cannot change.

The local additions are:

• EOT font, version 2.1 (fmt/DRAFT-EOT-2.1): EOT files whose version field is 0x00020001 (bytes 01 00 02 00 at offset 8), the version written by most converters for web fonts. PRONOM's fmt/1383 has the same label but expects the four bytes in the opposite order (02 00 01 00), and therefore never matches a real file; fmt/1382 has the same problem, and only fmt/1384 (0x00020002, compressed fonts) is correct. In a test of 26 distinct EOT fonts from 20 npm packages, 22 had version 2.1 and none had the bytes of fmt/1383. If PRONOM corrects fmt/1383, this local signature will match the same files and should be removed.
• SVG without an XML declaration: the PRONOM signatures for SVG require <?xml version="1.0" at the very start of the file, so SVG files that begin directly with <svg were not recognised. The added signature maps them to the generic SVG format fmt/91.
• Motion Photo (Android and Samsung): two signatures, for a JPEG and for a HEIC with an MP4 video appended (see below). They are two new formats, fmt/DRAFT-MotionPhoto-JPEG-1.0 and fmt/DRAFT-MotionPhoto-HEIF-1.0, and they come before the plain JPEG and HEIC formats in the result.
• JSON, JavaScript and CSS: twelve heuristic signatures (two for JSON, eight for JavaScript and two for CSS), because PRONOM has no binary signature for these text formats and web archives (WARC files) are full of them. The rules are strict, to avoid false labels, and are described below.

The additions are exactly the difference between the two files, and can be listed at any time with diff -u src/main/resources/DROID_SignatureFile_V125.xml src/main/resources/DROID_SignatureFile_V125_experimental.xml. Anyone who adds or changes a local signature should read SIGNATURE_DESIGN.md first; it gives the rules and how to test a change.

New releases of the signature file are published by PRONOM on the DROID signature files page. To use another signature file, load it with the File or InputStream constructor.

Motion Photo signatures

An Android Motion Photo (Google Pixel, recent Samsung phones) is a JPEG or HEIC still image with a short MP4 video appended to it, described by Google's Motion Photo format 1.0. To the PRONOM signatures such a file is an ordinary Exif JPEG or HEIC, so droid-light reports it as that. Two local signatures recognise the Motion Photo itself:

• JPEG (fmt/DRAFT-MotionPhoto-JPEG-1.0, MIME type image/jpeg): the file starts with FFD8FF and the text Semantic="MotionPhoto" occurs within the first 131,072 bytes. That text is the attribute of the item in the XMP container directory that describes the video, so the file says that it has a video. The offset of that text varies a lot between phones (about 11,000 to 57,000 in the sample files), because the Exif data comes before the XMP.
• HEIF (fmt/DRAFT-MotionPhoto-HEIF-1.0, MIME type image/heif): ftyp at byte 4 with the brand heic, heix or mif1, followed by the same text within the next 8,388,608 bytes (it is at byte 443,072 in the Samsung HEIC sample, after the image data).

The two formats have priority over the Exif, JFIF and raw JPEG formats (fmt/41, 42, 43, 44, 645, 1507 and x-fmt/390, 391, 398) and over HEIC (fmt/1101), so the Motion Photo is the first result and the plain format is still reported after it. A file that has the MotionPhoto flag in its XMP but no video item is not a Motion Photo and keeps its ordinary result. The samples are in src/test/resources/pronom-samples and are checked by PronomSamplesDetectionTest like all other samples: three files whose names start with fmt-DRAFT-MotionPhoto- (two JPEG, one HEIC; each must be detected as the Motion Photo format first and the plain format after it), and two fmt-41_ JPEG samples that must stay plain JPEG (a Motion Photo flag without a video item, and the legacy MicroVideo layout).

Not covered: Apple Live Photos (two separate files, a HEIC or JPEG and a MOV, where the still image cannot be told from any other HEIC), the older Google MicroVideo layout (MVIMG_*.jpg), Samsung photos from before the Google XMP was added, AVIF Motion Photos, and the Motion Photo variants of Xiaomi, OPPO, vivo and Huawei, for which no sample files were available.

Heuristic signatures for JSON, JavaScript and CSS

JSON, JavaScript and CSS are text formats without magic bytes, so these signatures only look for syntax that is typical of the format. They are deliberately strict: a missed file is better than a wrong label.

• JSON (fmt/817): a { or [ within the first 3 bytes. An object ({) must be followed directly by a quote or a whitespace character (space, tab, line feed or carriage return), and a quote must follow within the next 12 bytes. An array ([) must be followed within 12 bytes by {, a quote or [.

• JavaScript (x-fmt/423): one of these cues. Generic words such as return, function or var on their own are not enough.

  • module.exports or typeof within the first 3,000 bytes;
  • "use strict" or 'use strict', with quotes, within the first 1,000 bytes;
  • var within the first 64 bytes;
  • export within the first 1,000 bytes, directly followed by default, const , function, {, *, class , let , var or async ;
  • import { within the first 256 bytes, with } from within the next 160 bytes;
  • define( within the first 260 bytes, directly followed by [, a quote or function.

• CSS (x-fmt/224): two signatures, either of which is enough.

  • A { within the first 3,000 bytes, followed within 24 bytes by a full CSS property name (for example font-family, margin-top, background-color, color, width), with the : directly after the name. The names are listed longest first (font-family before font), which the matching needs; see SIGNATURE_DESIGN.md.
  • A { within the first 3,000 bytes, followed within 8 bytes by -- (a custom property), with a : within 40 bytes after the {.

  The rule requires whole property names with the colon right behind them because the earlier rule, which accepted property-name prefixes within 200 bytes of a brace, also matched code and text that use braces and colons: it labelled 100 of 2,454 source and text files as CSS (Python dictionaries such as {'color': ...}, Ruby hashes and prose).

The three formats are the last ones in the format list of the signature file, in the order JSON, JavaScript, CSS. When several formats match a file, they are reported in the order of the format list, except that a format that another matching format outranks by a PRONOM priority rule comes after the others. So a file that matches one of these three and a format with a real PRONOM signature is reported as the real format first, and a file that matches several of the three is reported as JSON before JavaScript before CSS.

The results are heuristic. Measured on 9,105 real files from 58 npm packages, the signatures identified 95.6% of the JavaScript files and 100% of the JSON files (about 0.06% false positives); 3 of 3,402 non-JavaScript text files were labelled JavaScript. For CSS, measured on 834 real CSS files from npm packages, the signatures identified 98.2% (819 files); 13 of 7,287 other npm files and 4 of 2,454 source and text files (Python, Ruby, PHP, C, Java, Go and prose) were labelled CSS. On an independent set of assets from Python packages they identified 100% of the JSON files, 91.9% of the 37 CSS files (34 files) and about 66% of the JavaScript files; most of the JavaScript files that are missed are generated files without any JavaScript keyword. The known CSS misses are style sheets with a very long header comment (Font Awesome) and style sheets that contain only @import lines (animate.css). No unidentified binary file is labelled CSS, and none of 288 binary files that no PRONOM signature identifies was labelled as CSS, JavaScript or JSON, and NoFalsePositivesOnBinaryDataTest checks that random bytes and the payload of the binary test samples never get one of the three formats. Known false positives are text files of other kinds that contain the same syntax, for example Django and Jinja template files. One case among the test samples: jQuery 1.7.1 (a minified JavaScript file) contains the text <!doctype html> and is therefore reported as HTML5 (fmt/471) first and as JavaScript second.

Limits

• No guessing. Only binary signatures are used, never file extensions. Only the formats that have a signature in the signature file are identified; plain text, for example, is not. CSS, JavaScript and JSON are identified by heuristic signatures (see above), which can be wrong.
• No container signatures. DROID also looks inside ZIP and OLE2 containers to tell, for example, a Word document from a spreadsheet. droid-light does not; such files are reported by their binary signature only.
• Damaged or truncated files are not identified when the signature needs something at the end of the file (the %%EOF of a PDF, the end of a PNG, the end marker of a JPEG). An empty result is the correct answer then.
• Search distance. A pattern with no offset limit in the signature file (for example an XMP marker that can sit anywhere) is looked for within the first or last 150,000 bytes of the file (MAX_ANCHOR_SEARCH_DISTANCE in DroidSignatureVerifier). A pattern with a limit of its own is searched exactly as far as that limit allows and no further; the end marker of a JPEG, for example, may be followed by up to 131,072 bytes of other data. 150,000 covers the largest distance found in a test of 46,000 real files (145,408 bytes); a larger value costs almost nothing, because only a signature whose other byte sequences already matched pays for the long search.

Signatures that need an exhaustive search

droid-light does not try every combination of fragment options when it checks a signature. For each step of a signature's fragment chain (the parts that sit before or after the main byte pattern, separated by gaps of variable length, often with several alternatives per step) it takes the first place where a fragment matches, and it does not go back to try other places if a later step then fails. DROID itself explores all combinations. We chose this on purpose: exploring all combinations can take a time that grows very quickly with the number of steps, the number of alternatives and the length of the gaps, and this is what makes DROID hang on some files (see digital-preservation/droid#1663). In our view this is a design flaw in the signature file format, not in the individual signatures: the format allows a signature to require an exhaustive search, and nothing limits how expensive that search can be. A signature that is meant to be checked quickly should not need one.

The consequence is that a few signatures that DROID matches are not matched by droid-light: FictionBook 2 (fmt/2106), FASTQ files that start with an empty read (fmt/2118), and the alternative RTF rule of fmt/45 (it would only be a second result). In a test with 27,780 real files these cases affected 10 files, and no common web format was affected. Protection against hangs and speed have priority over completeness here.

Building

Java 11 or later: mvn clean install. The signature file is bundled in the jar.

License

BSD 3-Clause, see LICENSE. DROID is released under the New BSD License as well. droid-light is an independent implementation that reads the same signature file; it is not affiliated with or endorsed by The National Archives.

Third-party material

The signature files. DROID_SignatureFile_V125.xml was created by the PRONOM project of The National Archives, which publishes its content under the Open Government Licence v3.0. DROID_SignatureFile_V125_experimental.xml is a copy of it with the signatures added by this project (see above). Neither file is covered by this project's BSD license.

Contains public sector information licensed under the Open Government Licence v3.0.

DROID. Parts of the matching behaviour follow DROID, the file format identification tool developed by The National Archives and released under the New BSD License.

Test samples. The files in src/test/resources/pronom-samples are unmodified files from other projects, or files written by ffmpeg from a clip; they are test data only and are not covered by this project's BSD license. MANIFEST.json records for every file where it comes from (repository, commit and path) and under which licence it is published. The Motion Photo samples come from immich-app/test-assets (phone photos; the repository's LICENSE is AGPL-3.0, its README says the assets are public domain) and Apache Tika (two small synthetic test files, used as negatives, and a HEIC; Apache-2.0), with the same provenance in MANIFEST.json. Newer samples come from these sources:

• Apache POI, Apache Tika, Apache Commons Compress and Apache Commons Imaging, and pdf.js: Apache-2.0. For some files the repository does not document where the file itself comes from; the manifest says so.
• openexr-images (OpenEXR images): BSD-3-Clause.
• AOMediaCodec/av1-avif: BSD-2-Clause.
• OpenImageIO-images: the licence differs per file, and the manifest gives it for each one: the libtiff licence, Apache-2.0 (the OpenImageIO project), or "no expressive content" (as stated in the repository's THIRD-PARTY.md).
• ianare/exif-samples: CC BY-SA 4.0 (user-contributed). The three JPEG files taken from it are unmodified and stay under that licence.
• Nokia heif_conformance and openjpeg-data: no licence is stated for the files taken from them.
• OpenMPT (tracker modules): BSD-3-Clause.
• Ruffle test SWFs: MIT OR Apache-2.0. The files under from_shumway come from the Mozilla Shumway test corpus and are Apache-2.0; the manifest gives the licence for each file.
• Big Buck Bunny clips from opencv_extra: the content is licensed CC BY 3.0, (c) Blender Foundation, www.bigbuckbunny.org. opencv_extra itself states no licence. Some media samples were made with ffmpeg from one such clip; for those the manifest gives the exact command line instead of a source path.
• CSS, JavaScript and JSON samples (these formats have no binary signature in PRONOM's file): pallets/flask (a tutorial stylesheet, BSD-3-Clause), kognise/water.css (a minified stylesheet, MIT), git/git-scm.com (a script and a minified jQuery 1.7.1; the repository's own code is MIT, and jQuery is MIT), samayo/country-json (MIT) and Fyrd/caniuse (data licensed CC BY 4.0).
• BibTeX and LaTeX samples: JabRef (a BibTeX database; MIT) and pgf (a chapter of the PGF/TikZ manual; LaTeX Project Public License and/or GNU Free Documentation License, as stated in the file header).

Removed: Aho-Corasick detector

An alternative detector based on the Aho-Corasick algorithm, which finds the start patterns of all signatures in a single pass, was removed. It meant maintaining two implementations of the same rules, and it only paid off when every signature had to be searched across very large parts of a file, which normal detection does not need. The last version is kept in the git tag aho-corasick-archive.