Skip to content

SOURCE.md

Latest commit

 

History

History
40 lines (24 loc) · 2.76 KB

SOURCE.md

File metadata and controls

40 lines (24 loc) · 2.76 KB
Raw

Source and verification

Where the models come from and how to reproduce or confirm them, so nothing has to be taken on trust.

Model

Both files are float16 ONNX exports of one OpenCLIP model.

  • Upstream: laion/CLIP-ViT-B-16-DataComp.XL-s13B-b90K, license MIT. A CLIP ViT-B/16 trained on DataComp-1B with OpenCLIP (the open_clip pretrained tag datacomp_xl_s13b_b90k). The CLIP architecture and the byte-level BPE tokenizer vocabulary originate from OpenAI CLIP (MIT).
  • Export: pip install open_clip_torch onnx onnxruntime, then python convert/convert_clip.py. The script loads the model through open_clip, exports each tower to float32 ONNX (torch.onnx.export, opset 17, legacy exporter), converts the weights to float16 with onnxruntime.transformers.float16.convert_float_to_float16(keep_io_types=True) so the inputs and outputs stay float32, and self-checks that the ONNX matches the reference model and that the float16 matches float32 (both cosines print 1.00000).

Input / output

  • clip_image.onnx: input pixel_values [1,3,224,224] float32 (RGB, NCHW); output image_embeds [1,512] float32.
  • clip_text.onnx: input input_ids [1,77] int32; output text_embeds [1,512] float32.

Both embeddings are returned unnormalized; the client L2-normalizes each before the cosine similarity (a dot product once both sides are unit length).

Image preprocessing the client must match: resize the shorter edge to fill 224, center-crop to 224x224, scale to 0..1, then normalize per channel with CLIP mean (0.48145466, 0.4578275, 0.40821073) and std (0.26862954, 0.26130258, 0.27577711), red-green-blue order, channel-planes first (NCHW). Text: the open_clip byte-level BPE tokenizer (49408 vocab, start-of-text 49406, end-of-text 49407), padded or truncated to 77 tokens.

Float16

The models are float16: half the size and memory footprint of float32, with no meaningful loss in retrieval quality. The inputs and outputs stay float32 (keep_io_types), so the client feeds ordinary float pixels and int token ids and reads float embeddings.

On mobile ONNX Runtime, load them with basic graph optimization (ORT_ENABLE_BASIC), which runs the model's own operators and their float16 CPU kernels. (The extended level fuses GELU into a com.microsoft.Gelu contrib op that has no mobile float16 kernel.)

Verify

sha256sum clip_image.onnx clip_text.onnx

Expected:

57bd1d3bc57e74d99a14a1ef0d1de717049dc2a14cb867a5927907a88c4eecaf  clip_image.onnx
2ca4794ab703fdccf8bd903ae6fa52835ba657c8b68e1d569fb59400a859d282  clip_text.onnx

Sizes: clip_image.onnx 172680414 bytes, clip_text.onnx 127267139 bytes.