Skip to content

faster Unique and AllUnique. Add algo_bench.cc - #3354

Merged
copybara-service[bot] merged 3 commits into
google:masterfrom
a-eee:faster-unique
Sep 8, 2026
Merged

faster Unique and AllUnique. Add algo_bench.cc#3354
copybara-service[bot] merged 3 commits into
google:masterfrom
a-eee:faster-unique

Conversation

@a-eee

@a-eee a-eee commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Previously, Unique used CompressBlendedStore, with the rationale that CompressStore couldn't be used because of in-place. But this is wrong, because always stores <= loads, so the next load can't be affected. Also I added 4x unroll, which hides latency.

For AllUnique I also added 4x unroll, so now AllFalse is performed once every 4N elements.

On apple m5:

All measurements Before: 
=== BenchAll for NEON_BF16:
NEON_BF16:    unique:  random-runs             0%:  u8 1000000    84324.8 ns,  11.858910 GB/s
NEON_BF16:    unique:  random-runs            10%:  u8 1000000   154445.9 ns,   6.474757 GB/s
NEON_BF16:    unique:  random-runs            20%:  u8 1000000   178771.8 ns,   5.593724 GB/s
NEON_BF16:    unique:  random-runs            50%:  u8 1000000   180329.1 ns,   5.545416 GB/s
NEON_BF16:    unique:  random-runs            80%:  u8 1000000   179799.1 ns,   5.561764 GB/s
NEON_BF16:    unique:  random-runs            90%:  u8 1000000   179533.7 ns,   5.569985 GB/s
NEON_BF16:    unique:  random-runs           100%:  u8 1000000   145673.8 ns,   6.864651 GB/s
NEON_BF16:    unique:  random-runs             0%: i16 1000000    90008.3 ns,  22.220168 GB/s
NEON_BF16:    unique:  random-runs            10%: i16 1000000   155386.9 ns,  12.871094 GB/s
NEON_BF16:    unique:  random-runs            20%: i16 1000000   264025.0 ns,   7.575040 GB/s
NEON_BF16:    unique:  random-runs            50%: i16 1000000   355787.8 ns,   5.621328 GB/s
NEON_BF16:    unique:  random-runs            80%: i16 1000000   356649.2 ns,   5.607751 GB/s
NEON_BF16:    unique:  random-runs            90%: i16 1000000   356161.7 ns,   5.615426 GB/s
NEON_BF16:    unique:  random-runs           100%: i16 1000000   254565.9 ns,   7.856511 GB/s
NEON_BF16:    unique:  random-runs             0%: i32 1000000   166195.0 ns,  24.068116 GB/s
NEON_BF16:    unique:  random-runs            10%: i32 1000000   184724.1 ns,  21.653919 GB/s
NEON_BF16:    unique:  random-runs            20%: i32 1000000   307879.0 ns,  12.992116 GB/s
NEON_BF16:    unique:  random-runs            50%: i32 1000000   675335.3 ns,   5.922984 GB/s
NEON_BF16:    unique:  random-runs            80%: i32 1000000   713984.9 ns,   5.602360 GB/s
NEON_BF16:    unique:  random-runs            90%: i32 1000000   714828.5 ns,   5.595748 GB/s
NEON_BF16:    unique:  random-runs           100%: i32 1000000   449108.3 ns,   8.906537 GB/s
NEON_BF16:    unique:  random-runs             0%: i64 1000000   311411.3 ns,  25.689495 GB/s
NEON_BF16:    unique:  random-runs            10%: i64 1000000   322907.3 ns,  24.774911 GB/s
NEON_BF16:    unique:  random-runs            20%: i64 1000000   353565.6 ns,  22.626637 GB/s
NEON_BF16:    unique:  random-runs            50%: i64 1000000   949546.2 ns,   8.425077 GB/s
NEON_BF16:    unique:  random-runs            80%: i64 1000000  1397022.9 ns,   5.726463 GB/s
NEON_BF16:    unique:  random-runs            90%: i64 1000000  1430712.2 ns,   5.591621 GB/s
NEON_BF16:    unique:  random-runs           100%: i64 1000000   661232.9 ns,  12.098611 GB/s
NEON_BF16:    unique:   fixed-runs    period =   1:  u8 1000000    84363.2 ns,  11.853510 GB/s
NEON_BF16:    unique:   fixed-runs    period =  31:  u8 1000000   181107.3 ns,   5.521588 GB/s
NEON_BF16:    unique:   fixed-runs    period =  63:  u8 1000000   179492.1 ns,   5.571275 GB/s
NEON_BF16:    unique:   fixed-runs    period = 127:  u8 1000000   179290.2 ns,   5.577551 GB/s
NEON_BF16:    unique:   fixed-runs    period = 255:  u8 1000000   179302.1 ns,   5.577179 GB/s
NEON_BF16:    unique:   fixed-runs    period = 511:  u8 1000000   179798.4 ns,   5.561785 GB/s
NEON_BF16:    unique:   fixed-runs    period =   1: i16 1000000    89870.2 ns,  22.254312 GB/s
NEON_BF16:    unique:   fixed-runs    period =  31: i16 1000000   365059.3 ns,   5.478561 GB/s
NEON_BF16:    unique:   fixed-runs    period =  63: i16 1000000   358315.6 ns,   5.581671 GB/s
NEON_BF16:    unique:   fixed-runs    period = 127: i16 1000000   358076.1 ns,   5.585405 GB/s
NEON_BF16:    unique:   fixed-runs    period = 255: i16 1000000   355447.5 ns,   5.626710 GB/s
NEON_BF16:    unique:   fixed-runs    period = 511: i16 1000000   356087.4 ns,   5.616599 GB/s
NEON_BF16:    unique:   fixed-runs    period =   1: i32 1000000   161208.4 ns,  24.812604 GB/s
NEON_BF16:    unique:   fixed-runs    period =  31: i32 1000000   728458.8 ns,   5.491045 GB/s
NEON_BF16:    unique:   fixed-runs    period =  63: i32 1000000   717432.7 ns,   5.575436 GB/s
NEON_BF16:    unique:   fixed-runs    period = 127: i32 1000000   721324.7 ns,   5.545353 GB/s
NEON_BF16:    unique:   fixed-runs    period = 255: i32 1000000   716491.8 ns,   5.582757 GB/s
NEON_BF16:    unique:   fixed-runs    period = 511: i32 1000000   718103.4 ns,   5.570229 GB/s
NEON_BF16:    unique:   fixed-runs    period =   1: i64 1000000   306151.2 ns,  26.130877 GB/s
NEON_BF16:    unique:   fixed-runs    period =  31: i64 1000000  1433033.4 ns,   5.582563 GB/s
NEON_BF16:    unique:   fixed-runs    period =  63: i64 1000000  1433885.9 ns,   5.579245 GB/s
NEON_BF16:    unique:   fixed-runs    period = 127: i64 1000000  1443005.2 ns,   5.543985 GB/s
NEON_BF16:    unique:   fixed-runs    period = 255: i64 1000000  1435549.0 ns,   5.572781 GB/s
NEON_BF16:    unique:   fixed-runs    period = 511: i64 1000000  1437488.8 ns,   5.565261 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%:  u8 1000000     5116.9 ns,  48.857525 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%:  u8 1000000    10214.3 ns,  48.951161 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%:  u8 1000000    15408.1 ns,  48.675586 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%:  u8 1000000    20389.9 ns,  49.043892 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%: i16 1000000    10256.9 ns,  48.747614 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%: i16 1000000    20581.7 ns,  48.586956 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%: i16 1000000    30719.5 ns,  48.828850 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%: i16 1000000    41941.4 ns,  47.685531 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%: i32 1000000    20631.3 ns,  48.469981 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%: i32 1000000    41347.1 ns,  48.370932 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%: i32 1000000    62596.8 ns,  47.925737 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%: i32 1000000    83511.8 ns,  47.897428 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%: i64 1000000    35464.1 ns,  56.395105 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%: i64 1000000    71216.1 ns,  56.167103 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%: i64 1000000   110960.8 ns,  54.073157 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%: i64 1000000   149158.5 ns,  53.634209 GB/s
=== BenchAll for NEON:
NEON:    unique:  random-runs             0%:  u8 1000000    86144.7 ns,  11.608372 GB/s
NEON:    unique:  random-runs            10%:  u8 1000000   157937.0 ns,   6.331638 GB/s
NEON:    unique:  random-runs            20%:  u8 1000000   179565.2 ns,   5.569008 GB/s
NEON:    unique:  random-runs            50%:  u8 1000000   180091.1 ns,   5.552746 GB/s
NEON:    unique:  random-runs            80%:  u8 1000000   179701.2 ns,   5.564792 GB/s
NEON:    unique:  random-runs            90%:  u8 1000000   181235.1 ns,   5.517694 GB/s
NEON:    unique:  random-runs           100%:  u8 1000000   146741.5 ns,   6.814703 GB/s
NEON:    unique:  random-runs             0%: i16 1000000    91755.6 ns,  21.797032 GB/s
NEON:    unique:  random-runs            10%: i16 1000000   155831.4 ns,  12.834386 GB/s
NEON:    unique:  random-runs            20%: i16 1000000   261639.9 ns,   7.644093 GB/s
NEON:    unique:  random-runs            50%: i16 1000000   356186.4 ns,   5.615037 GB/s
NEON:    unique:  random-runs            80%: i16 1000000   356951.8 ns,   5.602998 GB/s
NEON:    unique:  random-runs            90%: i16 1000000   356952.3 ns,   5.602990 GB/s
NEON:    unique:  random-runs           100%: i16 1000000   258118.6 ns,   7.748377 GB/s
NEON:    unique:  random-runs             0%: i32 1000000   165901.7 ns,  24.110657 GB/s
NEON:    unique:  random-runs            10%: i32 1000000   186548.4 ns,  21.442151 GB/s
NEON:    unique:  random-runs            20%: i32 1000000   311125.6 ns,  12.856542 GB/s
NEON:    unique:  random-runs            50%: i32 1000000   680345.7 ns,   5.879364 GB/s
NEON:    unique:  random-runs            80%: i32 1000000   718351.0 ns,   5.568309 GB/s
NEON:    unique:  random-runs            90%: i32 1000000   720354.7 ns,   5.552820 GB/s
NEON:    unique:  random-runs           100%: i32 1000000   456433.5 ns,   8.763599 GB/s
NEON:    unique:  random-runs             0%: i64 1000000   315516.7 ns,  25.355232 GB/s
NEON:    unique:  random-runs            10%: i64 1000000   328762.1 ns,  24.333708 GB/s
NEON:    unique:  random-runs            20%: i64 1000000   367818.6 ns,  21.749852 GB/s
NEON:    unique:  random-runs            50%: i64 1000000   890322.0 ns,   8.985513 GB/s
NEON:    unique:  random-runs            80%: i64 1000000  1381721.3 ns,   5.789880 GB/s
NEON:    unique:  random-runs            90%: i64 1000000  1421099.7 ns,   5.629443 GB/s
NEON:    unique:  random-runs           100%: i64 1000000   739132.2 ns,  10.823503 GB/s
NEON:    unique:   fixed-runs    period =   1:  u8 1000000    83983.8 ns,  11.907057 GB/s
NEON:    unique:   fixed-runs    period =  31:  u8 1000000   179370.3 ns,   5.575061 GB/s
NEON:    unique:   fixed-runs    period =  63:  u8 1000000   179153.5 ns,   5.581806 GB/s
NEON:    unique:   fixed-runs    period = 127:  u8 1000000   179280.2 ns,   5.577860 GB/s
NEON:    unique:   fixed-runs    period = 255:  u8 1000000   179300.6 ns,   5.577226 GB/s
NEON:    unique:   fixed-runs    period = 511:  u8 1000000   179592.8 ns,   5.568153 GB/s
NEON:    unique:   fixed-runs    period =   1: i16 1000000    89543.7 ns,  22.335453 GB/s
NEON:    unique:   fixed-runs    period =  31: i16 1000000   355812.0 ns,   5.620946 GB/s
NEON:    unique:   fixed-runs    period =  63: i16 1000000   355553.8 ns,   5.625027 GB/s
NEON:    unique:   fixed-runs    period = 127: i16 1000000   355144.2 ns,   5.631515 GB/s
NEON:    unique:   fixed-runs    period = 255: i16 1000000   355146.8 ns,   5.631473 GB/s
NEON:    unique:   fixed-runs    period = 511: i16 1000000   355338.3 ns,   5.628439 GB/s
NEON:    unique:   fixed-runs    period =   1: i32 1000000   162528.8 ns,  24.611026 GB/s
NEON:    unique:   fixed-runs    period =  31: i32 1000000   712491.3 ns,   5.614104 GB/s
NEON:    unique:   fixed-runs    period =  63: i32 1000000   711975.2 ns,   5.618173 GB/s
NEON:    unique:   fixed-runs    period = 127: i32 1000000   713911.5 ns,   5.602935 GB/s
NEON:    unique:   fixed-runs    period = 255: i32 1000000   710806.0 ns,   5.627414 GB/s
NEON:    unique:   fixed-runs    period = 511: i32 1000000   710528.8 ns,   5.629610 GB/s
NEON:    unique:   fixed-runs    period =   1: i64 1000000   307141.1 ns,  26.046658 GB/s
NEON:    unique:   fixed-runs    period =  31: i64 1000000  1424395.0 ns,   5.616419 GB/s
NEON:    unique:   fixed-runs    period =  63: i64 1000000  1424536.0 ns,   5.615864 GB/s
NEON:    unique:   fixed-runs    period = 127: i64 1000000  1425119.2 ns,   5.613565 GB/s
NEON:    unique:   fixed-runs    period = 255: i64 1000000  1422616.2 ns,   5.623442 GB/s
NEON:    unique:   fixed-runs    period = 511: i64 1000000  1421607.3 ns,   5.627433 GB/s
NEON: AllUnique:   alt-prefix            25%:  u8 1000000     5122.9 ns,  48.800418 GB/s
NEON: AllUnique:   alt-prefix            50%:  u8 1000000    10425.7 ns,  47.958509 GB/s
NEON: AllUnique:   alt-prefix            75%:  u8 1000000    15550.2 ns,  48.230991 GB/s
NEON: AllUnique:   alt-prefix           100%:  u8 1000000    20620.7 ns,  48.495030 GB/s
NEON: AllUnique:   alt-prefix            25%: i16 1000000    10354.2 ns,  48.289501 GB/s
NEON: AllUnique:   alt-prefix            50%: i16 1000000    20487.5 ns,  48.810287 GB/s
NEON: AllUnique:   alt-prefix            75%: i16 1000000    32082.5 ns,  46.754420 GB/s
NEON: AllUnique:   alt-prefix           100%: i16 1000000    42414.8 ns,  47.153379 GB/s
NEON: AllUnique:   alt-prefix            25%: i32 1000000    20516.4 ns,  48.741586 GB/s
NEON: AllUnique:   alt-prefix            50%: i32 1000000    42244.0 ns,  47.344057 GB/s
NEON: AllUnique:   alt-prefix            75%: i32 1000000    63509.1 ns,  47.237289 GB/s
NEON: AllUnique:   alt-prefix           100%: i32 1000000    84974.4 ns,  47.073009 GB/s
NEON: AllUnique:   alt-prefix            25%: i64 1000000    36607.1 ns,  54.634153 GB/s
NEON: AllUnique:   alt-prefix            50%: i64 1000000    73108.0 ns,  54.713607 GB/s
NEON: AllUnique:   alt-prefix            75%: i64 1000000   109322.7 ns,  54.883379 GB/s
NEON: AllUnique:   alt-prefix           100%: i64 1000000   148125.9 ns,  54.008117 GB/s
=== BenchAll for NEON_WITHOUT_AES:
NEON_WITHOUT_AES:    unique:  random-runs             0%:  u8 1000000    86734.0 ns,  11.529510 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%:  u8 1000000   155291.8 ns,   6.439491 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%:  u8 1000000   179479.8 ns,   5.571656 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%:  u8 1000000   181251.7 ns,   5.517191 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%:  u8 1000000   181757.5 ns,   5.501838 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%:  u8 1000000   182492.0 ns,   5.479693 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%:  u8 1000000   144711.9 ns,   6.910282 GB/s
NEON_WITHOUT_AES:    unique:  random-runs             0%: i16 1000000    92854.0 ns,  21.539194 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%: i16 1000000   159837.4 ns,  12.512717 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%: i16 1000000   265847.3 ns,   7.523115 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%: i16 1000000   357349.3 ns,   5.596765 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%: i16 1000000   358459.7 ns,   5.579427 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%: i16 1000000   356441.1 ns,   5.611024 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%: i16 1000000   253424.1 ns,   7.891910 GB/s
NEON_WITHOUT_AES:    unique:  random-runs             0%: i32 1000000   166872.9 ns,  23.970339 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%: i32 1000000   188606.0 ns,  21.208236 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%: i32 1000000   311332.3 ns,  12.848007 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%: i32 1000000   677622.3 ns,   5.902993 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%: i32 1000000   716572.8 ns,   5.582126 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%: i32 1000000   717174.4 ns,   5.577444 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%: i32 1000000   476437.2 ns,   8.395650 GB/s
NEON_WITHOUT_AES:    unique:  random-runs             0%: i64 1000000   315148.4 ns,  25.384867 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%: i64 1000000   330637.3 ns,  24.195694 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%: i64 1000000   371432.2 ns,  21.538251 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%: i64 1000000   898535.4 ns,   8.903378 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%: i64 1000000  1402607.6 ns,   5.703662 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%: i64 1000000  1423906.3 ns,   5.618347 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%: i64 1000000   705954.2 ns,  11.332181 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =   1:  u8 1000000    84224.3 ns,  11.873065 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  31:  u8 1000000   179443.6 ns,   5.572782 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  63:  u8 1000000   179178.9 ns,   5.581014 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 127:  u8 1000000   179363.8 ns,   5.575260 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 255:  u8 1000000   179310.8 ns,   5.576908 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 511:  u8 1000000   179578.5 ns,   5.568595 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =   1: i16 1000000    89825.5 ns,  22.265383 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  31: i16 1000000   358171.4 ns,   5.583918 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  63: i16 1000000   355846.0 ns,   5.620410 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 127: i16 1000000   355238.3 ns,   5.630024 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 255: i16 1000000   355058.5 ns,   5.632875 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 511: i16 1000000   355227.6 ns,   5.630192 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =   1: i32 1000000   163713.5 ns,  24.432922 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  31: i32 1000000   714305.2 ns,   5.599847 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  63: i32 1000000   713922.0 ns,   5.602853 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 127: i32 1000000   712744.5 ns,   5.612109 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 255: i32 1000000   715164.6 ns,   5.593118 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 511: i32 1000000   714567.0 ns,   5.597796 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =   1: i64 1000000   312256.7 ns,  25.619945 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  31: i64 1000000  1432338.6 ns,   5.585272 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period =  63: i64 1000000  1424970.2 ns,   5.614152 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 127: i64 1000000  1422386.6 ns,   5.624350 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 255: i64 1000000  1422503.0 ns,   5.623890 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs    period = 511: i64 1000000  1420695.3 ns,   5.631046 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%:  u8 1000000     5152.8 ns,  48.517659 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%:  u8 1000000    10436.4 ns,  47.909018 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%:  u8 1000000    15391.1 ns,  48.729566 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%:  u8 1000000    20486.1 ns,  48.813700 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%: i16 1000000    10333.2 ns,  48.387735 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%: i16 1000000    21022.4 ns,  47.568299 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%: i16 1000000    31383.0 ns,  47.796590 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%: i16 1000000    42188.2 ns,  47.406612 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%: i32 1000000    21027.8 ns,  47.556130 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%: i32 1000000    43039.0 ns,  46.469519 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%: i32 1000000    63677.0 ns,  47.112806 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%: i32 1000000    85158.0 ns,  46.971506 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%: i64 1000000    36042.0 ns,  55.490818 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%: i64 1000000    72224.4 ns,  55.382913 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%: i64 1000000   108691.7 ns,  55.202012 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%: i64 1000000   151839.6 ns,  52.687171 GB/s
=== BenchAll for EMU128:
EMU128:    unique:  random-runs             0%:  u8 1000000   793888.9 ns,   1.259622 GB/s
EMU128:    unique:  random-runs            10%:  u8 1000000  1921776.1 ns,   0.520352 GB/s
EMU128:    unique:  random-runs            20%:  u8 1000000  2613870.9 ns,   0.382574 GB/s
EMU128:    unique:  random-runs            50%:  u8 1000000  3972623.0 ns,   0.251723 GB/s
EMU128:    unique:  random-runs            80%:  u8 1000000  2527370.5 ns,   0.395668 GB/s
EMU128:    unique:  random-runs            90%:  u8 1000000  1723792.0 ns,   0.580116 GB/s
EMU128:    unique:  random-runs           100%:  u8 1000000   615735.1 ns,   1.624075 GB/s
EMU128:    unique:  random-runs             0%: i16 1000000   237438.9 ns,   8.423220 GB/s
EMU128:    unique:  random-runs            10%: i16 1000000   749580.1 ns,   2.668160 GB/s
EMU128:    unique:  random-runs            20%: i16 1000000  1154002.2 ns,   1.733099 GB/s
EMU128:    unique:  random-runs            50%: i16 1000000  2173653.0 ns,   0.920110 GB/s
EMU128:    unique:  random-runs            80%: i16 1000000  1128025.2 ns,   1.773010 GB/s
EMU128:    unique:  random-runs            90%: i16 1000000   709310.4 ns,   2.819640 GB/s
EMU128:    unique:  random-runs           100%: i16 1000000   222927.7 ns,   8.971520 GB/s
EMU128:    unique:  random-runs             0%: i32 1000000   268492.3 ns,  14.898008 GB/s
EMU128:    unique:  random-runs            10%: i32 1000000   663042.5 ns,   6.032796 GB/s
EMU128:    unique:  random-runs            20%: i32 1000000  1053228.4 ns,   3.797847 GB/s
EMU128:    unique:  random-runs            50%: i32 1000000  2083701.7 ns,   1.919661 GB/s
EMU128:    unique:  random-runs            80%: i32 1000000  1058569.2 ns,   3.778685 GB/s
EMU128:    unique:  random-runs            90%: i32 1000000   665682.0 ns,   6.008875 GB/s
EMU128:    unique:  random-runs           100%: i32 1000000   224409.1 ns,  17.824590 GB/s
EMU128:    unique:  random-runs             0%: i64 1000000   228497.2 ns,  35.011365 GB/s
EMU128:    unique:  random-runs            10%: i64 1000000   701728.8 ns,  11.400415 GB/s
EMU128:    unique:  random-runs            20%: i64 1000000  1107302.9 ns,   7.224762 GB/s
EMU128:    unique:  random-runs            50%: i64 1000000  2262730.3 ns,   3.535552 GB/s
EMU128:    unique:  random-runs            80%: i64 1000000  1091724.1 ns,   7.327858 GB/s
EMU128:    unique:  random-runs            90%: i64 1000000   655168.0 ns,  12.210609 GB/s
EMU128:    unique:  random-runs           100%: i64 1000000   172802.9 ns,  46.295520 GB/s
EMU128:    unique:   fixed-runs    period =   1:  u8 1000000   800860.5 ns,   1.248657 GB/s
EMU128:    unique:   fixed-runs    period =  31:  u8 1000000  1087393.5 ns,   0.919630 GB/s
EMU128:    unique:   fixed-runs    period =  63:  u8 1000000   863512.0 ns,   1.158061 GB/s
EMU128:    unique:   fixed-runs    period = 127:  u8 1000000   751343.1 ns,   1.330950 GB/s
EMU128:    unique:   fixed-runs    period = 255:  u8 1000000   692575.7 ns,   1.443885 GB/s
EMU128:    unique:   fixed-runs    period = 511:  u8 1000000   660805.0 ns,   1.513306 GB/s
EMU128:    unique:   fixed-runs    period =   1: i16 1000000   234530.1 ns,   8.527689 GB/s
EMU128:    unique:   fixed-runs    period =  31: i16 1000000   241389.4 ns,   8.285368 GB/s
EMU128:    unique:   fixed-runs    period =  63: i16 1000000   228693.9 ns,   8.745313 GB/s
EMU128:    unique:   fixed-runs    period = 127: i16 1000000   227603.1 ns,   8.787228 GB/s
EMU128:    unique:   fixed-runs    period = 255: i16 1000000   239093.1 ns,   8.364941 GB/s
EMU128:    unique:   fixed-runs    period = 511: i16 1000000   228169.4 ns,   8.765419 GB/s
EMU128:    unique:   fixed-runs    period =   1: i32 1000000   263647.8 ns,  15.171753 GB/s
EMU128:    unique:   fixed-runs    period =  31: i32 1000000   237104.2 ns,  16.870222 GB/s
EMU128:    unique:   fixed-runs    period =  63: i32 1000000   252494.5 ns,  15.841931 GB/s
EMU128:    unique:   fixed-runs    period = 127: i32 1000000   258787.5 ns,  15.456698 GB/s
EMU128:    unique:   fixed-runs    period = 255: i32 1000000   241060.3 ns,  16.593362 GB/s
EMU128:    unique:   fixed-runs    period = 511: i32 1000000   230887.1 ns,  17.324486 GB/s
EMU128:    unique:   fixed-runs    period =   1: i64 1000000   220971.0 ns,  36.203852 GB/s
EMU128:    unique:   fixed-runs    period =  31: i64 1000000   176046.0 ns,  45.442662 GB/s
EMU128:    unique:   fixed-runs    period =  63: i64 1000000   176428.7 ns,  45.344091 GB/s
EMU128:    unique:   fixed-runs    period = 127: i64 1000000   199243.5 ns,  40.151883 GB/s
EMU128:    unique:   fixed-runs    period = 255: i64 1000000   198907.4 ns,  40.219727 GB/s
EMU128:    unique:   fixed-runs    period = 511: i64 1000000   187925.2 ns,  42.570128 GB/s
EMU128: AllUnique:   alt-prefix            25%:  u8 1000000    59440.5 ns,   4.205885 GB/s
EMU128: AllUnique:   alt-prefix            50%:  u8 1000000   116316.0 ns,   4.298635 GB/s
EMU128: AllUnique:   alt-prefix            75%:  u8 1000000   177174.3 ns,   4.233119 GB/s
EMU128: AllUnique:   alt-prefix           100%:  u8 1000000   234366.1 ns,   4.266829 GB/s
EMU128: AllUnique:   alt-prefix            25%: i16 1000000    36692.1 ns,  13.626924 GB/s
EMU128: AllUnique:   alt-prefix            50%: i16 1000000    73837.8 ns,  13.543196 GB/s
EMU128: AllUnique:   alt-prefix            75%: i16 1000000   110592.8 ns,  13.563268 GB/s
EMU128: AllUnique:   alt-prefix           100%: i16 1000000   148973.9 ns,  13.425167 GB/s
EMU128: AllUnique:   alt-prefix            25%: i32 1000000    34285.8 ns,  29.166580 GB/s
EMU128: AllUnique:   alt-prefix            50%: i32 1000000    69290.9 ns,  28.863816 GB/s
EMU128: AllUnique:   alt-prefix            75%: i32 1000000   103203.1 ns,  29.068880 GB/s
EMU128: AllUnique:   alt-prefix           100%: i32 1000000   139175.9 ns,  28.740606 GB/s
EMU128: AllUnique:   alt-prefix            25%: i64 1000000    29901.2 ns,  66.887046 GB/s
EMU128: AllUnique:   alt-prefix            50%: i64 1000000    59855.6 ns,  66.827498 GB/s
EMU128: AllUnique:   alt-prefix            75%: i64 1000000    89208.1 ns,  67.258475 GB/s
EMU128: AllUnique:   alt-prefix           100%: i64 1000000   119013.2 ns,  67.219412 GB/s

Success.
Now:

=== BenchAll for NEON_BF16:
NEON_BF16:    unique:  random-runs             0%:  u8 1000000    65845.2 ns,  15.187141 GB/s
NEON_BF16:    unique:  random-runs            10%:  u8 1000000    66495.9 ns,  15.038513 GB/s
NEON_BF16:    unique:  random-runs            20%:  u8 1000000    66409.5 ns,  15.058086 GB/s
NEON_BF16:    unique:  random-runs            50%:  u8 1000000    65952.0 ns,  15.162536 GB/s
NEON_BF16:    unique:  random-runs            80%:  u8 1000000    66260.2 ns,  15.092026 GB/s
NEON_BF16:    unique:  random-runs            90%:  u8 1000000    66349.5 ns,  15.071708 GB/s
NEON_BF16:    unique:  random-runs           100%:  u8 1000000    65514.6 ns,  15.263779 GB/s
NEON_BF16:    unique:  random-runs             0%: i16 1000000    61328.3 ns,  32.611357 GB/s
NEON_BF16:    unique:  random-runs            10%: i16 1000000    63504.5 ns,  31.493834 GB/s
NEON_BF16:    unique:  random-runs            20%: i16 1000000    62998.0 ns,  31.747042 GB/s
NEON_BF16:    unique:  random-runs            50%: i16 1000000    62671.3 ns,  31.912518 GB/s
NEON_BF16:    unique:  random-runs            80%: i16 1000000    62585.5 ns,  31.956293 GB/s
NEON_BF16:    unique:  random-runs            90%: i16 1000000    62595.6 ns,  31.951102 GB/s
NEON_BF16:    unique:  random-runs           100%: i16 1000000    61370.2 ns,  32.589118 GB/s
NEON_BF16:    unique:  random-runs             0%: i32 1000000    99447.4 ns,  40.222261 GB/s
NEON_BF16:    unique:  random-runs            10%: i32 1000000   102878.2 ns,  38.880944 GB/s
NEON_BF16:    unique:  random-runs            20%: i32 1000000   100748.8 ns,  39.702702 GB/s
NEON_BF16:    unique:  random-runs            50%: i32 1000000   100784.2 ns,  39.688778 GB/s
NEON_BF16:    unique:  random-runs            80%: i32 1000000   100208.3 ns,  39.916845 GB/s
NEON_BF16:    unique:  random-runs            90%: i32 1000000   100238.6 ns,  39.904794 GB/s
NEON_BF16:    unique:  random-runs           100%: i32 1000000    99968.2 ns,  40.012705 GB/s
NEON_BF16:    unique:  random-runs             0%: i64 1000000   192540.2 ns,  41.549771 GB/s
NEON_BF16:    unique:  random-runs            10%: i64 1000000   196104.8 ns,  40.794506 GB/s
NEON_BF16:    unique:  random-runs            20%: i64 1000000   194449.0 ns,  41.141886 GB/s
NEON_BF16:    unique:  random-runs            50%: i64 1000000   192809.0 ns,  41.491836 GB/s
NEON_BF16:    unique:  random-runs            80%: i64 1000000   190116.4 ns,  42.079475 GB/s
NEON_BF16:    unique:  random-runs            90%: i64 1000000   191091.3 ns,  41.864808 GB/s
NEON_BF16:    unique:  random-runs           100%: i64 1000000   190300.6 ns,  42.038757 GB/s
NEON_BF16:    unique:   fixed-runs   period =   1:  u8 1000000    64571.5 ns,  15.486704 GB/s
NEON_BF16:    unique:   fixed-runs   period =  31:  u8 1000000    64783.8 ns,  15.435953 GB/s
NEON_BF16:    unique:   fixed-runs   period =  63:  u8 1000000    64864.4 ns,  15.416776 GB/s
NEON_BF16:    unique:   fixed-runs   period = 127:  u8 1000000    64782.7 ns,  15.436218 GB/s
NEON_BF16:    unique:   fixed-runs   period = 255:  u8 1000000    69012.5 ns,  14.490128 GB/s
NEON_BF16:    unique:   fixed-runs   period = 511:  u8 1000000    67139.7 ns,  14.894310 GB/s
NEON_BF16:    unique:   fixed-runs   period =   1: i16 1000000    60298.7 ns,  33.168204 GB/s
NEON_BF16:    unique:   fixed-runs   period =  31: i16 1000000    61393.7 ns,  32.576634 GB/s
NEON_BF16:    unique:   fixed-runs   period =  63: i16 1000000    61375.3 ns,  32.586409 GB/s
NEON_BF16:    unique:   fixed-runs   period = 127: i16 1000000    61055.1 ns,  32.757315 GB/s
NEON_BF16:    unique:   fixed-runs   period = 255: i16 1000000    65174.7 ns,  30.686778 GB/s
NEON_BF16:    unique:   fixed-runs   period = 511: i16 1000000    62705.2 ns,  31.895262 GB/s
NEON_BF16:    unique:   fixed-runs   period =   1: i32 1000000    98333.8 ns,  40.677782 GB/s
NEON_BF16:    unique:   fixed-runs   period =  31: i32 1000000    99020.1 ns,  40.395824 GB/s
NEON_BF16:    unique:   fixed-runs   period =  63: i32 1000000   101381.6 ns,  39.454904 GB/s
NEON_BF16:    unique:   fixed-runs   period = 127: i32 1000000   103202.7 ns,  38.758686 GB/s
NEON_BF16:    unique:   fixed-runs   period = 255: i32 1000000   108254.7 ns,  36.949889 GB/s
NEON_BF16:    unique:   fixed-runs   period = 511: i32 1000000   102945.8 ns,  38.855408 GB/s
NEON_BF16:    unique:   fixed-runs   period =   1: i64 1000000   191555.8 ns,  41.763287 GB/s
NEON_BF16:    unique:   fixed-runs   period =  31: i64 1000000   188435.9 ns,  42.454753 GB/s
NEON_BF16:    unique:   fixed-runs   period =  63: i64 1000000   197816.7 ns,  40.441490 GB/s
NEON_BF16:    unique:   fixed-runs   period = 127: i64 1000000   198655.6 ns,  40.270706 GB/s
NEON_BF16:    unique:   fixed-runs   period = 255: i64 1000000   205708.5 ns,  38.889981 GB/s
NEON_BF16:    unique:   fixed-runs   period = 511: i64 1000000   196940.8 ns,  40.621350 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%:  u8 1000000     3367.4 ns,  74.241772 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%:  u8 1000000     6732.4 ns,  74.267930 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%:  u8 1000000    10038.8 ns,  74.710024 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%:  u8 1000000    13474.5 ns,  74.214170 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%: i16 1000000     7208.7 ns,  69.360989 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%: i16 1000000    14310.2 ns,  69.880181 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%: i16 1000000    21392.7 ns,  70.117409 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%: i16 1000000    28470.1 ns,  70.249208 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%: i32 1000000    13473.5 ns,  74.219695 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%: i32 1000000    27251.4 ns,  73.390705 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%: i32 1000000    41866.6 ns,  71.656142 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%: i32 1000000    55673.3 ns,  71.847747 GB/s
NEON_BF16: AllUnique:   alt-prefix            25%: i64 1000000    25826.5 ns,  77.439745 GB/s
NEON_BF16: AllUnique:   alt-prefix            50%: i64 1000000    54801.1 ns,  72.991210 GB/s
NEON_BF16: AllUnique:   alt-prefix            75%: i64 1000000    83377.9 ns,  71.961530 GB/s
NEON_BF16: AllUnique:   alt-prefix           100%: i64 1000000   111876.9 ns,  71.507183 GB/s
=== BenchAll for NEON:
NEON:    unique:  random-runs             0%:  u8 1000000    67406.7 ns,  14.835317 GB/s
NEON:    unique:  random-runs            10%:  u8 1000000    67648.2 ns,  14.782367 GB/s
NEON:    unique:  random-runs            20%:  u8 1000000    67992.4 ns,  14.707536 GB/s
NEON:    unique:  random-runs            50%:  u8 1000000    67328.8 ns,  14.852486 GB/s
NEON:    unique:  random-runs            80%:  u8 1000000    67490.4 ns,  14.816931 GB/s
NEON:    unique:  random-runs            90%:  u8 1000000    67278.4 ns,  14.863619 GB/s
NEON:    unique:  random-runs           100%:  u8 1000000    67069.9 ns,  14.909820 GB/s
NEON:    unique:  random-runs             0%: i16 1000000    62254.7 ns,  32.126089 GB/s
NEON:    unique:  random-runs            10%: i16 1000000    64220.5 ns,  31.142709 GB/s
NEON:    unique:  random-runs            20%: i16 1000000    63981.1 ns,  31.259229 GB/s
NEON:    unique:  random-runs            50%: i16 1000000    63756.5 ns,  31.369354 GB/s
NEON:    unique:  random-runs            80%: i16 1000000    63680.6 ns,  31.406761 GB/s
NEON:    unique:  random-runs            90%: i16 1000000    63676.4 ns,  31.408794 GB/s
NEON:    unique:  random-runs           100%: i16 1000000    62161.0 ns,  32.174501 GB/s
NEON:    unique:  random-runs             0%: i32 1000000   100899.2 ns,  39.643538 GB/s
NEON:    unique:  random-runs            10%: i32 1000000   103768.3 ns,  38.547431 GB/s
NEON:    unique:  random-runs            20%: i32 1000000   103120.3 ns,  38.789658 GB/s
NEON:    unique:  random-runs            50%: i32 1000000   101255.7 ns,  39.503960 GB/s
NEON:    unique:  random-runs            80%: i32 1000000   101421.1 ns,  39.439518 GB/s
NEON:    unique:  random-runs            90%: i32 1000000   100963.6 ns,  39.618250 GB/s
NEON:    unique:  random-runs           100%: i32 1000000   100436.6 ns,  39.826118 GB/s
NEON:    unique:  random-runs             0%: i64 1000000   195140.0 ns,  40.996199 GB/s
NEON:    unique:  random-runs            10%: i64 1000000   197575.9 ns,  40.490774 GB/s
NEON:    unique:  random-runs            20%: i64 1000000   195513.4 ns,  40.917907 GB/s
NEON:    unique:  random-runs            50%: i64 1000000   194311.7 ns,  41.170962 GB/s
NEON:    unique:  random-runs            80%: i64 1000000   191987.2 ns,  41.669443 GB/s
NEON:    unique:  random-runs            90%: i64 1000000   191604.1 ns,  41.752762 GB/s
NEON:    unique:  random-runs           100%: i64 1000000   192078.6 ns,  41.649616 GB/s
NEON:    unique:   fixed-runs   period =   1:  u8 1000000    65989.9 ns,  15.153826 GB/s
NEON:    unique:   fixed-runs   period =  31:  u8 1000000    65953.4 ns,  15.162212 GB/s
NEON:    unique:   fixed-runs   period =  63:  u8 1000000    66166.1 ns,  15.113480 GB/s
NEON:    unique:   fixed-runs   period = 127:  u8 1000000    66113.7 ns,  15.125463 GB/s
NEON:    unique:   fixed-runs   period = 255:  u8 1000000    70194.3 ns,  14.246166 GB/s
NEON:    unique:   fixed-runs   period = 511:  u8 1000000    68111.4 ns,  14.681831 GB/s
NEON:    unique:   fixed-runs   period =   1: i16 1000000    61283.7 ns,  32.635100 GB/s
NEON:    unique:   fixed-runs   period =  31: i16 1000000    62882.5 ns,  31.805326 GB/s
NEON:    unique:   fixed-runs   period =  63: i16 1000000    62756.7 ns,  31.869103 GB/s
NEON:    unique:   fixed-runs   period = 127: i16 1000000    62274.5 ns,  32.115885 GB/s
NEON:    unique:   fixed-runs   period = 255: i16 1000000    66442.8 ns,  30.101100 GB/s
NEON:    unique:   fixed-runs   period = 511: i16 1000000    63888.8 ns,  31.304378 GB/s
NEON:    unique:   fixed-runs   period =   1: i32 1000000    99437.2 ns,  40.226397 GB/s
NEON:    unique:   fixed-runs   period =  31: i32 1000000   100081.8 ns,  39.967300 GB/s
NEON:    unique:   fixed-runs   period =  63: i32 1000000   102382.8 ns,  39.069052 GB/s
NEON:    unique:   fixed-runs   period = 127: i32 1000000   103931.1 ns,  38.487040 GB/s
NEON:    unique:   fixed-runs   period = 255: i32 1000000   109485.8 ns,  36.534433 GB/s
NEON:    unique:   fixed-runs   period = 511: i32 1000000   104377.9 ns,  38.322304 GB/s
NEON:    unique:   fixed-runs   period =   1: i64 1000000   193529.4 ns,  41.337388 GB/s
NEON:    unique:   fixed-runs   period =  31: i64 1000000   190931.7 ns,  41.899795 GB/s
NEON:    unique:   fixed-runs   period =  63: i64 1000000   199753.7 ns,  40.049314 GB/s
NEON:    unique:   fixed-runs   period = 127: i64 1000000   199667.5 ns,  40.066603 GB/s
NEON:    unique:   fixed-runs   period = 255: i64 1000000   207383.7 ns,  38.575832 GB/s
NEON:    unique:   fixed-runs   period = 511: i64 1000000   197972.4 ns,  40.409666 GB/s
NEON: AllUnique:   alt-prefix            25%:  u8 1000000     3407.6 ns,  73.365191 GB/s
NEON: AllUnique:   alt-prefix            50%:  u8 1000000     6793.4 ns,  73.601349 GB/s
NEON: AllUnique:   alt-prefix            75%:  u8 1000000    10172.1 ns,  73.730792 GB/s
NEON: AllUnique:   alt-prefix           100%:  u8 1000000    13510.6 ns,  74.016066 GB/s
NEON: AllUnique:   alt-prefix            25%: i16 1000000     7262.6 ns,  68.845403 GB/s
NEON: AllUnique:   alt-prefix            50%: i16 1000000    14429.6 ns,  69.302111 GB/s
NEON: AllUnique:   alt-prefix            75%: i16 1000000    21667.7 ns,  69.227577 GB/s
NEON: AllUnique:   alt-prefix           100%: i16 1000000    28805.9 ns,  69.430236 GB/s
NEON: AllUnique:   alt-prefix            25%: i32 1000000    13694.7 ns,  73.020802 GB/s
NEON: AllUnique:   alt-prefix            50%: i32 1000000    27376.9 ns,  73.054307 GB/s
NEON: AllUnique:   alt-prefix            75%: i32 1000000    42162.9 ns,  71.152663 GB/s
NEON: AllUnique:   alt-prefix           100%: i32 1000000    56005.3 ns,  71.421855 GB/s
NEON: AllUnique:   alt-prefix            25%: i64 1000000    25730.7 ns,  77.728051 GB/s
NEON: AllUnique:   alt-prefix            50%: i64 1000000    54861.4 ns,  72.911025 GB/s
NEON: AllUnique:   alt-prefix            75%: i64 1000000    83682.1 ns,  71.699944 GB/s
NEON: AllUnique:   alt-prefix           100%: i64 1000000   111940.5 ns,  71.466550 GB/s
=== BenchAll for NEON_WITHOUT_AES:
NEON_WITHOUT_AES:    unique:  random-runs             0%:  u8 1000000    67783.0 ns,  14.752952 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%:  u8 1000000    68153.6 ns,  14.672745 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%:  u8 1000000    68323.3 ns,  14.636300 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%:  u8 1000000    67924.4 ns,  14.722255 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%:  u8 1000000    67995.8 ns,  14.706787 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%:  u8 1000000    68139.6 ns,  14.675747 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%:  u8 1000000    67767.3 ns,  14.756388 GB/s
NEON_WITHOUT_AES:    unique:  random-runs             0%: i16 1000000    62573.9 ns,  31.962227 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%: i16 1000000    64555.7 ns,  30.981020 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%: i16 1000000    64035.6 ns,  31.232650 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%: i16 1000000    63826.2 ns,  31.335119 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%: i16 1000000    63760.7 ns,  31.367264 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%: i16 1000000    63930.3 ns,  31.284049 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%: i16 1000000    62578.8 ns,  31.959703 GB/s
NEON_WITHOUT_AES:    unique:  random-runs             0%: i32 1000000   100813.1 ns,  39.677371 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%: i32 1000000   104681.2 ns,  38.211251 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%: i32 1000000   102686.5 ns,  38.953529 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%: i32 1000000   102309.9 ns,  39.096902 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%: i32 1000000   101786.5 ns,  39.297944 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%: i32 1000000   101805.3 ns,  39.290676 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%: i32 1000000   101217.4 ns,  39.518903 GB/s
NEON_WITHOUT_AES:    unique:  random-runs             0%: i64 1000000   194939.1 ns,  41.038448 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            10%: i64 1000000   198363.3 ns,  40.330033 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            20%: i64 1000000   195618.6 ns,  40.895916 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            50%: i64 1000000   194590.5 ns,  41.111980 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            80%: i64 1000000   193154.6 ns,  41.417598 GB/s
NEON_WITHOUT_AES:    unique:  random-runs            90%: i64 1000000   192369.1 ns,  41.586721 GB/s
NEON_WITHOUT_AES:    unique:  random-runs           100%: i64 1000000   192421.8 ns,  41.575328 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =   1:  u8 1000000    66880.4 ns,  14.952067 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  31:  u8 1000000    66652.7 ns,  15.003133 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  63:  u8 1000000    66691.4 ns,  14.994433 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 127:  u8 1000000    66955.9 ns,  14.935193 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 255:  u8 1000000    71004.9 ns,  14.083537 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 511:  u8 1000000    68671.2 ns,  14.562138 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =   1: i16 1000000    61815.4 ns,  32.354381 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  31: i16 1000000    63311.2 ns,  31.590007 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  63: i16 1000000    63262.9 ns,  31.614117 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 127: i16 1000000    63022.7 ns,  31.734606 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 255: i16 1000000    67100.3 ns,  29.806137 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 511: i16 1000000    64002.6 ns,  31.248730 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =   1: i32 1000000   100393.4 ns,  39.843267 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  31: i32 1000000   100678.5 ns,  39.730446 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  63: i32 1000000   103006.1 ns,  38.832633 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 127: i32 1000000   104955.7 ns,  38.111315 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 255: i32 1000000   110302.1 ns,  36.264033 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 511: i32 1000000   104553.1 ns,  38.258078 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =   1: i64 1000000   193584.6 ns,  41.325599 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  31: i64 1000000   191677.6 ns,  41.736755 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period =  63: i64 1000000   200899.5 ns,  39.820902 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 127: i64 1000000   200627.7 ns,  39.874860 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 255: i64 1000000   208250.5 ns,  38.415275 GB/s
NEON_WITHOUT_AES:    unique:   fixed-runs   period = 511: i64 1000000   199532.3 ns,  40.093758 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%:  u8 1000000     3409.6 ns,  73.321745 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%:  u8 1000000     6798.9 ns,  73.540999 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%:  u8 1000000    10173.6 ns,  73.720391 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%:  u8 1000000    13542.8 ns,  73.839719 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%: i16 1000000     7271.7 ns,  68.759417 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%: i16 1000000    14472.4 ns,  69.097058 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%: i16 1000000    21690.1 ns,  69.155937 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%: i16 1000000    28879.7 ns,  69.252683 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%: i32 1000000    13726.9 ns,  72.849466 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%: i32 1000000    27434.2 ns,  72.901704 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%: i32 1000000    42104.6 ns,  71.251119 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%: i32 1000000    56102.4 ns,  71.298186 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            25%: i64 1000000    26089.1 ns,  76.660452 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            50%: i64 1000000    54784.1 ns,  73.013823 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix            75%: i64 1000000    84106.4 ns,  71.338250 GB/s
NEON_WITHOUT_AES: AllUnique:   alt-prefix           100%: i64 1000000   112157.5 ns,  71.328295 GB/s
=== BenchAll for EMU128:
EMU128:    unique:  random-runs             0%:  u8 1000000   447696.5 ns,   2.233656 GB/s
EMU128:    unique:  random-runs            10%:  u8 1000000   540215.3 ns,   1.851114 GB/s
EMU128:    unique:  random-runs            20%:  u8 1000000   535776.7 ns,   1.866449 GB/s
EMU128:    unique:  random-runs            50%:  u8 1000000   534754.0 ns,   1.870019 GB/s
EMU128:    unique:  random-runs            80%:  u8 1000000   538267.3 ns,   1.857813 GB/s
EMU128:    unique:  random-runs            90%:  u8 1000000   537868.7 ns,   1.859190 GB/s
EMU128:    unique:  random-runs           100%:  u8 1000000   454390.6 ns,   2.200750 GB/s
EMU128:    unique:  random-runs             0%: i16 1000000   212272.2 ns,   9.421866 GB/s
EMU128:    unique:  random-runs            10%: i16 1000000   610399.1 ns,   3.276545 GB/s
EMU128:    unique:  random-runs            20%: i16 1000000   983623.4 ns,   2.033298 GB/s
EMU128:    unique:  random-runs            50%: i16 1000000  1985526.1 ns,   1.007290 GB/s
EMU128:    unique:  random-runs            80%: i16 1000000   955103.8 ns,   2.094013 GB/s
EMU128:    unique:  random-runs            90%: i16 1000000   599804.1 ns,   3.334422 GB/s
EMU128:    unique:  random-runs           100%: i16 1000000   218592.1 ns,   9.149463 GB/s
EMU128:    unique:  random-runs             0%: i32 1000000   277981.3 ns,  14.389457 GB/s
EMU128:    unique:  random-runs            10%: i32 1000000   570252.2 ns,   7.014440 GB/s
EMU128:    unique:  random-runs            20%: i32 1000000   900735.6 ns,   4.440815 GB/s
EMU128:    unique:  random-runs            50%: i32 1000000  1828138.6 ns,   2.188018 GB/s
EMU128:    unique:  random-runs            80%: i32 1000000   903093.3 ns,   4.429221 GB/s
EMU128:    unique:  random-runs            90%: i32 1000000   573826.4 ns,   6.970749 GB/s
EMU128:    unique:  random-runs           100%: i32 1000000   199069.9 ns,  20.093448 GB/s
EMU128:    unique:  random-runs             0%: i64 1000000   163096.6 ns,  49.050686 GB/s
EMU128:    unique:  random-runs            10%: i64 1000000   555231.9 ns,  14.408394 GB/s
EMU128:    unique:  random-runs            20%: i64 1000000   893106.7 ns,   8.957497 GB/s
EMU128:    unique:  random-runs            50%: i64 1000000  1825342.8 ns,   4.382738 GB/s
EMU128:    unique:  random-runs            80%: i64 1000000   875241.1 ns,   9.140339 GB/s
EMU128:    unique:  random-runs            90%: i64 1000000   530969.5 ns,  15.066779 GB/s
EMU128:    unique:  random-runs           100%: i64 1000000   141285.6 ns,  56.622897 GB/s
EMU128:    unique:   fixed-runs   period =   1:  u8 1000000   447432.5 ns,   2.234974 GB/s
EMU128:    unique:   fixed-runs   period =  31:  u8 1000000   532067.1 ns,   1.879462 GB/s
EMU128:    unique:   fixed-runs   period =  63:  u8 1000000   526067.2 ns,   1.900898 GB/s
EMU128:    unique:   fixed-runs   period = 127:  u8 1000000   529411.2 ns,   1.888891 GB/s
EMU128:    unique:   fixed-runs   period = 255:  u8 1000000   511695.5 ns,   1.954287 GB/s
EMU128:    unique:   fixed-runs   period = 511:  u8 1000000   489715.9 ns,   2.042000 GB/s
EMU128:    unique:   fixed-runs   period =   1: i16 1000000   211005.8 ns,   9.478413 GB/s
EMU128:    unique:   fixed-runs   period =  31: i16 1000000   236315.8 ns,   8.463252 GB/s
EMU128:    unique:   fixed-runs   period =  63: i16 1000000   226719.4 ns,   8.821479 GB/s
EMU128:    unique:   fixed-runs   period = 127: i16 1000000   220972.0 ns,   9.050921 GB/s
EMU128:    unique:   fixed-runs   period = 255: i16 1000000   231374.3 ns,   8.644002 GB/s
EMU128:    unique:   fixed-runs   period = 511: i16 1000000   223120.9 ns,   8.963750 GB/s
EMU128:    unique:   fixed-runs   period =   1: i32 1000000   273629.4 ns,  14.618313 GB/s
EMU128:    unique:   fixed-runs   period =  31: i32 1000000   227240.0 ns,  17.602536 GB/s
EMU128:    unique:   fixed-runs   period =  63: i32 1000000   213413.5 ns,  18.742952 GB/s
EMU128:    unique:   fixed-runs   period = 127: i32 1000000   205040.8 ns,  19.508316 GB/s
EMU128:    unique:   fixed-runs   period = 255: i32 1000000   220416.2 ns,  18.147487 GB/s
EMU128:    unique:   fixed-runs   period = 511: i32 1000000   212709.4 ns,  18.804995 GB/s
EMU128:    unique:   fixed-runs   period =   1: i64 1000000   158497.3 ns,  50.474032 GB/s
EMU128:    unique:   fixed-runs   period =  31: i64 1000000   138652.7 ns,  57.698103 GB/s
EMU128:    unique:   fixed-runs   period =  63: i64 1000000   145225.9 ns,  55.086599 GB/s
EMU128:    unique:   fixed-runs   period = 127: i64 1000000   143506.0 ns,  55.746816 GB/s
EMU128:    unique:   fixed-runs   period = 255: i64 1000000   141089.4 ns,  56.701628 GB/s
EMU128:    unique:   fixed-runs   period = 511: i64 1000000   149324.5 ns,  53.574608 GB/s
EMU128: AllUnique:   alt-prefix            25%:  u8 1000000    51955.1 ns,   4.811846 GB/s
EMU128: AllUnique:   alt-prefix            50%:  u8 1000000   105045.7 ns,   4.759831 GB/s
EMU128: AllUnique:   alt-prefix            75%:  u8 1000000   158304.9 ns,   4.737694 GB/s
EMU128: AllUnique:   alt-prefix           100%:  u8 1000000   212751.2 ns,   4.700326 GB/s
EMU128: AllUnique:   alt-prefix            25%: i16 1000000    35799.1 ns,  13.966839 GB/s
EMU128: AllUnique:   alt-prefix            50%: i16 1000000    71937.5 ns,  13.900946 GB/s
EMU128: AllUnique:   alt-prefix            75%: i16 1000000   107756.4 ns,  13.920287 GB/s
EMU128: AllUnique:   alt-prefix           100%: i16 1000000   145164.5 ns,  13.777469 GB/s
EMU128: AllUnique:   alt-prefix            25%: i32 1000000    31845.6 ns,  31.401495 GB/s
EMU128: AllUnique:   alt-prefix            50%: i32 1000000    63689.0 ns,  31.402583 GB/s
EMU128: AllUnique:   alt-prefix            75%: i32 1000000    96070.1 ns,  31.227200 GB/s
EMU128: AllUnique:   alt-prefix           100%: i32 1000000   126929.3 ns,  31.513599 GB/s
EMU128: AllUnique:   alt-prefix            25%: i64 1000000    24402.0 ns,  81.960461 GB/s
EMU128: AllUnique:   alt-prefix            50%: i64 1000000    53690.5 ns,  74.501009 GB/s
EMU128: AllUnique:   alt-prefix            75%: i64 1000000    82926.1 ns,  72.353585 GB/s
EMU128: AllUnique:   alt-prefix           100%: i64 1000000   111025.4 ns,  72.055590 GB/s
For Unique this is up to 8x faster, for AllUnique 1.5x. In all cases this is better (but sometimes <5% slower because of noise).

For measurements I added algo_bench.cc. Currently it is only able to run AllUnique and Unique. In comments I describe how to add new.

@a-eee
a-eee marked this pull request as draft September 4, 2026 06:35
@a-eee
a-eee marked this pull request as ready for review September 4, 2026 08:21
jan-wassenberg
jan-wassenberg requested changes Sep 4, 2026
View reviewed changes

@jan-wassenberg jan-wassenberg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! Good point that CompressStore is safe within the vector-multiple loop; we only require CompressBlendedStore at the end. Nice speedup.

Comment thread hwy/contrib/algo/algo_bench.cc Outdated
Comment thread hwy/contrib/algo/algo_bench.cc
Comment thread hwy/contrib/algo/find-inl.h
@a-eee

a-eee commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

In CopyIf there are also CompressBlendedStore, it's possible to change it to CompressStore. But this works only if to has allocated enough space. If the user knew that from count = 20, to would have <= 10 elements and allocated 10, CompressStore would break. So, is it safe to change it? Current CopyIf seems to handle this case, but neon BlendedStore always writes the full register, so even with CompressBlendedStore this case isnt safe.

Some targets implements safe CompressBlendedStore, some - not. I think either we need to make all of them safe or allow to touch masked values.

@a-eee

a-eee commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

In x86_avx3-inl.h CompressStore for the partial vectors even the upper unused lanes of mask are used. Because of this CI fails on this PR. I added the guard for <16 bytes: And(mask, FirstN(d, HWY_MAX_LANES_D(D)))

@jan-wassenberg

jan-wassenberg commented Sep 8, 2026

Copy link
Copy Markdown
Member

Thanks for fixing AVX3 partial CompressStore.

CompressBlendedStore is documented to be implementable as LoadU+Blend+StoreU. I do not think we can guarantee anything beyond "won't SEGV if pointers are aligned". In msan builds (HWY_MEM_OPS_MIGHT_FAULT) even this isn't enough - msan would complain. We have SafeFillN and SafeCopyN which are safe regardless but slow. I intend to update those to use masking where native. Do you see any alternatives or improvements beyond that?

@copybara-service
copybara-service Bot merged commit e2eff5d into google:master Sep 8, 2026
52 checks passed
@a-eee

a-eee commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

So for num <= lanes it's possible to implement SafeCopyN = StoreN + LoadN. The comment above CopyIf doesnt guarantee that everything is OK if to was allocated for the exact count, but the tail handling has this comment:

// Avoid storing to `to` unless we know it should be kept - otherwise, we
// might overrun the end if it was allocated for the exact count.

If this case isn't required to be safe, it is safe to use CompressStore instead of blend. If safety is necessary: StoreN(compressed, count_true), but this will impact performance on neon. Other variants I've come up with:

  • two loops: the first would find how many elements remain, the second one would use CompressStore where it's safe.
  • CopyIf overload where the user guarantees the buffer allocation capacity.

@jan-wassenberg

jan-wassenberg commented Sep 8, 2026

Copy link
Copy Markdown
Member

Yes, I think we should aim for STL compatibility and not necessarily require padding.

Do you care enough about performance that it's worth providing a separate PaddedCopyIf (or a better name)?

@a-eee
a-eee deleted the faster-unique branch September 8, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jan-wassenberg jan-wassenberg jan-wassenberg approved these changes

Assignees

No one assigned

Labels

ready to pull

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@a-eee @jan-wassenberg