ANE/benchmarks/local_m2_results.md

Local Apple M2 ANE Benchmark Results

Date: 2026-06-05

Host:

• Mac mini (Mac14,3), Apple M2, 8-core CPU (4 performance + 4 efficiency), 8 GB memory
• macOS 26.5, build 25F71
• Apple clang 21.0.0 (clang-2100.1.1.101)
• ANE subtype reported by the private benchmark API: h14
• Repository commit tested: d91c9845c0 (origin/main)

Runtime notes:

• Results were measured from a clean worktree at /private/tmp/ANE-m2-clean to avoid local code changes affecting benchmark numbers.
• Benchmarks were run sequentially on the same machine because parallel ANE workloads contend for the accelerator and produce outliers.
• Tables below use the median of three runs where repeated. Raw logs were kept locally under /private/tmp/ane_m2_2026-06-05_clean/.
• Qwen3-0.6B dynamic training was not run on this 8 GB M2 machine; resident fp32 weights, gradients, Adam state, activations, transposed buffers, and IOSurfaces are expected to be memory-heavy.
• Two upstream output quirks were observed but not fixed in this results-only report: inmem_peak prints an invalid %peak value, and ane_int8_bench labels the h14 run as M4. The tables below use the raw TFLOPS/TOPS and host metadata instead.

In-Memory Baseline

Command:

xcrun clang -O2 -fobjc-arc -framework Foundation -framework IOSurface -ldl \
  -o inmem_bench inmem_bench.m
./inmem_bench

Median of three runs:

Config	Weight MB	ms/eval	TFLOPS
256ch x 64sp	0.1	0.136	0.06
512ch x 64sp	0.5	0.140	0.24
1024ch x 64sp	2.0	0.204	0.66
2048ch x 64sp	8.0	0.358	1.50
3072ch x 64sp	18.0	0.552	2.19
4096ch x 64sp	32.0	0.909	2.36

Peak-Style Conv Chain

Command:

xcrun clang -O2 -fobjc-arc -framework Foundation -framework CoreML \
  -framework IOSurface -ldl -o inmem_peak inmem_peak.m
./inmem_peak

Median of three runs. % peak below is computed against the M2 15.8 TFLOPS FP16 reference; the current program output prints an invalid %peak column.

Config	Weight MB	GFLOP	ms/eval	TFLOPS	% peak
32x conv 512ch sp64	16.0	1.07	0.239	4.50	28.5
48x conv 512ch sp64	24.0	1.61	0.280	5.74	36.3
64x conv 512ch sp64	32.0	2.15	0.301	7.13	45.1
96x conv 512ch sp64	48.0	3.22	0.404	7.98	50.5
128x conv 512ch sp64	64.0	4.29	0.537	7.99	50.6
64x conv 256ch sp64	8.0	0.54	0.160	3.35	21.2
128x conv 256ch sp64	16.0	1.07	0.222	4.84	30.6
256x conv 256ch sp64	32.0	2.15	0.340	6.32	40.0
64x conv 384ch sp64	18.0	1.21	0.245	4.94	31.3
128x conv 384ch sp64	36.0	2.42	0.345	7.00	44.3

INT8 W8A8

Command:

xcrun clang -O2 -fobjc-arc -framework Foundation -framework IOSurface -ldl \
  -o ane_int8_bench ane_int8_bench.m
./ane_int8_bench

Median of three runs:

Config	Precision	Weight MB	GOP	ms/eval	TOPS	Ratio
128x conv 512ch 64x64	FP16	64.0	274.88	22.847	12.03	-
128x conv 512ch 64x64	W8A8	32.0	274.88	23.046	11.93	1.01x
64x conv 512ch 64x64	FP16	32.0	137.44	12.442	11.05	-
64x conv 512ch 64x64	W8A8	16.0	137.44	11.861	11.59	1.06x
256x conv 256ch 64x64	FP16	32.0	137.44	12.984	10.59	-
256x conv 256ch 64x64	W8A8	16.0	137.44	13.272	10.36	1.05x
128x conv 256ch 64x64	FP16	16.0	68.72	6.801	10.10	-
128x conv 256ch 64x64	W8A8	8.0	68.72	6.348	10.83	1.11x
128x conv 384ch 64x64	FP16	36.0	154.62	14.220	10.87	-
128x conv 384ch 64x64	W8A8	18.0	154.62	13.770	11.23	1.08x

On this M2, W8A8 is approximately parity to a modest improvement for these kernels, not the larger M4 speedup reported upstream.

Dynamic Matmul

Command:

cd training
xcrun clang -O2 -Wall -DACCELERATE_NEW_LAPACK -fobjc-arc \
  -o test_dynamic_matmul test_dynamic_matmul.m \
  -framework Foundation -framework CoreML -framework IOSurface -ldl -framework Accelerate
./test_dynamic_matmul

Result:

• 64x64 identity correctness: PASS, max error 0.001938
• 64x64 scale-by-2 correctness: PASS, ratio 2.000
• 768x768x256 single dynamic matmul: 1.012 ms/eval, 298.4 GFLOP/s
• With weight IO: 0.871 ms/eval, 346.8 GFLOP/s
• vs cblas_sgemm: PASS, max error 0.014646

Tiled 768x768 matmul:

tile_oc	tiles	compile ms	eval ms	GFLOP/s
64	12	543	4.318	69.9
128	6	260	1.752	172.4
256	3	110	1.041	290.1
384	2	69	0.871	346.8
768	1	47	0.652	463.0

Dynamic Training

Data:

cd training
bash download_data.sh

The local run used tinystories_data00.bin: 20,658,981 tokens, 41.3 MB.

Build and run:

cd training/training_dynamic
make MODEL=stories110m
./train --scratch --steps 20 --accum 10 --warmup 2 --data ../tinystories_data00.bin

Result:

• Model: Stories110M, 109.5M parameters
• Active compact vocab: 9,205 tokens from 32,000
• One-time compile: 1,196 ms for 10 kernels
• Train time: 31,081 ms total
• Average train: 1,554.0 ms/step
• Wall time: 68.0 s
• Step 0 loss: 9.1105
• Step 10 loss: 8.6389

Notes:

• This is the dynamic weight pipeline, not the static pipeline used by the main cross-generation training table.
• The measured dynamic training run was IO-dominated on this 8 GB M2 machine.