mirror of https://github.com/maderix/ANE.git
Merge c3263bd618 into d91c9845c0
This commit is contained in:
Xiang Li
GitHub
commit
82c275d140
|
|
@ -94,6 +94,26 @@
|
|||
"peak_tflops_inmem": 12.17,
|
||||
"notes": "inmem_peak only, no training data submitted.",
|
||||
"contributor": "elijah-pelton"
|
||||
},
|
||||
{
|
||||
"chip": "M5 Max",
|
||||
"cores": "18-core (6P+12E)",
|
||||
"ram_gb": 128,
|
||||
"macos": "26.4.1",
|
||||
"ane_subtype": "h17",
|
||||
"ms_per_step": [89.7, 90.2],
|
||||
"ane_ms": [7.8, 8.1],
|
||||
"compile_ms": [3335, 3384],
|
||||
"ane_tflops": 1.03,
|
||||
"ane_util_pct": 5.4,
|
||||
"peak_tflops_inmem": 13.80,
|
||||
"peak_tflops_int8_w8a8": 35.61,
|
||||
"peak_tflops_fp16_64x64": 19.27,
|
||||
"ms_per_step_dynamic_stories110m": 73.5,
|
||||
"ms_per_step_dynamic_qwen3_06b": 320.0,
|
||||
"compile_ms_dynamic": 421,
|
||||
"notes": "First H17 ANE on record — distinct subtype from M4/M5 base (both H16). FP16/INT8 peak compute matches M4 within 4%; training gains over M5 base are CPU-driven (12P cores + Accelerate). See benchmarks/m5max_result.md for the full probe report.",
|
||||
"contributor": "lixiang.ict@gmail.com"
|
||||
}
|
||||
],
|
||||
"neural_engine_specs": {
|
||||
|
|
@ -108,6 +128,7 @@
|
|||
"M3_Ultra": {"ne_cores": 32, "rated_tops": 31.6},
|
||||
"M4": {"ne_cores": 16, "rated_tops": 38, "note": "INT8/mixed-precision spec"},
|
||||
"M4_Max": {"ne_cores": 16, "rated_tops": 38, "note": "INT8/mixed-precision spec"},
|
||||
"M5": {"ne_cores": 16, "rated_tops": null, "estimated_tops": 19}
|
||||
"M5": {"ne_cores": 16, "rated_tops": null, "estimated_tops": 19, "ane_subtype": "h16"},
|
||||
"M5_Max": {"ne_cores": 16, "rated_tops": null, "measured_fp16_tflops": 19.27, "measured_int8_tops": 35.61, "ane_subtype": "h17", "note": "First chip on record reporting H17 ANE subtype; peak math matches M4 H16."}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,229 @@
|
|||
# M5 Max ANE Probe & Training Benchmark
|
||||
|
||||
**Machine**: MacBook Pro · Apple M5 Max (6P + 12E CPU) · 128 GB RAM
|
||||
**macOS**: 26.4.1 (Darwin 25.4.0), Model Identifier `Mac17,7`
|
||||
**Date**: 2026-04-23
|
||||
**ANE Family**: **`h17`** (new — M4 and base M5 are `h16`)
|
||||
|
||||
All data gathered with the repo's probes and training harness as-is, no source
|
||||
changes. Compared against:
|
||||
- [README.md](../README.md) M4 reference figures
|
||||
- [training/m5result.md](../training/m5result.md) — base M5 (10-core, 16 GB) probe notes
|
||||
- [benchmarks/community_results.json](./community_results.json) — M1/M3/M4/M5 submissions
|
||||
|
||||
---
|
||||
|
||||
## Hardware identification
|
||||
|
||||
**Question**: Is the ANE in M5 Max the same silicon block as M4 / base M5?
|
||||
|
||||
**Result**: **No — `_ANEDeviceInfo.aneSubType` returns `h17`**, a version not
|
||||
seen in any community submission. The base M5 (per `m5result.md`) still reports
|
||||
`h16`, same as M4. M5 Max is the first `h17` on record.
|
||||
|
||||
```
|
||||
=== ANE INT8 W8A8 Benchmark (M4, h17) === ← header label is hardcoded "M4",
|
||||
the "h17" is read from the device.
|
||||
```
|
||||
|
||||
Everything else still works: `program(1.3)` MIL, `_ANEInMemoryModelDescriptor`,
|
||||
`constexpr_affine_dequantize`, `quantize` / `dequantize`. No API closures on
|
||||
macOS 26.4.1.
|
||||
|
||||
---
|
||||
|
||||
## inmem_peak — deep conv stacks (FP16)
|
||||
|
||||
**Question**: Peak FP16 throughput using the same 128-layer conv sweep
|
||||
([inmem_peak.m](../inmem_peak.m)) the README reports for M4.
|
||||
|
||||
```
|
||||
Config W(MB) GFLOP ms/eval TFLOPS
|
||||
-----------------------------------------------------------------
|
||||
32x conv 512ch sp64 16.0 1.07 0.135 ms 7.95
|
||||
48x conv 512ch sp64 24.0 1.61 0.171 ms 9.42
|
||||
64x conv 512ch sp64 32.0 2.15 0.206 ms 10.40
|
||||
96x conv 512ch sp64 48.0 3.22 0.266 ms 12.13
|
||||
128x conv 512ch sp64 64.0 4.29 0.311 ms 13.80 ← peak
|
||||
64x conv 256ch sp64 8.0 0.54 0.168 ms 3.19
|
||||
128x conv 256ch sp64 16.0 1.07 0.132 ms 8.16
|
||||
256x conv 256ch sp64 32.0 2.15 0.216 ms 9.94
|
||||
64x conv 384ch sp64 18.0 1.21 0.142 ms 8.52
|
||||
128x conv 384ch sp64 36.0 2.42 0.203 ms 11.91
|
||||
```
|
||||
|
||||
**Peak: 13.80 TFLOPS** at `128× conv 512ch sp=64`.
|
||||
|
||||
| Chip | inmem_peak FP16 (TFLOPS) |
|
||||
|------------|--------------------------|
|
||||
| M3 Pro | 9.98 |
|
||||
| M4 Pro | 12.57 |
|
||||
| M4 Max | 10.93 |
|
||||
| M5 (16 GB) | 12.17 |
|
||||
| M5 (32 GB) | 12.44 |
|
||||
| **M5 Max** | **13.80** |
|
||||
|
||||
---
|
||||
|
||||
## ane_int8_bench — FP16 vs INT8 W8A8 (larger spatial 64×64)
|
||||
|
||||
**Question**: How close does M5 Max come to the M4 blog's 19 TFLOPS / 35 TOPS
|
||||
figures when the conv is large enough to saturate the array?
|
||||
|
||||
```
|
||||
Config GOP ms/eval TOPS Ratio
|
||||
-----------------------------------------------------------------
|
||||
FP16 128x conv 512ch 64x64 274.88 14.263 ms 19.27
|
||||
W8A8 128x conv 512ch 64x64 274.88 7.720 ms 35.61 1.85x
|
||||
FP16 64x conv 512ch 64x64 137.44 7.153 ms 19.21
|
||||
W8A8 64x conv 512ch 64x64 137.44 3.824 ms 35.94 1.87x
|
||||
FP16 256x conv 256ch 64x64 137.44 7.318 ms 18.78
|
||||
W8A8 256x conv 256ch 64x64 137.44 4.118 ms 33.37 1.78x
|
||||
FP16 128x conv 256ch 64x64 68.72 3.696 ms 18.59
|
||||
W8A8 128x conv 256ch 64x64 68.72 2.112 ms 32.54 1.75x
|
||||
FP16 128x conv 384ch 64x64 154.62 8.154 ms 18.96
|
||||
W8A8 128x conv 384ch 64x64 154.62 4.389 ms 35.23 1.86x
|
||||
```
|
||||
|
||||
| Precision | M5 Max | M4 (README `H16G`) |
|
||||
|-----------|--------|---------------------|
|
||||
| FP16 peak | **19.27 TFLOPS** | 18.6 TFLOPS |
|
||||
| INT8 W8A8 peak | **35.61 TOPS** | 35.1 TOPS |
|
||||
| INT8/FP16 ratio | 1.85× | 1.88× |
|
||||
|
||||
**Implication**: the `h17` ANE's raw compute is **within 4 % of `h16`**
|
||||
(run-to-run noise). Apple kept the ~19 TFLOPS FP16 / ~35 TOPS INT8 ceiling
|
||||
across two chip generations. The "38 TOPS" spec remains the INT8 path.
|
||||
|
||||
---
|
||||
|
||||
## sram_bench — working-set cliff
|
||||
|
||||
**Question**: Where does the on-chip SRAM spill to DRAM?
|
||||
|
||||
```
|
||||
Config W(MB) Act(MB) Tot(MB) ms/eval TFLOPS
|
||||
---------------------------------------------------------------------
|
||||
256ch x 64sp 0.1 0.03 0.2 0.212 ms 0.04
|
||||
512ch x 64sp 0.5 0.06 0.6 0.085 ms 0.40
|
||||
1024ch x 64sp 2.0 0.12 2.2 0.335 ms 0.40
|
||||
2048ch x 64sp 8.0 0.25 8.5 0.141 ms 3.80
|
||||
3072ch x 64sp 18.0 0.38 18.8 0.204 ms 5.92
|
||||
4096ch x 64sp 32.0 0.50 33.0 0.300 ms 7.17
|
||||
5120ch x 64sp 50.0 0.62 51.2 0.432 ms 7.76
|
||||
6144ch x 64sp 72.0 0.75 73.5 0.565 ms 8.56
|
||||
8192ch x 32sp 128.0 0.50 129.0 0.965 ms 4.45
|
||||
```
|
||||
|
||||
**M4 in the blog shows the cliff around ~32 MB**. On M5 Max throughput is still
|
||||
climbing past **73 MB** and only breaks at **129 MB**. Caveat: the last row
|
||||
also halves `sp` from 64 to 32 — a pipeline-starvation confound we can't rule
|
||||
out without an independent probe. What's unambiguous: the effective SRAM
|
||||
working set is **at least as large as M4's**, plausibly larger.
|
||||
|
||||
---
|
||||
|
||||
## inmem_bench — single 1×1 conv latency scan
|
||||
|
||||
```
|
||||
Config W(MB) ms/eval TFLOPS
|
||||
--------------------------------------------
|
||||
256ch x64sp 0.1 0.088 ms 0.09
|
||||
512ch x64sp 0.5 0.089 ms 0.38
|
||||
1024ch x64sp 2.0 0.313 ms 0.43
|
||||
2048ch x64sp 8.0 0.131 ms 4.10
|
||||
3072ch x64sp 18.0 0.189 ms 6.38
|
||||
4096ch x64sp 32.0 0.302 ms 7.11
|
||||
```
|
||||
|
||||
Dispatch floor ≈ 0.09 ms, matching the M4 blog's ~0.095 ms XPC/IOKit overhead.
|
||||
|
||||
---
|
||||
|
||||
## Training — dynamic pipeline (`training_dynamic/train.m`)
|
||||
|
||||
**Synthetic token data** (5 M random uint16 in [0, 5000) to mimic a compressed
|
||||
TinyStories vocab), random init, `--accum 10`.
|
||||
|
||||
| Model | Params | Layers | Kernels compiled once | **M5 Max ms/step** |
|
||||
|-------|--------|--------|-----------------------|--------------------|
|
||||
| Stories110M (MHA 12/12) | 109 M | 12 | 421 ms | **73.5 ms** |
|
||||
| Qwen3-0.6B (GQA 16/8) | 596 M | 28 | 398 ms | **320.0 ms** |
|
||||
|
||||
Qwen3-0.6B per-step timing breakdown (stable from step 10+):
|
||||
|
||||
```
|
||||
ane_fwd=54.6 io_fwd=15.2 rms=4.5 ane_bwd=70.5 io_bwd=43.3
|
||||
silu=27.0 rms_bwd=12.4 cls=8.7 cblas_wait=0.0 dw_copy=9.9
|
||||
```
|
||||
|
||||
ANE time = 125 ms (39 %) · CPU time = 195 ms (61 %). Bottleneck is unchanged
|
||||
from the README's diagnosis: ANE is idle most of the step waiting for
|
||||
RMSNorm / SiLU / classifier / dW / Adam on CPU.
|
||||
|
||||
---
|
||||
|
||||
## Training — static pipeline (`train_large.m`)
|
||||
|
||||
For apples-to-apples with `community_results.json` (all existing entries use
|
||||
this path).
|
||||
|
||||
```
|
||||
[batch 10: compile=3384ms train=902.5ms (90.2ms/step) compiles=72]
|
||||
ane=8.0 io=2.8 cls=7.6 elem=11.7 rms=0.1 cblas_wait=0.0 ms/step
|
||||
[batch 20: compile=3335ms train=897.1ms (89.7ms/step) compiles=72]
|
||||
[batch 30: compile=3353ms train=900.6ms (90.1ms/step) compiles=72]
|
||||
Total steps: 30
|
||||
Wall time: 13.1 s
|
||||
Compile time: 10072 ms (76.9 %)
|
||||
Train time: 2700 ms (20.6 %)
|
||||
Avg train: 90.0 ms/step
|
||||
```
|
||||
|
||||
| Chip | ms/step | ane ms | compile / 10 |
|
||||
|-------------|---------|--------|--------------|
|
||||
| M1 Pro | 148–163 | 32–35 | 7.9–8.5 s |
|
||||
| M1 Max | 143–167 | 35–45 | ~7.1 s |
|
||||
| M3 Ultra\* | 91 | ~10 | ~3.7 s |
|
||||
| M4 Pro | 69–73 | 8.9 | ~3.5 s |
|
||||
| M4 Max | 64 | 10.2 | ~3.5 s |
|
||||
| M5 (16 GB) | 101–120 | 9.1–9.8| 3.2–3.4 s |
|
||||
| **M5 Max** | **90.0**| **8.0**| **~3.35 s** |
|
||||
|
||||
\* repo reference platform.
|
||||
|
||||
---
|
||||
|
||||
## Speedup summary — M5 Max vs baselines
|
||||
|
||||
| Metric | M4 (README) | M5 base | M5 Max | vs M4 | vs M5 base |
|
||||
|--------|-------------|---------|--------|-------|------------|
|
||||
| FP16 peak (TFLOPS) | 18.6 | 12.17–12.44 | **19.27** | 1.04× | 1.55× |
|
||||
| INT8 W8A8 (TOPS) | 35.1 | — | **35.61** | 1.01× | — |
|
||||
| Stories110M static (ms/step) | 91 | 101–120 | **90.0** | 1.01× | 1.22× |
|
||||
| Stories110M dynamic (ms/step)| — | — | **73.5** | — | — |
|
||||
| Qwen3-0.6B dynamic (ms/step) | 412| — | **320.0** | 1.29× | — |
|
||||
|
||||
**Takeaways**:
|
||||
|
||||
1. **Peak ANE compute has not moved between M4 and M5 Max** (≈ 19 TFLOPS FP16,
|
||||
≈ 35 TOPS INT8). The `h16 → h17` version bump does not show up in peak math.
|
||||
2. **Training gains of 1.22–1.29× are CPU-driven**, not ANE-driven. The 12
|
||||
performance cores plus Accelerate's `cblas_sgemm` on M5 Max close the gap
|
||||
that made base M5 (4P + 6E) slower than M4 Pro despite a newer ANE.
|
||||
3. **M5 Max's effective SRAM working set is ≥ M4's.** The `sram_bench` cliff
|
||||
sits past 70 MB where M4's was at ~32 MB, though a cleaner probe is needed
|
||||
(the 128 MB row changes two variables at once).
|
||||
|
||||
---
|
||||
|
||||
## Strategic implications
|
||||
|
||||
- Anyone optimizing training on this repo for M5 Max should focus on pushing
|
||||
RMSNorm / SiLU / classifier onto the ANE, not on peak-throughput MIL tricks —
|
||||
the ANE already has 60 % idle headroom per step.
|
||||
- `h17` is worth re-probing with the tests under `training/test_*.m` — the
|
||||
`m5result.md` findings (weight-reload fails, weightsBuffer is inert,
|
||||
procedureIndex is accepted but ignored, QoS has no effect) were recorded on
|
||||
`h16` and may or may not hold on `h17`.
|
||||
- No evidence that Apple has tightened the private-API surface on macOS 26.4.1.
|
||||
Loading…
Reference in New Issue