Core ML and Create ML

The last chapter covered the ideas behind deep learning. This chapter puts them to work with Apple’s two machine learning frameworks:

• Create ML trains models on your Mac.
• Core ML runs trained models on the CPU, the GPU, and the Apple Neural Engine (ANE).

Most tutorials start with the Create ML app, where you drag a folder of data onto a window and click Train. I skip the app. Every model in this chapter comes from a Swift program or a Python script that you can keep in git, review in a pull request, and run again next month with the same result. We move from high level training APIs down to the hardware:

1. Train a text classifier with the CreateML framework, and compare two tabular classifiers built with the composable CreateMLComponents framework.
2. Convert a Hugging Face sentence embedding model to Core ML with coremltools, with int8 weights.
3. Use the MLTensor API for tensor math and async predictions.
4. Ask Core ML which device runs each operation with MLComputePlan, and find the operations that fall back from the Neural Engine to the CPU.
5. Look at stateful models (a KV-cache held in MLState) and multifunction models.
6. Tie it together in a capstone: a local RAG system where a small Create ML classifier routes questions and refuses off topic ones before a Core ML embedding model and Apple’s on-device language model do the rest.

Everything runs on your Mac. No API keys, no network calls at inference time, no cost per query.

The source code for this chapter is in the directory source-code/CoreML_and_CreateML. It is one Swift package with five command line programs and a shared library:

The examples need macOS 26 or later and Xcode 26 or later. You also need uv to run the Python conversion script, since uv installs the pinned versions of coremltools and PyTorch for you. Build all the models with two commands:

Both write to the models directory, which git ignores, since anyone can rebuild it.

The CreateML framework offers one type per task: MLTextClassifier, MLImageClassifier, MLBoostedTreeRegressor, and so on. You hand it data and parameters, and it hands back a trained model that you can test and save as a Core ML .mlmodel file.

Our text classifier is the router for the capstone at the end of the chapter. It reads a question and picks one of five labels: chemistry, economics, health, sports, or off_topic. The training data is a CSV file of short questions I wrote, about twenty per label, in training/router_queries.csv:

The program adds each sentence from the four documents the RAG system will search, labeled with the document’s topic, so the router also learns the words those documents use. Then it trains two models with different algorithms and keeps the better one:

A few details matter for reproducible training:

• DataFrame comes from Apple’s TabularData framework. CreateML accepts it directly, so you don’t need the older MLDataTable type.
• randomSplit(by:seed:) takes a seed. With a fixed seed, every run tests on the same 20% of the data.
• .maxEnt is a classic maximum entropy model over word features. .transferLearning(.bertEmbedding, …) trains a small classifier on top of a BERT model that ships with macOS. You don’t download anything.

Here is the output:

With only 171 examples, word counting fails: the maximum entropy model scores 49%. The BERT features already know that “basketball” sits close to “sport”, so transfer learning reaches 91% after eight seconds of training. The program saves the winner with write(to:metadata:), and the metadata (author, description, version) shows up in Xcode’s model viewer.

The CreateMLComponents framework splits training into parts you can combine: feature transformers, estimators, and the models they fit. Each estimator has an async fitted(to:) method, and each fitted model has an async applied(to:) method. This gives you more control than the one type per task CreateML API.

We use the Wisconsin breast cancer data from the anomaly detection chapter, this time for supervised learning. Each example becomes an AnnotatedFeature that pairs an MLShapedArray of nine features with a label:

We shuffle with a seeded generator (SeededGenerator in MLSupport) so the split repeats, then fit two estimators on the same training data:

ClassificationMetrics computes accuracy, precision, and recall from (predicted, actual) pairs:

Any fitted transformer can export itself as a Core ML model. We export the neural network, then load the file with plain Core ML to check its interface:

The output:

On this split, logistic regression gets every test case right and the network makes three mistakes. With 130 test cases, both numbers say “this data is easy”, not “logistic regression is better”. Change the seed and the order can flip.

Create ML covers common tasks, but most models you will want to run come from the PyTorch world. Apple’s coremltools Python package converts them. The script python/make_models.py starts with a header that tells uv which packages to install:

Pinned versions matter here. coremltools supports a narrow range of PyTorch releases, and it prints a warning for a version it has not been tested with.

We convert sentence-transformers/all-MiniLM-L6-v2, a small BERT model that maps text to a 384 number vector. The Hugging Face model returns one vector per token, so we wrap it to average the vectors of the real tokens and scale the result to unit length. That puts the whole sentence embedding step inside the Core ML model:

We trace the model with torch.jit.trace, convert it with a fixed input length of 128 tokens and float16 math, and then store the weights as 8-bit integers:

The script then runs the same sentence through PyTorch and Core ML:

A cosine similarity of 0.998 means quantization barely changed the embeddings. The int8 file is 22 MB; the float32 copy described next is 86 MB.

The script also saves a second copy, MiniLMEmbeddingFlexible.mlpackage, with float32 math and a flexible input length from 1 to 512 tokens. It has the same weights and gives the same answers. We use it later to show how these two choices change where Core ML runs the model.

Core ML needs a tokenizer on the Swift side too. The script writes the model’s vocabulary to minilm_vocab.txt, and WordPieceTokenizer.swift in MLSupport implements BERT’s uncased WordPiece scheme in about 80 lines: lowercase, strip accents, split on spaces and punctuation, then match each word against the vocabulary longest piece first, with ## marking pieces inside a word.

Xcode compiles .mlmodel and .mlpackage files into .mlmodelc bundles when it builds an app. A Swift package command line tool gets no such step, so ModelStore.swift compiles each model at first use with MLModel.compileModel(at:) and caches the result:

MLModelConfiguration holds the two settings we use most: computeUnits, which limits the devices Core ML may use, and functionName, which picks one function from a multifunction model.

Older Core ML code builds inputs as MLMultiArray objects and reads outputs back through MLFeatureProvider. Since macOS 15 and iOS 18, MLTensor gives you a typed, multidimensional array with the operations you know from NumPy or PyTorch, and models accept and return MLTensor values directly. Run the examples with:

The first part builds two small matrices, multiplies them, adds a bias vector that broadcasts across rows, and applies softmax:

Note the await on each read. Core ML runs tensor operations asynchronously, and shapedArray(of:) waits for the result and copies it into an MLShapedArray. Shapes, on the other hand, are known at once, so product.shape needs no await.

Embedder.swift in MLSupport wraps the MiniLM model. It tokenizes the text, builds two Int32 tensors, and calls the MLTensor form of prediction(from:):

Since the model returns unit length vectors, a matrix product gives cosine similarities, and topK picks the best matches:

Neither sports sentence shares a word with the query, yet both rank far above the chemistry sentence.

One caution from testing on macOS 27: topK(k) stops the program with a precondition failure when k equals the size of the axis. The capstone sorts with argsort(descendingOrder:) instead, which has no such limit.

withMLTensorComputePolicy runs a closure with a compute policy for the tensor operations inside it:

The GPU is not always faster. On Apple Silicon, the CPU handles 1024 by 1024 matrices well, and the GPU path pays for dispatch and synchronization. Measure on your own hardware before you pick a policy. You can also build a policy from any MLComputeUnits value with MLComputePolicy(.cpuAndNeuralEngine).

The async prediction(from:) method suspends the calling task instead of blocking a thread, so a SwiftUI app stays responsive while a model runs. Apple documents this API as safe to call from many tasks at once on one model. Embedder relies on that for its batch method, which embeds texts in a task group:

MLModel is not marked Sendable, so Swift 6 rejects sending it across tasks. My first version made Embedder an actor, and the compiler still refused: “sending ‘self.model’ risks causing data races”. Since Embedder holds only constants and the prediction API is documented as thread safe, @unchecked Sendable is the honest fix. The task group tags each result with its index because results arrive in completion order, not submission order.

Running predictions at the same time saves about a quarter of the time here, and each embedding takes well under a millisecond.

Setting computeUnits = .all gives Core ML permission to use the Neural Engine. It does not promise that Core ML will use it. Core ML decides per operation, and it will move work to the CPU without telling you when an operation, a data type, or a shape doesn’t suit the ANE. A model that runs on the ANE draws less power and leaves the GPU free. A model that silently falls back can be several times slower.

MLComputePlan (macOS 14.4 and later) answers the question directly. For each operation in an ML Program, it reports the devices that can run the operation, the device Core ML prefers, and an estimated relative cost. ComputePlanInspector walks the model structure and summarizes the plan:

Constants have no device usage, so deviceUsage(for:) returns nil for them. Control flow operations such as cond hold nested blocks, so the function recurses. The structure itself comes from the plan:

For each operation that misses the ANE, the inspector reports why. If supported includes the ANE, Core ML chose another device. If it doesn’t, the ANE can’t run that operation at all.

Run it on every model with swift run ComputePlanInspector. Here is the fixed shape, int8 MiniLM model:

All six transformer layers run on the ANE. Sixteen operations don’t, and the report shows why:

• One gather carries 45.8% of the estimated cost. That is the word embedding lookup: a 30,522 by 384 table indexed by token ids. The plan says the ANE does not support it. My guess is that the estimate weights it high because it reads a large table, since the lookup itself does little math.
• The small cast, greater_equal, select, and add operations work on the Int32 attention_mask input as BERT turns it into an additive mask. The plan lists only CPU and GPU for these integer operations.

These ops sit at the start of the graph, so the CPU hands off to the ANE once per prediction. That is a good result. If you wanted all of the model on the ANE, you would move the embedding lookup into Swift and pass float16 embeddings and a float16 mask as inputs.

Now the flexible length, float32 copy of the same network:

Every operation runs on the CPU, the graph has 111 more operations, and Core ML gives no cost estimates. This is the silent fallback to watch for. The model loads and returns correct answers, and nothing in the prediction API tells you it skipped both the ANE and the GPU. The report shows two causes:

• A flexible input length adds shape, gather, concat, and reshape operations that compute shapes at run time, and the ANE supports none of the dynamic reshape or shape operations.
• 24 of the 36 linear operations list only CPU and GPU. The ANE computes in float16, and this copy uses float32.

Fixed shapes and float16 math put the model on the ANE. If you need several input lengths, coremltools also supports enumerated shapes (a short list of fixed sizes), which the ANE handles far better than an open range.

Two more runs show the limits of the tool:

Without the ANE, the whole embedding model moves to the GPU, gather included. The tiny character language model from the next section is a different case: every op except three casts could run on the ANE, but Core ML keeps them all on the CPU. Most likely Core ML judged that, for layers only 128 wide, moving data to the ANE costs more than the math saves. The same plan can be right for one model and wrong for another, which is why you inspect rather than guess.

Last, the Create ML text classifier reports “Failed to construct compute plan, model is not supported.” MLComputePlan covers ML Programs and neural network models, and a Create ML text classifier is neither.

Use this checklist when the report shows ops off the ANE:

1. Is the op’s supported list missing ANE? Then the op, its data type, or its shape is the problem. Look for integer tensors, float32 weights, and flexible shapes.
2. Does supported include ANE but preferred say CPU? Then Core ML judged the transfer not worth it. Check whether the op is small, or sits between CPU ops.
3. Where are the off-ANE ops? A few at the start or end of the graph cost one handoff. Ops scattered through the middle cost a handoff each time.

A language model generates text one token at a time. A plain model must reprocess the whole context for every new token. A KV-cache keeps the attention keys and values from earlier tokens, so each step only computes the new token. Since macOS 15, Core ML supports this with state: tensors the model reads and writes in place across calls, held in an MLState object that Swift owns.

To keep the example small and quick to train, make_models.py trains a two layer character level transformer on the four documents in source-code/data. Training takes well under a minute on a laptop CPU. The model is tiny and it memorizes its training text, but it has every part a real decoder has.

The stateful module registers the caches as buffers. coremltools turns buffers named in states= into Core ML state:

Each call takes one token and its position. A one-hot vector writes the new key and value into cache slot position. Another mask hides the slots past position from attention. Every shape stays fixed, which keeps the graph simple for the converter and for the ANE. Indexing the cache with position directly would not work, since torch.jit.trace would record the index from the trace run as a constant.

Before converting, the script checks the cache logic. It feeds a prompt one character at a time and compares the final logits with one full forward pass of the base model:

The conversion declares the two caches as float16 state:

The script converts a second, stateless view of the same network: score takes a full 128 character window and returns log probabilities for every position. Then it merges both into one package:

save_multifunction stores weights that the functions share only once. This is how you ship a base model with several LoRA adapters, or separate prefill and decode functions, without paying for the weights twice. In Swift, you pick a function with MLModelConfiguration.functionName.

StatefulGeneration loads the generate function, makes a state, and passes it to every prediction:

The loop feeds the prompt to fill the cache, then samples new characters with the same call. Swift owns the state, so you can keep one MLState per chat session and drop it when the session ends.

This is where my testing hit a wall. On macOS 27.0.1 with Xcode 27, Core ML refuses to load the generate function. The plain model fails with “Failed to build the model execution plan … error code: -14”, and the same error comes back for the minimal accumulator example from the coremltools documentation, with coremltools 9.0 and 9.1.dev1, for every deployment target, and on CPU only. Stateless models from the same toolchain load fine. When you select generate in the multifunction package, the error text changes to a misleading “functionName property must be nil unless the model type is ML Program.” I don’t know the root cause; I could only rule out my model, since Apple’s own minimal example fails the same way.

So the program catches the error and falls back to the score function. Without a cache, it recomputes attention over the whole 128 character window for each new character:

The text drifts from sports into the chemistry document, which is what a character model trained on 1,700 words does. When a later macOS release loads stateful models again, the program uses the cached path without any change.

The score function has a use of its own. The mean log probability per character measures how much a string looks like the training text:

The first sentence comes from the training text, the second is new text on a topic the model saw, and the third is noise. A domain specific language model scored this way makes a cheap filter for input that doesn’t belong.

The final program combines the pieces into a question answering system that runs on the Mac:

The router does two jobs. As a guardrail, it stops requests the system should not handle before they reach the language model: chit chat, other tasks, and prompt injection attempts. As a semantic router, it limits retrieval to one topic’s passages, so a health question never retrieves an economics passage that happens to share words. Both jobs cost about 7 ms and no tokens.

Create ML text classifiers load as Core ML models, but the NaturalLanguage framework’s NLModel adds label probabilities on top:

A question passes only if the top label is a topic and its probability is at least 0.6. The threshold matters: in an earlier training run with a different set of document sentences, “Write a haiku about cats” came out as health with probability 0.49. The off_topic label catches what the training data covers; the threshold catches the rest.

Documents.swift splits each text file into passages of whole sentences, up to about 60 words, that never cross a paragraph break. At startup, PassageIndex embeds all 41 passages with the concurrent batch method and keeps one MLTensor matrix per topic:

For a few thousand passages, a matrix product like this is all the vector database you need.

The answer step uses Apple’s Foundation Models framework, which the chapter on Apple Intelligence covers. The instructions tell the model to use only the passages and to say so when they lack the answer:

If Apple Intelligence is off, or you pass —no-llm, the program prints the best passage instead, so the retrieval part still works on any Mac.

Your answers will differ in wording, since the language model samples its output. A few things stand out:

• The first router call takes 48 ms because it pays a one time setup cost. After that, routing takes 7 ms and retrieval about 1 ms. The language model takes about a second, more than 99% of the total.
• The WHO question routes and retrieves correctly, and the model then says the passages lack the answer. That is right: health.txt never mentions the World Health Organization. Grounded refusal is what the instructions ask for. Note the low best cosine (0.317): a check on that score could skip the LLM call, which is the next guardrail I would add.
• Both off topic requests stop at the router, so the prompt injection attempt never reaches the language model.

We trained two kinds of models from code with Create ML, converted and quantized a Hugging Face model with coremltools, and ran all of them from Swift with MLTensor and async predictions. MLComputePlan showed which operations run on the Neural Engine, and it caught a model that ran entirely on the CPU while returning correct results. Stateful and multifunction models let one package hold a KV-cache decoder and a scorer that share weights, though the stateful half would not load on the macOS 27 build I tested.

The capstone shows a pattern worth reusing: put a small, fast, custom trained model in front of a large general one. The small model refuses what it should, routes what it can, and costs a few milliseconds. The large model then does only the work that needs it.