Preface

In previous articles, I have written about how to run the Llama 3.2 3B model locally based on Ollama and call it using Lobechat (see article:Home Data Center Series: Building a Private AI: A Detailed Tutorial on Building Open-Source Large Language Models Locally Based on Ollama). However, due to hardware limitations at the time, I could only deploy it in a CPU-only manner using Docker on an Intel CPU mini PC. Without GPU support, the output speed of 1-2 words per second when using the i5 13400 CPU was very frustrating, so at that time, I wondered what it would feel like to run Ollama on an M4 Mac mini.

After holding back for 2 weeks, I finally placed an order. Considering that I might need to run local large models frequently in the future and even learn some video-related operations, I gritted my teeth and ordered the base configuration version of the M4 Pro:

---

When running local large models (such as Llama 3.2), there will be performance differences between the M4 and M4 Pro, even though both are equipped with a 16-core Neural Engine. This is mainly because they differ in CPU and GPU performance:

CPU Performance: There are differences in the CPU core count and performance between the M4 and M4 Pro. The M4 Pro is equipped with more high-performance cores, while the M4 is limited in core count or performance. This means that for tasks requiring multi-threaded parallel processing (such as data preprocessing during deep learning inference), the M4 Pro will perform better than the M4.

GPU Performance: The M4 Pro has a higher GPU core count and higher computing power. In some AI model inference processes that require a large amount of matrix calculations, the M4 Pro's GPU can provide faster computing speeds. Therefore, the M4 Pro is typically faster when executing inference for large deep learning models (such as Llama 3.2).

Memory Bandwidth: The M4 Pro is equipped with larger memory bandwidth, which is also crucial for running large models. The larger the bandwidth, the faster data is transferred between the CPU, GPU, and Neural Engine, thereby reducing model loading and inference latency.

Neural Engine Utilization: Although the M4 and M4 Pro have the same number of Neural Engine cores, the optimization of the overall system architecture and data processing speed will affect their actual performance. Therefore, the M4 Pro will have more of an advantage when executing complex AI inference tasks.

So overall, although their Neural Engines are the same, the overall performance of the M4 Pro when executing large model inference will be superior to the M4, especially when handling more complex, resource-intensive tasks where the difference will be more pronounced. This also proves that my decision to grit my teeth and go for the M4 Pro was definitely not a waste of money~.

---

After it arrived, following 2 days of tinkering, the initialization of regular productivity tools was basically complete, and I could finally start verifying the effect of running Llama 3.2 using Ollama.

I originally thought that installation and running would be very simple, and there would be no need to write a new article; I could just set it up and add an animated GIF of the actual running effect to my previous article about Ollama. However, when I actually started doing it, I found quite a few pitfalls. In the end, I thought it would be better to write a dedicated article.

Mac version Ollama deployment process

Install Ollama and download and run the Llama 3.2 model

I mentioned the difference between Docker and source deployment in my previous article:

So this time, I will definitely deploy using the source method. On the official Ollama website (https://ollama.com/download), download the macOS installation file (it was version v0.4.1 when I downloaded it):

It is actually a directly executable file; double-clicking it copies it directly to the Applications folder, so I won't bother with screenshots here. After that, use the following commands to download the Llama 3.2 3B model and run it:

ollama pull llama3.2
ollama run llama3.2

ollama run llama3.2

After running successfully, you can already perform Q&A locally:Note 1: Directly running theollama run llama3.2command will also automatically pull the Llama 3.2 model file when Ollama finds it is not available locally, which is actually similar to和docker pull。

docker run

Note 2: You can also download other models. For specific models, versions, parameter scales, sizes, and commands, please refer to the figure below:

---

Note 3: Models in Ollama are all using 4-bit quantization by default (otherwise it's unplayable~).

---

Use ”locally deployed” Lobechat for calling

Additional Knowledge: Model ”quantization” is a technique that reduces model computation and memory consumption by lowering the precision of model weights and activation values. Traditionally, the weights and activation values of deep learning models are stored and calculated as 32-bit floating-point numbers (FP32), whereas quantization reduces the number of bits of these values to 16-bit, 8-bit, or even 4-bit or 2-bit. After quantization, the memory requirements and computational load of the model are significantly reduced, enabling large models that were originally difficult to run to operate on resource-constrained hardware (such as ordinary graphics cards or mobile phones). At the same time, quantization also minimizes the impact of precision loss through optimization algorithms.http://127.0.0.1:3210Note 1: The so-called "locally deployed" Lobechat here refers to the scenario where the device deploying Ollama, the device installing Lobechat, and the device accessing it as a client using a browser are all the same device, so the browser here uses

to access Lobechat.Docker Series: Detailed Deployment Tutorial Based on the Open Source Large Language Model UI Framework: LobeChat和Home Data Center Series: Unlocking the Full Potential of LobeChat: A Complete Guide from Setup to Practical ApplicationNote 2: For detailed deployment and settings of Lobechat, please refer to my other two articles:

. For the sake of brevity, this article will not expand on this in detail, assuming everyone is an experienced user.command will also automatically pull the Llama 3.2 model file when Ollama finds it is not available locally, which is actually similar toUse the following

docker run --name lobe-chat  -d --restart=always \
  -p 3210:3210 \
  -e ACCESS_CODE=xxx \
  lobehub/lobe-chat

Usehttp://127.0.0.1:3210docker run --name lobe-chat -d --restart=always \

-p 3210:3210 \

-e ACCESS_CODE=xxx \



lobehub/lobe-chat

to access it, and the "Check" in the Ollama section of the "Language Model" part can pass:

Note: I originally wanted to include a GIF converted from a screen recording when using CPU only, but a 1-minute-and-41-second screen recording converted to a GIF is just too large, so I won't include it. Anyway, the output speed is roughly about 2 words per second.

Let's look at the knowledge base training time of llama3.2:

Nearly a year ago, which is okay. After all, it's free, so it's already pretty good.

Up to now, for general users with modest requirements, actually using the defaulthttp://127.0.0.1:3210address to access ollama via the local version of LobeChat is already sufficient.

However, this also has a very big limitation: when accessing LobeChat, the browser address can only usehttp://127.0.0.1:3210to access it for the ollama language model check in LobeChat to pass; if you change the address, for example, if I use the actual IP address of the local network card,http://192.168.10.115:3210to access LobeChat, the check will fail:

This is because by default, ollama restricts cross-origin access for security reasons, only allowing calls from ”127.0.0.1". Therefore, when accessing the local LobeChat using the local loopback IP address (http://127.0.0.1:3210), the call to ollama can pass the check; however, when accessing the local LobeChat using the IP address of the local physical network card (http://192.168.10.130:3210), it won't work.

The trouble this causes is: if my ollama wants to provide services to LobeChat deployed on other hosts in the local area network, it won't work by default.

For general users, this might not be an issue, but the problem is: I have already deployed the server-side database version of LobeChat inside my home data center, and normal use will definitely involve calling it from that device. Therefore, for me, the cross-origin access issue must be resolved.

Resolve Ollama cross-origin access issues

Solutions to cross-origin issues in other deployment modes

Regarding the issue of allowing cross-origin access in ollama, I actually briefly mentioned it in a previous article. However, at that time, I was just using Docker deployment for testing, and it had no practical value, so I didn't study it deeply. Simply put, for Docker deployment, you only need to add environmental parameters during deployment:-e OLLAMA_ORIGINS="*"和-e OLLAMA_HOST="0.0.0.0"to solve the cross-origin issue.

If ollama is deployed on Linux using the source code method, you only need to follow these steps to allow cross-origin access:

Edit the corresponding service file for ollama

sudo systemctl edit ollama.service

Add the following content

[Service] 
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_ORIGINS=*"

Save and exit

Reload systemd and restart Ollama

sudo systemctl daemon-reload
sudo systemctl restart ollama

Setting up cross-origin for ollama on Windows is actually similar to Linux and is also simple. The general steps are as follows:

Resolve cross-origin issues when deploying Ollama from source on macOS

The key is how to solve the cross-origin issue on Mac. Actually, the official documentation also mentions that you only need to run the following commands and then restart the ollama application:

launchctl setenv OLLAMA_ORIGINS "*"
launchctl setenv OLLAMA_HOST "0.0.0.0"

Of course, if you only need to allow specific domains, run the following command:

launchctl setenv Ollama_ORIGINS "google.com,apple.com"

However, I found that it didn't take effect after trying it. But after I followed these steps:

export OLLAMA_ORIGINS="*"
export OLLAMA_HOST=0.0.0.0
ollama serve

it passed the check:

A bit confused, let me sort it out in order first.
This is before opening the ollama app, naturally blank:

Opening only the ollama app, there will be 3 processes:

At this time, if you run the commandNote 1: Directly running the, you get the following result:

Run the commandollama serveafterwards:

Run it againNote 1: Directly running the, and it will succeed:

And the processes at this time are as follows:

Kill all ollama-related processes to return to blank, and only run the commandollama serve：

Run it againNote 1: Directly running the, and it will succeed:

Therefore, we can basically conclude that the Ollama APP actually doesn't do much now after completing the installation task on its first run (some previous versions seemed to automatically run the command when opening the APP,ollama servewhich would still make sense, but what I observed here is that it doesn't run automatically). The key to running local large models subsequently is theollama servecommand and theollamm runThese 2 commands.

Note: When usingollama runcommand to run a local large model, the Ollama app will also start automatically. At this time, quitting the Ollama app will also quit the running large model, so saying that the Ollama app is completely useless afterwards is not quite correct either~, at least it can be used as a quick exit command for the large model~.

---

Additional knowledge: launchctl setenv is used in macOS to setsystem-level environment variablescommand. These environment variables will take effect for the entire system, including all user processes. Environment variables set using launchctl setenv can ”theoretically” be read by all applications and terminal sessions.

---

Then why did the previouslaunchctl setenv OLLAMA_ORIGINS "*"和launchctl setenv OLLAMA_HOST "0.0.0.0"settings fail?

On macOS, environment variables set using launchctl setenv are not always immediately recognized by all applications or command-line tools. This is because there are some specific limitations in macOS's environment variable management, especially for commands and services run directly from the terminal:

Scope of environment variables

Environment variables set by launchctl setenv are only valid inlaunchdstarted processes, but they are not directly passed to commands you start from the terminal. For example, using ollama serve to start the ollama process might not get environment variables from launchctl, but rather from the current terminal's session environment.

So if previously starting the ollama app would automatically run the commandollama serve, then using launchctl setenv was still effective. But now that starting the ollama app no longer automatically runs the commandollama serve, and you can only run it manually in the terminal yourself, the launchctl setenv method is naturally useless (at least for me).

Isolation of terminal sessions

When runningexport OLLAMA_ORIGINS="*"in the terminal, the variable is set in the environment of the current terminal session. When ollama is started from the terminal, it can directly obtain this environment variable without needing to read it from launchctl. Therefore, for commands run directly from the terminal, export is usually more effective.

---

Note: Usinglaunchctl setenvcommand to set ollama's environment variables might not always fail to take effect; it still depends on different macOS versions. So you can try it first. If it succeeds, you don't need to bother with theexportcommand.

---

In summary, if you are just temporarily starting ollama and need it to be called across domains, you only need to run the following commands in sequence in a terminal:

export OLLAMA_ORIGINS="*"
export OLLAMA_HOST=0.0.0.0
ollama serve
ollama run llama3.2

export OLLAMA_HOST=0.0.0.0exportollama serve

ollama run llama3.2If you need the environment variables to take effect for all terminals and started processes, you can add thecommands to your ~/.zshrc or ~/.bash_profile configuration file (depending on the shell type you use), so that the environment variables are automatically loaded every time you open a new terminal. You can refer to my operation steps.

echo $SHELL

zsh

, of course, you can also confirm it with the following command:

vim ~/.zshrc

For example, my Mac outputs the following:

export OLLAMA_ORIGINS="*"
export OLLAMA_HOST=0.0.0.0

vim ~/.zshrc

source ~/.zshrc

export OLLAMA_ORIGINS="*"ollama serve和ollama runexport OLLAMA_HOST=0.0.0.0exportcommand.

---

Finally, reload the configuration:

export OLLAMA_ORIGINS="*"
export OLLAMA_HOST=0.0.0.0
ollama serve
ollama run llama3.2

source ~/.zshrc

---

Afterwards, every time you open the terminal to run the

High-difficulty challenge

Llama 3.2 Vision model introduction

Llama 3.2 Vision, released by Meta, is a powerful open-source multimodal model that combines visual understanding and reasoning capabilities, making it suitable for tasks such as visual grounding, document question answering, and image-text retrieval. By applying Chain-of-Thought (CoT) reasoning, this model excels in visual reasoning. Architecturally, Llama 3.2 Vision is actually composed of the Llama 3.1 LLM, a vision tower, and an image adapter.

Llama 3.2 Vision supports a joint input mode of image and text, and can also process text-only input. In image-text mode, the model accepts English input; in text-only mode, it supports multilingual processing, including English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

Practical testing

Use the following command to run the llama 3.2 vision model and ensure it runs successfully:

ollama run llama3.2-vision

Then correctly configure it in LobeChat (as introduced earlier in the article, such as creating a new model and customizing model configurations in Ollama under LobeChat's language models):

And call it in the chat interface:

Then, let's try the same requirements as when testing llama3.2-3B earlier to see the generation speed:

It feels no different from the 3B model. Let's take a look at image recognition:

Not bad.

Note: I also kept an eye on the GPU utilization in the Activity Monitor. From receiving the instruction to outputting the text, the GPU utilization hovered between 0-33%. Since the utilization is dynamic and hard to screenshot, I didn't bother to capture it. But overall, the load is not high (after all, the questions asked weren't difficult), and there is still a lot of potential to be tapped.

Summary

Overall, the efficiency of running large models with Ollama on the M4 Pro is very satisfying to me. Llama 3.2 Vision is already sufficient for my use. If needed in the future, I will try models with higher parameters to see where the limit of the M4 Pro actually lies.