解决MacBook Pro M2关于inport tensorflow 出现的Illegal instruction: 4

关于MacBook Pro M2关于inport tensorflow 出现的Illegal instruction: 4

1. 问题描述：新版的MacBook pro在完整移植老版python环境后，未正确运行使用intel芯片的老版macbook的python程序。

命令行报错内容为：Illegal instruction: 4
Pycharm报错内容为：Process finished with exit code 132 (interrupted by signal 4: SIGILL)
最终确定是tensorflow导致，解决方案如下：

2.解决方案

前言：解决方案灵感来源为解决M1芯片的问题，详见github：
https://github.com/apple/tensorflow_macos/issues/81，点击进入tensorflow_macos code, 其readme.md提供了重要信息：
You can now leverage Apple’s tensorflow-metal PluggableDevice in TensorFlow v2.5 for accelerated training on Mac GPUs directly with Metal. Learn more here.。

2.1 系统要求

macOS 12.0+

2.2 安装步骤

Step 1: 设置环境

x86 : AMD架构
推荐创建虚拟环境如下:

python3 -m venv ~/tensorflow-metal
source ~/tensorflow-metal/bin/activate
python -m pip install -U pip

注意：要求python 3.8

arm64 : Apple Silicon（苹果芯片，如M2）
下载与安装 Conda 环境:

chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh
sh ~/Downloads/Miniforge3-MacOSX-arm64.sh
source ~/miniforge3/bin/activate

过程输出结果如下所示：

MacBook-Pro:~ liujunhui$ sh ~/Downloads/Miniforge3-MacOSX-arm64.sh

Welcome to Miniforge3 4.14.0-0

In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>> 
Miniforge installer code uses BSD-3-Clause license as stated below.

Binary packages that come with it have their own licensing terms
and by installing miniforge you agree to the licensing terms of individual
packages as well. They include different OSI-approved licenses including
the GNU General Public License and can be found in pkgs//info/licenses
folders.

Miniforge installer comes with a boostrapping executable that is used
when installing miniforge and is deleted after miniforge is installed.
The bootstrapping executable uses micromamba, cli11, cpp-filesystem,
curl, c-ares, krb5, libarchive, libev, lz4, nghttp2, openssl, libsolv,
nlohmann-json, reproc and zstd which are licensed under BSD-3-Clause,
MIT and OpenSSL licenses. Licenses and copyright notices of these
projects can be found at the following URL.
https://github.com/conda-forge/micromamba-feedstock/tree/master/recipe.

=============================================================================
...
Please answer 'yes' or 'no':'
>>> yes

Miniforge3 will now be installed into this location:
/Users/liujunhui/miniforge3

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below
[/Users/liujunhui/miniforge3] >>> 
PREFIX=/Users/liujunhui/miniforge3
Unpacking payload ...
Extracting python-3.10.6-hbce4517_0_cpython.tar.bz2
Extracting bzip2-1.0.8-h3422bc3_4.tar.bz2
Extracting ca-certificates-2022.6.15-h4653dfc_0.tar.bz2
Extracting libffi-3.4.2-h3422bc3_5.tar.bz2
Extracting libzlib-1.2.12-ha287fd2_2.tar.bz2
Extracting ncurses-6.3-h07bb92c_1.tar.bz2
Extracting tzdata-2022c-h191b570_0.tar.bz2
Extracting xz-5.2.6-h57fd34a_0.tar.bz2
Extracting yaml-0.2.5-h3422bc3_2.tar.bz2
Extracting libsqlite-3.39.2-h2c9beb0_1.tar.bz2
...
==> For changes to take effect, close and re-open your current shell. <==

If you'd prefer that conda's base environment not be activated on startup, 
   set the auto_activate_base parameter to false: 

conda config --set auto_activate_base false

Thank you for installing Miniforge3!
MacBook-Pro:~ liujunhui$ source ~/miniforge3/bin/activate

Step2：安装虚拟环境3.8或者3.9，本文安装了3.9（原因在最后）。

(base) MacBook-Pro:~ liujunhui$ conda create -n test_tf python==3.9
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/liujunhui/miniforge3/envs/test_tf

  added / updated specs:
    - python==3.9
...
#
# To activate this environment, use
#
#     $ conda activate test_tf
#
# To deactivate an active environment, use
#
#     $ conda deactivate

激活环境：

(base) MacBook-Pro:~ liujunhui$ conda activate test_tf

Step 3 : 安转tensorflow依赖

conda install -c apple tensorflow-deps

• 在升级新的基础tensorflow版本时，建议

# uninstall existing tensorflow-macos and tensorflow-metal
python -m pip uninstall tensorflow-macos
python -m pip uninstall tensorflow-metal
# Upgrade tensorflow-deps
conda install -c apple tensorflow-deps --force-reinstall
# or point to specific conda environment
conda install -c apple tensorflow-deps --force-reinstall -n my_env

• tensorflow-deps 的版本是跟随 基础TensorFlow版本的，所以:
  对于v2.5：

conda install -c apple tensorflow-deps==2.5.0

对于v2.6

conda install -c apple tensorflow-deps==2.6.0

注意：以上仅支持python 3.8或3.9
命令行输出如下：

(test_tf) MacBook-Pro:~ liujunhui$ python -m pip uninstall tensorflow-macos
WARNING: Skipping tensorflow-macos as it is not installed.
(test_tf) MacBook-Pro:~ liujunhui$ python -m pip uninstall tensorflow-metal
WARNING: Skipping tensorflow-metal as it is not installed.
(test_tf) MacBook-Pro:~ liujunhui$ conda install -c apple tensorflow-deps --force-reinstall
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/liujunhui/miniforge3/envs/test_tf

  added / updated specs:
    - tensorflow-deps

...
## Package Plan ##

  environment location: /Users/liujunhui/miniforge3/envs/test_tf

  added / updated specs:
    - tensorflow-deps==2.6.0
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... done

Step 4 : 安装基础TensorFlow

python -m pip install tensorflow-macos

注意: If using conda environment built against pre-macOS 11 SDK use: SYSTEM_VERSION_COMPAT=0 python -m pip install tensorflow-macos otherwise you will get errors like : “not a supported wheel on this platform”

Step 5: 安装tensorflow-metal plugin

python -m pip install tensorflow-metal

搞定！

3. 其他问题

3.1 注意python的版本

默认的python 3.10，是不成功的，会出现如下错误：

(base) MacBook-Pro:~ liujunhui$ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:22) [Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
Traceback (most recent call last):
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 0x0002): Library not loaded: @rpath/libcblas.3.dylib
  Referenced from: /Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so
  Reason: tried: '/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/../../../../libcblas.3.dylib' (no such file), '/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/../../../../libcblas.3.dylib' (no such file), '/Users/liujunhui/miniforge3/bin/../lib/libcblas.3.dylib' (no such file), '/Users/liujunhui/miniforge3/bin/../lib/libcblas.3.dylib' (no such file), '/usr/local/lib/libcblas.3.dylib' (no such file), '/usr/lib/libcblas.3.dylib' (no such file)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 37, in <module>
    from tensorflow.python.eager import context
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 26, in <module>
    import numpy as np
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/__init__.py", line 144, in <module>
    from . import core
  File "/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.10 from "/Users/liujunhui/miniforge3/bin/python"
  * The NumPy version is: "1.22.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 0x0002): Library not loaded: @rpath/libcblas.3.dylib
  Referenced from: /Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so
  Reason: tried: '/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/../../../../libcblas.3.dylib' (no such file), '/Users/liujunhui/miniforge3/lib/python3.10/site-packages/numpy/core/../../../../libcblas.3.dylib' (no such file), '/Users/liujunhui/miniforge3/bin/../lib/libcblas.3.dylib' (no such file), '/Users/liujunhui/miniforge3/bin/../lib/libcblas.3.dylib' (no such file), '/usr/local/lib/libcblas.3.dylib' (no such file), '/usr/lib/libcblas.3.dylib' (no such file)

3.2 注意python环境的位置

(test_tf) MacBook-Pro:src liujunhui$ conda env list
# conda environments:
#
LamostBinary             /Users/liujunhui/.conda/envs/LamostBinary
PARSEC_isochrones        /Users/liujunhui/.conda/envs/PARSEC_isochrones
Paralledtest             /Users/liujunhui/.conda/envs/Paralledtest
mock_spectra             /Users/liujunhui/.conda/envs/mock_spectra
pythonProject            /Users/liujunhui/.conda/envs/pythonProject
base                     /Users/liujunhui/miniforge3
test_tf               *  /Users/liujunhui/miniforge3/envs/test_tf 
                         /opt/anaconda3
                         /opt/anaconda3/envs/test_tf

如上，当前环境实际是在miniforge3下，而非在anaconda3下，因为在anaconda3下也未安装成功，原因待实验。

3.3 注意numpy版本问题：

例如，如下报错，按照提示降低numpy版本号即可

(test_tf) MacBook-Pro:~ liujunhui$ python -m pip install tensorflow-macos
...
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.6.0
    Uninstalling tensorboard-2.6.0:
      Successfully uninstalled tensorboard-2.6.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
scipy 1.7.3 requires numpy<1.23.0,>=1.16.5, but you have numpy 1.23.2 which is incompatible.
Successfully installed absl-py-1.2.0 flatbuffers-1.12 keras-2.9.0 libclang-14.0.6 numpy-1.23.2 packaging-21.3 protobuf-3.19.4 pyparsing-3.0.9 tensorboard-2.9.1 tensorflow-estimator-2.9.0 tensorflow-macos-2.9.2

降低numpy版本至1.32.0:

(test_tf) MacBook-Pro:~ liujunhui$ conda install test_tf numpy==1.32.0