在树莓派上安装kubeedge

1. 环境准备

• 树莓派（Raspberry Pi 4）arm64架构 Ubuntu 18.04
• 树莓派（Raspberry Pi 4）arm64架构 docker 19.03.12
• Ubuntu 16.04 x86架构

2. 环境安装

2.1 安装gcc

首先确保gcc是否安装

gcc -version

如果没有安装，则安装gcc

sudo apt install gcc

2.2 安装go语言（>=1.12）

到Go官网下载Go安装包，下载地址：https://golang.org/dl
国内比较快的下载链接，通过golang中文社区进行下载：Go安装包下载

通过Xftp上传到树莓派中：

解压缩：

tar -zxvf go1.14.4.linux-arm64.tar.gz 

配置Go语言环境变量：

sudo vim /etc/profile

source /etc/profile

验证

go version

3. 下载kubeedge源代码

3.1 arm架构

对于kubeedge1.1.0 官网提供了arm架构（32bit）的编译好的文件，只需要下载上传到树莓派上即可。

Kubeedge官网： https://github.com/kubeedge/kubeedge
选择release版本，下载arm架构下编译好的文件：

3.2 arm64架构

mkdir kubeedge
cd kubeedge
mkdir bin

下载源代码

git clone https://github.com/kubeedge/kubeedge.git ~/kubeedge/src/github.com/kubeedge/kubeedge

4. 编译源代码

4.1 云端

可以直接下载与系统版本匹配的编译好的文件，或者通过下载kubeedge源代码的方式：

git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
cd $GOPATH/src/github.com/kubeedge/kubeedge

生成证书

$GOPATH/src/github.com/kubeedge/kubeedge/build/tools/certgen.sh genCertAndKey edge

证书将会生成在/etc/kubeedge/ca 和 /etc/kubeedge/certs目录下。
生成的证书需要复制到边缘端。
编译源代码：

cd $GOPATH/src/github.com/kubeedge/kubeedge/
git checkout -b v1.1.0
make all WHAT=cloudcore

创建设备和CRD

cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/devices
kubectl create -f devices_v1alpha1_devicemodel.yaml
kubectl create -f devices_v1alpha1_device.yaml

复制cloudcore二进制文件和配置文件

# run edge controller
# `conf/` should be in the same directory as the cloned KubeEdge repository
# verify the configurations before running cloud(cloudcore)
mkdir -p ~/cmd/conf
cp cloudcore ~/cmd/
cp -rf conf/* ~/cmd/conf/

设置配置文件：

cd ~/cmd/conf
vim controller.yaml

controller:
  kube:
    master:     # kube-apiserver address (such as:http://localhost:8080)
    namespace: ""
    content_type: "application/vnd.kubernetes.protobuf"
    qps: 5
    burst: 10
    node_update_frequency: 10
    kubeconfig: "~/.kube/config"   #Enter path to kubeconfig file to enable https connection to k8s apiserver, if master and kubeconfig are both set, master will override any value in kubeconfig.
cloudhub:
  protocol_websocket: true # enable websocket protocol
  port: 10000 # open port for websocket server
  protocol_quic: true # enable quic protocol
  quic_port: 10001 # open prot for quic server
  max_incomingstreams: 10000 # the max incoming stream for quic server
  enable_uds: true # enable unix domain socket protocol
  uds_address: unix:///var/lib/kubeedge/kubeedge.sock # unix domain socket address
  address: 0.0.0.0
  ca: /etc/kubeedge/ca/rootCA.crt
  cert: /etc/kubeedge/certs/edge.crt
  key: /etc/kubeedge/certs/edge.key
  keepalive-interval: 30
  write-timeout: 30
  node-limit: 10
devicecontroller:
  kube:
    master:        # kube-apiserver address (such as:http://localhost:8080)
    namespace: ""
    content_type: "application/vnd.kubernetes.protobuf"
    qps: 5
    burst: 10
    kubeconfig: "~/.kube/config" #Enter path to kubeconfig file to enable https connection to k8s apiserver,if master and kubeconfig are both set, master will override any value in kubeconfig.

修改cloudcore.service配置文件：

cd /root/kubeedge/bin/cloud
vim cloudcore.service

[Unit]
Description=cloudcore.service

[Service]
Type=simple
ExecStart=/root/kubeedge/bin/cloud/cloudcore

[Install]
WantedBy=multi-user.target                        

cp /root/kubeedge/bin/cloud/cloudcore.service /etc/systemd/system

运行cloudcore

sudo systemctl enable cloudcore
sudo systemctl start cloudcore

添加edge节点

vim edge-node1.yaml

apiVersion: v1
kind: Node
metadata:
  name: edge-node1
  labels:
    name: edge-node1
    node-role.kubernetes.io/edge: ''

kubectl apply -f edge-node1.yaml
kubectl get nodes

最终结果：

4.2 树莓派上编译源代码

• 如果可以直接在树莓派上进行编译是最好的选择
• 如果出现OOM或者部分依赖包访问不了的情况可以使用交叉编译

在树莓派上直接编译：

cd $GOPATH/src/github.com/kubeedge/kubeedge
# 选择版本
git checkout -b v1.1.0
# 编译
make all WHAT=edgecore

4.3 交叉编译源代码

需要在x86平台上下载kubeedge源代码
配置交叉编译环境：X86架构下交叉编译arm64源代码

首先进入到：/kubeedge/src/github.com/kubeedge/kubeedge/edge 目录下

cd ~/kubeedge/src/github.com/kubeedge/kubeedge/edge
# 查看makefile文件
vim Makefile 

找到自己的arm架构版本：

根据自身arm架构版本，添加配置文件：

sudo vim /etc/profile
source /etc/profile

# 进行编译 
go build cmd/edgecore/edgecore.go

过一会后便生成了edgecore文件，为可在arm64架构下可执行的文件。

4.4 将交叉编译后的edgecore上传到arm64架构下

由于上传的文件为单个文件，默认是不可执行的，需要添加文件可执行权限

chmod 777 edgecore

5. 修改配置文件

vim edgecore.service

[Unit]
Description=edgecore.service

[Service]
Type=simple
ExecStart=/home/ubuntu/kubeedge/edge/edgecore

[Install]
WantedBy=multi-user.target

将配置文件拷贝到system下

sudo cp /home/ubuntu/kubeedge/edge/edgecore.service /etc/systemd/system/edgecore.service

cd /home/ubuntu/kubeedge/edge/conf
vim edge.yaml

mqtt:
    server: tcp://127.0.0.1:1883 # external mqtt broker url.
    internal-server: tcp://127.0.0.1:1884 # internal mqtt broker url.
    mode: 0 # 0: internal mqtt broker enable only. 1: internal and external mqtt broker enable. 2: external mqtt broker enable only.
    qos: 0 # 0: QOSAtMostOnce, 1: QOSAtLeastOnce, 2: QOSExactlyOnce.
    retain: false # if the flag set true, server will store the message and can be delivered to future subscribers.
    session-queue-size: 100 # A size of how many sessions will be handled. default to 100.

edgehub:
    websocket:
        url: wss://x.x.x.x:10000/e632aba927ea4ac2b575ec1603d56f10/fb4ebb70-2783-42b8-b3ef-63e2fd6d242e/events
        certfile: /etc/kubeedge/certs/edge.crt
        keyfile: /etc/kubeedge/certs/edge.key
        handshake-timeout: 30 #second
        write-deadline: 15 # second
        read-deadline: 15 # second
    quic:
        url: 127.0.0.1:10001
        cafile: /etc/kubeedge/ca/rootCA.crt
        certfile: /etc/kubeedge/certs/edge.crt
        keyfile: /etc/kubeedge/certs/edge.key
        handshake-timeout: 30 #second
        write-deadline: 15 # second
        read-deadline: 15 # second
    controller:
        protocol: websocket # websocket, quic
        heartbeat: 15  # second
        project-id: e632aba927ea4ac2b575ec1603d56f10
        node-id: edge-node1

edged:
    register-node-namespace: default
    hostname-override: edge-node1
    interface-name: eth0
    edged-memory-capacity-bytes: 7852396000
    node-status-update-frequency: 10 # second
    device-plugin-enabled: false
    gpu-plugin-enabled: false
    image-gc-high-threshold: 80 # percent
    image-gc-low-threshold: 40 # percent
    maximum-dead-containers-per-container: 1
    docker-address: unix:///var/run/docker.sock
    runtime-type: docker
    remote-runtime-endpoint: unix:///var/run/dockershim.sock
    remote-image-endpoint: unix:///var/run/dockershim.sock
    runtime-request-timeout: 2
    podsandbox-image: kubeedge/pause-arm64:3.1 # kubeedge/pause:3.1 for x86 arch , kubeedge/pause-arm:3.1 for arm arch, kubeedge/pause-arm64 for arm64 arch
    image-pull-progress-deadline: 60 # second
    cgroup-driver: cgroupfs
    node-ip: ""
    cluster-dns: ""
    cluster-domain: ""
    
mesh:
    loadbalance:
        strategy-name: RoundRobin

启动edgecore：

sudo systemctl enable edgecore # 开机启动

sudo systemctl start edgecore
sudo systemctl status edgecore

报错：

要想启动内存cgroup，需要修改/boot/cmdline.txt文件：
我在Ubuntu 18.04 上找不到cmdline.txt文件，我浏览了/ boot / firmware目录，看到了config.txt文件：

看起来好像是从“ nobtcmd.txt”导入了cmdline。在elevator=deadline之前添加了“cgroup_enable=cpuset cgroup_enable=memory”

cd /boot/firmware
vim nobtcmd.txt

cgroup_enable=cpuset cgroup_enable=memory

重新启动：

6. 部署测试

vim deployment-armv8.yaml

metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        name: edge-node1
      containers:
      - name: nginx
        image: arm64v8/nginx
        ports:
        - containerPort: 80
          hostPort: 80
                              

kubectl apply -f deployment-armv8.yaml