分立机器基站接入核心网并调试可用解决之路

问题描述

在两台机器上，分别为服务器A与主机B，其中服务器A上部署OAI核心网，主机B上部署容器版ueransim。两机器用一根网线连接。两台机器的连接如此：但是ueransim无法接入核心网。

解决思路1——机器网卡设置到192.168.70.0网段

将机器物理网卡网段设置到与核心网所在网段一致，192.168.70.x，ueransim直接接入。

发现1

两机器分别设置如下：

核心网A：

ens38:
                  addresses: [192.168.70.12/24]
                  dhcp4: no
                  gateway4: 192.168.70.1
                  nameservers:
                          addresses: [8.8.8.8,144.144.144.144,255.255.255.255]

主机B：

ens39:
                  addresses: [192.168.70.13/24]
                  dhcp4: no
                  gateway4: 192.168.70.1
                  nameservers:
                          addresses: [8.8.8.8,144.144.144.144,255.255.255.255]

两者可以互相ping通，但开启核心网后却无法互相ping通

进行如下修改，将对方网口作为自己的网关后，在开通核心网后可以ping通，但依旧无法ping通核心网

核心网A：

ens38:
                  addresses: [192.168.70.12/24]
                  dhcp4: no
                  gateway4: 192.168.70.12
                  nameservers:
                          addresses: [8.8.8.8,144.144.144.144,255.255.255.255]

主机B：

ens39:
                  addresses: [192.168.70.13/24]
                  dhcp4: no
                  gateway4: 192.168.70.13
                  nameservers:
                          addresses: [8.8.8.8,144.144.144.144,255.255.255.255]

经过在服务器A抓包，发现服务器A的192.168.70.12上有抓到来自192.168.70.13寻找192.168.70.1的ARP，但是却没有成功找到的返回
有来自192.168.70.13的寻找192.168.70.1的icmp包，但只有request， no response found

然后通过如下方法解决，让主机B可以ping到192.168.70.1：
在主机B上添加到192.168.70.0/24网段的路由：

sudo route add -net 192.168.70.0/24 gw 192.168.70.12

在服务器A添加到主机B的路由：

sudo route add -host 192.168.70.13 gw 192.168.70.12

然后启动ueransim测试：

无法启动，ueransim需要接入demo-oai等网桥，貌似无法接入另一台机器上的网桥。

但这样的网络配置可以作为非容器化ueransim的一种配置，中间不用换网段到192.168.12.x

解决思路2 docker-swarm

Docker Compose和Docker Swarm旨在实现完全集成，这意味着您可以将Compose应用程序指向Swarm集群，并使它们都像使用单个Docker主机一样正常工作。引用
manager是我们管理集群的入口，我们的docker命令都是在manager上执行，node节点上是不能执行dockr命令的

这里我将服务器A作为管理节点，主机B作为工作节点

在服务器A上，通过ssh接入想作为管理节点的服务器，然后执行swarm init命令，指定swarm管理机的IP地址：

#安装docker-ce
sudo apt-get update
sudo apt-get install docker-ce
#指定为管理节点，该节点ip为192.168.12.3
lab@lab-virtual-machine:~$ docker swarm init --advertise-addr 192.168.12.3

出现以下提示

Swarm initialized: current node (uowf6o6oftuh47y2qjgm7jqeh) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-2f3q3ajwud8mg8xw62zbs7k0hpw9ypss7qgqcyjdzw5dns4qxf-a9mnk42uf9tjh58yj8s00t4ta 192.168.12.3:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

其中这里表示如何加入工作节点：

docker swarm join --token SWMTKN-1-2f3q3ajwud8mg8xw62zbs7k0hpw9ypss7qgqcyjdzw5dns4qxf-a9mnk42uf9tjh58yj8s00t4ta 192.168.12.3:2377

在主机B上将主机B设置为工作节点：

docker swarm join --token SWMTKN-1-2f3q3ajwud8mg8xw62zbs7k0hpw9ypss7qgqcyjdzw5dns4qxf-a9mnk42uf9tjh58yj8s00t4ta 192.168.12.3:2377

出现以下提示：

This node joined a swarm as a worker.

到此搭建了一个简单的swarm集群。

查看接入的节点：

lab@lab-virtual-machine:~$ docker node ls

查看创建的网络：

docker network ls

创建一个新网络：
网络模式需要选择overlay

docker network create -d overlay testnet

可是OAI核心网的网络模式是bridge，是否可以将其变成overlay？

将所有docker-compose-basic-vpp-nrf.yaml中的网络由bridge改为overlay：

networks:
    public_net:
        driver: overlay
        name: demo-oai-public-net
        ipam:
            config:
                - subnet: 192.168.70.0/24
        driver_opts:
            com.docker.network.bridge.name: "demo-oai"
    public_net_access:
        driver: overlay
        name: oai-public-access
        ipam:
            config:
                - subnet: 192.168.72.0/24
        driver_opts:
            com.docker.network.bridge.name: "cn5g-access"
    public_net_core:
        driver: overlay
        name: oai-public-core
        ipam:
            config:
                - subnet: 192.168.73.0/24
        driver_opts:
            com.docker.network.bridge.name: "cn5g-core"

然后使用docker stack启动

之前 swarm 集群中docker service create一次只能部署一个微服务，我们可以使用 docker stack + compose 一次启动多个服务
stack相关操作：

deploy      Deploy a new stack or update an existing stack
ls          List stacks
ps          List the tasks in the stack
rm          Remove one or more stacks
services    List the services in the stack

lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ docker stack deploy --compose-file docker-compose-basic-vpp-nrf.yaml oai

查看运行的服务：

docker service ls

查看容器运行状态：

docker service ps 容器名字

启动ueransim;

lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ sudo docker stack deploy --compose-file docker-compose-ueransim-vpp.yaml ueransim
[sudo] password for lab: 
WARN[0000] network public_net_access: network.external.name is deprecated in favor of network.name 
WARN[0000] network public_net: network.external.name is deprecated in favor of network.name 
Ignoring unsupported options: privileged

Ignoring deprecated options:

container_name: Setting the container name is not supported.

Error response from daemon: 2 matches found based on name: network demo-oai-public-net is ambiguous

其中前两个警告WARN【0000】是由于yaml文件网络external的格式不对，更改为如下后会消失：

networks:
    public_net:
        external: true
        name: demo-oai-public-net
    public_net_access:
        external: true
        name: oai-public-access

后面的警告Error response from daemon: 2 matches found based on name: network demo-oai-public-net is ambiguous是由于机器上docker network里的demo-oai-public-net有local与swarm两种

将其中local的部分删掉：

docker network rm f6dd0838cf68
docker network rm 34b108d99783
docker network rm f9b3e7f7b442

再次启动ueransim：

lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ sudo docker stack deploy --compose-file docker-compose-ueransim-vpp.yaml ueransim
Ignoring unsupported options: privileged

Ignoring deprecated options:

container_name: Setting the container name is not supported.

Creating service ueransim_ueransim

但是查看ueransim的日志，发现未能启动成功：
甚至连gnb都没有成功接入OAI核心网

lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ docker service logs 5iarfozcjdvi
WARNING: Error loading config file: /home/lab/.docker/config.json: open /home/lab/.docker/config.json: permission denied
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | Now setting these variables '@GTP_IP@ @IGNORE_STREAM_IDS@ @LINK_IP@ @MCC@ @MNC@ @NCI@ @NGAP_IP@ @NGAP_PEER_IP@ @SD_0@ @SD_1@ @SD_2@ @SST_0@ @SST_1@ @SST_2@ @TAC@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | Now setting these variables '@AMF_VALUE@ @APN@ @GNB_IP_ADDRESS@ @IMEI@ @IMEI_SV@ @IMSI@ @KEY@ @MCC@ @MNC@ @OP@ @OP_TYPE@ @PDU_TYPE@ @SD_C@ @SD_D@ @SD_R@ @SST_C@ @SST_D@ @SST_R@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | Done setting the configuration
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | ### Running ueransim ###
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | Running gnb
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | UERANSIM v3.2.5
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:40.606] [rls-udp] [error] RLS failure [Socket bind failed: Cannot assign requested address]
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:40.607] [sctp] [info] Trying to establish SCTP connection... (192.168.70.132:38412)
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:40.607] [gtp] [error] GTP/UDP task could not be created. Socket bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | Now setting these variables '@GTP_IP@ @IGNORE_STREAM_IDS@ @LINK_IP@ @MCC@ @MNC@ @NCI@ @NGAP_IP@ @NGAP_PEER_IP@ @SD_0@ @SD_1@ @SD_2@ @SST_0@ @SST_1@ @SST_2@ @TAC@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:40.613] [sctp] [error] Binding to 192.168.70.141:0 failed. SCTP bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | Now setting these variables '@AMF_VALUE@ @APN@ @GNB_IP_ADDRESS@ @IMEI@ @IMEI_SV@ @IMSI@ @KEY@ @MCC@ @MNC@ @OP@ @OP_TYPE@ @PDU_TYPE@ @SD_C@ @SD_D@ @SD_R@ @SST_C@ @SST_D@ @SST_R@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | Running ue
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | Done setting the configuration
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | UERANSIM v3.2.5
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:41.589] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | ### Running ueransim ###
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:43.791] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:45.993] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | Running gnb
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:46.592] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | UERANSIM v3.2.5
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:46.592] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:30:47.094] [nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:53.735] [rls-udp] [error] RLS failure [Socket bind failed: Cannot assign requested address]
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:31:16.604] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:31:16.605] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:53.737] [sctp] [info] Trying to establish SCTP connection... (192.168.70.132:38412)
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine    | [2023-04-17 07:31:16.823] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:53.737] [gtp] [error] GTP/UDP task could not be created. Socket bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:53.743] [sctp] [error] Binding to 192.168.70.141:0 failed. SCTP bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | Running ue
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | UERANSIM v3.2.5
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:54.719] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:56.921] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:59.124] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:59.722] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:31:59.722] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:32:00.225] [nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine    | [2023-04-17 07:32:29.734] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage

删除oai与ueransim服务：

docker stack rm oai
docker stack rm ueransim

将ueransim与oai合并到一起启动：
创建文件1.yaml

version: '3.8'
services:
    mysql:
        container_name: "mysql"
        image: mysql:8.0
        volumes:
            - ./database/oai_db2.sql:/docker-entrypoint-initdb.d/oai_db.sql
            - ./healthscripts/mysql-healthcheck2.sh:/tmp/mysql-healthcheck.sh
        environment:
            - TZ=Europe/Paris
            - MYSQL_DATABASE=oai_db
            - MYSQL_USER=test
            - MYSQL_PASSWORD=test
            - MYSQL_ROOT_PASSWORD=linux
        healthcheck:
            test: /bin/bash -c "/tmp/mysql-healthcheck.sh"
            interval: 10s
            timeout: 5s
            retries: 30
        networks:
            public_net:
                ipv4_address: 192.168.70.131
    oai-udr:
        container_name: "oai-udr"
        image: oaisoftwarealliance/oai-udr:v1.5.0
        environment:
            - TZ=Europe/Paris
            - UDR_NAME=OAI_UDR
            - UDR_INTERFACE_NAME_FOR_NUDR=eth0
            - MYSQL_IPV4_ADDRESS=192.168.70.131
            - MYSQL_USER=test
            - MYSQL_PASS=test
            - MYSQL_DB=oai_db
            - WAIT_MYSQL=120
            - USE_FQDN_DNS=yes
            - REGISTER_NRF=yes
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
        depends_on:
            - mysql
            - oai-nrf
        networks:
            public_net:
                ipv4_address: 192.168.70.136
    oai-udm:
        container_name: "oai-udm"
        image: oaisoftwarealliance/oai-udm:v1.5.0
        environment:
            - TZ=Europe/Paris
            - UDM_NAME=OAI_UDM
            - SBI_IF_NAME=eth0
            - REGISTER_NRF=yes
            - USE_FQDN_DNS=yes
            - UDR_IP_ADDRESS=192.168.70.136
            - UDR_FQDN=oai-udr
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
        depends_on:
            - oai-udr
        networks:
            public_net:
                ipv4_address: 192.168.70.137
    oai-ausf:
        container_name: "oai-ausf"
        image: oaisoftwarealliance/oai-ausf:v1.5.0
        environment:
            - TZ=Europe/Paris
            - AUSF_NAME=OAI_AUSF
            - SBI_IF_NAME=eth0
            - USE_FQDN_DNS=yes
            - REGISTER_NRF=yes
            - UDM_IP_ADDRESS=192.168.70.137
            - UDM_FQDN=oai-udm
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
        depends_on:
            - oai-udm
        networks:
            public_net:
                ipv4_address: 192.168.70.138
    oai-nrf:
        container_name: "oai-nrf"
        image: oaisoftwarealliance/oai-nrf:v1.5.0
        environment:
            - TZ=Europe/Paris
            - NRF_INTERFACE_NAME_FOR_SBI=eth0
        networks:
            public_net:
                ipv4_address: 192.168.70.130
    oai-amf:
        container_name: "oai-amf"
        image: oaisoftwarealliance/oai-amf:v1.5.0
        environment:
            - TZ=Europe/paris
            - MCC=208
            - MNC=95
            - REGION_ID=128
            - AMF_SET_ID=1
            - SERVED_GUAMI_MCC_0=208
            - SERVED_GUAMI_MNC_0=95
            - SERVED_GUAMI_REGION_ID_0=128
            - SERVED_GUAMI_AMF_SET_ID_0=1
            - SERVED_GUAMI_MCC_1=460
            - SERVED_GUAMI_MNC_1=11
            - SERVED_GUAMI_REGION_ID_1=10
            - SERVED_GUAMI_AMF_SET_ID_1=1
            - PLMN_SUPPORT_MCC=208
            - PLMN_SUPPORT_MNC=95
            - PLMN_SUPPORT_TAC=0xa000
            # Slice 0 (222, 123)
            - SST_0=222
            - SD_0=123
            # Slice 0 (128, 12)
            - SST_1=128
            - SD_1=12
            - AMF_INTERFACE_NAME_FOR_NGAP=eth0
            - AMF_INTERFACE_NAME_FOR_N11=eth0
            # One single SMF instance
            - SMF_INSTANCE_ID_0=1
            - SMF_FQDN_0=oai-smf
            - SMF_IPV4_ADDR_0=192.168.70.133
            - SELECTED_0=true
            - NF_REGISTRATION=yes
            - SMF_SELECTION=yes
            - USE_FQDN_DNS=yes
            - EXTERNAL_AUSF=yes
            - EXTERNAL_UDM=no
            - EXTERNAL_NSSF=no
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
            - AUSF_IPV4_ADDRESS=192.168.70.138
            - AUSF_FQDN=oai-ausf
            - UDM_IPV4_ADDRESS=192.168.70.137
            - UDM_FQDN=oai-udm
            - INT_ALGO_LIST=[ "NIA1" , "NIA2"]
            - CIPH_ALGO_LIST=[ "NEA1" , "NEA2"]
        depends_on:
            - mysql
            - vpp-upf
            - oai-ext-dn
            - oai-ausf
        networks:
            public_net:
                ipv4_address: 192.168.70.132
    oai-smf:
        container_name: "oai-smf"
        image: oaisoftwarealliance/oai-smf:v1.5.0
        environment:
            - TZ=Europe/Paris
            - SMF_INTERFACE_NAME_FOR_N4=eth0
            - SMF_INTERFACE_NAME_FOR_SBI=eth0
            - DEFAULT_DNS_IPV4_ADDRESS=172.21.3.100
            - DEFAULT_DNS_SEC_IPV4_ADDRESS=8.8.8.8
            - AMF_IPV4_ADDRESS=192.168.70.132
            - AMF_FQDN=oai-amf
            - UDM_IPV4_ADDRESS=192.168.70.137
            - UDM_FQDN=oai-udm
            - UPF_IPV4_ADDRESS=192.168.70.201
            - UPF_FQDN_0=vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
            - DEFAULT_CSCF_IPV4_ADDRESS=127.0.0.1  # only needed when ims is being used
            - USE_LOCAL_SUBSCRIPTION_INFO=yes  #Set to yes if SMF uses local subscription information instead of from an UDM
            - REGISTER_NRF=yes
            - DISCOVER_UPF=yes
            - USE_FQDN_DNS=yes
            - USE_NETWORK_INSTANCE=yes
            - ENABLE_USAGE_REPORTING=yes
            # Slice 0 (1, 0xFFFFFF)
            - DNN_NI0=oai
            - TYPE0=IPv4
            - DNN_RANGE0=12.1.1.151 - 12.1.1.253
            - NSSAI_SST0=1
            - SESSION_AMBR_UL0=200Mbps
            - SESSION_AMBR_DL0=400Mbps
            # Slice 1 (1, 1)
            - DNN_NI1=oai.ipv4
            - TYPE1=IPv4
            - DNN_RANGE1=12.1.1.51 - 12.1.1.150
            - NSSAI_SST1=1
            - NSSAI_SD1=1
            - SESSION_AMBR_UL1=100Mbps
            - SESSION_AMBR_DL1=200Mbps
            # Slice 2 (222, 123)
            - DNN_NI2=default
            - TYPE2=IPv4
            - DNN_RANGE2=12.1.1.2 - 12.1.1.50
            - NSSAI_SST2=222
            - NSSAI_SD2=123
            - SESSION_AMBR_UL2=50Mbps
            - SESSION_AMBR_DL2=100Mbps
            # Slice 3 for ims
            - DNN_NI3=ims
            - TYPE3=IPv4v6
            - DNN_RANGE3=14.1.1.2 - 14.1.1.253
        extra_hosts:
            - "vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org:192.168.70.201"
        depends_on:
            - oai-amf
        networks:
            public_net:
                ipv4_address: 192.168.70.133
    vpp-upf:
        privileged: true
        container_name: "vpp-upf"
        image: oaisoftwarealliance/oai-upf-vpp:v1.5.0
        environment:
            - IF_1_IP=192.168.70.201
            - IF_1_TYPE=N4
            - IF_2_IP=192.168.72.201
            - IF_2_TYPE=N3
            - IF_2_NWI=access.oai.org
            - IF_3_IP=192.168.73.201
            - IF_3_TYPE=N6
            - IF_3_IP_REMOTE=192.168.73.135 # EXT-DN IP Address
            - IF_3_NWI=internet.oai.org
            - NAME=VPP-UPF
            - MNC=95
            - MCC=208
            - REALM=3gppnetwork.org
            - VPP_MAIN_CORE=0
            - VPP_CORE_WORKER=1
#           - VPP_PLUGIN_PATH=/usr/lib64/vpp_plugins/                # RHEL7
            - VPP_PLUGIN_PATH=/usr/lib/x86_64-linux-gnu/vpp_plugins/ # Ubntu18.04
            - SNSSAI_SD=123
            - SNSSAI_SST=222
            - DNN=default
            - REGISTER_NRF=yes
            - NRF_IP_ADDR=192.168.70.130
            - NRF_PORT=80
            - HTTP_VERSION=1
        depends_on:
            - oai-nrf
        healthcheck:
            test: /bin/bash -c "pgrep vpp"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.134
            public_net_access:
                ipv4_address: 192.168.72.134
            public_net_core:
                ipv4_address: 192.168.73.134
    oai-ext-dn:
        privileged: true
        init: true
        container_name: "oai-ext-dn"
        image: oaisoftwarealliance/trf-gen-cn5g:latest
        entrypoint: /bin/bash -c \
              "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE;"\
              "ip route add 12.1.1.0/24 via 192.168.73.201 dev eth0; ip route; sleep infinity"
        command: ["/bin/bash", "-c", "trap : SIGTERM SIGINT; sleep infinity & wait"]
        healthcheck:
            test: /bin/bash -c "iptables -L -t nat | grep MASQUERADE"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net_core:
                ipv4_address: 192.168.73.135
    ueransim:
        container_name: ueransim
        image: ueransim:latest
        privileged: true
        environment:
            # GNB Congig Parameters
            - MCC=208
            - MNC=95
            - NCI=0x000000010
            - TAC=0xa000
            - LINK_IP=192.168.70.141
            - NGAP_IP=192.168.70.141
            - GTP_IP=192.168.72.141
            - NGAP_PEER_IP=192.168.70.132
            - SST=222
            - SD=123
            - IGNORE_STREAM_IDS=true
            # UE Config Parameters
            - NUMBER_OF_UE=1
            - IMSI=208950000000041
            - KEY=0C0A34601D4F07677303652C0462535B
            - OP=63bfa50ee6523365ff14c1f45f88737d
            - OP_TYPE=OPC
            - AMF_VALUE=8000
            - IMEI=356938035643803
            - IMEI_SV=0035609204079514
            - GNB_IP_ADDRESS=192.168.70.141
            - PDU_TYPE=IPv4
            - APN=default
            - SST_0=222
            - SD_0=123
            - SST_C=222
            - SD_C=123
            - SST_D=222
            - SD_D=123
        healthcheck:
            test: /bin/bash -c "ifconfig uesimtun0"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.141
            public_net_access:
                ipv4_address: 192.168.72.141
networks:
    public_net:
        driver: overlay
        name: demo-oai-public-net
        ipam:
            config:
                - subnet: 192.168.70.0/24
        driver_opts:
            com.docker.network.bridge.name: "demo-oai"
    public_net_access:
        driver: overlay
        name: oai-public-access
        ipam:
            config:
                - subnet: 192.168.72.0/24
        driver_opts:
            com.docker.network.bridge.name: "cn5g-access"
    public_net_core:
        driver: overlay
        name: oai-public-core
        ipam:
            config:
                - subnet: 192.168.73.0/24
        driver_opts:
            com.docker.network.bridge.name: "cn5g-core"

启动1.yaml：

lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ docker stack deploy --compose-file 1.yaml oaiue
WARNING: Error loading config file: /home/lab/.docker/config.json: open /home/lab/.docker/config.json: permission denied
Ignoring unsupported options: privileged

Ignoring deprecated options:

container_name: Setting the container name is not supported.

Creating network oai-public-access
Creating network oai-public-core
Creating network demo-oai-public-net
Creating service oaiue_ueransim
Creating service oaiue_vpp-upf
Creating service oaiue_oai-amf
Creating service oaiue_oai-udr
Creating service oaiue_oai-udm
Creating service oaiue_oai-ausf
Creating service oaiue_oai-nrf
Creating service oaiue_mysql
Creating service oaiue_oai-ext-dn
Creating service oaiue_oai-smf

这样运行的容器乱七八糟的分布在多个机器上，需要通过如下方式指定不同容器运行的节点：

利用docker-compose.yml文件中的deploy.placement.constraints属性

#指定manage或worker
deploy:
  replicas: 1
     placement:
        constraints:
          ## 常用方式指定 manager或者work节点
          - node.role == worker 

#指定主机名
deploy:
  replicas: 1
     placement:
        constraints:
          - node.hostname == docker1 

指定标签的方法：
node label 可以给一个或多个机器打上一个标签，然后再compose文件中指定节点标签，就可以部署在打上对应标签的机器上。具体步骤如下：

现在主节点机器上对所有节点机器进行打标签操作，命令如下：

docker node update --label-add role=标签名称  主机名
 
例如我要对一个主机名为docker1的机器打上一个标签为db的标签
docker node update --label-add role=db  docker1

deploy:
  replicas: 1
     placement:
        constraints:
          - node.labels.role == db

除此以外还有指定节点ID的方式

这里我采用指定manager与worker的方式：
将核心网部署在manager节点上，将ueransim部署在worker节点上

version: '3.8'
services:
    mysql:
        container_name: "mysql"
        image: mysql:8.0
        volumes:
            - ./database/oai_db2.sql:/docker-entrypoint-initdb.d/oai_db.sql
            - ./healthscripts/mysql-healthcheck2.sh:/tmp/mysql-healthcheck.sh
        environment:
            - TZ=Europe/Paris
            - MYSQL_DATABASE=oai_db
            - MYSQL_USER=test
            - MYSQL_PASSWORD=test
            - MYSQL_ROOT_PASSWORD=linux
        healthcheck:
            test: /bin/bash -c "/tmp/mysql-healthcheck.sh"
            interval: 10s
            timeout: 5s
            retries: 30
        networks:
            public_net:
                ipv4_address: 192.168.70.131
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager


    oai-udr:
        container_name: "oai-udr"
        image: oaisoftwarealliance/oai-udr:v1.5.0
        environment:
            - TZ=Europe/Paris
            - UDR_NAME=OAI_UDR
            - UDR_INTERFACE_NAME_FOR_NUDR=eth0
            - MYSQL_IPV4_ADDRESS=192.168.70.131
            - MYSQL_USER=test
            - MYSQL_PASS=test
            - MYSQL_DB=oai_db
            - WAIT_MYSQL=120
            - USE_FQDN_DNS=yes
            - REGISTER_NRF=yes
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
        depends_on:
            - mysql
            - oai-nrf
        networks:
            public_net:
                ipv4_address: 192.168.70.136
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    oai-udm:
        container_name: "oai-udm"
        image: oaisoftwarealliance/oai-udm:v1.5.0
        environment:
            - TZ=Europe/Paris
            - UDM_NAME=OAI_UDM
            - SBI_IF_NAME=eth0
            - REGISTER_NRF=yes
            - USE_FQDN_DNS=yes
            - UDR_IP_ADDRESS=192.168.70.136
            - UDR_FQDN=oai-udr
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
        depends_on:
            - oai-udr
        networks:
            public_net:
                ipv4_address: 192.168.70.137
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    oai-ausf:
        container_name: "oai-ausf"
        image: oaisoftwarealliance/oai-ausf:v1.5.0
        environment:
            - TZ=Europe/Paris
            - AUSF_NAME=OAI_AUSF
            - SBI_IF_NAME=eth0
            - USE_FQDN_DNS=yes
            - REGISTER_NRF=yes
            - UDM_IP_ADDRESS=192.168.70.137
            - UDM_FQDN=oai-udm
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
        depends_on:
            - oai-udm
        networks:
            public_net:
                ipv4_address: 192.168.70.138
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    oai-nrf:
        container_name: "oai-nrf"
        image: oaisoftwarealliance/oai-nrf:v1.5.0
        environment:
            - TZ=Europe/Paris
            - NRF_INTERFACE_NAME_FOR_SBI=eth0
        networks:
            public_net:
                ipv4_address: 192.168.70.130
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    oai-amf:
        container_name: "oai-amf"
        image: oaisoftwarealliance/oai-amf:v1.5.0
        environment:
            - TZ=Europe/paris
            - MCC=208
            - MNC=95
            - REGION_ID=128
            - AMF_SET_ID=1
            - SERVED_GUAMI_MCC_0=208
            - SERVED_GUAMI_MNC_0=95
            - SERVED_GUAMI_REGION_ID_0=128
            - SERVED_GUAMI_AMF_SET_ID_0=1
            - SERVED_GUAMI_MCC_1=460
            - SERVED_GUAMI_MNC_1=11
            - SERVED_GUAMI_REGION_ID_1=10
            - SERVED_GUAMI_AMF_SET_ID_1=1
            - PLMN_SUPPORT_MCC=208
            - PLMN_SUPPORT_MNC=95
            - PLMN_SUPPORT_TAC=0xa000
            # Slice 0 (222, 123)
            - SST_0=222
            - SD_0=123
            # Slice 0 (128, 12)
            - SST_1=128
            - SD_1=12
            - AMF_INTERFACE_NAME_FOR_NGAP=eth0
            - AMF_INTERFACE_NAME_FOR_N11=eth0
            # One single SMF instance
            - SMF_INSTANCE_ID_0=1
            - SMF_FQDN_0=oai-smf
            - SMF_IPV4_ADDR_0=192.168.70.133
            - SELECTED_0=true
            - NF_REGISTRATION=yes
            - SMF_SELECTION=yes
            - USE_FQDN_DNS=yes
            - EXTERNAL_AUSF=yes
            - EXTERNAL_UDM=no
            - EXTERNAL_NSSF=no
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
            - AUSF_IPV4_ADDRESS=192.168.70.138
            - AUSF_FQDN=oai-ausf
            - UDM_IPV4_ADDRESS=192.168.70.137
            - UDM_FQDN=oai-udm
            - INT_ALGO_LIST=[ "NIA1" , "NIA2"]
            - CIPH_ALGO_LIST=[ "NEA1" , "NEA2"]
        depends_on:
            - mysql
            - vpp-upf
            - oai-ext-dn
            - oai-ausf
        networks:
            public_net:
                ipv4_address: 192.168.70.132
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    oai-smf:
        container_name: "oai-smf"
        image: oaisoftwarealliance/oai-smf:v1.5.0
        environment:
            - TZ=Europe/Paris
            - SMF_INTERFACE_NAME_FOR_N4=eth0
            - SMF_INTERFACE_NAME_FOR_SBI=eth0
            - DEFAULT_DNS_IPV4_ADDRESS=172.21.3.100
            - DEFAULT_DNS_SEC_IPV4_ADDRESS=8.8.8.8
            - AMF_IPV4_ADDRESS=192.168.70.132
            - AMF_FQDN=oai-amf
            - UDM_IPV4_ADDRESS=192.168.70.137
            - UDM_FQDN=oai-udm
            - UPF_IPV4_ADDRESS=192.168.70.201
            - UPF_FQDN_0=vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org
            - NRF_IPV4_ADDRESS=192.168.70.130
            - NRF_FQDN=oai-nrf
            - DEFAULT_CSCF_IPV4_ADDRESS=127.0.0.1  # only needed when ims is being used
            - USE_LOCAL_SUBSCRIPTION_INFO=yes  #Set to yes if SMF uses local subscription information instead of from an UDM
            - REGISTER_NRF=yes
            - DISCOVER_UPF=yes
            - USE_FQDN_DNS=yes
            - USE_NETWORK_INSTANCE=yes
            - ENABLE_USAGE_REPORTING=yes
            # Slice 0 (1, 0xFFFFFF)
            - DNN_NI0=oai
            - TYPE0=IPv4
            - DNN_RANGE0=12.1.1.151 - 12.1.1.253
            - NSSAI_SST0=1
            - SESSION_AMBR_UL0=200Mbps
            - SESSION_AMBR_DL0=400Mbps
            # Slice 1 (1, 1)
            - DNN_NI1=oai.ipv4
            - TYPE1=IPv4
            - DNN_RANGE1=12.1.1.51 - 12.1.1.150
            - NSSAI_SST1=1
            - NSSAI_SD1=1
            - SESSION_AMBR_UL1=100Mbps
            - SESSION_AMBR_DL1=200Mbps
            # Slice 2 (222, 123)
            - DNN_NI2=default
            - TYPE2=IPv4
            - DNN_RANGE2=12.1.1.2 - 12.1.1.50
            - NSSAI_SST2=222
            - NSSAI_SD2=123
            - SESSION_AMBR_UL2=50Mbps
            - SESSION_AMBR_DL2=100Mbps
            # Slice 3 for ims
            - DNN_NI3=ims
            - TYPE3=IPv4v6
            - DNN_RANGE3=14.1.1.2 - 14.1.1.253
        extra_hosts:
            - "vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org:192.168.70.201"
        depends_on:
            - oai-amf
        networks:
            public_net:
                ipv4_address: 192.168.70.133
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    vpp-upf:
        privileged: true
        container_name: "vpp-upf"
        image: oaisoftwarealliance/oai-upf-vpp:v1.5.0
        environment:
            - IF_1_IP=192.168.70.201
            - IF_1_TYPE=N4
            - IF_2_IP=192.168.72.201
            - IF_2_TYPE=N3
            - IF_2_NWI=access.oai.org
            - IF_3_IP=192.168.73.201
            - IF_3_TYPE=N6
            - IF_3_IP_REMOTE=192.168.73.135 # EXT-DN IP Address
            - IF_3_NWI=internet.oai.org
            - NAME=VPP-UPF
            - MNC=95
            - MCC=208
            - REALM=3gppnetwork.org
            - VPP_MAIN_CORE=0
            - VPP_CORE_WORKER=1
#           - VPP_PLUGIN_PATH=/usr/lib64/vpp_plugins/                # RHEL7
            - VPP_PLUGIN_PATH=/usr/lib/x86_64-linux-gnu/vpp_plugins/ # Ubntu18.04
            - SNSSAI_SD=123
            - SNSSAI_SST=222
            - DNN=default
            - REGISTER_NRF=yes
            - NRF_IP_ADDR=192.168.70.130
            - NRF_PORT=80
            - HTTP_VERSION=1
        depends_on:
            - oai-nrf
        healthcheck:
            test: /bin/bash -c "pgrep vpp"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.134
            public_net_access:
                ipv4_address: 192.168.72.134
            public_net_core:
                ipv4_address: 192.168.73.134
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    oai-ext-dn:
        privileged: true
        init: true
        container_name: "oai-ext-dn"
        image: oaisoftwarealliance/trf-gen-cn5g:latest
        entrypoint: /bin/bash -c \
              "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE;"\
              "ip route add 12.1.1.0/24 via 192.168.73.201 dev eth0; ip route; sleep infinity"
        command: ["/bin/bash", "-c", "trap : SIGTERM SIGINT; sleep infinity & wait"]
        healthcheck:
            test: /bin/bash -c "iptables -L -t nat | grep MASQUERADE"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net_core:
                ipv4_address: 192.168.73.135
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == manager

    ueransim:
        container_name: ueransim
        image: ueransim:latest
        privileged: true
        environment:
            # GNB Congig Parameters
            - MCC=208
            - MNC=95
            - NCI=0x000000010
            - TAC=0xa000
            - LINK_IP=192.168.70.141
            - NGAP_IP=192.168.70.141
            - GTP_IP=192.168.72.141
            - NGAP_PEER_IP=192.168.70.132
            - SST=222
            - SD=123
            - IGNORE_STREAM_IDS=true
            # UE Config Parameters
            - NUMBER_OF_UE=1
            - IMSI=208950000000041
            - KEY=0C0A34601D4F07677303652C0462535B
            - OP=63bfa50ee6523365ff14c1f45f88737d
            - OP_TYPE=OPC
            - AMF_VALUE=8000
            - IMEI=356938035643803
            - IMEI_SV=0035609204079514
            - GNB_IP_ADDRESS=192.168.70.141
            - PDU_TYPE=IPv4
            - APN=default
            - SST_0=222
            - SD_0=123
            - SST_C=222
            - SD_C=123
            - SST_D=222
            - SD_D=123
        healthcheck:
            test: /bin/bash -c "ifconfig uesimtun0"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.141
            public_net_access:
                ipv4_address: 192.168.72.141
        deploy:
          mode: global
          placement: 
             constraints: 
               - node.role == worker

networks:
    public_net:
        driver: overlay
        name: demo-oai-public-net
        ipam:
            config:
                - subnet: 192.168.70.0/24
        driver_opts:
            com.docker.network.bridge.name: "demo-oai"
    public_net_access:
        driver: overlay
        name: oai-public-access
        ipam:
            config:
                - subnet: 192.168.72.0/24
        driver_opts:
            com.docker.network.bridge.name: "cn5g-access"
    public_net_core:
        driver: overlay
        name: oai-public-core
        ipam:
            config:
                - subnet: 192.168.73.0/24
        driver_opts:
            com.docker.network.bridge.name: "cn5g-core"

经过较长时间的等待后，启动结果如下：
核心网网元正常启动，ueransim未成功

对ueransim进行重启：

docker service ls  
docker service update --force 07ms7mdsk907

重启后依旧无法接入

将ueransim与核心网都部署到服务器A上，依旧同样的报错

再尝试将所有容器全部部署到主机B上

依旧不行

结点服务的可视化界面管理visualizer：

docker pull dockersamples/visualizer:latest
docker run -itd --name visualizer -p 8099:8080 -e HOST=192.168.12.3 -e PORT=8080 -v /var/run/docker.sock:/var/run/docker.sock dockersamples/visualizer:latest

然后在浏览器输入 192.168.12.3：8090访问
reference1
reference2
reference3：docker-compose+swarm