【containerd错误解决系列】(202303)failed to create shim task, OCI runtime create failed, unable to retrieve OCI runtime error, runc did not terminate successfully

环境

1
2
3
4
5
# cat /etc/redhat-release
CentOS Linux release 8.0.1905 (Core) 

# uname -r
4.18.0-348.rt7.130.el8.x86_64

问题及现象

  1. pod的状态全部都是ContainerCreating的状态

image.png

  1. containerd进程有大量报错，主要有：

failed to create containerd task: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/c4847070fad34a8da9b16b5c20cdc38e28a15cfcf9913d712e4fe60d8c9029f7/log.json: no such file or directory): runc did not terminate successfully: exit status 127: unknown

image.png

解决方案

查看现有libseccomp版本

1
2
3
4
# sudo rpm -qa | grep libseccomp
libseccomp-2.3.3-3.el8.x86_64


卸载低版本libseccomp

1
2
3
4
# sudo rpm -e libseccomp-2.3.3-3.el8.x86_64 --nodeps

# sudo rpm -qa | grep libseccomp
#

安装高版本libseccomp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# yum provides libseccomp
Last metadata expiration check: 0:48:39 ago on Tue 28 Mar 2023 01:49:06 PM CST.
libseccomp-2.5.2-1.el8.i686 : Enhanced seccomp library
Repo        : Base
Matched from:
Provide    : libseccomp = 2.5.2-1.el8

libseccomp-2.5.2-1.el8.x86_64 : Enhanced seccomp library
Repo        : Base
Matched from:
Provide    : libseccomp = 2.5.2-1.el8


# yum install libseccomp-2.5.2-1.el8.x86_64
Last metadata expiration check: 0:49:46 ago on Tue 28 Mar 2023 01:49:06 PM CST.
Dependencies resolved.
======================================================================================================================================================================
 Package                                   Arch                                  Version                                    Repository                           Size
======================================================================================================================================================================
Installing:
 libseccomp                                x86_64                                2.5.2-1.el8                                Base                                 71 k

Transaction Summary
======================================================================================================================================================================
Install  1 Package

Total download size: 71 k
Installed size: 166 k
Is this ok [y/N]: y
Downloading Packages:
libseccomp-2.5.2-1.el8.x86_64.rpm                                                                                                      38 MB/s |  71 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                 7.0 MB/s |  71 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                              1/1 
  Installing       : libseccomp-2.5.2-1.el8.x86_64                                                                                                                1/1 
  Running scriptlet: libseccomp-2.5.2-1.el8.x86_64                                                                                                                1/1 
  Verifying        : libseccomp-2.5.2-1.el8.x86_64                                                                                                                1/1 

Installed:
  libseccomp-2.5.2-1.el8.x86_64                                                                                                                                       

Complete!


# sudo rpm -qa | grep libseccomp
libseccomp-2.5.2-1.el8.x86_64

解决后现象

  1. pod状态

安装之后，不用重启containerd进程，就看到了目前pod的状态都正常了

image.png

  1. runc中依赖的libseccomp
    libseccomp已经是高版本的了
1
2
3
4
5
6
# runc --version
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d1
spec: 1.0.2-dev
go: go1.18.10
libseccomp: 2.5.2

原理

libseccomp需要高于2.4版本

containerd.io 要求安装版本为 2.4.0 的 libseccomp

具体官方依据还未找到，后续找到补充

参考

k8s系列-06-containerd的基本操作 卸载安装libeseccomp
containerd.io depends libseccomp2 ( = 2.4.0) but 2.3.1-2.1ubuntu4 is to be installed

yum相关命令

1
2
3
4
5
6
yum clean all
yum makecache
yum list --showduplicates  | grep libseccomp
yum install --downloadonly --downloaddir . libseccomp
yum install --downloadonly --downloaddir . libseccomp*

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61


[root@ltypaas-xx24b003-zhyw03w temp]# yum install --downloadonly --downloaddir . libseccomp
Last metadata expiration check: 0:01:35 ago on Thu 15 Jun 2023 02:21:25 PM CST.
Package libseccomp-2.5.1-1.uelc20.01.aarch64 is already installed.
Dependencies resolved.
==================================================================================================================================================================
 Package                           Architecture                   Version                                      Repository                                    Size
==================================================================================================================================================================
Upgrading:
 libseccomp                        aarch64                        2.5.2-1.0.4.uelc20.01                        UniontechOS-20-BaseOS                         69 k

Transaction Summary
==================================================================================================================================================================
Upgrade  1 Package

Total download size: 69 k
YUM will only download packages for the transaction.
Is this ok [y/N]: N
Operation aborted.


[root@ltypaas-xx24b003-zhyw03w temp]# yum install --downloadonly --downloaddir . libseccomp*
Last metadata expiration check: 0:01:46 ago on Thu 15 Jun 2023 02:21:25 PM CST.
Package libseccomp-2.5.1-1.uelc20.01.aarch64 is already installed.
Dependencies resolved.
==================================================================================================================================================================
 Package                               Architecture                Version                                     Repository                                    Size
==================================================================================================================================================================
Installing:
 libseccomp-devel                      aarch64                     2.5.2-1.0.4.uelc20.01                       UniontechOS-20-AppStream                      56 k
 libseccomp-static                     aarch64                     2.5.2-1.0.4.uelc20.01                       UniontechOS-20-Plus                           67 k
Upgrading:
 libseccomp                            aarch64                     2.5.2-1.0.4.uelc20.01                       UniontechOS-20-BaseOS                         69 k

Transaction Summary
==================================================================================================================================================================
Install  2 Packages
Upgrade  1 Package

Total download size: 192 k
YUM will only download packages for the transaction.
Is this ok [y/N]: y
Downloading Packages:
(1/3): libseccomp-2.5.2-1.0.4.uelc20.01.aarch64.rpm                                                                                34 MB/s |  69 kB     00:00    
(2/3): libseccomp-devel-2.5.2-1.0.4.uelc20.01.aarch64.rpm                                                                         3.1 MB/s |  56 kB     00:00    
(3/3): libseccomp-static-2.5.2-1.0.4.uelc20.01.aarch64.rpm                                                                        286 kB/s |  67 kB     00:00    
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                             822 kB/s | 192 kB     00:00     
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.



[root@ltypaas-xx24b003-zhyw03w temp]# ll
total 200
-rw-r----- 1 root root 71080 Jun 15 14:23 libseccomp-2.5.2-1.0.4.uelc20.01.aarch64.rpm
-rw-r----- 1 root root 57748 Jun 15 14:23 libseccomp-devel-2.5.2-1.0.4.uelc20.01.aarch64.rpm
-rw-r----- 1 root root 68128 Jun 15 14:23 libseccomp-static-2.5.2-1.0.4.uelc20.01.aarch64.rpm
[root@ltypaas-xx24b003-zhyw03w temp]#
k8s
【containerd错误解决系列】(202303)failed to create shim task, OCI runtime create failed, unable to retrieve OCI runtime error, runc did not terminate successfully
http://example.com/2023/03/28/k8s/【containerd错误解决系列】(202303)failed to create shim task, OCI runtime create failed, unable to retrieve OCI runtime error, runc did not terminate successfully/
作者
ningan123
发布于
2023年3月28日
许可协议