部署k8s集群
集群部署可以参考kubeadm部署k8s
为k8s集群创建动态存储
部署动态存储参考K8s集群部署动态存储
k8s部署Jenkins master
Jenkins-rbac.yaml
apiVersion: v1
kind: Namespace
metadata:
name: jenkins
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: jenkins
rules:
- apiGroups:
- '*'
resources:
- statefulsets
- services
- replicationcontrollers
- replicasets
- podtemplates
- podsecuritypolicies
- pods
- pods/log
- pods/exec
- podpreset
- poddisruptionbudget
- persistentvolumes
- persistentvolumeclaims
- jobs
- endpoints
- deployments
- deployments/scale
- daemonsets
- cronjobs
- configmaps
- namespaces
- events
- secrets
verbs:
- create
- get
- watch
- delete
- list
- patch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts:jenkins
Jenkins-Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: jenkins
labels:
app: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: jenkins:2.412 # 自定义镜像，后面有介绍
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
#resources:
#limits:
#memory: 4Gi
#cpu: "2000m"
#requests:
#memory: 4Gi
#cpu: "2000m"
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-home
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-home
namespace: jenkins
spec:
storageClassName: "nfs-client"
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 5Gi
Jenkins-Service.yaml
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: jenkins
labels:
app: jenkins
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: web
nodePort: 30080
port: 8080
targetPort: web
- name: agent
nodePort: 30081
port: 50000
targetPort: agent
访问Jenkins
Jenkins升级
如果是低版本的jenkins, 要升级高版本的jenkins，可以先启动低版本jenkins镜像，下载高版本jenkins.war包，将该war包拷贝到低版本jenkins容器的/usr/share/jenkins下，替换低版本jenkins.war包；
# 1 启动低版本jenkins
docker run --name jenkins -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts
# 2 将下载好的高版本的jenkins.war包拷贝到容器中
docker cp jenkins.war jenkins:/usr/share/jenkins
# 3 使用当前容器生成镜像
docker commit jenkins:2.412 jenkins
配置Jenkins slave
5.1 制作Jenkins-salve镜像
准备好maven安装包、settings.xml、Dockerfile
settings.xml
/usr/local/apache-maven/repo
alimaven
aliyun maven
https://maven.aliyun.com/repository/public
central
Dockerfile
FROM jenkins/jnlp-slave:latest-jdk11
USER root
# 安装 maven
COPY apache-maven-3.6.2-bin.tar.gz .
RUN tar -zxf apache-maven-3.6.2-bin.tar.gz &&
mv apache-maven-3.6.2 /usr/local &&
rm -f apache-maven-3.6.2-bin.tar.gz &&
ln -s /usr/local/apache-maven-3.6.2/bin/mvn /usr/bin/mvn &&
ln -s /usr/local/apache-maven-3.6.2 /usr/local/apache-maven &&
mkdir -p /usr/local/apache-maven/repo
COPY settings.xml /usr/local/apache-maven/conf/settings.xml
USER jenkins
构建镜像
docker build -t jenkins-salve-maven:latest .
5.2 在Jenkins中创建k8s凭据
生成凭据
cat /root/.kube/config
# 获取certificate-authority-data中的内容，生成ca.crt
echo "certificate-authority-data中的内容" | base64 -d > ca.crt
# 获取client-certificate-data中的内容，生成client.crt
echo "client-certificate-data中的内容" | base64 -d > client.crt
# 获取client-key-data中的内容，生成client.key
echo "client-key-data中的内容" | base64 -d > client.key
# 生产jenkins凭据(需要输入4为以上密码，请记住改密码，后面会用到)
openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
在jenkins中创建凭据
安装Kubernetes插件
在Jenkins上配置Kubernetes Cloud
继续配置jenkins信息
5.3 创建harbor仓库的凭证和代码仓库的凭据
5.4 docker.sock升级访问权限
chmod 777 /var/run/docker.sock
5.5 nfs服务器上创建maven本地仓库
mkdir /data/nfs/maven
5.6 创建流水线
添加参数
添加脚本
podTemplate(label: 'jenkins-slave', cloud: 'k8s', containers: [
containerTemplate(
name: 'jnlp',
image: "10.18.101.186/library/jenkins-slave-maven:latest" # 该镜像为前面我们创建的jenkins-slave-maven:latest
),
containerTemplate(
name: 'docker',
image: "docker:stable",
ttyEnabled: true,
command: 'cat'
),
],
volumes: [
hostPathVolume(mountPath: '/usr/bin//docker', hostPath: '/usr/bin//docker'), # 挂载docker
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'), # 挂载docker.sock (所有节点的docker.sock需要给最大权限)
nfsVolume(mountPath: '/usr/local/apache-maven/repo', serverAddress: '192.168.101.143' , serverPath: '/data/nfs/maven'), # mountPath要与settings.xml中配置的一致
],
)
{
node("jenkins-slave"){
// 第一步
stage('拉取代码'){
git branch: 'master', credentialsId: 'gitApiToken', url: 'https://gitee.com/sunny_001/education.git'
}
// 第二步
stage('代码编译'){
//编译并安装公共工程
sh "mvn clean install -Dmaven.test.skip=true"
}
stage('构建镜像'){
sh "docker build -t 10.18.101.186/sunny/${imageName}:${imageTag} ."
container('docker') {
withCredentials([usernamePassword(credentialsId: 'harbor', usernameVariable: 'username', passwordVariable: 'password')]) {
sh "docker login -u $username -p $password 10.18.101.186"
sh "docker push 10.18.101.186/sunny/${imageName}:${imageTag}"
}
sh "docker rmi -f 10.18.101.186/sunny/${imageName}:${imageTag}"
}
}
}
}
构建测试
输入参数，开始构建
此时会发现多出来一个节点，流水线在该节点运行，当运行结束后，k8s会回收该pod, 该节点消失;
构建日志
Started by user admin
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: k8s jenkins/jenkins-slave-9sp3n-2r1vk
Agent jenkins-slave-9sp3n-2r1vk is provisioned from template jenkins-slave-9sp3n
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
buildUrl: "http://jenkins.jenkins.svc.cluster.local:8080/job/test2/27/"
runUrl: "job/test2/27/"
labels:
jenkins: "slave"
jenkins/label-digest: "03ddc3eddf95d5470d5c7fb6d2937abaeca3b79e"
jenkins/label: "jenkins-slave"
name: "jenkins-slave-9sp3n-2r1vk"
namespace: "jenkins"
spec:
containers:
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_TUNNEL"
value: "jenkins.jenkins:50000"
- name: "JENKINS_AGENT_NAME"
value: "jenkins-slave-9sp3n-2r1vk"
- name: "JENKINS_NAME"
value: "jenkins-slave-9sp3n-2r1vk"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.jenkins.svc.cluster.local:8080/"
image: "10.18.101.186/library/jenkins-slave-maven:latest"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
resources: {}
tty: false
volumeMounts:
- mountPath: "/usr/local/apache-maven/repo"
name: "volume-2"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-1"
readOnly: false
- mountPath: "/usr/bin/docker"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- command:
- "cat"
image: "docker:stable"
imagePullPolicy: "IfNotPresent"
name: "docker"
resources: {}
tty: true
volumeMounts:
- mountPath: "/usr/local/apache-maven/repo"
name: "volume-2"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-1"
readOnly: false
- mountPath: "/usr/bin/docker"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- hostPath:
path: "/usr/bin//docker"
name: "volume-0"
- name: "volume-2"
nfs:
path: "/data/nfs/maven"
readOnly: false
server: "192.168.101.143"
- hostPath:
path: "/var/run/docker.sock"
name: "volume-1"
- emptyDir:
medium: ""
name: "workspace-volume"
Running on jenkins-slave-9sp3n-2r1vk in /home/jenkins/agent/workspace/test2
[Pipeline] {
[Pipeline] stage
[Pipeline] { (拉取代码)
[Pipeline] git
The recommended git tool is: NONE
using credential gitApiToken
Cloning the remote Git repository
Cloning repository https://gitee.com/sunny_001/education.git
> git init /home/jenkins/agent/workspace/test2 # timeout=10
Fetching upstream changes from https://gitee.com/sunny_001/education.git
> git --version # timeout=10
> git --version # 'git version 2.20.1'
using GIT_ASKPASS to set credentials gitApiToken
> git fetch --tags --force --progress -- https://gitee.com/sunny_001/education.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 94dafb13b46754c91eef5d4bda6faded1bfd3de8 (refs/remotes/origin/master)
> git config remote.origin.url https://gitee.com/sunny_001/education.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git config core.sparsecheckout # timeout=10
> git checkout -f 94dafb13b46754c91eef5d4bda6faded1bfd3de8 # timeout=10
> git branch -a -v --no-abbrev # timeout=10
> git checkout -b master 94dafb13b46754c91eef5d4bda6faded1bfd3de8 # timeout=10
Commit message: "删除文件 README.md"
> git rev-list --no-walk 94dafb13b46754c91eef5d4bda6faded1bfd3de8 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (代码编译)
[Pipeline] sh
+ mvn clean install -Dmaven.test.skip=true
[WARNING]
[WARNING] Some problems were encountered while building the effective settings
[WARNING] expected START_TAG or END_TAG not TEXT (position: TEXT seen ...alimavennua0 ua0 ua0 ----------------------
[INFO] Building education-docker 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ education-docker ---
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ education-docker ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ education-docker ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 10 source files to /home/jenkins/agent/workspace/test2/target/classes
[INFO] /home/jenkins/agent/workspace/test2/src/main/java/com/gsww/university_search/util/ResultUtil.java: /home/jenkins/agent/workspace/test2/src/main/java/com/gsww/university_search/util/ResultUtil.java uses unchecked or unsafe operations.
[INFO] /home/jenkins/agent/workspace/test2/src/main/java/com/gsww/university_search/util/ResultUtil.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ education-docker ---
[INFO] Not copying test resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ education-docker ---
[INFO] Not compiling test sources
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ education-docker ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ education-docker ---
[INFO] Building jar: /home/jenkins/agent/workspace/test2/target/education-docker-1.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.6.RELEASE:repackage (repackage) @ education-docker ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ education-docker ---
[INFO] Installing /home/jenkins/agent/workspace/test2/target/education-docker-1.0.0-SNAPSHOT.jar to /usr/local/apache-maven/repo/com/gsww/education-docker/1.0.0-SNAPSHOT/education-docker-1.0.0-SNAPSHOT.jar
[INFO] Installing /home/jenkins/agent/workspace/test2/pom.xml to /usr/local/apache-maven/repo/com/gsww/education-docker/1.0.0-SNAPSHOT/education-docker-1.0.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.265 s
[INFO] Finished at: 2023-07-06T07:21:39Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (构建镜像)
[Pipeline] sh
+ docker build -t 10.18.101.186/sunny/edu:v3.0.0 .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
Install the buildx component to build images with BuildKit:
https://docs.docker.com/go/buildx/
Sending build context to Docker daemon 25.05MB
Step 1/4 : FROM anapsix/alpine-java:jre8
---> 2f5c81ce3b29
Step 2/4 : VOLUME /tmp
---> Using cache
---> eda7e6d35e3c
Step 3/4 : ADD ./target/*.jar app.jar
---> cf110dd613d1
Step 4/4 : ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-Duser.timezone=GMT+08","-jar","/app.jar"]
---> Running in 117a5f851957
Removing intermediate container 117a5f851957
---> 428735d083da
Successfully built 428735d083da
Successfully tagged 10.18.101.186/sunny/edu:v3.0.0
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $password
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [password]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ docker login -u admin -p **** 10.18.101.186
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[Pipeline] sh
+ docker push 10.18.101.186/sunny/edu:v3.0.0
The push refers to repository [10.18.101.186/sunny/edu]
163bc186bfeb: Preparing
91abebed283b: Preparing
4fe15f8d0ae6: Preparing
91abebed283b: Layer already exists
4fe15f8d0ae6: Layer already exists
163bc186bfeb: Pushed
v3.0.0: digest: sha256:f9bdca4791c8c5a3a45f840a2ddbf15b5dfd92907e572e4e23526198499d879c size: 952
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] sh
+ docker rmi -f 10.18.101.186/sunny/edu:v3.0.0
Untagged: 10.18.101.186/sunny/edu:v3.0.0
Untagged: 10.18.101.186/sunny/edu@sha256:f9bdca4791c8c5a3a45f840a2ddbf15b5dfd92907e572e4e23526198499d879c
Deleted: sha256:428735d083dab37b918bb5475dea55d28512cab21f646450cec910043404d388
Deleted: sha256:cf110dd613d1e40b3f07be86393fb886fc4c1e8ec733c757d4257fcb9c557499
Deleted: sha256:2b6cebbe832e63603afec40b5cedb49ef0ba51fee9eb042d7ad573395747ba56
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS
