들어가며
- 스파클링소다는 가상 컨테이너 기술로 파드라는 객체들로 응용 프로그램들을 실행하고 관리하고 있습니다.
- 그렇다면, 컨테이너들의 로그를 확인하고 저장하는 방법에 대해 알아보겠습니다.
로그 확인하기
기본적으로 kubelet 이라는 명령어를 통해 객체를 조회하는 작업을 수행하게 됩니다.
kubectl get po NAME READY STATUS RESTARTS AGE argo-server-99c4d5895-jw2tb 1/1 Running 1 42d devainexus-0 1/1 Running 3 19d frameworkcontroller-0 1/1 Running 5 42d gitlab-postgresql-0 1/1 Running 1 19d gitlab-redis-5c897c5fcd-l5m7h 1/1 Running 4 20d gitlab-server-966879df4-8wdtk 1/1 Running 14 20d kubewatch-654455fdc4-rnfcm 1/1 Running 1 19d mariadb-cluster-primary-0 1/1 Running 1 42d mariadb-cluster-secondary-0 1/1 Running 1 42d nfs-client-nfs-client-provisioner-75d89c7546-hhtd6 1/1 Running 9 42d node-gpus-svc-9vv4m 1/1 Running 2 4d23h polyaxon-admin-user-jvr8f 0/1 Completed 0 42d polyaxon-clean-runs-cvqtl 0/1 Completed 0 42d polyaxon-polyaxon-api-7f8b88d9b8-x9gv9 1/1 Running 5 20d polyaxon-polyaxon-gateway-7dccd97877-thflh 1/1 Running 2 19d polyaxon-polyaxon-operator-6d55464545-47mfb 1/1 Running 159 42d polyaxon-polyaxon-streams-64b8966656-dg44z 1/1 Running 4 21d polyaxon-postgresql-0 1/1 Running 1 42d polyaxon-sync-db-jmgxp 0/1 Completed 0 42d sodaflow-app-6fb4dc86d8-xpz5h 1/1 Running 0 4d23h sodaflow-app-ui-85679bc695-h8nlp 1/1 Running 1 26d sodaflow-init-job-jshmn 0/1 Completed 0 42d sodaflow-memcached-57cfb89f87-j7p22 1/1 Running 1 42d sodaflow-nginx-ingress-controller-7b55dd5954-jxh5m 1/1 Running 1 19d sodaflow-nginx-ingress-controller-7b55dd5954-mdhzf 1/1 Running 1 19d sodaflow-nginx-ingress-controller-7b55dd5954-vsr9n 1/1 Running 1 19d sodaflow-nginx-ingress-default-backend-64459cdb5d-7t4w2 1/1 Running 1 42d test-vscode03ea4-0 2/2 Running 0 5h workflow-controller-76b8c46fdb-m5hgw 1/1 Running 1 42d
조회의 pod들 중, 로그를 확인하고 싶은 pod의 이름을 복사해 로그를 조회할 수 있습니다.
kubectl logs node-gpus-svc-9vv4m Update node worker gpu info. Insert gpu GPU-c288e54f-8533-fe4e-5449-2efbea6d4553 info into node worker Update node worker gpu info. Insert gpu GPU-c288e54f-8533-fe4e-5449-2efbea6d4553 info into node worker * Serving Flask app 'app' (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on all addresses (0.0.0.0) WARNING: This is a development server. Do not use it in a production deployment. * Running on http://127.0.0.1:80 * Running on http://6.2.171.83:80 (Press CTRL+C to quit)
실시간 로그를 조회하고자 한다면, -f 옵션을 붙여 사용할 수 있습니다.
kubectl logs -f node-gpus-svc-9vv4m
조회된 로그를 파일로 저장하고자 한다면, 아래와 같이 수행하면 됩니다.
kubectl logs [pod name] > [log file name] kubectl logs node-gpus-svc-9vv4m > node-gpus-svc.log
또한, 현재 노드에서 구동중인 컨테이너들에 대한 로그가 저장되어 있는 디렉터리에서 로그를 확인할 수도 있습니다.
k8s의 pod 로그는, pod가 실행 중인 node에 저장됩니다.
cd /var/log/containers ls argo-server-99c4d5895-jw2tb_sodaflow_argo-server-1ebbb5d3b29f78f3d4a4029d4c642326ac03d42c6a39a89ea2fbccf5c2bbf74c.log calico-kube-controllers-6c8f7d75bc-w2nzd_kube-system_calico-kube-controllers-5eba4711226d8154b9876ce357524ccc9656d1c94559d4a177d0f92d2d32019c.log calico-kube-controllers-6c8f7d75bc-w2nzd_kube-system_calico-kube-controllers-721a737fba26daffb6a3541d3d7cac666aef1f6b4a32fbec3c9033c532fa1ff6.log calico-node-92cwx_kube-system_calico-node-59161dddfae116cf2292cfb09b37c530a89041b3745a4949923a26a1be30c247.log calico-node-92cwx_kube-system_flexvol-driver-28ce0bf3f9a2cd0686a6050be78a5d62fd11ae0de1044f9ffe25bb5779fd8bde.log calico-node-92cwx_kube-system_install-cni-ffa067d27031d9935de27d1c3ccf1f965fdbffa27698641f2660f449d8be9f06.log calico-node-92cwx_kube-system_upgrade-ipam-f0336090871b61353d5f16e264b06de79959c869c29d062f57f94d8354f85eb7.log coredns-74f5cc8f99-26k2z_kube-system_coredns-c8c97a3f7535d56eb39c0a2c60d509ac9374733f3a2270adffceb59c41137201.log coredns-74f5cc8f99-jjs5r_kube-system_coredns-527c787585741b236c8256cad1066d1efd595160351778e861bade369ceb09ef.log etcd-master_kube-system_etcd-9685759836d693e3cdb999b2351f1911a476e3083d99f010b9a864f3ae89f079.log gitlab-postgresql-0_sodaflow_gitlab-postgresql-007ed1ca48d93877c4f6b6a0e651a64e6d46d76ee49d27dcf22456d8b5fac279.log gitlab-postgresql-0_sodaflow_init-chmod-data-cc8c1d2f0d8b08de655ea632ad8dff10fcd8941d563a03e3762637466c419b15.log kube-apiserver-master_kube-system_kube-apiserver-398c26a67f517fc5a5467f99c3123187b096ee62f38564b192bbcdb0b71f0f97.log kube-controller-manager-master_kube-system_kube-controller-manager-47f0645f659d5762d6058874141442ee2eff29cac9335e44484582c29f2ac61f.log kube-controller-manager-master_kube-system_kube-controller-manager-91f6b3dd089a9523bb89a825c1725701766c3b2105e936747aac7856ce16da32.log kube-prometheus-stack-grafana-644477786c-8m2r6_monitoring_grafana-399ecc5e62d633918e047166592394c0b8e49dcfb2a3d144054ff4a8f384e0e4.log kube-prometheus-stack-grafana-644477786c-8m2r6_monitoring_grafana-sc-dashboard-4d7d2038c2bbaf409d37023955e66cf22422f58bf8eb1f60f8a3132ce655fcfc.log kube-prometheus-stack-grafana-644477786c-8m2r6_monitoring_grafana-sc-datasources-98945451d9d4f7041ec2a0a97754f846bf4cf79357a8ea831aaeab16bc89ff6e.log kube-prometheus-stack-prometheus-node-exporter-wltm9_monitoring_node-exporter-07b82443bab4c3220b7608560d6843f39d4a861a53c70cc1a5e763e1f62b3707.log kube-proxy-tk4bq_kube-system_kube-proxy-c9cb71833a9a3d541648d4d154a87b02ce9c6c979bc4c08a5f4cba154a63f1e9.log kube-scheduler-master_kube-system_kube-scheduler-1bd466f19ac2d1197fcd7264bf87b0af851779163c3728680b23359a84efe0fc.log kubewatch-654455fdc4-rnfcm_sodaflow_kubewatch-302eacdf304c3dc6291dff7317039fc3fb05593c7fa8da4296c18ac5141d03f8.log mariadb-cluster-primary-0_sodaflow_mariadb-6bbc86cf29559bd817d5d159f1296e5837e9439e5a2f76caaa91bb325b5dec62.log mariadb-cluster-primary-0_sodaflow_volume-permissions-3877a5a31b786faceee13da43ccb887b6639d097f5c6530b4149c95710ca60d4.log mariadb-cluster-secondary-0_sodaflow_mariadb-f7794f42dfd093242b452f68153d3d3366c93e4227f3694095be7aa6f62d3ade.log mariadb-cluster-secondary-0_sodaflow_volume-permissions-924c32d32098eae1115114bdc7e57564a041942b78922b952b0d25734e1be631.log minio-74f77cf779-qm48c_minio_minio-8dbcac16e4dcd74988cca47eef14a65a83f4bb4b09494dd9a1dcd0461aba3e3a.log nfs-client-nfs-client-provisioner-75d89c7546-hhtd6_sodaflow_nfs-client-provisioner-29137512fea0c8b6b387596a8c46219ae845073cdab01fd941cfafa43f5b7921.log nfs-client-nfs-client-provisioner-75d89c7546-hhtd6_sodaflow_nfs-client-provisioner-f226d06acb94800f561a13eac872e7e9283b142b0086f7d11c863f331f836b88.log nvidia-device-plugin-daemonset-vkp87_kube-system_nvidia-device-plugin-ctr-de8f6df193885f0b37e80965c8a6d4e2c3ea4016e7b13151f185fd2cdbc183a8.log polyaxon-polyaxon-gateway-7dccd97877-thflh_sodaflow_polyaxon-gateway-be2fbd3d8fd94930991073fbb26a4de4af91144e39e58f025606c450e5ee9625.log polyaxon-polyaxon-operator-6d55464545-47mfb_sodaflow_polyaxon-operator-025d789ccc43f16117ac307e9add10f5acab96b14318368d494b1b280e2454d9.log polyaxon-polyaxon-operator-6d55464545-47mfb_sodaflow_polyaxon-operator-ef64642d495294b91c49712092af47e7d9e97a80e2ed77d0e812abde60aa7d3d.log polyaxon-postgresql-0_sodaflow_init-chmod-data-b4513dd266af7f55e18948159c7528d56880a5100b8a5ab6b9b6acc9be91cb37.log polyaxon-postgresql-0_sodaflow_polyaxon-postgresql-09219c1bd3aa5cc5d41d3ecec0cf37002ce340324e2b3d4873b677d58bc86aa9.log sodaflow-app-6fb4dc86d8-xpz5h_sodaflow_sodaflow-4426fd87cd40c623019fb37092158bb97dc3be8b2783b87459a1dbd5165ac876.log sodaflow-app-ui-85679bc695-h8nlp_sodaflow_sodaflow-ui-1954b854755bc4f79480f59f12cf67822b18e00ef316cc266ffbdc3f6959c90a.log sodaflow-memcached-57cfb89f87-j7p22_sodaflow_sodaflow-9c2a4a087c4c8639368bfdf37dc9b337a1e9659f9a6e3830c21bc53f4fa60e4e.log sodaflow-nginx-ingress-controller-7b55dd5954-jxh5m_sodaflow_nginx-ingress-controller-5d18686e6e96a76ca4ac0ff9704645caec757115b736367621151fb980223e2a.log sodaflow-nginx-ingress-controller-7b55dd5954-mdhzf_sodaflow_nginx-ingress-controller-92d4dc2fdb25e8a9b76029bb952439656f3785690e1914ad29c2ad40856d79bc.log sodaflow-nginx-ingress-controller-7b55dd5954-vsr9n_sodaflow_nginx-ingress-controller-cd402d384f97a515e1e289d47ae0a54c9136ddb6ac0d109d9d74ed70b93b4296.log sodaflow-nginx-ingress-default-backend-64459cdb5d-7t4w2_sodaflow_nginx-ingress-default-backend-a4bb0cbc43ed518995842ca5522139373194586c5832ad88674631c832fcd4c0.log workflow-controller-76b8c46fdb-m5hgw_sodaflow_workflow-controller-cfd83a5cd20594bfe51c5cc829a4f11cdb2e4cbe1c7dd87c99c6584ed48527f0.log
해당 파일의 이름은 {pod name}-{namespace}-{container name}.log 의 형식입니다.
마무리
- k8s의 pod들에 대한 로그를 확인하고 저장하는 방법을 알아보았습니다.
아티클이 유용했나요?
훌륭합니다!
피드백을 제공해 주셔서 감사합니다.
도움이 되지 못해 죄송합니다!
피드백을 제공해 주셔서 감사합니다.
피드백 전송
소중한 의견을 수렴하여 아티클을 개선하도록 노력하겠습니다.