{"id":1487,"date":"2023-05-26T09:51:49","date_gmt":"2023-05-26T09:51:49","guid":{"rendered":"http:\/\/patriciocerda.com\/?p=1487"},"modified":"2023-05-26T09:51:49","modified_gmt":"2023-05-26T09:51:49","slug":"openshift-etcd-backup-and-restore-with-kasten-k10-part-3-final","status":"publish","type":"post","link":"https:\/\/patriciocerda.com\/?p=1487","title":{"rendered":"OpenShift ETCD backup and Restore with Kasten K10 &#8211; Part 3-Final"},"content":{"rendered":"<p>Hi all and welcome back.\u00a0 This is the third and last part of the previous blog posts of backing up and restore ETCD in an OpenShift cluster using Kasten K10.\u00a0 I&#8217;ve already explained<a title=\"OpenShift ETCD backup and Restore with Kasten K10 \u2013 Part 1\" href=\"https:\/\/patriciocerda.com\/?p=1467\" target=\"_blank\" rel=\"noopener\"> what is ETCD<\/a> and why in some scenarios could be good to have an ETCD backup.\u00a0 The <a title=\"OpenShift ETCD backup and Restore with Kasten K10 \u2013 Part 2\" href=\"https:\/\/patriciocerda.com\/?p=1475\" target=\"_blank\" rel=\"noopener\">ETCD backup process with Kasten<\/a> was also explained in the last post, so, in this post we will describe the process to restore the ETCD\u00a0 backup using Kasten and how to use it to restore an OpenShift cluster to a previous state.<\/p>\n<p>When using Kasten we can <strong>restore the ETCD backup<\/strong> previously created, and use this backup to restore a Kubernetes cluster, in this OpenShift, to a previous state using the standard OpenShift procedure to restore ETCD with minor modifications.\u00a0\u00a0 Here is important to understand that Kasten <strong>won&#8217;t revert the Kubernetes\/OpenShift cluster state<\/strong> using the ETCD Backup, but all Kasten will do is to restore the ETCD backup file itself from the Location Profile (repository) to a location in one of the Control Plane nodes.\u00a0 After that you need to follow the required steps to use that backup file to revert the cluster to the desired state.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Important<\/strong>: Remember the RedHat Openshift Warning\u00a0 &#8220;<em>Restoring to a previous cluster state <strong>is a destructive and destablizing action<\/strong> to take on a running cluster. This should only be used as a <strong>last resort<\/strong>.&#8221;<\/em><\/p>\n<p>&nbsp;<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Contenidos<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/patriciocerda.com\/?p=1487\/#What_we_need\" >What we need:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/patriciocerda.com\/?p=1487\/#Restore_the_ETCD_backup_using_Kasten\" >Restore the ETCD backup using Kasten<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/patriciocerda.com\/?p=1487\/#Revert_the_OpenShift_cluster_state_using_ETCD_Backup\" >Revert the OpenShift cluster state using ETCD Backup<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"What_we_need\"><\/span>What we need:<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li>We need the cluster-restore.sh script provided by OpenShift with some modifications to work with the file restored by Kasten.\u00a0\u00a0 Kasten provides a modified version of this script, which can be downloaded from the following link: <a href=\"https:\/\/github.com\/kanisterio\/kanister\/blob\/master\/examples\/etcd\/etcd-in-cluster\/ocp\/cluster-ocp-restore.sh\" target=\"_blank\" rel=\"noopener\">https:\/\/github.com\/kanisterio\/kanister\/blob\/master\/examples\/etcd\/etcd-in-cluster\/ocp\/cluster-ocp-restore.sh<\/a><\/li>\n<li>Create a namespace where the K10 restore will be executed:<\/li>\n<\/ul>\n<style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-1 > .CodeMirror, .fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-1 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_1\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_1\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_1\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"\" data-theme=\"default\" data-mode=\"text\/x-sh\">oc create namespace etcd-restore<\/textarea><\/div>\n<ul>\n<li>We need to create a Persistent Volume and a Persistent Volume Clain in the namespace we created in the previous step.\u00a0 These PV and PVC will be used to copy the ETCD backup file to the Control Plane (master) node we choose to perform the restore operations.\u00a0 We can use the following commands to create both resources:<\/li>\n<\/ul>\n<style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-2 > .CodeMirror, .fusion-syntax-highlighter-2 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-2 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-2 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-2 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_2\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_2\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_2\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"\" data-theme=\"default\" data-mode=\"text\/x-sh\">oc --namespace etcd-restore apply -f \\\n     https:\/\/raw.githubusercontent.com\/kanisterio\/kanister\/0.91.0\/examples\/etcd\/etcd-in-cluster\/ocp\/blueprint-v2\/pv-etcd-backup.yaml\n\noc --namespace etcd-restore apply -f \\\n     https:\/\/raw.githubusercontent.com\/kanisterio\/kanister\/0.91.0\/examples\/etcd\/etcd-in-cluster\/ocp\/blueprint-v2\/pvc-etcd-backup.yaml<\/textarea><\/div>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1488\" src=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-300x50.png\" alt=\"\" width=\"450\" height=\"75\" srcset=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-200x33.png 200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-300x50.png 300w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-400x66.png 400w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-600x99.png 600w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-768x127.png 768w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-800x133.png 800w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-1024x170.png 1024w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-1200x199.png 1200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-16.09.35-1536x255.png 1536w\" sizes=\"(max-width: 450px) 100vw, 450px\" \/><\/a><\/p>\n<ul>\n<li>We need SSH connectivity to all master nodes (Control plane).<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Restore_the_ETCD_backup_using_Kasten\"><\/span>Restore the ETCD backup using Kasten<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Now that we have all the pre-requisites, we need to restore the ETCD backup file from the Kasten Location Profile (repository) to the Control Plane node we choose as the &#8220;Restore Node&#8221; (in my case master-0).\u00a0 So of course the first step is to choose one of the Control plane nodes, and then <strong>we add a label<\/strong> to that node, so Kasten can restore the ETCD backup file to that chosen node exactly.\u00a0 We will use the &#8220;etcd-restore=true&#8221; label using the following command:<\/p>\n<p><a href=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter  wp-image-1496\" src=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-300x48.png\" alt=\"\" width=\"381\" height=\"61\" srcset=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-200x32.png 200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-300x48.png 300w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-400x64.png 400w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-600x96.png 600w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-768x123.png 768w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-800x129.png 800w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-1024x165.png 1024w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-1200x193.png 1200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-14.41.46-1536x247.png 1536w\" sizes=\"(max-width: 381px) 100vw, 381px\" \/><\/a><\/p>\n<style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-3 > .CodeMirror, .fusion-syntax-highlighter-3 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-3 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-3 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-3 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_3\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_3\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_3\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"\" data-theme=\"default\" data-mode=\"text\/x-sh\">oc label node <your-leader-node-name> etcd-restore=true<\/textarea><\/div>\n<p>&nbsp;<\/p>\n<p>Why we need this label?\u00a0\u00a0 Well, if we remember from <a title=\"OpenShift ETCD backup and Restore with Kasten K10 \u2013 Part 2\" href=\"https:\/\/patriciocerda.com\/?p=1475\" target=\"_blank\" rel=\"noopener\">previous post<\/a>, Kasten will use a Kanister blueprint to backup and restore ETCD.\u00a0\u00a0 As we can see bellow, the &#8220;restore&#8221; section of the Kanister blueprint uses this label as the <strong>nodeSelector<\/strong> to run a Pod in the Master node chosen as the <strong>Restore node <\/strong>(in my case master-0), including also some Tolerations to allow the Pod to run in this node (by default no Pod can be scheduled in the Master nodes).\u00a0\u00a0\u00a0\u00a0 As the same blueprint describes, the pod is used to download the backup file from the object store and copy it to the <strong>\/mnt\/data<\/strong> location of the PV mapped to PVC `pvc-etcd` created previously as part of the pre-requisites.\u00a0\u00a0 The PV&#8217;s mount path is <strong>\/mnt\/data on leader node<\/strong> where the cluster-ocp-restore.sh script would be executed.<\/p>\n<style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-4 > .CodeMirror, .fusion-syntax-highlighter-4 > .CodeMirror .CodeMirror-gutters {background-color:var(--awb-color1);}.fusion-syntax-highlighter-4 > .CodeMirror .CodeMirror-gutters { background-color: var(--awb-color2); }.fusion-syntax-highlighter-4 > .CodeMirror .CodeMirror-linenumber { color: var(--awb-color8); }<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-4 fusion-syntax-highlighter-theme-light\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:var(--awb-color3);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_4\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_4\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_4\" data-readOnly=\"nocursor\" data-lineNumbers=\"1\" data-lineWrapping=\"\" data-theme=\"default\" data-mode=\"text\/x-sh\">  restore:\n    # This phase is not actualy performing restore of the etcd data store but is used\n    # to copy backup data to one of the leader nodes. It spins a pod on a leader node\n    # having label `etcd-restore`. The pod is used to download the backup file from the\n    # object store and copy it to the \/mnt\/data location of the PV mapped to PVC `pvc-etcd`.\n    # The PV's mount path is \/mnt\/data on leader node where the cluster-ocp-restore.sh\n    # script would be executed. \n    inputArtifactNames:\n    - etcdBackup\n    phases:\n    - func: PrepareData\n      name: copyFromObjectStore\n      args:\n        image: \"ghcr.io\/kanisterio\/kanister-tools:0.91.0\"\n        namespace: \"{{ .Object.metadata.namespace }}\"\n        podOverride:\n          nodeSelector:\n            etcd-restore: \"true\"\n          tolerations:\n          - key: \"node-role.kubernetes.io\/master\"\n            operator: \"Exists\"\n            effect: \"NoSchedule\"\n          containers:\n          - name: container\n            securityContext:    \n              privileged: true\n        volumes:\n          pvc-etcd: \"\/mnt\/data\"\n        command:\n        - bash\n        - -o\n        - errexit\n        - -o\n        - pipefail\n        - -c\n        - |\n          BACKUP_LOCATION='etcd-backup.db.gz'\n          kopia_snap='{{ .ArtifactsIn.etcdBackup.KopiaSnapshot }}'\n          kando location pull --profile '{{ toJson .Profile }}' --path \"${BACKUP_LOCATION}\" --kopia-snapshot \"${kopia_snap}\" - | gzip -d >> \/tmp\/etcd-backup.db\n          cp \/tmp\/etcd-backup.db \/mnt\/data<\/textarea><\/div>\n<p>&nbsp;<\/p>\n<p>Now, what we need to do next is to restore the ETCD backup file to the Master\/Leader node using Kasten.<\/p>\n<ul>\n<li>In Kasten dashboard, we can go to Applications and look for the Namespace &#8220;etcd-backup&#8221; which was created for the ETCD backup as described in the previous post.\u00a0\u00a0 We click in &#8220;restore&#8221; to chose one restore point:<a href=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter  wp-image-1491\" src=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-300x216.png\" alt=\"\" width=\"375\" height=\"270\" srcset=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-200x144.png 200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-300x216.png 300w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-400x288.png 400w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-600x432.png 600w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-768x554.png 768w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-800x577.png 800w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-1024x738.png 1024w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-1200x865.png 1200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.26.48-1536x1107.png 1536w\" sizes=\"(max-width: 375px) 100vw, 375px\" \/><\/a><\/li>\n<li>We choose one the restore points available.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-1492\" src=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-300x250.png\" alt=\"\" width=\"300\" height=\"250\" srcset=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-200x167.png 200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-300x250.png 300w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-400x334.png 400w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-600x500.png 600w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-768x641.png 768w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-800x667.png 800w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-1024x854.png 1024w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-1200x1001.png 1200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.00-1536x1281.png 1536w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<ul>\n<li>For the restore, please select the Namespace <strong>etcd-restore<\/strong> created previously as part of the pre-requisites and click in <strong>Restore<\/strong>.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter  wp-image-1493\" src=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-260x300.png\" alt=\"\" width=\"322\" height=\"372\" srcset=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-200x231.png 200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-260x300.png 260w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-400x461.png 400w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-600x692.png 600w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-768x886.png 768w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-800x923.png 800w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-888x1024.png 888w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-1200x1384.png 1200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-1331x1536.png 1331w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1-1775x2048.png 1775w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-18-a-las-10.27.18-1.png 1914w\" sizes=\"(max-width: 322px) 100vw, 322px\" \/><\/a><\/p>\n<ul>\n<li>We can go back to the Dashboard and monitor the restore task until is completed successfully.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter  wp-image-1494\" src=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-300x32.png\" alt=\"\" width=\"459\" height=\"49\" srcset=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-200x22.png 200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-300x32.png 300w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-400x43.png 400w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-600x65.png 600w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-768x83.png 768w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-800x86.png 800w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-1024x110.png 1024w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-1200x129.png 1200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.40.30-1536x165.png 1536w\" sizes=\"(max-width: 459px) 100vw, 459px\" \/><\/a><\/p>\n<ul>\n<li>Now we connect via SSH to the Master node chosen as the <strong>Restore node <\/strong>(in my case master-0), and we check the <strong>\/mnt\/data<\/strong> location for the ETCD backup file as we can see in the following image.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter  wp-image-1495\" src=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-300x47.png\" alt=\"\" width=\"345\" height=\"54\" srcset=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-200x31.png 200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-300x47.png 300w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-400x62.png 400w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-600x93.png 600w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-768x119.png 768w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-800x124.png 800w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-1024x159.png 1024w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-1200x186.png 1200w, https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/Captura-de-pantalla-2023-05-19-a-las-12.42.42-1536x238.png 1536w\" sizes=\"(max-width: 345px) 100vw, 345px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Revert_the_OpenShift_cluster_state_using_ETCD_Backup\"><\/span>Revert the OpenShift cluster state using ETCD Backup<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In this point, we have restored the ETCD backup file using Kasten, and all the remaining steps to revert the cluster state will follow the standard OpenShift ETCD restore process as described in the RedHat OpenShift documentation with some minor modifications:\u00a0 <a href=\"https:\/\/docs.openshift.com\/container-platform\/4.13\/backup_and_restore\/control_plane_backup_and_restore\/disaster_recovery\/scenario-2-restoring-cluster-state.html\" target=\"_blank\" rel=\"noopener\">https:\/\/docs.openshift.com\/container-platform\/4.13\/backup_and_restore\/control_plane_backup_and_restore\/disaster_recovery\/scenario-2-restoring-cluster-state.html<\/a><\/p>\n<p>The overall ETCD restore process includes:<\/p>\n<ul>\n<li aria-level=\"1\">Stop the static pods on any other control plane nodes but the restore node.\u00a0 <strong>Important<\/strong>: don&#8217;t stop the static pods in the restore node.<\/li>\n<li aria-level=\"1\">Running the restore script \u201c<i>cluster-restore.sh<\/i>\u201d in the restore node (<strong>use the modified <a href=\"https:\/\/github.com\/kanisterio\/kanister\/blob\/master\/examples\/etcd\/etcd-in-cluster\/ocp\/cluster-ocp-restore.sh\" target=\"_blank\" rel=\"noopener\">cluster-ocp-restore.sh script<\/a> mentioned before<\/strong> instead of the standard script provided by OpenShift)<\/li>\n<li aria-level=\"1\">Checking if the nodes are in the \u201c<strong>ready<\/strong>\u201d state<\/li>\n<li aria-level=\"1\">Restarting the <i>kubelet <\/i>service for all of the control plane hosts, including the restore node<\/li>\n<li aria-level=\"1\">Approving pending CSRs.<\/li>\n<li aria-level=\"1\">Verify that the single member control plane has started successfully.\u00a0\u00a0 From the restore node, verify that the etcd container is running.<\/li>\n<li aria-level=\"1\">Deleting and recreating all of the control plane machines, with the exclusion of the one chosen for recovery purposes (restore node).\n<ul>\n<li aria-level=\"1\">After these machines are recreated, a new revision is forced and etcd scales up automatically.<\/li>\n<li aria-level=\"1\">If you are running installer-provisioned infrastructure, or you used the Machine API to create your machines, the nodes will be re-created automatically.<\/li>\n<li aria-level=\"1\">Otherwise, you must create the new control plane node using the same method that was used to originally create it.<\/li>\n<\/ul>\n<\/li>\n<li>In a separate terminal window, log in to the cluster as a user with the <code>cluster-admin<\/code> role.\u00a0 Then force the etcd redeployment<\/li>\n<li aria-level=\"1\">Verifying if the nodes are up to date<\/li>\n<li aria-level=\"1\">After etcd is redeployed, force new rollouts for the control plane (this should reinstall Kubernetes API to all of the nodes since an internal load balancer is used to connect the kubelet to the API server)<\/li>\n<li aria-level=\"1\">Verify that all of the newly installed control plane hosts are working and joined the cluster.<\/li>\n<\/ul>\n<p>The same process but a bit summarized (in case you don&#8217;t want to follow the instructions from OpenShift documentation) it&#8217;s available in Kasten documentation: <a href=\"https:\/\/docs.kasten.io\/latest\/kanister\/etcd\/ocp\/install.html#restore\" target=\"_blank\" rel=\"noopener\">https:\/\/docs.kasten.io\/latest\/kanister\/etcd\/ocp\/install.html#restore<\/a><\/p>\n<p>&nbsp;<\/p>\n<p>This finalizes this Blog posts series about OpenShift ETCD Backup and Restore using Kasten.\u00a0 As we could see:<\/p>\n<ul>\n<li>Restoring ETCD backup should only be used as the last resort, as it&#8217;s a very destructive and destablizing action.<\/li>\n<li>With Kasten it&#8217;s possible to design a Backup and Disaster Recovery strategy to protect all the applications and cluster-wide resources <strong>without the need of taking ETCD backups<\/strong>, and avoiding this cumbersome process to revert the cluster state using a ETCD backup.\u00a0 If you want to know more about Backup and Disaster Recovery strategies for Kubernetes using Kasten, please check this whitepaper: <a href=\"https:\/\/fromthearchitect.net\/wp-content\/uploads\/2022\/09\/Designing_a_Kubernetes_DR_strategy.pdf\" target=\"_blank\" rel=\"noopener\">https:\/\/fromthearchitect.net\/wp-content\/uploads\/2022\/09\/Designing_a_Kubernetes_DR_strategy.pdf<\/a><\/li>\n<li>If you still want to take ETCD backups, you can leverage Kasten to:\n<ul>\n<li>Automate the ETCD backup and send it to a Location Profile (repository).\u00a0 Remember by default the ETCD backup is stored locally when created and it&#8217;s not automated natively (it requires a script for automation).<\/li>\n<li>Restore the ETCD backup file directly to the Master node chosen as Restore node.<\/li>\n<li>Keep multiple ETCD backups (restore points) according to the required retention policy.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi all and welcome back.\u00a0 This is the third and<\/p>\n","protected":false},"author":1,"featured_media":1471,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[25,67,27,68,43],"tags":[37,83,33,32,29,31,75,69,38,30],"aioseo_notices":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>OpenShift ETCD backup and Restore with Kasten K10 - Part 3-Final - vLatam<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/patriciocerda.com\/?p=1487\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"OpenShift ETCD backup and Restore with Kasten K10 - Part 3-Final - vLatam\" \/>\n<meta property=\"og:description\" content=\"Hi all and welcome back.\u00a0 This is the third and\" \/>\n<meta property=\"og:url\" content=\"https:\/\/patriciocerda.com\/?p=1487\" \/>\n<meta property=\"og:site_name\" content=\"vLatam\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-26T09:51:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"pcerda\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"pcerda\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/patriciocerda.com\/?p=1487\",\"url\":\"https:\/\/patriciocerda.com\/?p=1487\",\"name\":\"OpenShift ETCD backup and Restore with Kasten K10 - Part 3-Final - vLatam\",\"isPartOf\":{\"@id\":\"https:\/\/patriciocerda.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/patriciocerda.com\/?p=1487#primaryimage\"},\"image\":{\"@id\":\"https:\/\/patriciocerda.com\/?p=1487#primaryimage\"},\"thumbnailUrl\":\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png\",\"datePublished\":\"2023-05-26T09:51:49+00:00\",\"author\":{\"@id\":\"https:\/\/patriciocerda.com\/#\/schema\/person\/613aa192d7dbcedcd20e08318891aa2a\"},\"breadcrumb\":{\"@id\":\"https:\/\/patriciocerda.com\/?p=1487#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/patriciocerda.com\/?p=1487\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/patriciocerda.com\/?p=1487#primaryimage\",\"url\":\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png\",\"contentUrl\":\"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png\",\"width\":1920,\"height\":1080},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/patriciocerda.com\/?p=1487#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/patriciocerda.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"OpenShift ETCD backup and Restore with Kasten K10 &#8211; Part 3-Final\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/patriciocerda.com\/#website\",\"url\":\"https:\/\/patriciocerda.com\/\",\"name\":\"vLatam\",\"description\":\"El Blog de Patricio Cerda\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/patriciocerda.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/patriciocerda.com\/#\/schema\/person\/613aa192d7dbcedcd20e08318891aa2a\",\"name\":\"pcerda\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/patriciocerda.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/patriciocerda.com\/wp-content\/litespeed\/avatar\/ce92117b9294544adeaad229a8fbac13.jpg?ver=1776772888\",\"contentUrl\":\"https:\/\/patriciocerda.com\/wp-content\/litespeed\/avatar\/ce92117b9294544adeaad229a8fbac13.jpg?ver=1776772888\",\"caption\":\"pcerda\"},\"url\":\"https:\/\/patriciocerda.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"OpenShift ETCD backup and Restore with Kasten K10 - Part 3-Final - vLatam","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/patriciocerda.com\/?p=1487","og_locale":"en_US","og_type":"article","og_title":"OpenShift ETCD backup and Restore with Kasten K10 - Part 3-Final - vLatam","og_description":"Hi all and welcome back.\u00a0 This is the third and","og_url":"https:\/\/patriciocerda.com\/?p=1487","og_site_name":"vLatam","article_published_time":"2023-05-26T09:51:49+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png","type":"image\/png"}],"author":"pcerda","twitter_card":"summary_large_image","twitter_misc":{"Written by":"pcerda","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/patriciocerda.com\/?p=1487","url":"https:\/\/patriciocerda.com\/?p=1487","name":"OpenShift ETCD backup and Restore with Kasten K10 - Part 3-Final - vLatam","isPartOf":{"@id":"https:\/\/patriciocerda.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/patriciocerda.com\/?p=1487#primaryimage"},"image":{"@id":"https:\/\/patriciocerda.com\/?p=1487#primaryimage"},"thumbnailUrl":"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png","datePublished":"2023-05-26T09:51:49+00:00","author":{"@id":"https:\/\/patriciocerda.com\/#\/schema\/person\/613aa192d7dbcedcd20e08318891aa2a"},"breadcrumb":{"@id":"https:\/\/patriciocerda.com\/?p=1487#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/patriciocerda.com\/?p=1487"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/patriciocerda.com\/?p=1487#primaryimage","url":"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png","contentUrl":"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png","width":1920,"height":1080},{"@type":"BreadcrumbList","@id":"https:\/\/patriciocerda.com\/?p=1487#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/patriciocerda.com\/"},{"@type":"ListItem","position":2,"name":"OpenShift ETCD backup and Restore with Kasten K10 &#8211; Part 3-Final"}]},{"@type":"WebSite","@id":"https:\/\/patriciocerda.com\/#website","url":"https:\/\/patriciocerda.com\/","name":"vLatam","description":"El Blog de Patricio Cerda","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/patriciocerda.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/patriciocerda.com\/#\/schema\/person\/613aa192d7dbcedcd20e08318891aa2a","name":"pcerda","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/patriciocerda.com\/#\/schema\/person\/image\/","url":"https:\/\/patriciocerda.com\/wp-content\/litespeed\/avatar\/ce92117b9294544adeaad229a8fbac13.jpg?ver=1776772888","contentUrl":"https:\/\/patriciocerda.com\/wp-content\/litespeed\/avatar\/ce92117b9294544adeaad229a8fbac13.jpg?ver=1776772888","caption":"pcerda"},"url":"https:\/\/patriciocerda.com\/?author=1"}]}},"jetpack_featured_media_url":"https:\/\/patriciocerda.com\/wp-content\/uploads\/2023\/05\/IntroBlog1467.png","_links":{"self":[{"href":"https:\/\/patriciocerda.com\/index.php?rest_route=\/wp\/v2\/posts\/1487"}],"collection":[{"href":"https:\/\/patriciocerda.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/patriciocerda.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/patriciocerda.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/patriciocerda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1487"}],"version-history":[{"count":2,"href":"https:\/\/patriciocerda.com\/index.php?rest_route=\/wp\/v2\/posts\/1487\/revisions"}],"predecessor-version":[{"id":1497,"href":"https:\/\/patriciocerda.com\/index.php?rest_route=\/wp\/v2\/posts\/1487\/revisions\/1497"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/patriciocerda.com\/index.php?rest_route=\/wp\/v2\/media\/1471"}],"wp:attachment":[{"href":"https:\/\/patriciocerda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1487"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/patriciocerda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1487"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/patriciocerda.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1487"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}