Bulk Renaming With Vim's :cdo

Last week we ended by piecing together the following shell commands to make a json_*_error_log_table/ directory for every json_*_access_log_table/ also present.

~/big_query$ ls -1
json_a_access_log_table/
json_a_error_log_table/
json_b_access_log_table/
json_b_error_log_table/
json_c_access_log_table/
json_c_error_log_table/
json_d_access_log_table/
json_d_error_log_table/
json_e_access_log_table/
json_e_error_log_table/
json_f_access_log_table/
json_f_error_log_table/
json_g_access_log_table/
json_g_error_log_table/
json_h_access_log_table/
json_h_error_log_table/
json_i_access_log_table/
json_i_error_log_table/
json_j_access_log_table/
json_j_error_log_table/
json_k_access_log_table/
json_k_error_log_table/
native_a_access_log_table/
native_b_access_log_table/
native_c_access_log_table/
native_e_access_log_table/
native_f_access_log_table/
native_g_access_log_table/
native_i_access_log_table/
native_j_access_log_table/
native_k_access_log_table/
native_m_access_log_table/
native_n_access_log_table/
native_q_access_log_table/
native_r_access_log_table/
native_u_access_log_table/
native_v_access_log_table/
native_y_access_log_table/
native_z_access_log_table/

Neat! Now we just need to copy over some files into the newly created directories and make some minor edits.

Step 1: Copying JSON Files.

After creating our error_log_table directories in last week’s post we basically have the following scenario. Our access_logs_table directories have table_def.json files that we need to copy over to corresponding error_logs_table directories.

.
├── native_*_access_logs_table
│   ├── create.sql
│   └── insert.sql
├── json_*_error_logs_table
└── json_*_access_logs_table
    └── table_def.json

We can easily copy over these files using a simple for-loop in our shell.

Step 2: Vim’s :cdo command

Now we can get down to business. Below are the contents of the json_a_error_logs_table/table_def.json file. You’ll notice that there are 3 lines in the JSON that need to be updated (as denoted by the obnoxious not-valid-json I added).

{
  "tableReference": {
    "projectId": "my_project",
    "datasetId": "logs",
    "tableId": "json_a_access_logs"     <------- wrong tableId
  },
  "externalDataConfiguration": {
    "autodetect": false,
    "schema": {
      "fields": [
          {
              "name": "json",
              "type": "STRING"
          }
      ]
    },
    "sourceUris": [
      "gs://fake-test-bucket/topics/json_a_access_logs/*"   <------- wrong sourceUris
    ],
    "hivePartitioningOptions": {
      "mode": "CUSTOM",
      "sourceUriPrefix": "gs://fake-test-bucket/topics/json_a_access_logs/{dt:DATE}/{hour:INTEGER}",    <------- wrong sourceUriPrefix
      "requirePartitionFilter": true
    },
    "sourceFormat": "CSV",
    "compression": "GZIP",
    "ignoreUnknownValues": true
  }
}

Just like before, this wouldn’t be that much of a hassle if it was only this file that needed updating, but in this case we have table_def.json’s for directories a - k. Imagine having to manually repeat this for potentially even more 😓. Vim’s cdo command can be leveraged easily here! For those who are not aware, see the quick snippet from the vim help documentation:

:cdo[!] {cmd}		Execute {cmd} in each valid entry in the quickfix list.

All we have to do now is populate vim’s quickfixlist with vim’s :grep command. (If needed, look up any of the millions of tutorials already available covering :grep)

First, let’s find all instances of “access_logs” in all the json files in the new error_log_table directories.


The following command will populate vim’s quickfixlist with all grep matches.


You can use `:clist` to see these matches.


Execute this substitution for every entry in the quickfixlist.

:cdo s/access_/error_/g | update will replace every instance of access_ with error_ and update will write the changes to each file.

#TODO data engineering

Bulk Renaming With Vim's :cdo