Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | {{info}} | ||
| 2 | 2018-12-20: Version 0.5 released. | ||
| 3 | {{/info}} | ||
| 4 | |||
| 5 | {{warning title="Disclaimer"}} | ||
| 6 | As with any copy/sync tool, make sure to test thoroughly with data that can be lost. | ||
| 7 | |||
| 8 | Test with the ##~-~-dry## and ##~-~-dry-rsync## options to verify that the operations match what you want to do. | ||
| 9 | {{/warning}} | ||
| 10 | |||
| 11 | {{toc-local/}} | ||
| 12 | |||
| 13 | = Introduction = | ||
| 14 | |||
| 15 | rmirr provides a **safe** and easy to use way to set up and execute file storage mirroring instructions. And because it uses rsync to actually perform the synchronization, you get all the performance and availability benefits of rsync. | ||
| 16 | |||
| 17 | Features: | ||
| 18 | |||
| 19 | * specify only the source file object (file, directory) to mirror | ||
| 20 | * mirror settings selected by best match path | ||
| 21 | * support one or more mirror destinations | ||
| 22 | * ensure mirroring from specific sources only | ||
| 23 | * ensure single instance of mirroring operation per mirror path | ||
| 24 | * log history of mirroring invocations | ||
| 25 | * save to file report of mirrored objects | ||
| 26 | * email report file | ||
| 27 | |||
| 28 | = Why not just use rsync directly? = | ||
| 29 | |||
| 30 | Simple cases of mirroring files/directories can easily be done by using rsync directly. But, it does not provide any safeguards. For example, the following are not the same: | ||
| 31 | |||
| 32 | {{noformat}} | ||
| 33 | rsync -avz --delete /a/b/c mach2:/a/b/c | ||
| 34 | {{/noformat}} | ||
| 35 | |||
| 36 | and: | ||
| 37 | |||
| 38 | {{noformat}} | ||
| 39 | rsync -avz --delete /a/b/c/ mach2:/a/b/c | ||
| 40 | {{/noformat}} | ||
| 41 | |||
| 42 | Assuming a directory is being synced, call #2 is correct, as it uses a trailing / on the source path. rmirr ensures that will be done properly. | ||
| 43 | |||
| 44 | = Usage = | ||
| 45 | |||
| 46 | {{noformat}} | ||
| 47 | usage: rmirr [<options>] [-n <name>] -p <path> | ||
| 48 | rmirr [<options>] [-n <name>] -s <suitename> | ||
| 49 | rmirr [<options>] -n <name> | ||
| 50 | rmirr -l | ||
| 51 | |||
| 52 | Mirror file objects which can be specified by <path> (for which a | ||
| 53 | mirror is selected), <suite> (representing one or more paths), or | ||
| 54 | <name> (a mirror configuration). | ||
| 55 | |||
| 56 | Where: | ||
| 57 | -l List mirror configurations. | ||
| 58 | -n <name> | ||
| 59 | Mirror name. | ||
| 60 | -p <path> | ||
| 61 | Mirror path. | ||
| 62 | -s <suitename> | ||
| 63 | Alias for one or more paths. | ||
| 64 | |||
| 65 | Options: | ||
| 66 | -c <path> | ||
| 67 | Location of paths.json configuration file. | ||
| 68 | -d <hostname>[,...] | ||
| 69 | Destinations to mirror to from list in configuration. | ||
| 70 | --debug | ||
| 71 | Enable debugging. | ||
| 72 | --delete | ||
| 73 | Allow file object deletion. | ||
| 74 | --dry Dry run. Do not execute. | ||
| 75 | --dry-rsync | ||
| 76 | Dry run for rsync. | ||
| 77 | --mailto <emailaddr>[,...] | ||
| 78 | Set/override recipients when mailing report. | ||
| 79 | --mailreport | ||
| 80 | Mail report. | ||
| 81 | --nolock | ||
| 82 | Do not use/require lock to run. | ||
| 83 | --safeoff | ||
| 84 | Disable safemode. | ||
| 85 | --showreport | ||
| 86 | Show report on console. | ||
| 87 | --verbose | ||
| 88 | Enable verbosity. | ||
| 89 | -y Do not ask for confirmation before executing. | ||
| 90 | {{/noformat}} | ||
| 91 | |||
| 92 | = Configuration = | ||
| 93 | |||
| 94 | Configuration file is located at ##~~/.rmirr/rmirr.json##. | ||
| 95 | |||
| 96 | (% class="wrapped" %) | ||
| 97 | |=((( | ||
| 98 | Section | ||
| 99 | )))|=((( | ||
| 100 | Name | ||
| 101 | )))|=((( | ||
| 102 | Description | ||
| 103 | ))) | ||
| 104 | |(% rowspan="2" %)(% rowspan="2" %) | ||
| 105 | ((( | ||
| 106 | defaults | ||
| 107 | )))|((( | ||
| 108 | email_recipients | ||
| 109 | )))|((( | ||
| 110 | List of email recipients when ~-~-mailreport is used. | ||
| 111 | ))) | ||
| 112 | |(% colspan="1" %)(% colspan="1" %) | ||
| 113 | ((( | ||
| 114 | smtp_server | ||
| 115 | )))|(% colspan="1" %)(% colspan="1" %) | ||
| 116 | ((( | ||
| 117 | Set SMTP server. Default is to use "localhost". | ||
| 118 | ))) | ||
| 119 | |(% colspan="1" %)(% colspan="1" %) | ||
| 120 | ((( | ||
| 121 | suites | ||
| 122 | )))|(% colspan="1" %)(% colspan="1" %) | ||
| 123 | ((( | ||
| 124 | <suitename> | ||
| 125 | )))|(% colspan="1" %)(% colspan="1" %) | ||
| 126 | ((( | ||
| 127 | List of paths to be mirrored. | ||
| 128 | ))) | ||
| 129 | |(% rowspan="6" %)(% rowspan="6" %) | ||
| 130 | ((( | ||
| 131 | mirrors | ||
| 132 | )))|((( | ||
| 133 | comment | ||
| 134 | )))|((( | ||
| 135 | Description of what is being mirrored. | ||
| 136 | ))) | ||
| 137 | |(% colspan="1" %)(% colspan="1" %) | ||
| 138 | ((( | ||
| 139 | destinations | ||
| 140 | )))|(% colspan="1" %)(% colspan="1" %) | ||
| 141 | ((( | ||
| 142 | List of destinations to which the source is to be mirrored each of the format [<user>@]]<host>:<path>. | ||
| 143 | ))) | ||
| 144 | |((( | ||
| 145 | email_recipients | ||
| 146 | )))|((( | ||
| 147 | List of email recipients when ~-~-mailreport is used. Overrides the value from the defaults section. | ||
| 148 | ))) | ||
| 149 | |((( | ||
| 150 | excludes | ||
| 151 | )))|((( | ||
| 152 | List of file/dir names to exclude from the mirroring. | ||
| 153 | ))) | ||
| 154 | |((( | ||
| 155 | name | ||
| 156 | )))|((( | ||
| 157 | Mirror configuration name. | ||
| 158 | ))) | ||
| 159 | |((( | ||
| 160 | source | ||
| 161 | )))|((( | ||
| 162 | Source of the mirror: [<user>@]<host>:<path>. | ||
| 163 | ))) | ||
| 164 | |||
| 165 | = Examples = | ||
| 166 | |||
| 167 | Given the configuration at ##~~/.rmirr/rmirr.json##: | ||
| 168 | |||
| 169 | {{code language="none"}} | ||
| 170 | { | ||
| 171 | "defaults": { | ||
| 172 | "email_recipients": [] | ||
| 173 | }, | ||
| 174 | "suites": { | ||
| 175 | "all": [ | ||
| 176 | "/data/main", | ||
| 177 | "/data/finance", | ||
| 178 | "/data/hr" | ||
| 179 | ] | ||
| 180 | }, | ||
| 181 | "mirrors": [ | ||
| 182 | { | ||
| 183 | "name": "data-main--home--finance-hr", | ||
| 184 | "comment": "data main: home -> finance, hr", | ||
| 185 | "source": "data@home.abc.xyz:/data/main", | ||
| 186 | "excludes": [".snapshot", ".snapshots"], | ||
| 187 | "destinations": ["finance.abc.xyz", "hr.abc.xyz"] | ||
| 188 | }, | ||
| 189 | { | ||
| 190 | "name": "data-finance--home--finance", | ||
| 191 | "comment": "data finance: home -> finance", | ||
| 192 | "email_recipients": ["bob@finance.abc.xyz"], | ||
| 193 | "source": "data@home.abc.xyz:/data/finance", | ||
| 194 | "excludes": [".snapshot", ".snapshots"], | ||
| 195 | "destinations": ["finance.abc.xyz"] | ||
| 196 | }, | ||
| 197 | { | ||
| 198 | "name": "data-hr--home--hr", | ||
| 199 | "comment": "data hr: home -> hr", | ||
| 200 | "source": "data@home.abc.xyz:/data/hr", | ||
| 201 | "excludes": [".snapshot", ".snapshots"], | ||
| 202 | "destinations": ["hr.abc.xyz"] | ||
| 203 | } | ||
| 204 | ] | ||
| 205 | } | ||
| 206 | {{/code}} | ||
| 207 | |||
| 208 | List suite and mirror settings: | ||
| 209 | |||
| 210 | {{noformat}} | ||
| 211 | rmirr -l | ||
| 212 | {{/noformat}} | ||
| 213 | |||
| 214 | Mirror ##/data/main## from home.abc.xyz: | ||
| 215 | |||
| 216 | {{noformat}} | ||
| 217 | rmirr -p /data/main | ||
| 218 | {{/noformat}} | ||
| 219 | |||
| 220 | Notes: | ||
| 221 | |||
| 222 | * mirrors to both finance.abc.xyz and hr.abc.xyz | ||
| 223 | * .snapshot and .snapshots are excluded | ||
| 224 | * by default, mirroring **does not** delete at the destination | ||
| 225 | * by default, mirroring **requires** user interaction ("y" to proceed) | ||
| 226 | |||
| 227 | Show what rmirr would do **without** executing it (dry-run): | ||
| 228 | |||
| 229 | {{noformat}} | ||
| 230 | rmirr --dry -p /data/main | ||
| 231 | {{/noformat}} | ||
| 232 | |||
| 233 | Mirror /data/main/opt: | ||
| 234 | |||
| 235 | {{noformat}} | ||
| 236 | rmirr -p /data/main/opt | ||
| 237 | {{/noformat}} | ||
| 238 | |||
| 239 | Notes: | ||
| 240 | |||
| 241 | * because ##/data/main/opt## is under ##/data/main##, the settings for ##/data/main## are used | ||
| 242 | * only ##/data/main/opt## is mirrored | ||
| 243 | |||
| 244 | Mirror /data/finance and allow deletion at the destination: | ||
| 245 | |||
| 246 | {{noformat}} | ||
| 247 | rmirr --delete -p /data/finance | ||
| 248 | {{/noformat}} | ||
| 249 | |||
| 250 | Do a dry run (at rsync level) to mirror /data/finance with deletion and no user interaction: | ||
| 251 | |||
| 252 | {{noformat}} | ||
| 253 | rmirr --dry-rsync --delete -y -p /data/finance | ||
| 254 | {{/noformat}} | ||
| 255 | |||
| 256 | Mirror /data/main to specific destination (hr.abc.xyz): | ||
| 257 | |||
| 258 | {{noformat}} | ||
| 259 | rmirr --delete -y -d hr.abc.xyz -p /data/main | ||
| 260 | {{/noformat}} | ||
| 261 | |||
| 262 | Request report be emailed: | ||
| 263 | |||
| 264 | {{noformat}} | ||
| 265 | rmirr -p /data/main --mailreport | ||
| 266 | {{/noformat}} | ||
| 267 | |||
| 268 | Request report be output to console: | ||
| 269 | |||
| 270 | {{noformat}} | ||
| 271 | rmirr -p /data/main --showreport | ||
| 272 | {{/noformat}} | ||
| 273 | |||
| 274 | Mirror the "all" suite: | ||
| 275 | |||
| 276 | {{noformat}} | ||
| 277 | rmirr -s all | ||
| 278 | {{/noformat}} | ||
| 279 | |||
| 280 | = Mirroring Selected File Objects = | ||
| 281 | |||
| 282 | Sometimes only a subset of file objects (files, directories) need to be mirrored rather than a full tree. rmirr supports this with the names setting in the configuration file. | ||
| 283 | |||
| 284 | Configuration for mirroring bash files: | ||
| 285 | |||
| 286 | {{code language="none"}} | ||
| 287 | { | ||
| 288 | "suites": { | ||
| 289 | "bashfiles": [ | ||
| 290 | "~/.bash_completion.d", | ||
| 291 | "~/.bash_profile", | ||
| 292 | "~/.bash_logout", | ||
| 293 | "~/.bashrc", | ||
| 294 | "~/.profile", | ||
| 295 | "~/.profile.d" | ||
| 296 | ] | ||
| 297 | }, | ||
| 298 | "mirrors": [ | ||
| 299 | { | ||
| 300 | "name": "my-bash-files", | ||
| 301 | "comment": "my .bash files", | ||
| 302 | "source": "jdm@home.abc.xyz:~", | ||
| 303 | "excludes": [ | ||
| 304 | ".snapshot", | ||
| 305 | ".snapshots" | ||
| 306 | ], | ||
| 307 | "destinations": [ | ||
| 308 | "finance.abc.xyz", | ||
| 309 | "hr.abc.xyz" | ||
| 310 | ] | ||
| 311 | } | ||
| 312 | ] | ||
| 313 | } | ||
| 314 | {{/code}} | ||
| 315 | |||
| 316 | Notes: | ||
| 317 | |||
| 318 | * the suites section defines a suite to reference those bash-specific file objects by a single name | ||
| 319 | * of course, any changes/tweaks done at the destinations will be overwritten when rmirr is called | ||
| 320 | |||
| 321 | To mirror all the file objects of the "bashfiles" suite: | ||
| 322 | |||
| 323 | {{noformat}} | ||
| 324 | rmirr -s bashfiles | ||
| 325 | {{/noformat}} | ||
| 326 | |||
| 327 | To mirror specific file objects: | ||
| 328 | |||
| 329 | {{noformat}} | ||
| 330 | rmirr -p /home/jdm/.bash_completion.d | ||
| 331 | {{/noformat}} |