Backing Up Your Files: A Comprehensive Guide for CG Artists and Users Alike
Quick Index:
Creating an Automated Backup System: Why?
Adding A DOSRAR Environment Variable
Getting to Know DOSRAR
Writing A DOS Batch Script to Backup Your Files
Archiving Compressed File Types: An Easy Solution
Backing up with Parity and Splitting Archives
Syncing and Updating Backup Archives
Automating Batch Files
Conclusion
As artists who work in such an intangible medium, we cannot afford to lose any of our work. The thought of losing work is just unthinkable -but it happens. frequently. It is hard to be proactive, especially when you have never had a drive fail, or lost any data, but you must be vigilant, because all hard disks eventually fail.
I recently had a RAID failure. Because multiple drives failed (2), I was unable to rebuild the lost data through parity, and I lost a lot of work. This is the first time I have ever really lost a lot of data. I back up somewhat regularly, and in the past when a drive, I just rebuilt it with parity data. [RAID parity explained]
In this tutorial I will talk about the available methods of backing up your files automatically, and explain how to create your own automated backup system using RAR compression and the built in DOS scheduler in windows NT4/2K/XP.
Creating An Automated Backup System: Why?
There are many programs out there that will help you back up your files. Any of these that are automated usually must run 24/7 and take up shop in your taskbar. I wanted a simple automated system that used tools already on my computer to back up files automatically, not install more 3rd party apps and services.
NTBackup: The first option along this route is the NTBackup [Click to open] capability built into Windows. You can also open NTBackup by clicking the Start menu and selecting [Run...], then typing ntbackup into the field and clicking [OK]. It is a very decent, and practically unknown way of backing up your files. It is my opinion that if people knew it existed, they wouldn't purchase any 3rd party software to do the same task, it also has it's own built in scheduler. If you are a novice user, and do not care to learn anything about writing DOS batch files to automate tasks, or unlocking the power of the RAR command line options, then stop here, but let me tell you that many problems you will run into with Maya, MTOR, and other high end apps could be solved or worked around with a little batch script know-how.. NTBackup has some issues, mainly that it uses a proprietary compression method: BKF, and only Windows can read that and deal with those files, it doesn't appear to compress that well at all, and does not allow for parity.
I decided to attempt to create an automated backup system that was light (very simple, with little or no mem footprint), and used a tried and true compression algorithm that allowed for parity and could be manipulated via batch/command line.
WinRAR: Without a doubt, WinRAR is one of, if not the best, file compression utilities/algorithms on the market. If you are a CG Artist and have never heard of this program/algorithm, you really need to read up on it. A lot of people do not understand the need for file compression, at the left you see a Lightwave object that has been compressed with ZIP and RAR, for a long time now ZIP has been the compression of choice on the PC platform, and to an extent it still continues to be today, but RAR is faster, and compresses files significantly smaller. [Comprehensive WinRAR FAQ/Tutorial]
Adding A DOSRAR Environment Variable
Once you have installed WinRAR, you will need to set an environment variable to allow any scripts you write to access it from anywhere they are run on your PC. This is fairly easy: right click on your computer (via the desktop) and select 'Properties', this will open the system preferences, you can also get here by clicking the Start menu and going to Settings>Control Panel>System. Next, click on the [Advanced] tab, at the bottom you will see a button that says [Environment Variables], click it. Under 'System Variables you will see the 'Path' variable, double click it. BE CAREFUL not to delete or change any of the current variables, but add the location of your winrar folder followed by a semicolon to the list of Paths. You may have a pretty long Path variable already, and it may help to copy it and paste it into a text editor if you think you might mess something up, here is what a common Path variable could look like:
C:\Program Files\Alias\Maya6.0\bin;C:\Program Files\j2sdk_nb\j2sdk1.4.2\jre\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem
You can see that Maya and the Java Runtime Environment have added themselves to the original Windows 'Path' variable, allowing the programs to be called from any command line. This is what allows you to type 'render file.mb' and start a Maya render from any location on your machine via command line. (Something else you should already know how to do if you are still using the Maya render engine) Your path can be a lot longer if you have use DOS a lot or a PRMAN compliant renderer. Anyhow, you will be adding the path to the installation folder of WinRAR somewhere into the Path variable, I installed WinRar into C:\Utilities\WinRAR;, the result will look something like this:
C:\Utilities\WinRAR;C:\Program Files\Alias\Maya6.0\bin;C:\Program Files\j2sdk_nb\j2sdk1.4.2\jre\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem rar a -rr10 -s h:\backupC.rar @backupC.lst
The above line does a lot of things, it saves all the files and folders specified in the list 'backupC.lst' to drive h: as backupC.rar. It also writes a 10% Recovery Record or parity amount (-rr10). This way up to 10% of the compressed archive can become corrupt or unusable, and it will be able to recover the data through parity. The flag -s makes the archive a Solid Archive using a special compression method which sees all files as one continuous data stream. This is particularly advantageous when packing a large number of small files. The file backupC.lst is a file I created that resides in the same folder as the batch I will be writing, it is a text file that lists the files I would like backed up on the c: drive, here is mine as an example:
C:\Documents and Settings\User\Desktop
C:\Documents and Settings\User\Favorites
C:\Documents and Settings\User\My Documents\maya
C:\Documents and Settings\User\*.cfg
C:\Documents and Settings\All Users\Start Menu
C:\Documents and Settings\User\Local Settings\Application Data\Microsoft\Outlook
C:\Internet\FlashFXP\*.dat
C:\Internet\mirc\*.ini
C:\Internet\Soulseek\*.cfg
You can see that I backed up my Maya user folder, Lightwave configs, desktop, email, etc. Now you could set this to back up everything in one go, but I have decided to backup everything by drive, this should make things much more manageable. Some drives, like c:, have a lot of files I do not need to back up, others, like my RAID array, I need most or all of the files, so creating a list of only certain directories to backup is very powerful.
Writing A DOS Batch Script to Backup Your Files
I will now talk a little bit about batch files. DOS batch files, in their simplest forms, are scripts that can enter commands into DOS like you are actually entering them. You type them and line by line, they will execute the commands you have entered. They are text files with the extension .BAT (example: autoexec.bat)
As you may have noticed when reading the man pages for rar, you can comment the RAR files. You can comment them either manually, or from a comment text file. I will show you how to create a text file that can then add a comment with the current date and time of the backup to the archive. Here is the sample batch script:
rar a -rr5 -s h:\backupC.rar @backupC.lst
rar c -z_date_.txt h:\backupC.rar
del _date_.txt
In the above example, I echo the current date and time into a text file (_date_.txt), I then create the archive, as shown above, and append the comment "created [date] [time]' to the archive, reflecting the current date and time. The _date_.txt is then deleted. The above text, when put in a text file and saved with the extension .bat (filename.bat), will create a script that:
1) Creates a text file named _date_.txt containing the date and time
2) Creates an archive named backupC.rar in the root of the h:\ drive containing all the files listed in backupC.lst
3) Comment the archive that was just created (h:\backupC.rar) with a comment reflecting the current time and date
4) Delete the file _date_.txt
This script will be executed whenever the file is double clicked or run. Later we will talk about using MS Scheduler or the DOS 'AT' command to automate these kinds of scripts, but first, let's talk more about DOSRAR and compression.
Archiving Compressed File Types: An Easy Solution
By now if you have been using RAR or any other file compression for a while, you may have noticed that compressed file types -just don't compress! Not only that, but they chug away for what seems an eternity while your compressor completes it's futile attempt. Files are compressed by finding and removing repetitious code inside of the file, if all the repetitious code has already been removed, the file cannot be compressed anymore. To the right you can see some examples. The 17MB MP3 took 54 seconds to compress on an Atlas 10K HD with a 2.2Ghz Athlon processor (that is a long time). Some files, like the Windows Media clip are so well compressed already, that compressing them adds a header, and only increases their size. The Hell Boy DivX was barely compressed at all. If you are rendering to a RAW or uncompressed file type, then RAR is great, but many people aren't. I for one used to render to PNG a lot, and now even with EXR you can render HDR files with great lossless compression. RLE, ZIPS, and ZIP are a few lossless compression algorithms that are available for things like EXR, TGA, etc...
The flag -m0 is useful when creating a RAR archive, this flag denotes the compression method of 'Store', where it does not compress the files at all. You may think this odd, but in this way, you backup all your compressed files very quickly -in about the time it takes to copy them to the new location, and it allows you to add parity to the file, or create a parity set (recovery volume). The other great thing is that you can play back any files inside the archive with little or no processor usage.
WinRAR can creat hybrid archives that have both compressed files and 'Stored' files! This is very important. It will compress files that compress well (MB, OBJ, LWO, DXF, BMP, TGA, EXR, AVI, PSD, TXT, DOC) etc.. and can be set to 'Store' file types that don't (MP3, PNG, EXR, AVI, ZIP, RAR, WMA, WMV), in the same archive! And before I get a load of mail, things like TGA,EXR,AVI etc, are uncompressed by default, but a lot of people compress them, which is why they are on both lists. To create these hybrid archives, you must invoke the flag -ms[list], here is an example:
rar a -msjpg;mp3;exr;avi;zip;rar;wma;wmv h:\backupC.rar @backupC.lst
This will do the same as before in terms of backing up the contents of backupC.lst to h:\backupC.rar, but it will not compress those listed file types!
Backing up With Parity and Splitting Archives
I have mentioned parity throughout this tutorial, we have talked about RAID arrays that have parity, creating RAR archives with parity, (although RAID parity and checksum parity are very different) and now I will discuss parity itself in more detail. Parity data is data that is used to reconstruct files that are corrupt or missing. There is another use for parity, and those are Parity Files. You can use a program like FSRaid to generate parity files of any file set, much like the Recovery Record that WinRAR uses. [FSRaid FAQ] These Parity Files are referred to as 'PAR files'. PAR files need multiple files to generate data from and the best thing to do is compress files into multiple volumes, and then create a parity set of said volumes. WinRAR now offers this feature with the -rv[N] flag, but you would first have to generate a RAR that was split up into many volumes with this flag: -v<size>[k|b|f|m|M]. This is actually how most advanced file swappers share files on the net, especially on usenet and places where you frequently lose parts of files due to corrupt data. Here is a DOSRAR example:
This will create a set of solid volumes 1440 bytes in length, and 4 'Recovery Volumes', the interesting thing about Parity files, is that these 4 recovery volumed can replace any 4 of the RAR volumes that may become corrupt.
Now there are some other very good reasons for splitting RAR archives. I almost always split large archives. Think about it: if you are backing up a hard drive, or a large data set, it's a lot more managable in smaller parts, not to mention, if you later need to back up this data to media like DVDR. The ISO standard itself disallows any files larger than 2GB, so if that is a concern, you should split to 2GB volumes. I usually split to 4GB, as I burn UDF DVDRs, but you get the idea. The way parity works with programs like FSRaid, you will generate a par set for the data set you are backing up, here is an example: You backup your data into 25 2GB RAR volumes. You then generate 3 2GB PAR files with FSRaid. Those 3 files can completely replace any 3of the 25 volumes that may become corrupt. A WinRAR Recovery Volume works about the same way.
Syncing and Updating Backup Archives
Now is an especially good time to talk about syncing and updating RAR archives. If you have used the Store option especially to backup a large amount of compressed data, or if you have backed up an entire hard disk, it could take hours to recompress everything, this is where the Update command or -u flag can help a lot. Here is an example:
rar a -u h:\backupC.rar @backupC.lst
The above adds files not already in the archive and updates files changed since they were packed to the archive. This saves an incredible amount of time when dealing with large volumes. Another option is 'Freshen' with the -f flag, which only updates current files and does not copy over newly created ones.
Automating Batch Files
The easiest way to automate a task under WindowsXP is probably the 'Task Scheduler', which you can find in the Start menu under Settings>Control Panel>System>Scheduled Tasks>Add A Scheduled Task. You can also add tasks manually at the command prompt with the AT command, again, you can pull up it's man page or help file by typing 'AT /?' at the prompt. This allows you to add tasks to the scheduler from batch files as well. Here is an example:
This will run the batch file backupC.bat, located in the folder h:\backup, every morning at 5am.
Conclusion
I have taken you through creating an inexpensive system for backing up your files on one hard drive that can easily be extrapolated to back up all your data. The system uses the best compression program/algorithm on the market, and allows you to use parity to safeguard that backup against data corruption, you can even employ a software RAID parity set (FSRaid) which can be stored on yet another drive. We talked about compressing already compressed data -finally, because no one talks about that, and you understand how to schedule DOS Batch Files to run automatically. If you lose any data from here on out, ignorance is definitely not an excuse. If I messed something up, or you have any questions, send me an email.
