Retention Rules vs. Maximum # of Online Generations

May 2024 · 8 minute read
Retention Rules vs. Maximum # of Online Generations

Retention Rules vs. Maximum # of Online Generations

Creation Date: October 09, 2007

Revision Date: April 07, 2010

Product: DS‑Client

This information is only provided as a sample of different strategies. Each customer is ultimately responsible for the storage and deletion policies for their data.

This article outlines the differences between setting up maximum generations on a backup set and using retention rules. It also describes how these two functions work together.

The main point is that the customer must choose what mechanism will be in charge of deleting generations from online storage:

Maximum Generations are passive (see “Maximum Generations”) - they allow generations to be overwritten whenever necessary (on the DS-System online storage).Retention Rules are active (see “Retention Rules”) - they must be applied to a backup set, then enforced (either on schedule or on-demand) to physically delete generations from the DS-System online storage or move generations from DS-System online storage to BLM Archiver.

The default setting from DS-Client installation is to use maximum generations, meaning the oldest version of a file will be overwritten if you have backed up the maximum allowed number of generations for a file.

Generations are created on the online storage when the backup detects that a file has changed. Files that never change have only one generation on the DS-System online storage.

See Also

Maximum Generations

The maximum generations parameter is set up in the backup set properties, and it must be a number between 1-9999. This means that this backup set will save up to this maximum number of restorable generations on DS-System online storage. It will not keep more than this number on the online storage (when it reaches the limit, the oldest generation is overwritten). Overwriting is done during the backup process and does not take into consideration any retention rules that might apply, since retention rules run separately from the backup.

Best uses for maximum generations are:

for CDP (Continuous Data Protection) backup sets, where files are constantly changing and immediately being backed up, and the client would like to control the size of the online storage.

when there are no retention rules applied, and online storage size is a concern.

All cumulative backups (on-demand and scheduled) will keep up to the maximum generation number. In case a low number of maximum generations is set up for a backup intended to run on schedule, be aware that if you run an on-demand backup, what is kept online may not be what you expect.

For example, you create a backup set with maximum generations set to 7, and schedule it to run daily (to keep the daily changes for a week). Assuming all backup files change daily, you would expect one generation for each day. If an on-demand backup is run on this set, it will disrupt the pattern, and you will have 2 generations from the same day (and restorable generations going back 6 days, instead of 7). [This assumes that the file was changed before the on-demand backup, and before the scheduled backup.]

Retention Rules

Retention rules apply to the online generations every time they are enforced on a backup set. This can happen in two ways:

On-demand, separately from the backup process.

Within a schedule, independently or right after a backup task for this backup set.

When using retention rules, the “enforce retention” process works with what is saved in the online storage by the backup process. The following considerations should be made when applying Retention Rules to a backup set:

Retention Rule compatibility with Maximum number of generations

Ensure the Retention Rules work with the maximum number of generations for the backup set.

If the maximum generations is set to a low number (e.g. 7), then the retention rules may not produce the intended result. (The oldest generations are likely to be overwritten by the backup process, before any of the generations qualify for deletion by the retention rules.)

Retention Rule compatibility with its own rules (Delete options, Time-based online Retention, and Moving old data to BLM Archiver)

Retention Rules work best if the maximum generations parameter is set to the maximum (9999) or to a high number (e.g. 300). The retention rules help by deleting what you define to be unnecessary, and to ‘retain’ online what you define to be critical. As a result, you will most likely never reach a high number of generations for the same file on the DS-System online storage.

NOTE:  Results depend on the Retention Rules and how often they are enforced.

Delete options

The following refers to the option Deletion of files removed from source / HSMed data:

a)Once this retention rule is enforced on a backup set, it will check the options configured for the files that are removed from source, or for data that has been moved by a third-party HSM/ILM solution to a remote location.

a)DS-Client can be instructed to keep [x] number of generations for each file in the backup set after a certain period of time, since the file has been removed from the source machine.

NOTE:  DS-Client detects that files have been removed from source when the backup activity is running.

If a placeholder / stub is detected on the source machine, (Windows) DS-Client can be instructed to delete all file generations backed up before the stub / placeholder was detected, or to delete all generations that are not stubs / placeholders and were backed up in an earlier backup session than the stub.

By default, enforcing any Retention Rule will cause a deletion of all data that is defined as obsolete on from the DS-System online storage. However, a copy of this data can be sent to BLM Archiver before it is removed completely from the DS-System online storage.

Time-based online retention

The following refers to the deletion of files from the DS-System online storage based on user-specified time-based rules.

Keep most recent generations has a default value of 1. All generations of a file will be removed from the DS-System online storage, with the exception of the latest (most recent) generation. If this number is increased to a very high amount (e.g. 9999), all the other time-based rules specified in this Retention Rule configuration will not have any effect or the required effect (because they will be over-ruled by this particular rule).

Keep all generations for [x] interval of time for [y] interval of time. Time-based rules should be as specific as possible. When a retention rule has more details, it will work better and be much more precise than a general rule. For more details, see “Retention Rule Examples”.

Archive old data to BLM Archiver

Specify the age of the files at which DS-Client will move that data to BLM Archiver. The algorithm will compare the last modification date with the backup session date, and will apply the oldest of the two for the archiving rules.

For example, if all files older than one (1) month should be sent to BLM, DS-Client will compare the “last modification time” and “backup time” of a file; it will take the oldest value to calculate the age of the file and will execute the corresponding action (archive / do not archive).

Local Storage Retention

This only applies to the files saved in the DS-Client Local Storage Buffer(s).

To have the DS-Client save multiple generations on the local storage, the backup set must be assigned to a retention rule that has the Local Storage Retention options enabled.

Local Storage can only keep data on the local storage buffer if that same data (exact generation) exists on the DS-System online storage as well.

Summary

Customers have a choice between 2 methods of deleting online generations.

Passive

Maximum generations: overwrites oldest generation whenever necessary (no user intervention required). If set to a low number, this will make the backup set's retention rule inefficient. It is best used with a CDP backup, because it is enforced every time a backup runs (once maximum generations have been reached).

Active

Retention rules: to be used correctly, the following best practices apply:

Maximum generations of the backup set should be set to the maximum, or another high number (several hundreds/thousands).

Keep most recent generations should be set to a low number, such as 1, if time-based rules are also specified in the same Retention Rule.

Time-based rules should be as specific as possible (e.g. ‘Keep one generation every week on or before Sunday at 11:59PM’ for 1 Month) instead of being generic (like ‘Keep one generation every week for 1 Month).

Manual Delete: (Sets > Delete > Selective Delete) can be performed on-demand at any time to free up space on the DS-System online storage.

NOTE:  Because of the many configurations that are possible, it must be emphasized that the customer is responsible for configuring the storage and retention policies. Treat retention as if you are performing a manual delete of data: once data is deleted from online, you can never get it back.

Every time you delete data, you have the option to send a copy of the generations that are about to be deleted to BLM. When BLM is selected for a backup set, you can effectively save a copy of every backup generations forever.


ncG1vNJzZmirY2Ourq3ZqKWar6NjsLC5jpqqop%2Bilnqlu8KupJ6mpJbBqrvNaH%2BepKBkkZR5oqWgnqakWn9xlMSlp2h8g2KQrbXEp6uYgJWhvXCexK2cp6yZpLugntSlnKyXpqh7oJnAsaCmrZ2UrKC7xZiGp6SZo7Kgk8SnnKuZpJ57qcDMpQ%3D%3D