Zope Pointers: Database Packing

by Mark Nenadov (Written September 24, 2005. Last revised October 29, 2007)

Note: If you already have some experience with Zope but are looking to learn more, you should consider purchasing Zope Bible by Wiley & Sons.

Introduction

Before starting, I want to make it clear that this is aimed to help you make intelligent decisions about packing. It isn't intended to be a comprehensive tutorial.

In Zope lingo, "packing" is your Zope Data.fs file (which is a ZODB object database) and pruning old revisions of objects. Packing can be done automatically or via the Zope management interface on an irregular basis. Auto-packing is good in that it chugs along whether or not you forget to do it. Auto-packing is bad in that it can't possibly understand when packing may be against your best interests (ie. when it eliminates an object revision that you wish to undo).

The Need for Packing and the Dangers of Packing

When you pack your Data.fs file, you supply Zope with a number of days (0 or a positive integer) and it deletes all OUTDATED object revisions which are older than that. Notice how I emphasized OUTDATED. If you supply 10 days, it doesn't delete EVERY object older than 10 days, but instead all objects that are older than 10 days AND have a newer duplicate copy (this is significant because Zope just keeps on appending new revisions to the end of the file).

Warning: When you pack a database, some older revisions of objects will be lost. This is normally not a problem (since the newest copies are kept), but in the event that you want to be able to undo or see an old revision, you will notice that it is not there after the pack!

Deciding How To Pack

How should one handle packing? It really depends on what sort of Zope instance you are are working with. Here are some potentially helpful generalizations:

• If your instance is under heavy development, consider packing less frequently.
• If disk space is an issue, consider packing more frequently (and deleting the .old file left in the Zope "var" folder, once you've confirmed everything is OK).
• If performance is an issue, consider packing more frequently.
• If data is frequently deleted or modified, consider packing more frequently.
• If the data modification or deletion processes are error prone, consider packing less frequently.

You will need to weigh these variables in order to determine how often packing should occur. Beyond just setting how often you pack, you should also consider the number of days or "threshold" to set for revisions. If losing revisions would be very problematic, consider setting it to over 30 days, but in some cases 5 days should do the trick.

In the event that ALL of the following is true about your environment, consider not packing at all:

• Your instance is actively developed by two or more programmers
• Data is frequently modified and deleted by non-programmers.
• Server disk space is not an issue
• Zope continues to perform efficiently on your system.
• You find that the "undo" functionality (or even the revision history) are used frequently

Those presence of those 5 factors would indicate that you have more to lose from packing than you could gain. But, for most instances, there is great advantage in packing.

Be Gentle

The best advise I can give is to be careful in implementing a new packing strategy--proceed thoughtfully. Here are some tips on how to make your implementation of packing to be more "gentle":

• When you execute a pack, a .old file will appear in your Zope "var" directory. This is a backup of the state of your database before the pack. Keep it around until you are confident of the integrity of your Data.fs. There is a chance that packing could cause some corruption. So, be careful and don't delete that file prematurely.
• Back Data.fs up frequently. That way if pack eliminates a revision you later need, you can always fish it out from the backup!
• Start with a large lapse of time between packs and then slowly decrease it as you become more confident with it.
• Start with a higher number of days that you supply to Zope for the pack ("removing previous revisions of objects that are older than X days") and then slowly decrease it as you get confident that you are not losing revisions that you need.