Module help

source code

PYBAG - Crossplatform files synchronization and backup portable tools.

About.

This program implements portable bag and is intended for fast synchronization and backup.

It let's you use a portable digital storage devace to carry your electronic documents similar to the way you can use a bag to carry paper documents.

You can synchronize the bag with your original files easily. If a synchronization conflict occurs, it will be reported. You can specify rules for automatic conflict resolution.

With PYBAG, you can backup files and synchronize any changes made to the original files with the bag. The synchronization process will only copy changed files.

The program is cross-platform and independent from the OS and filesystem. You may easily synchronize files between Windows and Linux, for example. Symbolic links are supported on all systems (if the OS or filesystem does not support symlinks, then they are emulated).

This program has a GUI and a command line interface.

Project page: http://pybag.sourceforge.net

Bugs trucker: http://sourceforge.net/tracker/?group_id=258722&atid=1129796

Requiraments.

Python 2.5.1 ... 2.7 ....

For GUI also required:

wxPython 2.8.10 +

General.

PYBAG copy all you files and folders into "./pybagfiles" folder. Information about all entities in "pybagfiles" folder written into DB "./pybag.db" - ziped text file. In DB saves files in bag size, modify date, crc32 for all files in bag. In DB files placed in next format:

   DB_VERSION_MARKER DB_VERSION
   F SIZE MDATE CRC32 path_to_file1_in_pybagfiles link_point1
   F SIZE MDATE CRC32 path_to_file2_in_pybagfiles link_point2
   ... and so on.

All lines must end with newline, where:

• "F" - flag: d-directory, f-file, s-symlink, ''-is empty entry.
• "SIZE" - file size in decimal or "-" for directories.
• "MDATE" - modification date in YYYY/MM/DD-HH:mm:ss.
• "CRC32" - crc32 in hex if exist or "-".
• "path_to_file" - bag path to item into pybagfiles.
• "link_point" - if item link then it'spoint, otherwise '-'.

Also have "pybag.cfg" file where saved preferences. Format is:

• [root] global preference:
  • ignore = global ignores default = ''-process all files. Separated commas ','. Pattern as fnmatch.
  • symlink = copy | ignore | copynoemul| ignorenoemul. default = copy.
  • compare = stat | crc32. Compare method, "stat" - by size and modify date, "crc32" - by crc32, default = stat.
  • backup = 0 | 1. Default = 0
  • mddelta = Time differense in seconds for assign files equal then compare modify date (float). Default is 1.0 sec.
  • nextid = next free sequence id (integer).
  • r### = F path where "F" - flag (f=file, d=directory, s=symlink), "path" - path to root ###, where ### is root id. Last part of path is a start path in pybagfiles folder.
  • emulation = 0 | 1.
  • verbosity - verbosity level (-1 .. 5).
  • debug = 0 | 1.
  • logging = 0 | 1.
  • colorize = 0 | 1.
  • outfilter - outfilter for sync command.
• [r###_pref] - local preference if need default none:
  • ignore = ignore list. Separated commas ','. Pattern as fnmatch.
  • symlink = as above.
  • compare = as above.
  • backup = as above.
  • selectdefault = 0 | 1.
• [GUI] - GUI preference:
  • writecolsatexit = Write column position when exit from program ( only if emuletion turn off). 0 | 1. Default is 0.
  • output - Show output window. 0 | 1. Default is 1.
  • showtips - Show tips window at start up. 0 | 1. Default is 1.
  • column# - Columns preferences. "#" is a column order. Format is: column0 = name, visible, width. Where cfgname - is a column name, visible - column visibility 0 or 1, width - column width in pixels ( from 0 to 2000). Allowed column names is next:
    • stateicon - Item state icon.
    • actionicon - Item action icon.
    • err - Item errors mark (e - error, c - conflict, w - warning).
    • forced - Forced state (! - item action is forced).
    • dir - Action direction:
      • <<< - New in origin, copy to bag.
      • >>> - New in bag, copy o origin.
      • <-- - Changed in origin, copy to bag.
      • --> - Changed in bag, copy to origin.
      • xxx - Delete item.
      • --- - Ignored item.
      • === - Unchanged item.
      • <== - Unchanged item, time shift in bag.
      • ==> - Unchanged item, time shift in origin.
      • *** - Error in filter and programm. Note author if this situation appears.
    • path - Item path in bag.
    • desc - Description.
    • type - Item type:
      • f - file,
      • d - directory,
      • s - symlink,
      • empty - unknown (or may be ignored).
    • mdatedb - Item date in DB, float.
    • gmdatedb - Item date in DB, GMT.
    • lmdatedb - Item date in DB, Local.
    • crc32db - Item CRC32 sum in DB.
    • sizedb - Item size in DB, bytes.
    • symlinkdb - Symlink in DB.
    • mdatebag - Item date in bag, float.
    • gmdatebag - Item date in bag, GMT.
    • lmdatebag - Item date in bag, Local.
    • crc32bag - Item CRC32 sum in bag.
    • sizebag - Item size in bag, bytes.
    • symlinkbag - Symlink in bag.
    • mdateorigin - Item date in origin, float.
    • gmdateorigin - Item date in origin, GMT.
    • lmdateorigin - Item date in origin, Local.
    • crc32origin - Item CRC32 sum in origin.
    • sizeorigin - Item size in origin, bytes.
    • symlinkorigin - Symlink in origin.
    • timeshift - Time shift:
      • b - Item in bag is time shifted,
      • o - Item in origin is timeshifted,
      • empty - Item not time shifted.
  • filter_#### = 0 or 1. Filter with name #### allowed or disabled. Filters may be next:
    • filter_ok - synchronization OK,
    • filter_bag - Item action to bag,
    • filter_orig - Item action to origin,
    • filter_del - Delete item,
    • filter_ch - Item was changed,
    • filter_new - Item is new,
    • filter_err - Error in item,
    • filterconf - Conflict in item,
    • filter_warn - Warning in item,
    • filter_forced - Item action is forced,
    • filter_unch - Item is unchanged,
    • filter_file - Item type is file,
    • filter_dir - Item type is directory,
    • filter_sym - Item type is symlink,
    • filter_unk - Item type is unknown,
    • filter_ign - Item is ignored.
    • filter_ts - Item is timeshifted.

Directory structure.

• doc - Documentation directory.
  • api - API documentation.
  • manual - User manual.
• pybagfiles - Directory where files saved in bag.
• pybag.db - File list database, ziped text.
• pybag.cfg - Configuration file.
• pybag.log - Log file.
• ...~ - File ends with tilda "~" is backup.

Timeshift.

In several conditions (for example when synchronize Windows and Flash-FAT) you may have problems with daylight saving. Details - you synchronize all you files in evening and in night was changet time for daylight saving. In the morning you found that all you files was changed (modification date between bag and origin different about 3600 seconds)!!! This is bad do copy all files. You can use "timeshift" operation instead real copying. Timeshift is only change modification (and access) date. "Timeshift to bag" mean that for all marked files in bag will be set modification date same as in origin. "Timeshift to origin" mean that for all marked files in origin will be set modification date same as in bag.

In CLI interface you can use "-y" option for set timeshift manualy or use action "tsbag", "tsorigin", "tsreset" for set (reset) timeshift by template. In GUI you can use menu "Action->Timeshift-> ... " for set timeshift by template. Set timeshift by template means that you specify file which needs in timeshift and it become template - calculated time difference from this file used for all other files.

Command line.

pybag.py --command -option1 -option2 argument1 argument2 ...

Warning: At once may be executed only one command.

Recomendation: Use emulation mode -m1 befor real files operation (exclude --dist). For real file operation you need use option -m0.

Commands.

• --gui

  Open GUI. Without commands also run GUI.

• --add {-k -s -e -b -x -rDIR} path1 {path2 ...}

  Add new roots to bag. If option -k -e -s -b is not specified then will be used it's default values (symlink ignore, all files, stat comparison, no backup). In next step you must synchronize pybag with sync. Options in this command set properties for added roots and for next synchronizations. In the future you can manualy edit pybag.cfg for change root properties or use GUI interface.

  Options:

  • -kFLAG

    Symlink action:

    • c or copy - copy symlink, emulate symlink.
    • i or ignore - ignore symlink, emulate symlink. It is default.
    • cn or copynoemul - copy symlink, do not emulate symlink.
    • in or ignorenoemul - ignore symlink, do not emulate symlink.
  • -sFLAG

    Set compare method for files comparison. FLAG is comparison method:

    • stat or no flag or empty - Use only stat information (size, modify date) - default. For directory modify date do not used.
    • crc32 - Use CRC32 check then syncronize.
  • -eFILEMASK

    Exclude from process all files and folders liked FILEMASK:

    • Wildcards (as fnmatch) separated commas: "*.bak,*.pyc".
    • RegExp if FILEMASK starts with "!": "!(.*\.bak$)|(.*\.pyc$)".

    You may get few optins "-e". Default all files processed.

  • -bFLAG

    Backup mode for this root:

    • 0 - No backup (default).
    • 1 - This is will be backup copy of origin. In synchronization always used direction "bagall". Backup copy synchronize in up 2 times faster then standart.
  • -xDEF

    Mark roots as default:

    • 1 - Mark as default.
    • 0 - Do not mark.

    If not specified then mark all roots as default.

  • -rDIR

    Add all items from directory DIR as roots. You may get few optins "-r".

• --remove root1 {root2 ...}

  Remove roots from bag.

• --dist {-z} {-i} {-x} {-a} {-rROOT1 ...} {path1}

  Distribute PYBAG program to specified location. This do clear distribute without any files in current bag. If options -r or -a is given then in destination path also adjust pybag.cfg to point original to this (current) point. If you point "path1" into other pybag then adding all to it. If you initialize -i then you may not specify "path1". With other options "path1" is required.

  Options:

  • -z

    Copy all additional files (manual, license and etc.). Create folder "pybagfiles" if need. Not copied roots.

  • -i

    Initialize - Create "pybagfiles" directory, configuration file and DB file. If "path1" not specified then initialize current location, overwise into specified location. Create only absent items, do not change exists. If "path1" not exists then try create this directory.

  • -a

    Copy all roots (if given this option, then options "-r" mark roots for miss).

  • -xDEF

    Select roots for distribute:

    • all - All roots.
    • default - Default roots.

    If not specified then select only default roots. This options ignored when specified "-a" or "-r" option.

  • -rROOT

    Distribute only specified roots (if option -a given, then mark not copied roots).

• --sync { -fFILE } {-xDEF} {-dDIR} {-dDIR} {-sMDDELTA} { -pA } { -bFLAG } { -gOUTFILTER } { -yFLAGVALUE } {path1}

  Synchronize current bag with original files. If "path1" specified then syncronize with specified location (path1). If no "f" option given then try sync all pybag roots in path "path1/root". if specified "f" then "path1" must correspond path in option. After comparing you will be prompt for continue synchronization and need answer y for continue or n for cancel.

  Options:

  • -fFILE

    Syncronize or copy only specified file "FILE", where FILE is a relative path to "/pybagfiles".

    For example: you have file "pybag/pybagfiles/project/bin/file1.py", for synchronize you must write pybag --sync -f"project/bin/file1.py".

  • -xDEF

    Select roots for synchronization:

    • all - All roots.
    • default - Default roots.

    If not specified then select only default roots. This options ignored when specified "-f" option.

  • -dDIR

    Direction for synchronization copy if conflicts occurs:

    • bag - write changed files into bag.
    • origin - write changed files into copy or original location.
    • newer - newer files overwrite older.
    • older - older files overwrite newer.
    • no - do not change files. Is default.
    • bagall - write all changes from origin to bag.
    • originall - write all changes from bag to origin.
    • bagcopy - copy all changes to bag, but do not delete from bag anything.
    • origincopy - copy all changes to origin, but do not delete from origin anything.

    newer and older may be given with bag and origin simulteniously (for example -dbag -dnewer ). First will be checked newer and older, after - used bag and origin.

    ATTENTION: Do not mix other direction flags, result may be umbigous.

  • -sMDDELTA

    Set modify date delta in seconds (float number). If dates different less then MDDELTA then they assign equal. Overrade configuration file settings.

  • -bFLAG

    Backup option:

    • 0 - Standart backup, is default, read only origin. This mode is fast - read only origin files.
    • 1 - Synchronization mode, read origin and bag and synchronize.
  • -pA

    Autoanswer for question ( "Continue synchronization?"):

    • y - yes, continue.
    • n - no, cancel.
  • If no autoanswer specified then you can use interactive menu:
    • d DIR DIR - change rules for autoconflict resolution (after change direction comparison will be repeated, and menu displayed again).
    • m FLAG - change emulation level.
    • t FLAG - change debug level.
    • v FLAG - change verbosity level.
    • y - continue syncronization.
    • n - cancel syncronization.
    • g OUTFILTER - change output filter.
    • r - repeat report.
    • a #ACTION FILTER - manual force action ACTION with paths FILTER. Action may be next:
      • delete - delete item,
      • bag - copy to bag or ignore if origin deleted,
      • origin - copy to origin or ignore if bag deleted,
      • skip - skip action (mark unchanged),
      • restore - restore automatic action.
      • tsbag - timeshift to bag, FILTER used as template.
      • tsorigin - timeshift to origin, FILTER used as template.
      • tsreset - reset timeshift.

      "#" mean "+" or "-" or nothing. "+" - is interactive action, ask for each path user confirmation. "-" - do not list paths and not need in confirmation. Nothing - is list paths but not need in confirmation. After using ts.... action items will be recompared and all actions will be reset. Use first ts... action and after other actions.

  • -gOUTFILTER

    Output filter - difine which operation will be printed. Default is -gsbonhdftwce1234 - display in report all operation with new, copied, deleted, forced and not forced files, errors, conflicts and warnings for all direction in short format. Filter options:

    • x - Do not show report.
    • l - Long format of report: '%%(forced)s %%(dir)s %%(err)s [%%(path)s]\n\t%%(desc)s'.
    • s - Short format of report: '%%(forced)s %%(dir)s %%(err)s [%%(path)s]'.
    • p - Old (as in previous version) format: '\t[%(path)s]-->\n\t == "%%(desc)s".'.
    • m..FORMAT.. - Manual specified format as print function. ".." - is separator (may be any two symbols). Available keywords is:
      • err - "w", "c", "e" or " " - warning, conflict, error or nothing.
      • forced - "!" or " " - Operation forced or not.
      • path - Item path (in OS format).
      • desc - Description.
      • type - "f", "d", "s" ,"u" - Item type, file, directory, symlink or unknown.
      • mdatedb - Modification date in DB, seconds.
      • gmdatedb - GMT modification date in DB, "YYYY-DD-MM HH-MM-SS".
      • lmdatedb - Local modification date in DB, "YYYY-DD-MM HH-MM-SS".
      • crc32db - CRC32 in DB, "0xFFFFFFFF".
      • sizedb - Size in DB.
      • symlinkdb - Iflink point DB, "-" if not symlink.
      • mdatebag - Modification date in bag, seconds.
      • gmdatebag - GMT modification date in bag, "YYYY-DD-MM HH-MM-SS".
      • lmdatebag - Local modification date in bag, "YYYY-DD-MM HH-MM-SS".
      • crc32bag - CRC32 in bag, "0xFFFFFFFF".
      • sizebag - Size in bag.
      • symlinkbag - Iflink point bag, "-" if not symlink.
      • mdateorigin - Modification date in origin, seconds.
      • gmdateorigin - GMT modification date in origin, "YYYY-DD-MM HH-MM-SS".
      • lmdateorigin - Local modification date in origin, "YYYY-DD-MM HH-MM-SS".
      • crc32origin - CRC32 in origin, "0xFFFFFFFF".
      • sizeorigin - Size in origin.
      • symlinkorigin - If link point origin, "-" if not symlink.
      • dir - Direction, displayed by next symbols:
        • <<< - New in origin, copy to bag.
        • >>> - New in bag, copy o origin.
        • <-- - Changed in origin, copy to bag.
        • --> - Changed in bag, copy to origin.
        • xxx - Delete item.
        • --- - Ignored item.
        • === - Unchanged item.
        • <== - Unchanged item, time shift in bag.
        • ==> - Unchanged item, time shift in origin.
        • *** - Error in filter and programm. Note author if this situation appears.
      • timeshift - Time shift:
        • b - Item in bag is time shifted,
        • o - Item in origin is time shifted,
        • empty - Item not shifted.
    • z00 - List report by n items per page.
    • b - Operation with direction to bag.
    • o - Operation with direction to origin.
    • n - Operation with new files.
    • h - Operation with changed (copied) files.
    • d - Operation with deleted files.
    • y - Timeshift operations.
    • i - Ignored files.
    • u - Unchanged files.
    • f - Forced action.
    • w - Warnings.
    • c - Conflicts.
    • e - Errors.
    • 1 - Files.
    • 2 - Directories.
    • 3 - Symlinks.
    • 4 - Unknown (normaly this is must not be).
    • r..PATT.. - Path wilds or regexp filter. If not specified then all displayed.
  • -yFLAGVALUE

    Timeshift option - set predifined time shift. FLAG is next:

    • b - Time shift in bag.
    • o - Time shift in origin.

    VALUE may be next:

    • #FILTER, where FILTER is "wilds" or "!regexp" filter to find item which will be used as template for time shift. Example: -yb#* - time shift to bag and use first found file as template.
    • BAGSHIFT#ORIGINSHIFT - Set up manualy time shifts for bag and origin. Example: -yb-3600#0 - time shift to bag and set bag shift to "-3600" and origin to "0.0". BAGSHIFT and ORIGINSHIFT used for get modification date for files. For files in bag modification date calculated as (bagfile.mdate + BAGSHIFT), for files in origin modification date calculated as (originfile.mdate + ORIGINSHIFT). If no timeshift set then modification date equal file.mdate.
• --relocate -rROOT=NEWPATH { -rROOT=NEWPATH ...}

  Realocate root. Replace origin path for ROOT by new NEWPATH.

• --stat {-xDEF} {-rROOT1 ...}

  Show statistic about bag files for all roots or only for specified with option -r.

  In statistic shown for each root and for all summary roots at end:

  • Files count,
  • Directory count,
  • Symlink count,
  • Summary items count,
  • Summary items size,
  • Database size (for roots summary).

  Options:

  • -rROOT

    Show statistic only about specified roots.

  • -xDEF

    Select roots:

    • all - All roots.
    • default - Default roots.

    If not specified then select all roots. This options ignored when specified "-r" option.

• --cleandb

  Clean DB from old records (old roots which already removed).

• --help, -h, --usage

  Display help information.

• --version

  Show version.

• --author

  Show author.

• --copyright

  Show copyright.

• --license

  Show license.

• --about

  About information.

• --debug {name ...}

  Set PYBAG_DEBUG and PYBAG_LOG to True and call cli._debug function. For debug purpouse only. Must be first in options list. If get name then call only this methods ("_test_name") if exists. Right tests will be only into source distribution. In copy files (for example distributed with --dist) number tests will be fail.

Common options:

This options may be used for all command.

• -vLEVEL

  Set verbosity level (0 - no verbosity) 0 ... 4. Default assign 3.

  • -1 - No output. Only log file.
  • 0 - Only global errors.
  • 1 - Very small output, only about success and errors.
  • 2 - As 1 and conflicts.
  • 3 - Normal output. As 2 and warnings.
  • 4 - Large output. As 3 and file operation.
  • 5 - Huge output. As 4 and logging.
• -lFLAG

  Enable 1 or disable 0 logging. FLAG may be 0 or 1. Default assign 1.

• -tFLAG

  Enable 1 or disable 0 debug output. FLAG may be 0 or 1. Default assign 0.

• -mFLAG

  Enable 1 or disable 0 emulation. FLAG may be 0 or 1. Do not realy perform write operation. Do all actions as real command but do not change any files. Default assign 0. Usually this option use with -l or -t options for logging emulation result.

• -cFLAG

  Enable 1 or disable 0 colorize output. FLAG may be 0 or 1. Default assign 1.

GUI.

See GUI description in manual.

Imports: __year__, __year_start__