Module pybag
[hide private]
[frames] | no frames]

Module pybag

source code

PYBAG - Crossplatform files synchronization and backup portable tools.

About.

This program implements portable bag and is intended for fast synchronization and backup.

It let's you use a portable digital storage devace to carry your electronic documents similar to the way you can use a bag to carry paper documents.

You can synchronize the bag with your original files easily. If a synchronization conflict occurs, it will be reported. You can specify rules for automatic conflict resolution.

With PYBAG, you can backup files and synchronize any changes made to the original files with the bag. The synchronization process will only copy changed files.

The program is cross-platform and independent from the OS and filesystem. You may easily synchronize files between Windows and Linux, for example. Symbolic links are supported on all systems (if the OS or filesystem does not support symlinks, then they are emulated).

This program has a GUI and a command line interface.

Project page: http://pybag.sourceforge.net

Bugs trucker: http://sourceforge.net/tracker/?group_id=258722&atid=1129796

Requiraments.

Python 2.5.1 ... 2.7 ....

For GUI also required:

wxPython 2.8.10 +

General.

PYBAG copy all you files and folders into "./pybagfiles" folder. Information about all entities in "pybagfiles" folder written into DB "./pybag.db" - ziped text file. In DB saves files in bag size, modify date, crc32 for all files in bag. In DB files placed in next format:

   DB_VERSION_MARKER DB_VERSION
   F SIZE MDATE CRC32 path_to_file1_in_pybagfiles link_point1
   F SIZE MDATE CRC32 path_to_file2_in_pybagfiles link_point2
   ... and so on.

All lines must end with newline, where:

Also have "pybag.cfg" file where saved preferences. Format is:

Directory structure.

Timeshift.

In several conditions (for example when synchronize Windows and Flash-FAT) you may have problems with daylight saving. Details - you synchronize all you files in evening and in night was changet time for daylight saving. In the morning you found that all you files was changed (modification date between bag and origin different about 3600 seconds)!!! This is bad do copy all files. You can use "timeshift" operation instead real copying. Timeshift is only change modification (and access) date. "Timeshift to bag" mean that for all marked files in bag will be set modification date same as in origin. "Timeshift to origin" mean that for all marked files in origin will be set modification date same as in bag.

In CLI interface you can use "-y" option for set timeshift manualy or use action "tsbag", "tsorigin", "tsreset" for set (reset) timeshift by template. In GUI you can use menu "Action->Timeshift-> ... " for set timeshift by template. Set timeshift by template means that you specify file which needs in timeshift and it become template - calculated time difference from this file used for all other files.

Command line.

pybag.py --command -option1 -option2 argument1 argument2 ...

Warning: At once may be executed only one command.

Recomendation: Use emulation mode -m1 befor real files operation (exclude --dist). For real file operation you need use option -m0.

Commands.

Common options:

This options may be used for all command.

GUI.

See GUI description in manual.


Version: 0.4.4

Author: Mazhugin Aleksey

Copyright: Copyright (c) 2017 Mazhugin Aleksey

License: Copyright (c) 2007 .. 2017, Mazhugin Aleksey All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Mazhugin Aleksey nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Classes [hide private]
  lock_list
Class target for easy multithreading access to variables (in list).
  command_class
  _gp_class
Global preferences saver class.
  _r
Class for result return from commands.
Functions [hide private]
 
init_dirs()
Initialize directories.
source code
 
_u(a, enc=None)
Convert a to unicode if a is not unicode.
source code
None
log(*arg)
Print all concatenated _u(arg) if PYBAG_LOG True.
source code
None
dbg(*arg)
Print all concatenated _u(arg) if PYBAG_DEBUG True.
source code
 
log_writeln(*arg)
Used only in first call, and replace themself by log_writeln_1 if log_init pass, or log_writeln_0 otherwise.
source code
None
log_writeln_1(*arg)
Main writeln function for logging.
source code
 
log_writeln_0(*arg)
Used if log_init was fail in log_writeln and do nothing.
source code
 
log_init()
Initialize file output stream.
source code
 
log_clear()
Do clean up for fstream: close if need.
source code
 
prt(*arg, **kwd)
Print all arg consecutively with print builtin function.
source code
 
prt0(*arg, **kwd)
Print arguments arg with prt with verbose = 0.
source code
 
prt1(*arg, **kwd)
Print arguments arg with prt with verbose = 1.
source code
 
prt2(*arg, **kwd)
Print arguments arg with prt with verbose = 2.
source code
 
prt3(*arg, **kwd)
Print arguments arg with prt with verbose = 3.
source code
 
prt4(*arg, **kwd)
Print arguments arg with prt with verbose = 4.
source code
 
prt5(*arg, **kwd)
Print arguments arg with prt with verbose = 5.
source code
 
initGUI() source code
 
cmd_gui(cmd)
Start GUI.
source code
 
command(cmd)
Start command interface.
source code
Class object or bool.
parse_command()
Start command parser.
source code
bool
test_init()
Test if programm location already initialized.
source code
bool
init_program()
Initialize program environment.
source code
bool
initpath(path)
Initialize bag in path.
source code
bool
initdb(path, db={})
Initialize database file in specified path.
source code
bool
initconfig(path)
Initialize configuration file in specified path.
source code
bool
initpybagfiles(path)
Initialize pybagfiles directory in specified path.
source code
 
deinit_program()
Deinitialize program structures and close opened files.
source code
str
transreport(opt=None)
Translate report option -g into global variables.
source code
dict
dbread(path=None)
Read database file pybag.db (gzipped).
source code
 
dbsave(dbdict, path=None, patht=None)
Save database to file.
source code
 
dbdumprow(dbrow, tab=u' ....->')
Dump DB row to string.
source code
 
dbdump(db, tab=u'DBDUMP:')
Dump DB to string.
source code
 
dbnormalize(db)
Normalize DB - convert values to right type and past default if absent.
source code
list
splitpaths(path)
Return list of path's parts.
source code
 
splitpathsdb(path)
See splitpaths.
source code
 
getpathdirdb(path)
For DB format paths return parent directory for given path.
source code
 
dbstartswith(dbpath, idx)
Test if dbpath starts with idx.
source code
 
path2db(path)
Convert path to DB format (separate by DBSEP).
source code
 
path2os(path)
Convert path to OS format from DB.
source code
 
walk(path, onerror=None)
Top-down directory walker.
source code
 
symlinktest(path, useemul=None)
Test file for symlink emulation or real.
source code
 
readlink(path, useemul=None)
Read symbolic link point.
source code
 
symlink(link, path, useemul=None)
Save symbolic link point.
source code
 
rmdirr(path)
Delete directory recursively.
source code
 
copytree(src, dst, symlinks=False, exclude=[])
Copy files tree from "src" to "dst".
source code
list or None
validatepaths(root, bagpath=None, syncpath=None)
Validate paths.
source code
 
getcrc32(path)
Calculate CRC32 for given file.
source code
tuple
dbwalkfiles(db, root, path, isbag, dbindex)
Walk throw bagpath files from pybagfiles and syncpath for original files.
source code
str
transopt(opt, cmd)
Translate option in command.
source code
 
cmd_(cmd)
No command, only option.
source code
 
cmd_unknown(cmd)
Unknown command.
source code
 
f_stat(cmd)
Show info about bag files.
source code
 
f_dist(cmd)
Distribute PYBAG program to specified location.
source code
 
cmd_dist(cmd)
Distribute PYBAG program to specified location.
source code
 
cmd_stat(cmd)
Show statistic about bag files.
source code
 
f_usage(cmd) source code
 
cmd_usage(cmd)
Print usage information.
source code
 
cmd_version(cmd)
Show version.
source code
 
cmd_author(cmd)
Show author.
source code
 
cmd_license(cmd)
Show license.
source code
 
cmd_copyright(cmd)
Show copyright.
source code
 
cmd_about(cmd)
Show about.
source code
 
f_help(cmd) source code
 
cmd_help(cmd)
Print usage information.
source code
 
cmd_add(cmd) source code
 
f_add(cmd) source code
 
cmd_remove(cmd)
Remove roots from bag.
source code
 
cmd_cleandb(cmd)
Remove not exists paths/roots from DB.
source code
 
cmd_relocate(cmd)
Relocate specified root (change its origin into configuration file).
source code
 
outreport(db=None, ks=None)
Output comparison report from database db in console.
source code
stirng.
do_action(action, filter, interactive=True, listpaths=True)
Perform changes into db[DBDIR] to specified action.
source code
 
cmd_sync(cmd)
Synchronize files.
source code
bool
f_syncread(cmd)
Read DB, files.
source code
 
f_synccompare(direction)
Compare DB.
source code
 
f_syncsync()
Syncronize DB.
source code
 
f_syncclear()
Clear DB.
source code
 
main()
Main finction.
source code
 
_debug()
Debug function.
source code
 
_test_debugTrue()
Only for test
source code
 
__test_debugFalse()
Only for test
source code
 
_test_gp_class() source code
 
_test_dbnormalize() source code
 
_test_splitpaths() source code
 
_test_dbread() source code
 
_test_findroot() source code
 
_test_validatepaths() source code
 
_test_dbwalkfiles() source code
 
_test_transopt() source code
 
_test_path2db() source code
 
_test_path2os() source code
 
_test_readlink() source code
 
_test_symlink() source code
 
_test_getcrc32() source code
 
_test_transreport() source code
 
init_encoding()
Get system enconings.
source code
 
mainstart() source code
 
loadimages() source code
Variables [hide private]
  wx = None
  __version__ = '0.4.4'
Version.
  __author__ = u'Mazhugin Aleksey'
Author.
  __copyright__ = u'Copyright (c) 2017 Mazhugin Aleksey'
Copyright.
  __license__ = ...
License.
  __about__ = ...
About.
  __file__ = os.getcwdu()
  _db_version = u'0.2'
Database version.
  _db_marker = u'PYBAG DB version = '
Database marker.
  _db_marker_len = 19
Database marker length.
  _pbls = u'pybaglink start:'
Symlink emulation start marker.
  _pble = u':pybaglink end'
Symlink emulation end marker.
  _pblmaxlen = 4096
Maximum symlink emulation file length.
  _app = None
Application.
  _gui = None
GUI top frame.
  PYBAG_DEBUG = False
Debug mode.
  PYBAG_DEBUG_DEFAULT = u'0'
Default value of PYBAG_DEBUG.
  PYBAG_LOG = True
Logging mode.
  PYBAG_LOG_DEFAULT = u'1'
Default value of PYBAG_LOG.
  PYBAG_EMUL = True
Emulation mode.
  PYBAG_EMUL_DEFAULT = u'1'
Default value of PYBAG_EMUL.
  TSDIRNONE = 0
No timeshift.
  TSDIRBAG = 1
Timeshift to bag.
  TSDIRORIG = 2
Timeshift to origin
  TSBAG = 0
Timeshift for bag.
  TSORIG = 0
Timeshift for origin.
  TSDIR = 0
Timeshift direction
  VERBOSE = 3
Verbosity level.
  VERBOSE_DEFAULT = u'3'
Default value of VERBOSE.
  CRC32BUF = 1048575
CRC32 read buffer (2^20 == 1'048'576).
  DBDUMPMAX = 500
DBDUMP count restrict.
  LOOPN = 10000
Loops in cicles between CANCEL testing.
  BACKUPMODE = False
Backup mode in synchronization, normal.
  TIMEFORMAT = u'%Y-%m-%d %H:%M:%S'
  IMGINDEX = {}
Image list index for GUI
  WRITECOLSATEXIT = False
If true - write gui columns at exit in config.
  SHOWTIPS = 0
If true then show tips at start up.
  TIPSNUMBER = 0
Next tips index for show.
  P_OUTPUT = u'output'
  P_SHOWTIPS = u'showtips'
  P_WRITECOLSATEXIT = u'writecolsatexit'
  P_TIPSNUMBER = u'tipsnumber'
  P_GUI = u'GUI'
  P_DEBUG = u'debug'
  P_EMUL = u'emulation'
  P_LOGGING = u'logging'
  P_VERB = u'verbosity'
  P_OUTFILTER = u'outfilter'
  AppState = {'ff_bag': True, 'ff_ch': True, 'ff_conf': True, 'f...
Application state for GUI UI update.
  _filter = {u'ff_bag': u'filter_bag', u'ff_ch': u'filter_change...
  _listks = []
Temporary keys when update list filter.
  IsDBReaded = False
Flag marked if DB readed.
  IsDBCompared = False
Flag marked if DB compared.
  IsDBSynced = False
Flag marked if DB synchronized.
  LI_CANCEL = 0
  LI_ERROR = 1
  LI_PERCENT = 2
Percents of done for current function.
  LI_PERCENT100 = 3
100% percents for current function.
  LI_DESC = 4
Action description.
  LI_PERCENTPART = 5
Percents of global of current function
  LI_PERCENTGLOBAL = 6
Global percents done
  LI_FUN2 = 7
Function which must run after thread exited.
  LI_UPDATEFILTER = 8
True if need update list by filter.
  LI_FILTERRUNNING = 9
True if filter running.
  LI_THREAD = 10
True if thread running.
  LI_LENGTH = 11
  _outlock = <thread.lock object>
Lock object for gui print.
  _outtxt = u''
Text printed from GUI.
  DBPATH = 0
Path as is in DB.
  IBAG = 100
Bag group index.
  ISYNC = 200
Original group index.
  IDB = 300
DB group index.
  IFLAG = 1
Flag subgroup.
  ISIZE = 2
Size subgroup.
  IMDATE = 3
Modify date subgroup.
  ICRC32 = 4
CRC32 subgroup.
  IREAD = 5
Read subgroup.
  IPATH = 6
Path subgroup.
  ILINK = 7
Link point subgroup.
  DBBAGFLAG = 101
  DBBAGSIZE = 102
  DBBAGMDATE = 103
  DBBAGCRC32 = 104
  DBBAGREAD = 105
  DBBAGPATH = 106
  DBBAGLINK = 107
  DBDBFLAG = 301
  DBDBSIZE = 302
  DBDBMDATE = 303
  DBDBCRC32 = 304
  DBDBREAD = 305
  DBDBLINK = 307
  DBDBPATH = 306
  DBORIGFLAG = 201
  DBORIGSIZE = 202
  DBORIGMDATE = 203
  DBORIGCRC32 = 204
  DBORIGREAD = 205
  DBORIGPATH = 206
  DBORIGLINK = 207
  DBDIR = 15
Direction of syncronization.
  DBSYNC = 17
Syncronization status.
  DBCOMPARE = 18
Compare property from config.
  DBSYMLINK = 19
Symlink property from config.
  DBDESC = 20
Description of comparison, syncronization.
  DBDIRMAN = 21
Manual overwrite (GUI).
  DBDESCMAN = 22
Manual description (GUI).
  DBITEMID = 23
ItemId into treeCtrl (GUI).
  DBDIRREADED = 24
As DBDIR, but after reading.
  DBDESCREADED = 25
As DBDESC, but after reading.
  DBSYMLINKEMUL = 26
Emulate symlink or not for current item.
  DBBACKUP = 27
Backuped item.
  DBBACKUPYES = True
Backuped item.
  DBBACKUPNO = False
Not backuped item.
  DBREADYES = True
Item was tried read.
  DBREADNONE = False
Item not tried read.
  DBSYMEMULNONE = False
  DBFLAGNONE = u''
No flag.
  DBFLAGDIR = u'd'
Directory.
  DBFLAGFILE = u'f'
Regular file.
  DBFLAGSYMLINK = u's'
Symlink.
  DBFLAGUNKNOWN = u'u'
Unknown item type.
  DBMDATENONE = 0.0
None modify date.
  DBNONE = 0
Empty/None item.
  DBPATHNONE = u''
Empty path.
  DBDESCNONE = u'Not compared.'
Empty description.
  DBDESCMANNONE = u''
Empty description for manual (GUI).
  DBDESCOK = u'Ok.'
Successfull description.
  DBLINKNONE = u'-'
Fill string for item in DB if it's not a link.
  DBITEMNONE = None
  DBDIRNONE = 0
Direction None.
  DBDIRBAG = 1
Direction left (to bag).
  DBDIRORIG = 2
Direction right (to orig).
  DBDIRDB = 4
Direction to DB.
  DBDIRCOPY = 8
Copy to.
  DBDIRDELETE = 16
Delete.
  DBDIRWARNING = 32
If Warning.
  DBDIRCONFLICT = 64
If conflict.
  DBDIRERROR = 128
If error occur.
  DBDIRNEW = 256
New item.
  DBDIRUNCHANGED = 512
Unchanged item.
  DBDIRFORCED = 1024
Indicate that file action is forced.
  DBDIRIGNORED = 2048
indicate ignored files.
  DBDIRDIFF = 4096
Different direction.
  DBDIROK = 8192
Ok flag.
  DBDIRRESTORE = 16384
Used for restore compared direction.
  DBDIRTS = 32768
Indicate timeshift.
  DBSYNCNONE = 0
Syncronization None.
  DBSYNCERRREADBAG = 1
Error then read from bag.
  DBSYNCERRREADSYNC = 2
Error then read from orig.
  DBSYNCERRREAD = 3
Error then read items.
  DBSYNCERRWRITEBAG = 4
Error then write to bag.
  DBSYNCERRWRITESYNC = 8
Error then write to orig.
  DBSYNCERRWRITE = 12
Error then write items.
  DBSYNCOK = 16
Syncronization saccessfull.
  DBSYNCERROTHER = 32
Other error then sync.
  DBSYNCERRCOMPARE = 64
Error from compare.
  DBSYNCERROR = 111
All errors.
  DBSYNCCOMPARE = 128
Compare and syncronize this record.
  DBSYNCIGNORED = 256
Ignored.
  DBSYNCCONFLICT = 512
Conflict.
  DBFORCEDIRNONE = 0
Do nothing.
  DBFORCEDIRBAG = 1
In all conflicts copy to bag.
  DBFORCEDIRORIGIN = 2
In all conflicts copy to origin.
  DBFORCEDIROLDER = 4
In all conflicts use older.
  DBFORCEDIRNEWER = 8
In all conflicts use newer.
  DBFORCEDIRBAGALL = 16
All changes from origin to bag.
  DBFORCEDIRORIGINALL = 32
All changes from bag to origin.
  DBFORCEDIRORIGINCOPY = 64
Copying changes from bag to origin (do not delete from origin).
  DBFORCEDIRBAGCOPY = 128
Copying changes from origin to bag (do not delete from bag).
  BACKUPDIR = 16
Backup mode direction.
  _dbforce = {u'bag': 1, u'bagall': 16, u'bagcopy': 128, u'newer...
Direction force array.
  _action_choise = [(u'No', 0), (u'To bag', 1), (u'To origin', 2...
Actions choise for GUI.
  _column_choise_default = []
Columns in default order for GUI list.
  _column_choise = []
Columns.
  COL_NAME = 0
  COL_HEADING = 1
  COL_VISIBLE = 2
  COL_WIDTH = 3
  COL_FORMAT = 4
  P_IGNORE = u'ignore'
Property ignore.
  P_SYMLINK = u'symlink'
Property symbolic link.
  P_COMPARE = u'compare'
Property compare.
  P_BACKUP = u'backup'
Property backup.
  P_IGNORENONE = u''
No ignores.
  P_SYMLINKIGNORE = u'ignore'
Ignore symlink.
  P_SYMLINKIGNORENOEMUL = u'ignorenoemul'
Ignore symlink, no emulation.
  P_SYMLINKCOPY = u'copy'
Copy symlink.
  P_SYMLINKCOPYNOEMUL = u'copynoemul'
Copy symlink, no emulation.
  P_COMPARESTAT = u'stat'
Compare by statistic.
  P_COMPARECRC32 = u'crc32'
Compare by CRC32.
  P_BACKUPYES = u'1'
Backup.
  P_BACKUPNO = u'0'
No backup.
  P_SYMLINK_IGNORE = (u'ignore', u'ignorenoemul')
  P_SYMLINK_EMUL = (u'ignore', u'copy')
  _symlink_choice = [u'ignore', u'ignorenoemul', u'copy', u'copy...
  _compare_choice = [u'stat', u'crc32']
  P_DEFIGNORE = u''
Default ignores - No ignores.
  P_DEFSYMLINK = u'copy'
Default symlink action - Copy symlinks.
  P_DEFCOMPARE = u'stat'
Default compare method - Compare only by statistic.
  P_DEFBACKUP = u'0'
Default not backup root.
  DBSEP = u'/'
DB path separator.
  P_MDDELTA = u'mddelta'
  MDDELTA_DEFAULT = u'5.0'
  MDDELTA = u'5.0'
Modify time available difference for equality.
  REPORTDIR_DEFAULT = u'bonhdyfwce'
Report direction filter default settings.
  REPORTFORMAT_DEFAULT = u's'
Report format default.
  REPORTFORMAT = u's'
Report format, short by default.
  REPORTPATT_DEFAULT = u'!.*'
Report pattern default.
  REPORTPATT = u'!.*'
Report wilds or RegExp filter for paths.
  REPORTPATTCMP = None
Compiled report pattern.
  REPORTTYPE_DEFAULT = u'1234'
Report type default.
  REPORTTYPE = u'1234'
Report type, all by default.
  REPORTPAGE_DEFAULT = u'0'
Report page size default.
  REPORTPAGE = u'0'
Report page size in items, unlimit by default.
  REPORTFILTER_DEFAULT = u'sbonhdyfwce1234'
Report filter default settings.
  REPORTFILTER = u'sbonhdyfwce1234'
Report filter settings.
  REPORTSHOW = True
  O_REPORTFORMAT = {u'l': u'%(forced)s %(dir)s %(err)s [%(path...
Report formats option.
  O_REPORTFILTER = {u'b': 1, u'c': 64, u'd': 16, u'e': 128, u'f'...
Report filter options.
  O_REPORTTYPE = {u'1': u'f', u'2': u'd', u'3': u's', u'4': u'u'}
Report filter options.
  _backupmode = {u'0': 128, u'1': 16}
Backup mode === direction.
  O_ACTION = [u'bag', u'origin', u'delete', u'skip', u'restore',...
Available actions in cli menu.
  O_K_LIST = [u'c', u'i', u'copy', u'ignore', u'cn', u'in', u'co...
  O_K_DICT = {u'c': u'copy', u'cn': u'copynoemul', u'copy': u'co...
  O_S_LIST = [u'stat', u'crc32']
  O_S_DICT = {u'crc32': u'crc32', u'stat': u'stat'}
  _db_formatlist = [(101, <type 'unicode'>, u'', u'DBBAGFLAG'), ...
Format list of tuples (index, type, default) for database.
  SYMLINKEMUL = True
If True then used symlink emulation if no native support.
  DBESCAPE = u' %'
Escape symbols.
  DBESCESC = u'#'
Escape symbol/prefix for hex value.
  OSSEP = '/'
Os path separator.
  OSASEP = None
Alternative os path separator.
  OSWIN = False
  fstream = None
Output log stream.
  _cp = None
ConfigParser instance.
  _db = {}
Database.
  _ks = []
List of sorted in ascending order _db keys.
  _re_rootsec = re.compile(r'(r\d+)')
RegExp for root options names.
  _usage = ...
Usage information.
  climenu = ...
Menu for display in command line version in synchronization.
  climenuhelp = ...
Menu help for display in command line version in synchronization.
  pagemenu = ...
  _tips = [u'All you changes and preferences will be saved only ...
  dir_root = u''
Current path to pybag location (folder).
  dir_files = u''
Files folder.
  dir_db = u''
Data base file path.
  dir_dbt = u''
Data base temp file path.
  dir_cfg = u''
Configuration file path.
  dir_cfgt = u''
Configuration temp file path.
  _loglock = <thread.lock object>
Lock object for threading output.
  LOCK = <pybag.lock_list object>
Initialize LOCK instance.
  _epdl = None
  _epde = None
  _epse = None
  _inep = None
  _outep = None
  catalog = {}
  index = []
  __package__ = None

Imports: os, shutil, sys, time, getopt, atexit, ConfigParser, codecs, locale, re, gzip, binascii, traceback, webbrowser, platform, _threading


Function Details [hide private]

_u(a, enc=None)

source code 

Convert a to unicode if a is not unicode. Use default locale encoding if not given enc.

log(*arg)

source code 

Print all concatenated _u(arg) if PYBAG_LOG True.

Parameters:
  • arg (*object) - Object to output. All objects convert to string str() function.
Returns: None
None.

dbg(*arg)

source code 

Print all concatenated _u(arg) if PYBAG_DEBUG True.

Parameters:
  • arg (*object) - Object to output. All objects convert to string str() function.
Returns: None
None.

log_writeln_1(*arg)

source code 

Main writeln function for logging. Write concatenated arguments into outstream and append new line ('\n').

Parameters:
  • arg (*object) - Object to output. All objects convert to string _u() function.
Returns: None
None.

log_init()

source code 

Initialize file output stream.

Default is "pybag.log".

prt(*arg, **kwd)

source code 

Print all arg consecutively with print builtin function. Also log all output.

Parameters:
  • verbose - Verbosity level. {default VERBOSE == 3}.
  • place - Function name from called print.
  • error - Error description or exception.

parse_command()

source code 

Start command parser.

Returns: Class object or bool.
Object ret if OK, else False. ret contains next attributes:
  • cmd - command name, str.
  • fs - files list, list.
  • option name - (option, values), list.

test_init()

source code 

Test if programm location already initialized.

Returns: bool
True if initialized.

init_program()

source code 

Initialize program environment.

Test current program location for initialization. Ask user for initialization if need and initialize if user agree. In initialization time (if user agree) set emulation off. After return it in previous state. Create needed files and folders if it not exist. Read configuration file.

Returns: bool
True if initialization successful, otherwise False.

initpath(path)

source code 

Initialize bag in path. Create path if need. Do not change already exists files.

Parameters:
  • path (str) - Path to initialize bag.
Returns: bool
True if successfull or False otherwise.

initdb(path, db={})

source code 

Initialize database file in specified path. If not exists then create DB == db (by default is empty DB).

Parameters:
  • path (str) - Path to initialize DB.
  • db (dict = {}) - DB for initializing.
Returns: bool
True if successfull, False overwise.

initconfig(path)

source code 

Initialize configuration file in specified path.

If not exists then create next configuration file:

   # PYBAG configuraton file.
   
   [root]
   
   nextid = 1
   ignore =
   compare = stat
   symlink = copy
   
   # Subroots sections.
Parameters:
  • path (str) - Path to initialize configuration file.
Returns: bool
True if successfull, False overwise.

initpybagfiles(path)

source code 

Initialize pybagfiles directory in specified path.

If not exists then create empty directory "pybagfiles".

Parameters:
  • path (str) - Path to initialize pybagfiles directory.
Returns: bool
True if successfull, False overwise.

transreport(opt=None)

source code 

Translate report option -g into global variables.

Parameters:
  • opt (str) - Option value.
Returns: str
u'' - empty string if all OK. Error string otherwise.

dbread(path=None)

source code 

Read database file pybag.db (gzipped). If errors occurs then setup _r.e to error message. In DB files placed in next format:

  • DB_VERSION_MARKER DB_VERSION
  • F SIZE MDATE CRC32 path_to_file1_in_pybagfiles link_point1
  • F SIZE MDATE CRC32 path_to_file2_in_pybagfiles link_point2
  • ... and so on.

All lines must end with newline.:

  • "F" - flag: d-directory, f-file, s-symlink, ''-is empty entry.
  • "SIZE" - file size in decimal or "-" for directories.
  • "MDATE" - modification date in YYYY/MM/DD-HH:mm:ss.
  • "CRC32" - crc32 in hex if exist or "-".
  • "path_to_file" - bag path to item into pybagfiles id "db" format.
  • "link_point" - if item link then it's point in "db" format, otherwise '-'.

All lines must end with newline. Read only DB information.

Parameters:
  • path (str = None) - Path from reading DB. If None then use dir_db.
Returns: dict
DB dictionary. In _r return output and errors. In returned dbdictionary key is normalized/cased path (relative to pybag), value is dict of next items (see in pybag description).

dbsave(dbdict, path=None, patht=None)

source code 

Save database to file. If errors occurs hen setup _r.e to error message. In DB files placed in next format:

  • DB_VERSION_MARKER DB_VERSION
  • F SIZE MDATE CRC32 path_to_file1_in_pybagfiles link_point1
  • F SIZE MDATE CRC32 path_to_file2_in_pybagfiles link_point2
  • ... and so on.

For details format see dbread.

Parameters:
  • path (str = None) - Path from save DB. If None then use dir_db.
  • patht (str = None) - Path from save backup DB. If None then use "path"+"~".
  • dbdict (dict) - DB dictionary. See dbread for details.

dbdumprow(dbrow, tab=u' ....->')

source code 

Dump DB row to string. Dump only items from _db_formatlist.

dbdump(db, tab=u'DBDUMP:')

source code 

Dump DB to string. Dump only items from _db_formatlist.

dbnormalize(db)

source code 

Normalize DB - convert values to right type and past default if absent. If no such index or wrong values or type then create and set to default.

Returns:
Output and errors in _r and changed input db.

splitpaths(path)

source code 

Return list of path's parts.

Parameters:
  • path (str) - Path.
Returns: list
List of splited parts of path. If path empty then empty list.

Warning: Normalize path before splitting. See os.path.normpath().

splitpathsdb(path)

source code 

See splitpaths. Used for DB format paths.

getpathdirdb(path)

source code 

For DB format paths return parent directory for given path. If no parent return empty string u''.

path2db(path)

source code 

Convert path to DB format (separate by DBSEP).

Warning: Path started from root "/" converted wrongly - "/" character changed to os separator. For example "/abc/def.ghi" converted to "\/abc/def.ghi" under Windows and "#002F/abc/def.ghi" under unix.

walk(path, onerror=None)

source code 

Top-down directory walker. return in iterate (r,d,f), as python standard os.walk. Note: Default python os.walk some times generate error in mixing encodings in file names ... may be it bug?

symlinktest(path, useemul=None)

source code 

Test file for symlink emulation or real. Return True if symlink, False overwise.

Parameters:
  • useemul (bool) - If true then use symlink emulation if no native support.

readlink(path, useemul=None)

source code 

Read symbolic link point. Use native function if support or emulate. If filesystem not support link then also emulate it.

Emulate file format is: "pybaglink start:xxxxxxxxxx:pybaglink end". File encoding is "utf_8".

Parameters:
  • useemul (bool) - If true then use symlink emulation if no native support.
Raises:
  • IOError - if error then read link.

symlink(link, path, useemul=None)

source code 

Save symbolic link point. Use native function if support or emulate. If filesystem not support link then also emulate it. Before create new link you must remove previous.

Emulate file format is: "pybaglink start:xxxxxxxxxx:pybaglink end". File encoding is "utf_8".

Parameters:
  • useemul (bool) - If true then use symlink emulation if no native support.
Raises:
  • IOError - if error then save link.

rmdirr(path)

source code 

Delete directory recursively. Delete everything reachable from the directory named in 'path', (do not follow for symbolic link).

Warning: This is dangerous! For example, if path == '/home', it could delete all your home files.

!!! ROOT FOLDERS ('/' and 'C:') CAN NOT BE DELETED THIS FUNCTION!!!

If you want delete root folder you must delete it manualy.

copytree(src, dst, symlinks=False, exclude=[])

source code 

Copy files tree from "src" to "dst". "dst" must be a directory. If "symlinks" True then also copy symlinks (default False). If symlinks not supported then they emulated. If "dst" not exists then create it. From copying exluded items equal "exclude" list.

validatepaths(root, bagpath=None, syncpath=None)

source code 

Validate paths.

Parameters:
  • root (str) - Root (r1, r2, ...).
  • bagpath (str { = None }) - Root path in pybag. If None then get from root config.
  • syncpath (str { = None }) - Syncronization path. If empty then used base (config) path.
Returns: list or None
list [root, bagpath, syncpath, dbindex] or None if path wrong. All path returned in absolute form, dbindex is normalized.

getcrc32(path)

source code 

Calculate CRC32 for given file. Return 0 for other situation (directory, link, absent).

dbwalkfiles(db, root, path, isbag, dbindex)

source code 

Walk throw bagpath files from pybagfiles and syncpath for original files.

Parameters:
  • db (dict) - Database dictionary.
  • root (str) - Root name for read configuration.
  • path (str) - Absolut path for walking. Path must exists.
  • dbindex (str) - Index into DB equivalent path.
  • isbag (bool) - True if walk throw bag, False for origin.
Returns: tuple
(success, ignored) tuple and changed input db:
  • success - bool, True if Ok, False if error or path not exists.
  • ignored - int, number of ignored items.
Raises:
  • ValueError - Raise if path is not absolut.

Warning: path must be in absolute normalized format and must exists. Usually this function get arguments that returned by validatepaths.

transopt(opt, cmd)

source code 

Translate option in command. Get from "cmd" list "cmd.x" where x - is "opt". Get value from dictionary O_X_DICT[O_X_LIST[cmd.x[-1]]]. If no "cmd.x" then return O_X_DICT[O_X_LIST[0]]. If not supported option then return empty string ''.

Parameters:
  • opt (str) - Option symbol: 'k', 's'.
  • cmd (object) - Command object.
Returns: str
Translated option value or empty string.

cmd_unknown(cmd)

source code 

Unknown command. Print message about unknown command.

f_stat(cmd)

source code 

Show info about bag files.

Returns:
String with statistics.

outreport(db=None, ks=None)

source code 

Output comparison report from database db in console.

Filter output

do_action(action, filter, interactive=True, listpaths=True)

source code 

Perform changes into db[DBDIR] to specified action.

Parameters:
  • action (basestr) - Action string O_ACTION.
  • filter (string.) - Filter string, "wilds" or "!regexp". If not specified then used "!.*" - match all paths.
  • interactive (bool) - If True then ask from user confirmation.
  • listpaths (bool) - If True then list all affected paths.
Returns: stirng.
String with error or empty string if all ok.

cmd_sync(cmd)

source code 

Synchronize files.

Algorithm:

  • Test for attributes.
  • Append paths to lists.
  • dbread.
  • Loop for all paths:
    • dbwalkfiles.
  • dbcomparefiles.
  • dbsyncfiles.
  • dbcleardb
  • Print results.
  • Save DB.

f_syncread(cmd)

source code 

Read DB, files.

Returns: bool
Number of ignores. Result in _r.

f_synccompare(direction)

source code 

Compare DB.

Return tuple of number modified items.

Parameters:
  • direction (int { DBFORCEDIR*** constants }) - Forced direction in conflicts.

Warning: All paths (BAG, Origin) must be filled. And DB must be normalized.

f_syncsync()

source code 

Syncronize DB.

Return tuple of modified and errored items.

f_syncclear()

source code 

Clear DB. Remove from DB unused record (successfully deleted and ignored items).

_debug()

source code 

Debug function.

Test functions.


Variables Details [hide private]

__license__

License.

Value:
u'''
Copyright (c) 2007 .. 2017, Mazhugin Aleksey

All rights reserved.

Redistribution and use in source and binary forms, with or without mod\
ification,
are permitted provided that the following conditions are met:
...

__about__

About.

Value:
u'''
    ###########################################
    #           ***   P Y B A G   ***         #
    #              version:  0.4.4            #
    #  Copyright (c) 2017, Mazhugin Aleksey   #
    ###########################################

PYBAG - Crossplatform files synchronization and backup portable tools.
...

PYBAG_DEBUG

Debug mode. If True (default False) then out all debug messages. Prepend to message "-->" string.

Value:
False

PYBAG_LOG

Logging mode. If True (by default) then out all log messages. Prepend to message "#" string.

Value:
True

PYBAG_EMUL

Emulation mode. If True (by default) then do not real changes (emulation mode).

Value:
True

AppState

Application state for GUI UI update.

Value:
{'ff_bag': True,
 'ff_ch': True,
 'ff_conf': True,
 'ff_del': True,
 'ff_dir': True,
 'ff_err': True,
 'ff_file': True,
 'ff_forced': True,
...

_filter

Value:
{u'ff_bag': u'filter_bag',
 u'ff_ch': u'filter_changed',
 u'ff_conf': u'filter_conflict',
 u'ff_del': u'filter_delete',
 u'ff_dir': u'filter_directory',
 u'ff_err': u'filter_error',
 u'ff_file': u'filter_file',
 u'ff_forced': u'filter_forced',
...

DBFORCEDIROLDER

In all conflicts use older. May be used with "NEWER", "OLDER" flags.

Value:
4

DBFORCEDIRNEWER

In all conflicts use newer. May be used with "NEWER", "OLDER" flags.

Value:
8

_dbforce

Direction force array.

Value:
{u'bag': 1,
 u'bagall': 16,
 u'bagcopy': 128,
 u'newer': 8,
 u'no': 0,
 u'older': 4,
 u'origin': 2,
 u'originall': 32,
...

_action_choise

Actions choise for GUI.

Value:
[(u'No', 0),
 (u'To bag', 1),
 (u'To origin', 2),
 (u'Newer', 8),
 (u'Older', 4),
 (u'Newer, to bag', 9),
 (u'Newer, to origin', 10),
 (u'Older, to bag', 5),
...

_symlink_choice

Value:
[u'ignore', u'ignorenoemul', u'copy', u'copynoemul']

MDDELTA

Modify time available difference for equality. Default is 5.0 second.

Value:
u'5.0'

O_REPORTFORMAT

Report formats option.

  • l - Long format of report.
  • s - Short format of report.
  • p - Old (as in previous version) format.
  • m - Manual specified format.
  • z00 - List report by n items per page.
Value:
{u'l': u'''%(forced)s %(dir)s %(err)s [%(path)s]
	%(desc)s''',
 u'p': u'''	[%(path)s]-->
	  == "%(desc)s".''',
 u's': u'%(forced)s %(dir)s %(err)s [%(path)s]'}

O_REPORTFILTER

Report filter options.

  • b - Operation with direction to bag.
  • o - Operation with direction to origin.
  • n - Operation with new files.
  • h - Operation with changed (copied) files.
  • d - Operation with deleted files.
  • y - Timeshift operations.
  • i - Ignored files.
  • u - Unchanged files.
  • f - Forced action.
  • w - Warnings.
  • c - Conflicts.
  • e - Errors.
Value:
{u'b': 1,
 u'c': 64,
 u'd': 16,
 u'e': 128,
 u'f': 1024,
 u'h': 8,
 u'i': 2048,
 u'n': 256,
...

O_REPORTTYPE

Report filter options.

  • 1 - Files.
  • 2 - Directories.
  • 3 - Symlinks.
  • 4 - Unknown.
  • r - Path wilds or regexp filter. If not specified then all paths displayed.
Value:
{u'1': u'f', u'2': u'd', u'3': u's', u'4': u'u'}

O_ACTION

Available actions in cli menu.

Value:
[u'bag',
 u'origin',
 u'delete',
 u'skip',
 u'restore',
 u'tsbag',
 u'tsorigin',
 u'tsreset']

O_K_LIST

Value:
[u'c',
 u'i',
 u'copy',
 u'ignore',
 u'cn',
 u'in',
 u'copynoemul',
 u'ignorenoemul']

O_K_DICT

Value:
{u'c': u'copy',
 u'cn': u'copynoemul',
 u'copy': u'copy',
 u'copynoemul': u'copynoemul',
 u'i': u'ignore',
 u'ignore': u'ignore',
 u'ignorenoemul': u'ignorenoemul',
 u'in': u'ignorenoemul'}

_db_formatlist

Format list of tuples (index, type, default) for database.

Value:
[(101, <type 'unicode'>, u'', u'DBBAGFLAG'),
 (102, <type 'int'>, 0, u'DBBAGSIZE'),
 (103, <type 'float'>, 0.0, u'DBBAGMDATE'),
 (104, <type 'int'>, 0, u'DBBAGCRC32'),
 (105, <type 'bool'>, False, u'DBBAGREAD'),
 (106, <type 'unicode'>, u'', u'DBBAGPATH'),
 (107, <type 'unicode'>, u'-', u'DBBAGLINK'),
 (301, <type 'unicode'>, u'', u'DBDBFLAG'),
...

SYMLINKEMUL

If True then used symlink emulation if no native support. Used as default parameter in symlink functions (symlink, readlink, symlinktest).

Value:
True

DBESCAPE

Escape symbols. DBESCESC and DBSEP is autoadded in start of secuence. Used in previous to 0.2.1 versions. Now used inline symbols = space only.

Value:
u' %'

_usage

Usage information.

Value:
u'''
    #############################################
    #           ***   P Y B A G   ***           #
    #              version:  0.4.4              #
    #  Copyright (c) 2011, Mazhugin Aleksey     #
    #############################################

                  Usage.
...

climenu

Menu for display in command line version in synchronization.

Value:
u'''
Enter action (h - for help):

'''

climenuhelp

Menu help for display in command line version in synchronization.

Value:
u'''
Menu help:
-------------
    d DIR DIR - change rules for autoconflict resolution
        (after change direction comparison will be repeated,
        and menu displayed again). DIR is next:
        [bag, origin, no, newer, older, bagall, originall,
        bagcopy, origincopy].
...

pagemenu

Value:
u'''
ENTER - next page,  "NUMBER" - page size, "q" - quit report, "a" - lis\
t all.
'''

_tips

Value:
[u'All you changes and preferences will be saved only if "Emulation mo\
de" turn off.',
 u'You can hide/show columns in list in menu when click right mouse bu\
tton on the columns header.',
 u'If you use any root only for backup then you can mark it in prefere\
nces as "Backup" and synchronization will be up 2 times faster.',
 u'In preferences you can adjust default filters for items list.',
 u'If you click right mouse button on column header then you can move \
...

LOCK

Initialize LOCK instance. It used in multithreading

Value:
<pybag.lock_list object>